MLXHub AI Benchmark · Protocol 6
One score. Every part explained.
How well does your iPhone or iPad run local AI? We measure the work it can do, the memory it can use, and how performance holds up over time.
A performance benchmark. It does not grade the intelligence or writing quality of a model.
01 · What happens on your device
The same work. Eight different runs.
One complete benchmark measures memory, then runs a short and a long conversation with each model below. The prompts and generation settings come from a shared protocol.
| Model | Precision | What it adds |
|---|---|---|
| Gemma 4 E2B Instruct | 4-bit | A lighter resident starting point. |
| Gemma 4 E4B Instruct | 4-bit | A larger resident workload. |
| Gemma 4 E2B Instruct | 8-bit | The E2B family at a different precision and memory footprint. |
| LFM2.5 8B A1B | 8-bit | A mixture-of-experts workload that exercises expert streaming. |
4-bit and 8-bit describe how model weights are stored. They are different workloads, not quality ratings. LFM streams experts from storage; the other three model configurations are resident in memory.
Short prompts represent a lighter input. Long prompts exercise accumulated context and its processing cost. Testing both exposes different bottlenecks; it does not isolate every difference between models.
Find the usable memory ceiling
The memory test gradually allocates memory and observes pressure warnings and the peak footprint. A safety margin turns those observations into a recommended usable amount for this run. More capacity can support larger local workloads.
Load the model
Each scenario loads its model. Model bytes divided by load time gives throughput in GB/s. This captures the cost of getting a model ready, separately from the speed of generating a reply.
Process a fresh prompt
The model reads the short or long prompt with an empty prompt cache. Real prompt tokens divided by processing time measures cold prefill: how quickly it processes the input before generating.
Check prompt reuse
A warm prompt pass shows whether retained context can be reused. Its speed is kept as a diagnostic when there is a real cache hit. It adds no separate points and never replaces the cold measurement.
Four minutes of sustained work
The model completes successive turns of up to 32 tokens. The first 10 seconds are warm-up. After that, only measured generation intervals count toward tokens per second; time spent reading another prompt is not counted as slow generation.
Track sustained performance and heat
The test compares early and late generation and records thermal conditions. It also measures expert-reading throughput for the streamed model. The next phase waits for the required thermal conditions before continuing.
Why the full test takes longer than 32 minutes
Eight scenarios × four minutes gives 32 minutes of nominal sustained phases. Downloads, loading, cold prompt processing, the memory test and cooling add time. A scenario may extend by up to one minute solely to collect enough evidence: at least two independent turns and five scored seconds.
Give the measurements a fair start
Use a Release build, leave Low Power Mode off, unplug external power and let the device cool. The app checks these conditions and marks thermally contaminated phases. Keep the app in the foreground. Locking or leaving it currently interrupts the run; resume is not yet available.
You can select memory and whole model tests before starting. Partial runs keep their local measurements, but need the complete suite to produce a publishable global score.
02 · From measurements to one number
Six measures. A deliberate balance.
The weights prioritize generation and usable memory while retaining preparation speed, sustained performance and expert reading. They are an explicit product decision, not statistically fitted or scientifically optimal weights.
Generation
30%Tokens per second during scored generation. Its 30% weight prioritizes the speed of the output you read.
Usable memory
22%Recommended safe memory from this run. Its 22% weight recognizes capacity for models and context; memory is measured once, against both references.
Cold prompt processing
15%Input tokens per second with an empty prompt cache. Its 15% weight reflects the wait before a response, especially with longer prompts.
Endurance
15%A balance of speed retention and thermal comfort. Its 15% weight rewards performance that lasts beyond a quick burst.
Expert reading
10%MiB per reading second in LFM’s expert stream. Its 10% weight captures workloads beyond resident memory. Reads may use system cache; this is not raw flash speed.
Model loading
8%Model size divided by loading time, in decimal GB/s. Its 8% weight counts preparation without letting less frequent cold starts dominate the score.
Compare first. Combine second.
- For each scenario, divide its measurement by the matching reference. A ratio of 1 matches the reference; 2 is twice its measured value. Every ratio has a 0.05 floor and no upper cap.
- Use the geometric mean — a multiplicative average — within each measure. Generation, prompt processing, loading and endurance each combine eight scenarios equally. Expert reading combines only the two LFM scenarios.
- Memory uses the same measured amount twice: once against the 4-bit memory reference and once against the 8-bit reference. Combine those two ratios geometrically, then apply its 22% weight once.
D: generation · P: cold prompt · L: loading · M: memory · E: endurance · S: expert reading
This is not a sum of points or a 50/50 average of the two category scores. Doubling generation alone multiplies the global score by 2⁰·³⁰, about 1.231 — a 23% increase.
The exact details
How endurance is calculated
Speed retention is the last generation window divided by the first, capped at 1. Windows use the first and last 15% of observed generation, split between distinct completed turns. Thermal comfort is a time-weighted average: nominal 1, fair 0.85, serious 0.45 and critical 0.15. Endurance is the square root of retention × comfort. Heating during a valid phase lowers comfort; it does not automatically invalidate the run.
How the memory safety margin works
With warning footprint W and critical footprint K, recommended memory is the smaller of 0.95 × W and 0.85 × K. Without a critical event, it is 0.85 × W. Values truncate to whole bytes. If no warning arrives, peak footprint supplies W.
What changes when you filter the ranking
Model and conversation filters restrict the measurements being combined. An individual metric shows 1000 × its aggregated ratio. For a combined score without expert streaming, the storage factor is omitted; the remaining weights are not rescaled. A 4-bit score and an 8-bit score therefore describe their own workloads.
What happens when memory runs out
A correctly documented memory limit keeps completed phases and assigns a ratio of 0.05 to the interrupted phase and the phases depending on it. A load failure penalizes all scenario metrics; a cold-prompt failure keeps loading; a warm-prompt or generation failure keeps loading and cold prefill. The scenario stays in the average, so running out of memory never earns an advantage by disappearing. Cancelled, omitted, thermally contaminated or invalid tests do not receive this treatment.
Explore the formula
Change a measure. See its real weight.
Choose a measured example, then adjust it. Performance controls multiply every scenario’s original ratio before the 0.05 floor. Memory is one shared amount compared with both references. This explains the mathematics; it does not predict a real device.
× the selected example
GiB of recommended memory
Illustrated global score
iPhone 17 Pro Max
All intermediate values stay unrounded. The displayed score is rounded only here. Changes in one control are isolated for learning; real measurements often influence one another.
03 · A measured starting point
1000 is an anchor. Not a promise.
Protocol 6 uses the iPhone 17 Pro Max run for its 4-bit references and the 8 GB iPad Pro 12.9-inch (5th generation, M1) run for its 8-bit references. Both completed all eight scenarios without a pause.
| Device | 4-bit score | 8-bit score | Global score |
|---|---|---|---|
| iPhone 17 Pro Max | 1000 | 1654.93 | 1349.51 |
| iPad Pro M1 | 739.15 | 1000 | 859.74 |
Each device scores 1000 in the category it anchors. Neither is required to score 1000 globally: the global formula spans both sets of measurements.
Measured on app Release 2.2.1 (84), protocol 6. iPhone: iOS 27.0 build 24A437; iPad: iPadOS 27.0 build 24A435. “Release” describes the app build, not a claim that the operating system was a public stable release.
Download the protocol 6 reference values (JSON)These are reference values used in scoring, not a live ranking. The same benchmark protocol and calibration must be used for a meaningful comparison.
04 · Checks, with honest limits
Harder to inflate. Still not proof.
The ranking recalculates scores from measurements instead of trusting a submitted score. Several checks look for inconsistent data. None can guarantee that a carefully fabricated measurement is genuine.
A shared measurement protocol
Fixed scenarios, settings, warm-up, timing and independent generation windows limit cherry-picking. Partial selection cannot masquerade as a complete global benchmark.
Recalculate the score
The app validates the payload’s digest and recomputes scores against the current calibration. A verdict must match the exact submission and configuration; editing a number does not preserve that match.
Check that the numbers agree
The optional batch reviewer compares tokens with scored time, measured time with elapsed time, expert throughput with available read counters, and safe memory with its exported pressure readings. Scores are independently recomputed again.
Compare like with like
Statistical groups separate hardware, physical RAM, major OS version, protocol, model, context and metric. A median-based check can flag unusual values when at least three samples exist. A statistical outlier asks for review; it is not a fraud verdict.
Publishing does not wait for review
Eligible results appear immediately. Your account can keep an independent best result for each device and protocol. A new run replaces that device’s published result only when its global score improves. Results from an iPad do not overwrite an iPhone’s result.
What the labels mean
- Not reviewed
- Published and checked by the app; no administrative review verdict. Included in the default ranking.
- Reviewed
- Algorithmic consistency checks were performed. There may still be too few comparable samples for a population check. This is not proof of authenticity.
- Verified
- Reserved for a separate independent verifier, which is not implemented. A statistical review does not grant this status.
Batch review is an operator-run tool, not an automatic publication gate or an always-running service. Flags and rejections require explicit handling; results are not silently deleted.
What these checks cannot establish
- They do not prove that the device, account or measurements are genuine. Hashes protect an exact payload; they do not certify that its contents are true.
- Model weights are not yet cryptographically pinned to every submission. Model repositories and operating-system builds can change; sharing a model name does not prove identical weights.
- Expert reads may come from system cache. Ambient temperature, every background activity and all OS changes are not controlled. The reviewer does not rerun inference or reconstruct every metric from raw traces.
- Internally consistent invented data and manipulated comparison groups can pass lightweight checks. A very high score alone is not an error, and an unusual result alone is not evidence of cheating.