3.5 Bits Beat 4 Bits: Formats, Not Precision

The number on the model card is bits per weight, so that is what everyone compares: Q4 is smaller than Q8, Q8 is smaller than FP16, and smaller should be faster. Our board has rows that obey that rule and rows that break it, and the ones that break it are the interesting ones — because when you pick a quantization you are not really picking a precision. You are picking a kernel: a specific dequantize-and-multiply path that some engine, somewhere, has or has not optimised for your hardware.

Three data sets, each holding the engine, the hardware and the context fixed while the format changes.

As of 2026-09-15. All rows are imported sourced measurements on our board, each carrying engine version, quantization and context; we did not run these tests. Sources at the end.

Ladder 1: 3.5 bits beats 4 bits by 2×

DGX Spark ×4 running the 552B-class DeepSeek-V4.1-Flash, all rows on vLLM at a 300,000-token window — same engine, same rig, same context, three different weight formats:

Read the top and bottom rows together. The 3.5-bit format is 2.0–2.2× faster than the 4-bit format on the same hardware, same engine, same window. A higher precision ran faster than a lower one, which is the opposite of what bpw predicts.

Nothing about this is mysterious once you look at what each format actually is. MXFP4 is a microscaling format — groups of 32 weights share an exponent — and serving it means the engine must apply a scale per group on every matmul. EXL3 (ExLlamaV3) is a 3.5-bit format whose dequantization path was written for exactly this kind of serving; the numbers here say the vLLM build on that box had a considerably better path for it than for MXFP4. Both rows are "4 bits more or less". Only one of them is fast.

Ladder 2: the same engine, the same card, FP8 vs NVFP4

RTX PRO 6000 Blackwell running Muse-Glimmer-30B, both rows on vLLM at a 32,768-token window:

Here the ordering is the intuitive one — the narrower format wins — but the size of the win is the point: 1.43× for halving the weight width. If you were budgeting on a "half the bits, twice the speed" rule of thumb you would be disappointed; if you were budgeting on "same engine, both formats supported, here is the actual gap" you would have predicted this.

The same card, same model, on llama.cpp at a 131,072-token window shows the ordinary precision ladder doing its job, which is the sanity check that the format story is not just noise:

1.67× from 8-bit to 16-bit, on one engine, one rig, one window. This row is what "bits per weight behaves" looks like — and it is not the ladder people expect to be interesting.

Ladder 3: the 5.9× that is not about bits at all

One more pair from the DGX Spark family, and this one is a warning rather than a lesson. DGX Spark ×2 running Qwen3.8-Flash-Next on vLLM at a 262,144-token window:

These are separate model entries on the board because they are separate checkpoints, not two quantizations of one file: the hibrid48 variant mixes precisions by layer, and its build ships against a different vLLM (0.29 versus a patched nightly). Treat the 5.9× as a build-and-checkpoint result, not a format result. It is in this article because it is the cleanest illustration of the trap: two rows whose names differ by a suffix, one number five times the other, and nothing in the bpw tells you which is which.

Which is why our board keeps them apart. Rows with different engine versions or checkpoint identities are different rows, and the entity pages show the full provenance for exactly this reason.

How to choose a quantization, given all this

1. Start from the engine's support matrix, not the model's bit depth. The question is not "how many bits" but "does this engine have a fast path for this format on this GPU family". EXL3 at 3.5 bpw beat two 4-bit formats on the hardware where it had a path.
2. Compare within one engine. Across engines you are measuring the engine: the RTX 5090 rows for one 27B model on our board span 16.8 to 435 tok/s across vllm, llama.cpp, tensorrt-llm and a custom runtime, at contexts from 2K to 251K. That spread is not the model, and it is not the quantization.
3. Prefer the format your stack has measured. A row on our board with the engine version attached is a better predictor of your result than a whitepaper's bpw-vs-quality curve.
4. Expect the speed win to be smaller than the size win. Halving weight width bought 1.43× (NVFP4 vs FP8) and 1.67× (Q8 vs BF16) in the clean pairs above. Size and speed are related, not proportional — and at long context, where the KV cache and prefill matter more, even that relationship softens.
5. Recheck after engine upgrades. These gaps are build-specific. A format that loses today can win after a kernel lands, and the version column is what lets you see it happen.

FAQ

Is a lower-bit quantization always faster?

No. In our DGX Spark ×4 rows for DeepSeek-V4.1-Flash, EXL3 at 3.5 bits per weight ran at 82 tok/s while MXFP4 at 4 bits ran at 41 — the higher-precision format was twice as fast, on the same engine, rig and 300K context.

Why would a higher precision be faster than a lower one?

Because throughput is decided by the dequantization and matmul kernels the engine has for that format, not by the stored width. A format with a well-optimised serving path moves less compute per token than one that has to scale and unpack on the fly.

How much faster is a 4-bit model than an 8-bit one?

In our cleanest same-engine pairs: NVFP4 was 1.43× FP8 on an RTX PRO 6000 (both on vLLM, 32K context), and Q8_K_XL was 1.67× BF16 on the same card under llama.cpp at 131K context. Halving the weight width does not halve generation time.

What should I check before switching quantization formats?

Whether your engine has a fast path for the target format on your GPU generation, and whether a row with your engine version exists on the board. Format support is per-build: the same engine can be fast on one format and slow on another.

Does the quantization affect quality the same way it affects speed?

No — speed and quality are separate axes. The format decides whether your hardware has a fast path; the bits decide how much of the original weights survive. Our quantization guide covers the quality side.

Sources

• Ladder 1 — DGX Spark ×4 × DeepSeek-V4.1-Flash on vLLM at 300K context, formats EXL3 3.5bpw / FP4+FP8 native / MXFP4: imported and sourced on the board; see DGX Spark and DeepSeek-V4.1-Flash.
• Ladder 2 — RTX PRO 6000 Blackwell × Muse-Glimmer-30B, vLLM FP8-block vs NVFP4 at 32,768 context, and llama.cpp Q8_K_XL vs BF16 at 131,072 context: RTX PRO 6000 Blackwell · Muse-Glimmer-30B.
• Ladder 3 — DGX Spark ×2 × Qwen3.8-Flash-Next checkpoints on vLLM at 262,144 context, source: @vr8vr8.
• Board and provenance rules: benchmark board · methodology.

Imported upstream measurements as of 2026-09-14. DiggTop did not run these tests; each row carries its engine version, quantization and context, and states the fields the methodology rules require.