Quantization for Local LLMs, Explained
Data as of 2026-10-07. Distribution numbers are from the DiggTop board — 336 sourced rows (partial rows are excluded from every count on this page).
TL;DR
| Quant | Typical VRAM per billion params | Notes |
|---|---|---|
| Q8_0 | ~1.0 GB | Near-lossless GGUF; the safe default when VRAM allows |
| Q6_K | ~0.75 GB | Good quality; common on 24GB cards |
| Q4_K_M | ~0.55 GB | The community workhorse — 60 of our 336 sourced rows |
| IQ4_XS | ~0.5 GB | Better quality-per-bit than plain Q4 on llama.cpp |
| NVFP4 | ~0.5 GB | Blackwell-only 4-bit; fastest 27B runs on our board |
| FP8 | ~1.0 GB | Datacenter/Blackwell serving quant (vLLM/SGLang) |
Why quantization decides everything
Local inference speed is dominated by one question: does the model fit in VRAM, with room for KV cache and context? Quantization shrinks the weights so a bigger model fits a smaller card. The tradeoff is quality — and the formats above trade quality against size differently.
A rough planning rule at Q4: ~0.6GB per billion parameters of weights, plus 0.1–0.2GB per billion for context and runtime overhead — so ~0.7–0.8GB per billion all-in. A 27B model therefore lands around 17–20GB at Q4, which is why 24GB cards are the community sweet spot.
What the community actually runs
Our board's quant distribution (2026-10-07, 336 sourced rows):
| Quant | Runs | Family |
|---|---|---|
| Q4_K_M | 60 | GGUF K-quant |
| Q8_0 | 21 | GGUF |
| NVFP4 | 27 | NVIDIA Blackwell |
| FP8 | 15 | vLLM / datacenter |
| MQ4 | 15 | vLLM Marlin 4-bit |
| Q6_K | 6 | GGUF |
| IQ4_XS | 6 | GGUF IQ-quant |
| W4A16 | 7 | AWQ-style 4-bit |
Q4_K_M is the default for a reason: it is the llama.cpp mid-size K-quant with the best size/quality balance on 16–24GB cards. NVFP4's share (27 runs) is driven entirely by Blackwell owners. The fastest sourced 27B row is Qwen3.8-27B at IQ4_XS on an RTX 5090 (435.1 tok/s); a 4,546.8 tok/s NVFP4/sglang row exists in the database but is still partial.
Which quant for your hardware
| Your card | Recommended | Why |
|---|---|---|
| 12–16GB | Q4_K_M, IQ4_XS | Fit first, then pick quality |
| 24GB | Q4_K_M for 27B; Q6_K for 13B and under | Room for context matters more than 2% perplexity |
| Blackwell 32GB+ (5090, PRO 6000) | NVFP4 for 27B-class | Hardware 4-bit: half the footprint, tensor-core speed |
| Datacenter / serving (H100, V100 x4) | FP8, W4A16, MQ4 | vLLM/SGLang deployment formats |
Backend support: llama.cpp (and LM Studio) run GGUF K-quants and IQ-quants; vLLM runs FP8/W4A16/MQ4 (Marlin); SGLang covers FP8/FP4 paths. The same model can land in every row of our compatibility matrix via different quants.
Quality vs speed: how to verify
- Start with Q8_0, measure quality on your workload, then step down (Q6_K → Q4_K_M → IQ4_XS) until you hit the quality cliff.
- Quantization hurts some tasks more than others — coding and long-context reasoning degrade faster than chat.
- Every run on our board carries quant, engine version and context, so you can compare the same model at different quants: benchmarks · methodology.
Sources
- Quant distribution and runs: DiggTop benchmarks board · compatibility matrix
- Methodology rules: Benchmark Methodology
- llama.cpp quantization docs: github.com/ggml-org/llama.cpp
FAQ
Which quant should I start with?
Q4_K_M. It is the community default: quality close to Q5 with meaningful VRAM headroom left over. If you are VRAM-starved, drop to IQ4_XS when the perplexity delta stays small; if you have headroom and quality matters (RAG, long documents), move up to Q6_K or Q8_0.
What is the real quality difference between Q4 and Q8?
Small for most general chat — per-model perplexity deltas are typically well under 0.5, and many users cannot tell them apart. The exceptions are math- and code-heavy workloads and smaller models under ~10 B, where Q8 can be noticeably better. Run your own check before deciding, as the quality section describes.
What are NVFP4 and FP8 for?
They are the 50-series / vLLM-side formats: NVFP4 is roughly 4-bit with near-FP8 quality on supported models, and an FP8 KV cache cuts context VRAM cost. If you are on a 5090-class stack running vLLM, use them where the model supports it; on llama.cpp with a 3090-class card, stay in GGUF land.
Do quant names mean the same thing across engines?
No. Q4_K_M is a GGUF (llama.cpp) name, NVFP4 is a 5090 format, IQ4_XS is an importance-matrix GGUF. A "4-bit" on one engine is not the same artifact as a Q4_K_M GGUF — compare tok/s and quality within the same engine, per the methodology.
Numbers are snapshots — submit your own run with one prompt to add a data point.