llama.cpp Optimization Cheat Sheet
llama.cpp is the default inference path for local LLMs, and 90% of "my rig is slow" cases come from a handful of settings. This is the short list that matters, in order of impact.
As of 2026-10-07. These are engine settings, not hardware prices, so they age slowly — but quant, context and batch defaults move with llama.cpp releases. Check your build's flags against the docs before trusting an old command line.
1. Get the weights fully into VRAM
-ngl 999 (offload all layers). Every layer left on the CPU costs an order of magnitude in tokens/s. Check VRAM with nvidia-smi mid-generation — not at load time.
Rule: weights + KV cache ≤ ~90% of VRAM. If it doesn't fit, drop the quant or the context before you drop layers — spilling layers to the CPU is the VRAM cliff, and it costs an order of magnitude.
2. Pick the quant for the model size, not for the bragging rights
| Model class | Default pick | Why |
|---|---|---|
| ≤ 8B | Q6_K / Q8_0 | Quality loss is ~0, VRAM headroom is cheap |
| 14B–32B | Q5_K_M / Q4_K_M | Sweet spot; Q4_K_M at 27–32B is still strong |
| 70B | Q4_K_M | Q3 starts to hurt; Q5 needs 56GB+ |
| MoE (A3B/A22B class) | Q4_K_M / FP8 | Small active params → big context headroom |
Quant choice is part of the benchmark's identity — our board records it per row, and the same model at Q4 vs Q8 is a different result.
3. Enable flash attention
-fa 1 (on by default in recent builds). Cuts KV memory and usually adds a few percent to generation speed. No downside for standard contexts.
4. Size the context to the job
Context is the silent VRAM hog: a 27B Q4 model at 4K context is a different VRAM footprint than at 32K. If you don't need long context, cap it (-c 4096) and the freed VRAM buys you a bigger model or batch.
5. Batch and prompt handling
-b(batch) and-ub(upload batch) mostly affect prompt processing speed, not generation tok/s.- For agent workloads, raising batch helps prompt reprocessing; for chat, defaults are fine.
6. Measure before and after
Run llama-bench with the exact commit and quant you'll ship:
llama-bench -m model.gguf -ngl 999 -fa 1 -n 512 -b 512
Record: commit, quant, context, GPU, driver. That tuple is a benchmark row — submit it to our board and it gets attributed to your test.
What doesn't move the needle (despite the forums)
- "Undervolting for speed" — power draw changes, throughput doesn't.
- Swapping PCIe slot positions — negligible after model load.
- Exotic schedulers/overlays — measure them; they rarely beat a correct baseline.
Related
- Quant choice is half the battle: quantization guide
- Which card for which quant: compatibility matrix
Sources
- llama.cpp documentation: https://github.com/ggml-org/llama.cpp
- Real-world runs with methodology: DiggTop benchmarks · methodology