Quantization
Fewer bits per weight, more model per card — and what it costs in quality.
Quantization is the single biggest lever on which models fit: 4-bit weights are roughly a third the size of FP16, which is the difference between a 70B model on one 48 GB card and nothing. These pieces compare the formats that matter in practice, from GGUF k-quants to NVFP4 and FP8 on newer silicon.
Quantization for Local LLMs: Q4_K_M vs NVFP4
How quantization cuts VRAM for local LLM inference: GGUF K-quants, IQ quants and NVFP4 compared with community data from real GPU runs.
- Sep 15, 2026 · 10 min readX local-model bench digest — Sep 15, 2026Local-LLM throughput measured on X: 23 posts and 78 runs, with hardware, quantization, engine, context and source links.
- Sep 15, 2026 · 6 min read3.5 Bits Beat 4 Bits: Formats, Not PrecisionEXL3 at 3.5 bpw outran MXFP4 by 2× on the same 552B MoE. Three clean ladders where only the format changed — and how to pick a quantization.
- Oct 16, 2026 · 10 min readX local-model bench digest — Oct 16, 2026Local-LLM throughput measured on X: 27 posts, 54 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 8, 2026 · 10 min readX local-model bench digest — Oct 8, 2026Local-LLM throughput measured on X: 25 posts and 69 runs, with hardware, quantization, engine, context and source links.
- Oct 2, 2026 · 11 min readX local-model bench digest — Oct 2, 2026Local-LLM throughput measured on X: 47 posts, 137 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 21, 2026 · 9 min readX local-model bench digest — Sep 21, 2026Local-LLM throughput measured on X: 16 posts and 49 runs, with hardware, quantization, engine, context and source links.
- Sep 19, 2026 · 10 min readX local-model bench digest — Sep 19, 2026Local-LLM throughput measured on X: 25 posts, 91 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 30, 2026 · 10 min readX local-model bench digest — Sep 30, 2026Local-LLM throughput measured on X: 32 posts, 90 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 29, 2026 · 11 min readX local-model bench digest — Sep 29, 2026Local-LLM throughput measured on X: 37 posts, 111 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 17, 2026 · 10 min readX local-model bench digest — Sep 17, 2026Local-LLM throughput measured on X: 30 posts and 68 runs, with hardware, quantization, engine, context and source links.
- Sep 16, 2026 · 9 min readX local-model bench digest — Sep 16, 2026Local-LLM throughput measured on X: 18 posts, 48 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 13, 2026 · 9 min readX local-model bench digest — Sep 13, 2026Local-LLM throughput measured on X: 14 posts, 51 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 7, 2026 · 6 min readQwen3.8-27B Uncensored: MTP Kept, Cost MeasuredHeretic abliteration on Qwen3.8-27B: refusals 98 to 12 per 100 prompts, ~0.5 capability points cost, MTP tensors kept. PPL, sizes and tok/s, linked.
- Oct 7, 2026 · 11 min readX local-model bench digest — Oct 7, 2026Local-LLM throughput measured on X: 35 posts and 83 runs, with hardware, quantization, engine, context and source links.
- Oct 1, 2026 · 10 min readX local-model bench digest — Oct 1, 2026Local-LLM throughput measured on X: 28 posts, 97 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 28, 2026 · 8 min readQwen3.8-27B on 22 GPUs: 44 Sourced RunsOne model, 22 GPUs, 44 sourced runs, 16.8 to 435 tok/s. The card matters less than the quant, the engine and the context you run it at.
- Sep 26, 2026 · 9 min readX local-model bench digest — Sep 26, 2026Local-LLM throughput measured on X: 27 posts, 74 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 22, 2026 · 10 min readX local-model bench digest — Sep 22, 2026Local-LLM throughput measured on X: 29 posts, 79 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 20, 2026 · 9 min readX local-model bench digest — Sep 20, 2026Local-LLM throughput measured on X: 21 posts and 85 runs, with hardware, quantization, engine, context and source links.
- Sep 12, 2026 · 9 min readX local-model bench digest — Sep 12, 2026Local-LLM throughput measured on X: 25 posts, 81 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 14, 2026 · 11 min readX local-model bench digest — Oct 14, 2026Local-LLM throughput measured on X: 34 posts and 111 runs, with hardware, quantization, engine, context and source links.
- Oct 11, 2026 · 9 min readX local-model bench digest — Oct 11, 2026Local-LLM throughput measured on X: 27 posts and 66 runs, with hardware, quantization, engine, context and source links.
- Oct 9, 2026 · 10 min readX local-model bench digest — Oct 9, 2026Local-LLM throughput measured on X: 32 posts, 79 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 4, 2026 · 10 min readX local-model bench digest — Oct 4, 2026Local-LLM throughput measured on X: 28 posts, 60 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 28, 2026 · 9 min readX local-model bench digest — Sep 28, 2026Local-LLM throughput measured on X: 17 posts and 68 runs, with hardware, quantization, engine, context and source links.
- Sep 28, 2026 · 9 min readOne 180B MoE, Five Formats: 41 Sourced RunsQwen3.8-Flash-Next's split entities merged into one page: 41 sourced rows across three checkpoints, 20.3 to 474 tok/s, and what fits a 24GB card.
- Sep 25, 2026 · 9 min readX local-model bench digest — Sep 25, 2026Local-LLM throughput measured on X: 23 posts, 51 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 23, 2026 · 8 min readX local-model bench digest — Sep 23, 2026Local-LLM throughput measured on X: 20 posts and 56 runs, with hardware, quantization, engine, context and source links.
- Sep 15, 2026 · 7 min readTwo RTX 3060s: 16 tok/s Dense, 94 tok/s MoEThe same dual-3060 rig runs a 30B dense model at 16 tok/s and a 30B mixture-of-experts at 94. That 6x spread decides whether the build is worth it.
- Sep 5, 2026 · 5 min readRTX 5090 vs RTX 3090 for Local LLMsRTX 5090 vs used RTX 3090 for running LLMs locally: community tok/s data, VRAM math, street prices and which card actually wins.
- Oct 15, 2026 · 10 min readX local-model bench digest — Oct 15, 2026Local-LLM throughput measured on X: 34 posts and 84 runs, with hardware, quantization, engine, context and source links.
- Oct 13, 2026 · 10 min readX local-model bench digest — Oct 13, 2026Local-LLM throughput measured on X: 33 posts and 89 runs, with hardware, quantization, engine, context and source links.
- Oct 6, 2026 · 10 min readX local-model bench digest — Oct 6, 2026Local-LLM throughput measured on X: 34 posts, 83 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 3, 2026 · 10 min readX local-model bench digest — Oct 3, 2026Local-LLM throughput measured on X: 26 posts and 88 runs, with hardware, quantization, engine, context and source links.
- Sep 28, 2026 · 9 min readCMP 170HX, B70, V100: Cheap Cards Punching UpSame model, same W4A16 quant, same vLLM engine: a $1,050 CMP 170HX runs 255.9 tok/s and a $9,000 RTX 5090 gets 137.9.
- Sep 18, 2026 · 10 min readX local-model bench digest — Sep 18, 2026Local-LLM throughput measured on X: 30 posts and 77 runs, with hardware, quantization, engine, context and source links.
- Oct 18, 2026 · 8 min readX local-model bench digest — Oct 18, 2026Local-LLM throughput measured on X: 10 posts and 34 runs, with hardware, quantization, engine, context and source links.
- Oct 17, 2026 · 9 min readX local-model bench digest — Oct 17, 2026Local-LLM throughput measured on X: 29 posts, 67 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Oct 10, 2026 · 9 min readX local-model bench digest — Oct 10, 2026Local-LLM throughput measured on X: 29 posts, 74 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 27, 2026 · 10 min readX local-model bench digest — Sep 27, 2026Local-LLM throughput measured on X: 26 posts, 80 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 5, 2026 · 3 min readBest GPU for Qwen3: VRAM Tiers per ModelWhich GPU runs which Qwen3 model — 0.5B to 180B mapped to VRAM tiers, with real community tok/s from 143 sourced DiggTop benchmark runs.
- Sep 4, 2026 · 3 min readllama.cpp Optimization Cheat SheetThe settings that actually move llama.cpp throughput: quant choice, offload, flash attention, context, batch — with the flags and the reasoning.
- Oct 12, 2026 · 9 min readX local-model bench digest — Oct 12, 2026Local-LLM throughput measured on X: 40 posts and 109 runs, with hardware, quantization, engine, context and source links.
- Oct 5, 2026 · 10 min readX local-model bench digest — Oct 5, 2026Local-LLM throughput measured on X: 25 posts and 83 runs, with hardware, quantization, engine, context and source links.
- Sep 14, 2026 · 8 min readX local-model bench digest — Sep 14, 2026Local-LLM throughput measured on X: 12 posts, 25 runs — hardware, quantization, engine and tok/s with a source link for every number.
- Sep 5, 2026 · 3 min readThe 48GB VRAM Local LLM BuildHow to get 48GB of VRAM for 70B-class local LLMs: dual RTX 3090 vs used A6000 48GB — power, cooling, PCIe and real community tok/s.
- Sep 24, 2026 · 8 min readX local-model bench digest — Sep 24, 2026Local-LLM throughput measured on X: 15 posts and 44 runs, with hardware, quantization, engine, context and source links.
- Sep 4, 2026 · 4 min readBudget Build Monthly — September 2026First monthly build review: three tiers vs. the 2026-09-04 price snapshot, plus value picks — Intel's 24GB at $599 and AMD's 32GB at $1,329.
- Sep 4, 2026 · 8 min readLocal LLM GPU Buying Guide (2026)What to buy to run local LLMs in September 2026, chosen by VRAM tier with current US prices, used-market alternatives and honest tradeoffs.
- Sep 11, 2026 · 9 min readMoE Expert Offload: Same Weights, 26× ApartSame weights, same box: 3.3 tok/s with whole-layer expert offload vs 150 with row-selective UVA reads, and what decides big-MoE speed on consumer GPUs.
- Sep 5, 2026 · 3 min readQwen3.8-27B on a Single RTX 3060 (12GB)Qwen3.8-27B on a 12GB RTX 3060: two IQ2_XS community builds — 158K context at 22 tok/s, or 82K with MTP draft at 38 tok/s. Full llama.cpp configs.