Benchmark Methodology

Every number on this site is either measured by us or attributed to a public source — with the engine, version, quantization and context attached. This page defines the bar.

What we record

Each benchmark row must carry:

• Hardware — exact GPU (and count, if multi-GPU)
• Model + quantization — e.g. Qwen3-27B @ Q4_K_M
• Backend and version — llama.cpp commit / build, vllm version, mlx version
• Context length — the prompt + generation window actually used
• Output tok/s (mean) and, where reported, prompt tok/s, peak VRAM
• Source — our own test, a user submission, or an import (with the original URL)

What the row labels mean

Every label below is derived from what the row itself contains. Nothing on this site is labelled on the strength of a review, because there is no review step: a row is stored exactly as its author stated it, with the source link attached.

• ✓ sourced — the row names the engine and its version, the quantization, the context length and a specific hardware configuration, and links to the author's original post or run page. That link is the check: open it and read the number yourself. 336 rows — 334 attributed claims plus 2 measurements of our own.
• self-measured — DiggTop ran it on its own hardware. Those 2 rows are documented on self-measured runs, including the parameters we failed to record.
• … partial — the row is on the board and in nothing else: excluded from every average, ranking and count. 132 rows, for one of two reasons, and the badge names which: a required field is missing (115 rows state no engine version; 6 sit on hardware that is not identified), or the row states every field but was left unpromoted because nothing corroborates it (13 rows, including a 1,392 tok/s CMP 170HX claim).

What these labels are not. DiggTop does not re-run other people's measurements, so none of them means "independently confirmed". Of the 468 rows on the board (2026-10-07): 334 sourced claims, 2 of our own, 132 partial — and the attributed ones come from exactly two places, localmaxxing (361 rows) and public X posts (105). We cannot confirm a third-party number by finding it twice either. Only 4 (model, GPU) combinations on the board have two independent sources at all, and none of them agree closely enough to call one another confirmed; control for quantization, engine and context — the three things that actually move throughput — and the number of configurations with a second independent source is zero. There is no pair of rows on this board that corroborates another. That is the whole reason every row links to its author instead of carrying a seal. The fastest number on the board (4,546.8 tok/s) is partial for exactly this reason — the source never stated an engine version.

Read ✓ as "a complete, dated, reproducible claim with a source", not as "we checked it".

If you can re-run one of these rows, submit it: a second independent source is the only thing that would let us mark anything confirmed.

How imported data is handled

We import from public datasets (currently localmaxxing) with per-row attribution: every imported row links back to its original run page. We keep the source's own wording for its status and never convert it into a DiggTop claim. We do not re-label third-party numbers as our own tests.

What we deliberately do not do

• No anonymous "top 100" without methodology links.
• No blended numbers across different contexts, quantizations, or batch sizes.
• No scraping of vendor claim pages as "measured" results — vendor claims are labeled as claims.
• No "independently verified" badge: we cannot independently verify a measurement we did not run.

Reproducing a run

If you want to reproduce a row, the backend version + quantization + context length in the table give you the starting point. For llama.cpp, the relevant command is:

llama-bench -m model.gguf -ngl 100 -n 512 -b 512

Matching the exact commit matters: throughput changes between releases.