Context Is Not a Cliff: 790× the Window, +18%

Ask anyone who runs local models what happens when the context window fills up and you get one answer: it gets slower. Ask the benchmark rows on this site and you get something stranger. On a single DGX Spark running Qwen3.8-Flash-Next (180B) at NVFP4 under vLLM, we have ten measurements spanning a context window from 327 to 258,790 tokens — a 790× range. Decode throughput across that range goes from 37.4 to 44.2 tok/s. The largest window is the fastest row in the set.

That is not a typo, and it is not one lucky run. It is the same model, the same checkpoint, the same engine, and the same box — the only variable that changes between rows is the context length the run was configured with.

As of 2026-09-15. Every number below is an imported, sourced row on our board, with the engine version and context attached; we did not run these tests ourselves. Sources are linked at the end.

The flat line, in full

From 8K to 258K — a 31× increase in the window — the spread is 41–44 tok/s. That is measurement noise, not a trend. The 327-token row, the only one that stands apart, is slower, which is the opposite of the folk claim.

Why this is less mysterious than it looks

Decode throughput is not one number. It is the sum of two costs that scale very differently:

• Weight traffic — every token reads the model's active weights. For a dense model that is the full parameter count; for a mixture-of-experts model it is only the routed experts, but either way it is a fixed cost per token that does not care how long your conversation is.
• KV cache traffic — every token reads the attention keys and values for everything before it. This does grow linearly with context.

At 327 tokens the KV cache is too small to matter and per-token overheads (kernel launches, sampling, the scheduler) dominate — which is why the short row is the slow one. Somewhere in the thousands the pipeline warms up and the number settles. Past that, weights are still the bulk of the traffic, and the KV stream grows into a rounding error rather than a bottleneck. On this model, at 258K tokens, the KV traffic has grown enough to matter — and it still does not move the number outside noise.

That is the part most benchmark tables get wrong. A row that quotes 180B at 40 tok/s and omits the context tells you nothing about whether the same rig will do 40 tok/s in your 100K-token agent session. The good news from this data set is that it will.

Where context does cost you

It costs you in two other places, and they are the ones people feel.

Prefill — the cost of reading the prompt. Filling a 200K-token context is a dense matrix operation over the whole prompt: it is the one part of the pipeline that scales with context length and cannot hide behind a fixed weight stream. We have measured prefill ladders separately in the board's prompt tok/s column precisely because a long-context workload is dominated by prefill, not decode. If a run reports 40 tok/s decode and your agent reads 100K tokens before answering, your wall-clock time is the prefill, not the decode.

Speculative decoding, whose benefit is context-dependent. The second data set on our board makes this visible. On DGX Spark ×2 running the same 552B-class DeepSeek-V4.1-Flash under EXL3 3.5bpw with DSpark speculative decoding (k=3, TP=2) — same engine version, same day:

Same box, same weights, same engine — a 5× drop from 4K to 17K, and then… almost nothing. From 17K to 559K (a 33× window increase) the number moves 6%. The cliff is in the short range, and it is a speculative-decoding cliff: the helper drafts tokens ahead, and its acceptance rate depends on how much the model can lean on local pattern repetition. Short contexts decode into a well-primed pattern and the drafter lands often; past ~17K the acceptance rate falls to something steady, and from there the per-token cost is dominated by the fixed weight stream again — the same regime as the flat line above.

So the "long context is slow" intuition is really two effects wearing one coat: prefill scales, decode mostly does not, and speculation gains depend on where you are in the prompt. Neither is visible if your benchmark row only carries a tok/s number.

What to do with this

• When you read a tok/s figure, find the context first. On our board it is a column, and the entity pages show it per row. Two rows for the same card and model can differ 5× with the same weights on the same silicon.
• Judge long-context workloads by prefill, not decode. If your prompt is 100K tokens, the question "how fast does it generate" is secondary to "how long until the first token".
• Treat speculative decoding as a workload optimisation, not a hardware one. It is a 2–4× win on short, patterned generations and close to a wash on long context — and it can be a loss on prose, as the GB300 test in our MoE offload coverage showed: 10–18% slower at k=5 on real text.
• A single point is not a curve. Both sets here come from one submitter each, on one machine each. They are cleanly specified, which is why they are on the board, but they are one machine's behaviour. The pattern is what generalises; the exact numbers are that box.

FAQ

Does local LLM decode speed drop as context grows?

Not materially, on current engines — decode reads the model's weights every token at a fixed cost, while the KV cache it also reads stays small relative to that until extreme lengths. On our DGX Spark Qwen3.8-Flash-Next data, an 8K to 258K increase in the window left throughput flat at 41–44 tok/s.

Why does the shortest context show the slowest throughput?

Per-token overheads — scheduling, kernel launches, sampling — are a fixed cost per token that is amortised better on longer runs. The 327-token row at 37.4 tok/s is the only point in that set outside the noise band, and it is the short one.

What actually gets slower with long context?

Prefill, the phase that reads your prompt before the first token comes out. It scales with prompt length and dominates wall-clock time for long-context workloads. The board records prompt tok/s separately from output tok/s for that reason.

Is speculative decoding worth it for long-context work?

It depends on acceptance, which falls as context grows. The EXL3 + DSpark ladder on the 552B DeepSeek model shows 154 tok/s at 4K falling to 30–35 tok/s and staying there through 559K.

Why does our board show such different tok/s for the same GPU and model?

Because the rows differ in engine, quantization and context — and sometimes all three. The RTX 5090 rows for one 27B model span 16.8 to 435 tok/s across vllm, llama.cpp, tensorrt-llm and a custom engine, at contexts from 2K to 251K. Comparing across engines is comparing nothing; the methodology page explains which fields we require before a row is eligible.

Sources

• Context ladder, DGX Spark × Qwen3.8-Flash-Next NVFP4 under vLLM (10 rows, 327 → 258,790 tokens): imported and sourced on the board, DGX Spark entity page.
• Speculative-decoding ladder, DGX Spark ×2 × DeepSeek-V4.1-Flash EXL3 3.5bpw, exl3 v1.4.2, TP=2, DSpark k=3 (7 rows, 1K → 559K), source: @vr8vr8.
• Model and hardware pages: Qwen3.8-Flash-Next · DeepSeek-V4.1-Flash · full benchmark board.

Imported upstream measurements as of 2026-09-14. DiggTop did not run these tests; each row carries its engine version, quantization and context, and states the fields the methodology rules require.