Mixed lengths at full speed
Real traffic mixes short chats with long documents. Before paging, 14 streams spread over seven lengths, from 1,000 to 64,000 tokens, decoded at 12.2 tokens per second each, against 30.4 when all 14 had the same length. With paging the mixed case runs at 31.4: as fast as uniform traffic, which itself is unchanged.
The same effect on one GPU, measured as the time one decoding step takes with requests at 512 to 8,000 tokens of context:
| Model, concurrent requests | Before, ms per step | Paged, ms per step | Faster |
|---|---|---|---|
| Qwen3-8B, 4 | 25.1 | 7.2 | 3.5× |
| Qwen3-8B, 16 | 26.6 | 10.7 | 2.5× |
| Qwen3.6-35B-A3B, 4 | 39.7 | 10.4 | 3.8× |
| Qwen3.6-35B-A3B, 16 | 41.2 | 14.2 | 2.9× |
Memory reserved closer to need
Pages are small, so a request reserves little more than it asks for. Across the 23,536 requests of our API test suite, the memory reserved exceeded what the requests asked for by 3%. Under the previous scheme the same requests would have taken 15% extra. On GLM-4.7-Flash, servers could now fill their whole cache with full-length requests, which the previous build refused.
What it costs
- A little per step. In our 21-model release sweep, single-stream decoding is 4 to 8% slower on most dense models. The hybrid Qwen3.5, Qwen3.6 and Qwen3.8 models are unchanged, and the two 27B hybrids gain 4 to 5% at 64 streams.
- Three models are slower. On Qwen3-14B, Llama-3.2-3B and GLM-4.7-Flash, whose attention layout our fastest paged kernel does not cover yet, decoding is 13 to 31% slower. Closing that gap is ongoing work.
How we measured
- Mixed lengths. Qwen3.8-27B, 4-bit with an 8-bit attention cache, one server on two RTX 5090. 14 concurrent streams sent to the server directly, without the API: two each at maximum lengths of 1K, 2K, 4K, 8K, 16K, 32K and 64K tokens, a short prompt and up to 935 generated tokens, greedy decoding. The uniform case runs all 14 at 8K. The last build before paging and the first paged build, 13 October 2026, one run per case.
- One GPU. Qwen3-8B and Qwen3.6-35B-A3B on one RTX 5090, the same day, comparing the paged attention path with the previous one.
- Memory. Computed from the reservations logged by the paged build during the API test suite, with the previous scheme's rule applied to the same requests. The suite's median request is 924 tokens.
- Sweep. 21 models on four RTX 5090, the same two builds on the same day, one run per model.
Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching