What changed
A request on ARK passes through every GPU that holds a part of the model, once for each token it generates. How those steps are driven decides how much of the time the GPUs compute and how much they wait. ARK 2.0 inverts the processing loops that drive them, so the GPUs spend their time computing. The details stay with us; the effect is measurable, and it is below.
What it improved
Every chat model in the release benchmark decodes faster on a single stream, from 1.13× (Qwen3-32B) to 2.55× (Llama-3.2-1B). The gain is largest where the GPU's own work per token is smallest: small models and mixture-of-experts models such as gpt-oss-20b and Qwen3.6-35B-A3B, where the time saved between steps is a larger share of every token.
| Model | ARK 1.1, tok/s | ARK 2.0, tok/s | Speed-up |
|---|---|---|---|
| Llama-3.2-1B | 237.0 | 603.4 | 2.55× |
| Llama-3.2-3B | 170.2 | 326.8 | 1.92× |
| gpt-oss-20b | 139.4 | 256.1 | 1.84× |
| Qwen3-Coder-30B | 103.1 | 174.0 | 1.69× |
| Qwen3.5-4B | 89.2 | 143.9 | 1.61× |
| Llama-3.1-8B | 130.6 | 209.1 | 1.60× |
| Ministral-3-8B | 127.3 | 199.1 | 1.56× |
| Qwen3.6-35B | 65.7 | 102.1 | 1.55× |
| Qwen3-8B | 118.3 | 172.5 | 1.46× |
| Ministral-3-14B | 98.3 | 140.2 | 1.43× |
| Qwen3-14B | 90.4 | 117.7 | 1.30× |
| Devstral-24B | 72.3 | 92.7 | 1.28× |
| Qwen3.8-27B | 38.1 | 48.1 | 1.26× |
| Qwen3.6-27B | 38.2 | 48.2 | 1.26× |
| GLM-4.7-Flash | 63.2 | 76.0 | 1.20× |
| Qwen3-32B | 52.2 | 58.8 | 1.13× |
Faster tokens also cost less energy: on one stream, the median model uses 23% less energy per token than on ARK 1.1.
What it did not change
- Many concurrent streams. On its own, the new loop structure did not raise total throughput under load. It was the precondition for decode batching, which did.
- Reading the prompt. Prefill speed was unchanged by this work.
How we measured
- Releases. ARK 1.1.0 and ARK 2.0.0, measured on 1 and 2 October 2026 on the same test system: four RTX 5090 in two machines. Both releases ran the same model files and the same version of the inference engine; each model had the same GPUs in both runs.
- Measured. Decode speed of one stream over a run of about two minutes, from the release benchmark's single-stream variation. Energy per token from the machines' power draw.
- Models. The 16 chat models that ran on the same number of GPUs in both releases. Qwen3.5-122B-A10B and Llama-3.3-70B ran on different GPU counts and are left out.
- Other changes. ARK 2.0 also brought decode batching and fused kernels. They raise throughput with many concurrent streams and do not speed up a single one; the single-stream gain is the loops'.
- Noise. One run per model, so differences of a few percent are within noise.
Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching