ARK 1.0 to ARK 2.0, full load, median of 15 models
9.3×
total throughput: 294 to 2,733 tokens per second
−90%
energy per token: 0.95 to 0.10 Wh per 1,000 tokens
1.74×
faster per stream, while serving 64 streams instead of 12

How it works

Decoding is limited by memory, not arithmetic: each step streams the model's weights through the GPU, and with a single request most of the GPU's compute sits idle while it waits for them. In a batched step, every request in flight shares that one pass over the weights. Adding a request adds a little arithmetic, not another pass. The GPU's spare compute becomes throughput.

Throughput that scales

On ARK 2.0, total throughput keeps rising as streams are added. Qwen3.6-35B-A3B on one machine with two RTX 5090, measured through the API: 87 tokens per second for one stream, 1,341 for sixty-four, at a 4,000-token context.

Total decode throughput of ARK 2.0 as concurrent streams grow, Qwen3.6-35B-A3B on two RTX 5090 0 500 1,000 1,500 2,000 1 4 8 16 24 32 48 64 Concurrent streams Tokens per second, all streams 1,599 1K context 1,341 4K context
Total decode throughput of ARK 2.0 by number of concurrent streams, at 1,000- and 4,000-token contexts. Qwen3.6-35B-A3B on two RTX 5090, through the HTTP API.

ARK 1.0 against ARK 2.0

ARK 1.0 was tested at up to 12 concurrent streams on our four-GPU test system; ARK 2.0 at 64. At those loads, every model's total throughput rose between 6.8 and 12.5 times, and the energy each token costs fell by 81 to 93%: the machines draw about the same power while producing nine times the tokens.

Total throughput at full load, ARK 1.0 (12 streams) and ARK 2.0 (64 streams) 0 1,250 2,500 3,750 5,000 480 3,273 gpt-oss-20b 424 3,596 Qwen3-8B 345 2,927 Qwen3-14B 204 1,940 Qwen3-32B 173 1,717 Qwen3.5-122B 235 2,687 Qwen3.6-35B 114 1,391 Llama-3.3-70B Tokens per second, all streams ARK 1.0 ARK 2.0
Total throughput at full load, ARK 1.0 (12 streams) and ARK 2.0 (64 streams), seven of the fifteen models.
ModelARK 1.0, tok/sARK 2.0, tok/sGainWh per 1k tokens
Qwen3.6-27B1361,69712.5×2.49 → 0.19
Qwen3.8-27B1371,69412.4×2.49 → 0.19
Llama-3.3-70B1141,39112.2×4.78 → 0.34
Qwen3.5-4B3063,54611.6×0.72 → 0.07
Qwen3.6-35B2352,68711.4×0.91 → 0.09
Qwen3.5-122B1731,7179.9×1.56 → 0.19
Qwen3-32B2041,9409.5×2.20 → 0.19
Devstral-24B2942,7339.3×1.51 → 0.14
Ministral-3-8B4443,8198.6×0.70 → 0.07
Qwen3-Coder-30B3883,2958.5×0.58 → 0.07
Qwen3-14B3452,9278.5×1.14 → 0.10
Qwen3-8B4243,5968.5×0.68 → 0.07
Ministral-3-14B3813,2288.5×0.95 → 0.10
GLM-4.7-Flash2351,8788.0×0.95 → 0.14
gpt-oss-20b4803,2736.8×0.47 → 0.09

These are release-to-release figures: ARK 2.0's throughput comes from batching together with fused kernels, which keep a batched step fast for quantized models, on top of inverted processing loops, which made batching possible.

What it costs

  • Speed per stream falls as streams are added. In the scaling run above, one stream decodes at 86.8 tokens per second; with 32 streams each gets 38.3, with 64 each gets 20.9. Total throughput is what grows.
  • Long prompts arriving together wait for each other before their first token. That part of the pipeline is the subject of our prefill work.

How we measured

  • Release comparison. The ARK release benchmark on four RTX 5090 in two machines: ARK 1.0.0 on 22 September 2026 at 12 streams, the most it was tested at (10 for Qwen3.5-4B), and ARK 2.0.0 on 1 October 2026 at 64 streams. Each variation runs about two minutes, once.
  • Energy. Watt-hours per 1,000 tokens from the machines' power draw, CPUs included.
  • Scaling. Qwen3.6-35B-A3B on one machine with two RTX 5090, ARK 2.0.0, through the HTTP API, 1 October 2026; no request failed. Total throughput is each stream's decode speed times the number of streams.

Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching

← All research · The ARK 2.0 release benchmark