ARK 2.0 to ARK 2.1, Qwen3.5-122B-A10B
1.8×
faster first token for a 64,000-token prompt: 8.4 to 4.7 seconds
1.6×
combined prompt reading with 32 prompts of 16,000 tokens at once: 9,980 to 15,607 tokens per second
191/192
answered within 90 seconds when 64 prompts of 16,000 tokens arrive together, against 50

What changed

A model as large as Qwen3.5-122B-A10B is split across two machines, and every part of a prompt has to pass through the GPUs of both. ARK 2.1 reads the prompt in pieces that flow through those GPUs like an assembly line: while one GPU works on a piece, the next one works on the piece before it. Hand-overs between the machines cost less, and the gaps where a GPU waited for its next piece are gone. Every GPU stays busy while a prompt is read.

One prompt

The longer the prompt, the larger the gain: a 4,000-token prompt gets its first token in 0.5 seconds instead of 0.64, a 16,000-token prompt in 1.2 instead of 2.0, and a 64,000-token prompt in 4.7 instead of 8.4.

Time to first token for one prompt, by prompt length, ARK 2.0 and ARK 2.1 0 2.5 5 7.5 10 0.6 0.5 4K tokens 2.0 1.2 16K tokens 8.4 4.7 64K tokens Seconds (lower is better) ARK 2.0 ARK 2.1
Time to first token for a single prompt (lower is better), ARK 2.0 and ARK 2.1. Qwen3.5-122B-A10B on four RTX 5090 in two machines.

Many prompts at once

When prompts arrive together, what counts is the combined rate at which the deployment reads them. With 16,000-token prompts, ARK 2.0 tops out at about 10,000 prompt tokens per second, however many arrive. ARK 2.1 reads 15,600 to 16,400 per second from two prompts upward, so the last prompt of a burst gets its first token much sooner: with 32 prompts at once, after 33 seconds instead of 52.

Combined prompt-reading rate for 16,000-token prompts arriving together, ARK 2.0 and ARK 2.1 0 5,000 10,000 15,000 20,000 8,225 14,082 1 at once 9,364 16,427 2 at once 9,630 16,437 4 at once 9,814 16,250 8 at once 9,866 15,840 16 at once 9,980 15,607 32 at once Prompt tokens per second, all prompts ARK 2.0 ARK 2.1
Combined prompt-reading rate when 1 to 32 prompts of 16,000 tokens arrive at the same moment, ARK 2.0 and ARK 2.1.
Prompt × at onceARK 2.0, tok/sARK 2.1, tok/sGainSlowest TTFT, s
4K × 16,2988,1561.29×0.6 → 0.5
4K × 27,95911,7431.48×1.0 → 0.7
4K × 48,84312,3221.39×1.8 → 1.3
4K × 89,35511,4671.23×3.5 → 2.9
4K × 169,57811,5351.20×6.8 → 5.6
4K × 329,63411,2191.16×13.6 → 11.6
4K × 649,67311,0381.14×27.0 → 23.6
16K × 18,22514,0821.71×2.0 → 1.2
16K × 29,36416,4271.75×3.5 → 2.0
16K × 49,63016,4371.71×6.8 → 4.0
16K × 89,81416,2501.66×13.3 → 8.0
16K × 169,86615,8401.61×26.5 → 16.5
16K × 329,98015,6071.56×52.2 → 33.4
16K × 64within 90 s: 50 → 191 of 192–
64K × 17,71613,7941.79×8.4 → 4.7
64K × 28,52014,2601.67×15.3 → 9.1
64K × 48,59013,6821.59×30.4 → 19.1
64K × 88,38313,8121.65×62.1 → 37.8
64K × 16within 90 s: 0 → 10 of 48–

At the limit

The heaviest bursts put about one million prompt tokens on the deployment at once, as much as its attention memory holds. Both runs used a 90-second limit on how long the API waits for a request's first token. With 64 prompts of 16,000 tokens, ARK 2.0 answered 50 of 192 requests within it; ARK 2.1 answered 191, the last after 66 seconds, and the API's limit on concurrent requests refused the one remaining. With 16 prompts of 64,000 tokens, neither version reads everything within 90 seconds: ARK 2.0 answered none of 48 requests in time, ARK 2.1 answered 10.

What this does not show

  • The first long prompt after a server starts takes longer while the GPUs prepare for the new piece size: on this model, 12.3 seconds instead of 4.7 for a 64,000-token prompt, once.
  • Long answers generated at the same time. Each request here generates only 8 tokens. How conversations that are writing long answers fare while prompts are read is a separate measurement.

How we measured

  • Deployment. Qwen3.5-122B-A10B quantized to 4 bits, split over two machines with two RTX 5090 each, an 8-bit attention cache with room for 1,048,576 tokens per server, the same API, inference engine and settings in both runs. Only the ARK runtime changed. We confirmed from the servers' logs which build served each run.
  • Runs. ARK 2.0 and ARK 2.1 on 20 October 2026, one after the other on the same machines.
  • Prompts. Bursts of 1 to 64 requests released at the same moment, with prompts of 4,000, 16,000 and 64,000 tokens through the HTTP API. Every prompt is different from its first line, so nothing could come from the prompt cache; the API reported 0 cached tokens throughout. Both runs read exactly the same prompts.
  • Measured. Time to first token per request. The combined rate is the prompt tokens of all answered requests divided by the time until the last of them got its first token. Each cell ran three times; the table shows medians, and TTFT is the time to first token of the slowest request in the burst. Bursts above one million tokens (64,000-token prompts, 32 and 64 at once) were not run.

Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching

← All research