ARK and vLLM on the same GPUs

ARK 2.1 against vLLM 0.27.1, 4,000-token prompts, 64 users

2.5×

total decode throughput: 1,655 against 652 tokens per second

7.0×

sooner first token: 6.8 against 47.9 seconds (median)

failed requests on either engine in every cell published here

Throughput as users are added

With one user vLLM generates faster, 160 tokens per second against 88, and at four users it is still 11% ahead. From 8 users, ARK's total keeps climbing while vLLM's levels off: at 32 users ARK generates 1,296 tokens per second against vLLM's 601, at 64 users 1,655 against 652.

Total decode throughput, all users together, by number of concurrent users. Qwen3.6-35B-A3B, 4,000-token prompts, two RTX 5090.

First token

ARK reads the prompts faster at every load: the median first token comes 4.7 to 7.1 times sooner across both prompt lengths, from 0.33 against 1.58 seconds for a single user to 6.8 against 47.9 seconds with 64 users at 4,000 tokens. With its default settings vLLM takes a burst of new prompts in about one at a time, so its first-token time grows in step with the number of users.

Median time to first token by number of concurrent users (lower is better), 4,000-token prompts.

Where vLLM is ahead

A single user decodes faster on vLLM: 160 tokens per second against ARK's 88 at 4,000 tokens. vLLM keeps a per-user lead up to 4 users at 4,000 tokens and up to 16 users at 16,000. ARK is built for many users sharing the GPUs: beyond those points, each ARK user gets more tokens per second than on vLLM.

Decode speed each user sees, by number of concurrent users, 4,000-token prompts. Logarithmic scale.

All results

4,000-token prompts

Users	First token, ARK	vLLM	Per user, ARK	vLLM	Total, ARK	vLLM
1	0.33 s	1.58 s	88.3	159.6	88	160
4	0.63 s	4.05 s	78.4	86.7	313	347
8	1.08 s	7.01 s	64.5	55.1	516	441
16	1.95 s	12.88 s	47.7	31.9	764	511
24	2.78 s	18.73 s	44.3	23.6	1,064	567
32	3.53 s	24.56 s	40.5	18.8	1,296	601
48	5.11 s	36.24 s	31.5	13.0	1,511	624
64	6.82 s	47.92 s	25.9	10.2	1,655	652

16,000-token prompts

Users	First token, ARK	vLLM	Per user, ARK	vLLM	Total, ARK	vLLM
1	0.88 s	6.07 s	83.0	159.2	83	159
4	2.57 s	15.20 s	65.6	96.1	262	384
8	4.85 s	27.03 s	50.3	63.7	403	509
16	9.35 s	50.63 s	30.9	39.6	494	633
24	13.87 s	74.21 s	31.8	29.3	762	703

First token: median seconds. Per user and total: decode tokens per second. At 16,000 tokens the benchmark ran up to 24 users; there vLLM leads on total throughput up to 16 users, and at 24 users ARK leads, 762 against 703 tokens per second.

What makes it a fair comparison, and what does not

Same: the model (Qwen3.6-35B-A3B), 4-bit weights and a 16-bit attention cache on both engines, prompt caching on in both, two RTX 5090 in one machine, the same day, one engine after the other. The benchmark script unchanged, the same prompts (4,119 and 16,407 tokens), the same limit of 256 new tokens, thinking off on both, requests through each engine's OpenAI-compatible API.
Default settings, on purpose. Both engines ran as installed. Tuning one of them, for example letting vLLM take in more prompt tokens per step, would mean tuning both; that is a separate comparison.
Answer length. Each engine stops where the model ends its answer: ARK's answers averaged about 200 tokens, vLLM's about 145. The decode figures are rates, not totals per request. A measure that ignores answer length, the median gap between tokens, puts the gap at 64 users slightly wider: 25.4 against 8.7 tokens per second per user.
Prompt cache. During decoding, the benchmark sends the same prompts again. ARK served 99% of each prompt from its cache; vLLM, which caches whole blocks of 1,056 tokens, 77% at 4,000 tokens and 97% at 16,000.
16 bits, on purpose. ARK 2.1 can keep its attention cache in 8 bits, twice the context in the same memory (see quantized attention). Against vLLM's 16-bit cache that would not be like for like, so both engines ran with 16-bit caches here.

How we measured

ARK. ARK 2.1.0 serving Qwen3.6-35B-A3B on one machine with two RTX 5090, through the ARK API, 20 October 2026.
vLLM. vLLM 0.27.1 with tensor parallelism over both GPUs, the model's AWQ 4-bit weights and the benchmark authors' settings, thinking off, on the same machine, 20 October 2026.
Benchmark. A third-party script, unchanged. For each prompt length and number of users it first measures time to first token with fresh prompts, then decode speed on the same prompts sent again: each request's tokens divided by the time from its first token to its last. Two repeats per cell; the total is the per-user speed times the number of users. No request failed on either engine.

← All research · Benchmarks