Benchmarks

Measured on our GPUs, published with the data.

Every figure on this page comes from a measured run: what was sent, to which model, on which GPUs, and what came back. The full data sits in a table under each chart.

ARK Core · October 2026

ARK Core under real traffic.

We sent summarisation requests to our own ARK Core deployments, through the public ARK Cloud API: texts of about 1,100 tokens, each summarised in about a tenth of its length, streamed, from 1 up to 60 requests at a time. That is the shape of most production traffic: much more reading than writing.

Throughput

Input + output tokens per second across all requests in flight.

Total throughput on ARK Cloud as concurrent requests grow 0 2,500 5,000 7,500 10,000 1 8 16 24 32 48 60 Concurrent requests Tokens per second (input + output) 7,439 Qwen3.6-35B 5,726 Qwen3.5-122B
Qwen3.5-122B-A10B · 4× RTX 5090Qwen3.6-35B-A3B · 4× RTX 4070 Ti SUPER

Time to first token

Median seconds from sending a request to its first streamed token.

Median time to first token on ARK Cloud as concurrent requests grow 0.0 1.0 2.0 3.0 4.0 1 8 16 24 32 48 60 Concurrent requests Seconds to first token (median) 3.41 s Qwen3.5-122B 1.66 s Qwen3.6-35B
Qwen3.5-122B-A10BQwen3.6-35B-A3B

Qwen3.5-122B-A10B on four RTX 5090

15 October 2026 · public API · 730 requests

ConcurrentInput tok/sOutput tok/sTotal tok/sTTFT p50TTFT p95Decode per requestCompleted
1433424750.45 s0.52 s50.4 tok/s10/10
2791758660.46 s0.58 s45.4 tok/s20/20
41,3251291,4540.48 s0.58 s37.6 tok/s40/40
82,0611982,2600.52 s0.95 s28.6 tok/s60/60
122,6192492,8680.52 s1.24 s23.9 tok/s80/80
163,2582883,5460.55 s1.79 s20.9 tok/s100/100
243,8813464,2280.57 s2.56 s16.7 tok/s110/110
324,5294024,9310.76 s3.56 s14.8 tok/s110/110
485,2154605,6751.53 s5.81 s12.8 tok/s100/100
605,2634635,7263.41 s7.47 s13.6 tok/s100/100

Qwen3.6-35B-A3B on four RTX 4070 Ti SUPER

16 October 2026 · public API · 730 requests; two responses were lost on the network path after the API had completed them

ConcurrentInput tok/sOutput tok/sTotal tok/sTTFT p50TTFT p95Decode per requestCompleted
1699667650.27 s0.42 s82.8 tok/s10/10
21,1401101,2490.29 s0.39 s67.9 tok/s20/20
42,0481942,2420.32 s0.51 s57.9 tok/s40/40
83,0002823,2820.32 s0.69 s43.4 tok/s59/60
123,7463534,0990.32 s1.04 s33.6 tok/s80/80
164,3504104,7600.35 s1.33 s29.9 tok/s100/100
245,5235206,0430.54 s2.11 s27.9 tok/s109/110
326,1165786,6930.84 s2.96 s24.2 tok/s110/110
486,8076417,4481.21 s4.92 s19.6 tok/s100/100
606,7966437,4391.66 s6.16 s16.2 tok/s100/100
Two more production models on smaller GPUs

Qwen3.5-4B on two RTX 4060 Ti

16 October 2026 · production API · up to 32 concurrent requests

ConcurrentInput tok/sOutput tok/sTotal tok/sTTFT p50TTFT p95Decode per requestCompleted
1846478940.34 s1.18 s61.2 tok/s10/10
21,281911,3720.35 s0.49 s53.9 tok/s20/20
41,8091461,9550.37 s0.61 s43.4 tok/s40/40
82,6482202,8680.36 s1.07 s32.0 tok/s60/60
123,0572603,3170.45 s1.74 s25.4 tok/s80/80
163,5303013,8300.50 s2.42 s21.7 tok/s100/100
243,7763264,1010.64 s4.02 s16.1 tok/s110/110
323,8563354,1921.07 s5.89 s13.5 tok/s110/110

Bielik-11B-v3.0-Instruct on four RTX 3060

16 October 2026 · production API · throughput levels off from 16 concurrent requests on these 12 GB cards

ConcurrentInput tok/sOutput tok/sTotal tok/sTTFT p50TTFT p95Decode per requestCompleted
1304293330.49 s0.67 s44.1 tok/s10/10
2482475290.56 s0.99 s33.6 tok/s20/20
4698687650.73 s1.38 s25.3 tok/s40/40
8935911,0260.95 s2.11 s15.1 tok/s60/60
121,0341001,1331.05 s3.09 s10.7 tok/s80/80
161,1081071,2151.48 s4.27 s8.8 tok/s100/100
241,1391101,2492.11 s6.81 s6.3 tok/s110/110
321,1701131,2833.70 s9.54 s5.2 tok/s110/110
481,2171181,3355.96 s15.24 s3.9 tok/s100/100
601,2211181,3398.67 s19.61 s3.6 tok/s100/100
Output-heavy · October 2026

Long answers, many users.

The opposite traffic shape: short prompts of 100 tokens, each answered with 2,000 tokens, streamed through the HTTP API. Qwen3.5-122B-A10B on four RTX 5090 with ARK 2.0, on a test system set up like production.

  • From one stream to 64, total output grows 28-fold.
  • Every stream still receives a token every 35 ms (median) at 64 streams.
  • No request failed and none came back empty.
  • Against direct access to the same servers, the API layer delivered 97 to 101% of the throughput at every level.

Output tokens per second

All streams together, by number of concurrent streams.

Output tokens per second, Qwen3.5-122B writing 2,000-token answers 0 500 1,000 1,500 2,000 52 1 102 2 200 4 542 16 831 32 1,455 64 Concurrent streams Output tokens per second
Concurrent streamsOutput tok/sPer streamTTFT p50Inter-token p50Failed
15252.8 tok/s0.19 s17.2 ms0
210251.4 tok/s0.20 s18.1 ms0
420050.4 tok/s0.25 s18.3 ms0
1654234.4 tok/s0.23 s25.8 ms0
3283126.5 tok/s0.16 s35.5 ms0
641,45524.3 tok/s0.23 s35.4 ms0
ARK Core · October 2026

Energy per token.

Measured on our own ARK Core deployments, the ones that serve ARK Cloud, under input-heavy traffic, with the GPUs and CPUs of every serving machine counted, idle draw included.

  • 21 Wh per million tokens on Qwen3.6-35B-A3B (four RTX 4070 Ti SUPER), 22 on Qwen3.5-4B (two RTX 4060 Ti), 76 on Qwen3.5-122B-A10B (four RTX 5090).
  • About 0.025 Wh for a request of 1,200 tokens on Qwen3.6-35B-A3B.
  • 1.8 to 4.7 times less energy per token with 16 or more users than with one.

The full report

Wh per million tokens

One request at a time and 16 or more at once. Lower is better.

Energy per million tokens processed on ARK Cloud, one request at a time and 16 or more at once 0 125 250 375 500 100 21 Qwen3.6-35B-A3B 39 22 Qwen3.5-4B 191 76 Qwen3.5-122B-A10B 272 90 Bielik-11B Wh per million tokens (lower is better) 1 at a time 16+ at once
ARK 2.0 release run · 1 October 2026

ARK 2.0 across 19 models.

The release benchmark for self-hosted deployments: 19 chat models on four RTX 5090 across two machines, measured directly against the ARK runtime, one stream and 64 streams at a time.

Single stream
1.54× fastermedian over 15 models, ARK 2.0 against ARK 1.0
Concurrency
64 streamswhere ARK 1.0 was tested at 12
Total throughput
9.3×median, each version at its full test load
Energy per token
−90%median at full load, GPUs and CPUs included

Total decode throughput at 64 concurrent streams

Tokens per second across all streams, ARK 2.0, 4× RTX 5090.

Aggregate decode throughput at 64 concurrent streams, ARK 2.0, 4x RTX 5090 0 2,500 5,000 7,500 10,000 Llama-3.2-1B 5,500 Llama-3.2-3B 4,644 Llama-3.1-8B 3,911 Ministral-3-8B 3,819 Qwen3-8B 3,596 Qwen3.5-4B 3,546 Qwen3-Coder-30B-A3B 3,295 gpt-oss-20b 3,273 Bielik-11B-v3.0 3,265 Ministral-3-14B 3,228 Qwen3-14B 2,927 Devstral-Small-2-24B 2,733 Qwen3.6-35B-A3B 2,687 Qwen3-32B 1,940 GLM-4.7-Flash 1,878 Qwen3.5-122B-A10B 1,717 Qwen3.6-27B 1,697 Qwen3.8-27B 1,694 Llama-3.3-70B 1,391
ModelOne stream64 streams, total64 streams, eachEnergy at 64 streams
Llama-3.2-1B603.4 tok/s5,500 tok/s85.9 tok/s0.03 Wh / 1k tokens
Llama-3.2-3B326.8 tok/s4,644 tok/s72.6 tok/s0.04 Wh / 1k tokens
Llama-3.1-8B209.1 tok/s3,911 tok/s61.1 tok/s0.07 Wh / 1k tokens
Ministral-3-8B199.1 tok/s3,819 tok/s59.7 tok/s0.07 Wh / 1k tokens
Qwen3-8B172.5 tok/s3,596 tok/s56.2 tok/s0.07 Wh / 1k tokens
Qwen3.5-4B143.9 tok/s3,546 tok/s55.4 tok/s0.07 Wh / 1k tokens
Qwen3-Coder-30B-A3B174.0 tok/s3,295 tok/s51.5 tok/s0.07 Wh / 1k tokens
gpt-oss-20b256.1 tok/s3,273 tok/s51.1 tok/s0.09 Wh / 1k tokens
Bielik-11B-v3.0143.5 tok/s3,265 tok/s51.0 tok/s0.08 Wh / 1k tokens
Ministral-3-14B140.2 tok/s3,228 tok/s50.4 tok/s0.10 Wh / 1k tokens
Qwen3-14B117.7 tok/s2,927 tok/s45.7 tok/s0.10 Wh / 1k tokens
Devstral-Small-2-24B92.7 tok/s2,733 tok/s42.7 tok/s0.14 Wh / 1k tokens
Qwen3.6-35B-A3B102.1 tok/s2,687 tok/s42.0 tok/s0.09 Wh / 1k tokens
Qwen3-32B58.8 tok/s1,940 tok/s30.3 tok/s0.19 Wh / 1k tokens
GLM-4.7-Flash76.0 tok/s1,878 tok/s29.3 tok/s0.14 Wh / 1k tokens
Qwen3.5-122B-A10B52.4 tok/s on 4 GPUs1,717 tok/s26.8 tok/s0.19 Wh / 1k tokens
Qwen3.6-27B48.2 tok/s1,697 tok/s26.5 tok/s0.19 Wh / 1k tokens
Qwen3.8-27B48.1 tok/s1,694 tok/s26.5 tok/s0.19 Wh / 1k tokens
Llama-3.3-70B31.0 tok/s on 4 GPUs1,391 tok/s21.7 tok/s0.34 Wh / 1k tokens

Most models were measured on a single GPU for one stream; Qwen3.5-122B-A10B and Llama-3.3-70B span four. Energy is the power drawn by both machines, GPUs and CPUs, divided by the tokens produced.

ARK 2.1 and vLLM · October 2026

Same GPUs, same model, same benchmark.

Qwen3.6-35B-A3B at 4 bits on two RTX 5090, driven by the same third-party benchmark script with the same prompts against ARK 2.1 and vLLM 0.27.1, on the same day, both at default settings.

  • At 64 users with 4,000-token prompts, ARK generates 1,655 tokens per second against vLLM's 652.
  • The first token comes 4.7 to 7.1 times sooner on ARK at every load.
  • vLLM is faster for a single user: 160 against 88 tokens per second.

The full comparison

Total decode throughput

All users together, 4,000-token prompts, by number of concurrent users.

Total decode throughput by number of users, ARK 2.1 and vLLM 0.27.1, 4K-token prompts 0 500 1,000 1,500 2,000 1 4 8 16 24 32 48 64 Concurrent users Tokens per second, all users 1,655 ARK 2.1 652 vLLM
Method

How we measure.

  • Throughput is tokens processed per second of wall-clock time across all requests in flight. Input tokens count when the prompt is read; output tokens as they are generated.
  • Time to first token is measured on the client, from sending the request to receiving the first streamed token, network included.
  • Decode per request is a request's output tokens divided by the time between its first and last token.
  • Reliability counts requests that completed with a non-empty answer. A failed or empty response counts against us.
  • Hardware is NVIDIA consumer GPUs: RTX 3060, 4060 Ti, 4070 Ti SUPER and 5090. Results on other GPUs differ.
  • Versions: the real-traffic and long-answer runs used ARK 2.0 as deployed in October 2026; the release run used the ARK 2.0.0 release. We rerun these benchmarks as ARK changes and update this page.

Want to measure ARK on your own workload? Talk to us about a proof of concept on your hardware.

See it on your own traffic.

Point your application at ARK Cloud, or ask us for a proof of concept on your GPUs.

Get an API key Talk to us