Qwen3.8-27B, 4-bit, one RTX 5090, fused against standard kernels
2.6×
decoding 16 requests at once: 234 to 611 tokens per second
2.3×
decoding 64 requests at once: 739 to 1,677 tokens per second
+45%
reading short prompts of 256 tokens

Why standard kernels stall

The standard 4-bit kernels are tuned for one request at a time. They keep up as requests are added, then stop: from 8 to 16 concurrent requests, decoding with them rises only from 206 to 234 tokens per second. The fused kernel unpacks the weights and multiplies them in a single pass shaped for a batch, so it keeps scaling: 312 tokens per second at 8 requests, 611 at 16 and 1,677 at 64.

Decode throughput by number of concurrent requests, fused kernels off and on 0 500 1,000 1,500 2,000 47 45 1 request 206 312 8 requests 234 611 16 requests 739 1,677 64 requests Tokens per second, all requests Unfused Fused
Decode throughput by number of concurrent requests, standard (unfused) and fused 4-bit kernels. Qwen3.8-27B on one RTX 5090.

That is what lets decode batching pay off on quantized models: without it, the batch would be held back by the kernels long before the GPU runs out of compute.

Reading the prompt

Prompt reading gains too, most on short prompts. From 2,000 tokens on, the two are within 8%:

Prompt lengthStandard, tok/sFused, tok/sChange
256 tokens2,1533,132+45%
1,024 tokens3,1593,603+14%
2,048 tokens3,3543,633+8%
8,192 tokens3,3163,397+2%

What it costs

  • One request at a time runs about as fast as before: 45 tokens per second fused, 47 standard on this model.
  • Start-up. The fused kernels tune themselves to the GPU when a server starts. We limit that tuning so that a server still starts in seconds.
  • Scope. The fused kernels serve ARK's quantized chat models. Unquantized models, such as Whisper and BGE-M3, are unaffected.

How we measured

  • Setup. Qwen3.8-27B quantized to 4 bits, one RTX 5090, the inference engine on its own with ARK's settings: no network, no API, so the kernels are the only difference. Fused kernels switched on and off, 16 October 2026.
  • Measured. Decode tokens per second for 1, 8, 16 and 64 concurrent requests, and prompt tokens per second for prompts of 256 to 8,192 tokens. Median of three runs.
  • In ARK. ARK switches the fused kernels on whenever a server batches its requests, which is the default.

Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching

← All research