Twice the context
With Qwen3-8B on one RTX 5090, the largest load that fit with a 16-bit cache was 16 concurrent conversations of 8,192 tokens: 131,072 tokens in all. With the 8-bit cache, 16 conversations of 16,384 tokens fit: 262,144 tokens on the same card. The same halving holds for every model whose cache is quantized, which is what lets ARK offer long contexts on GPUs with 16 or 32 GB.
Faster where the cache is large
Each decoding step reads the cache of every request in the batch. Half the bytes means less to read, so the 8-bit cache is faster exactly where the cache is big: 21% faster per step at 16 concurrent 4,000-token contexts, 30% at 8,000 tokens. At 16 contexts of 16,000 tokens, which the 16-bit cache cannot hold, a step takes 18.2 ms: about what the 16-bit cache needs at half that context. Reading the prompt takes about as long as before: 2% longer on Qwen3-8B, 5% shorter on Qwen3.6-35B-A3B.
In our 21-model release sweep on four RTX 5090 at 64 concurrent streams, most models are within 5% of the 16-bit release or faster: gpt-oss-20b by 10%, Qwen3-8B by 5%, Qwen3.6-35B-A3B by 4%.
Quality
We compared the 8-bit cache with the 16-bit one token by token, feeding both the same text:
| Model | Same top token | 16-bit run against itself | Perplexity |
|---|---|---|---|
| Qwen3-8B, 792 tokens | 99.24% | 99.87% | unchanged |
| Qwen3.6-35B-A3B, 727 tokens | 99.31% | 99.59% | +0.3% |
Both models recalled every fact planted in a long text.
What it costs
- Short single requests are up to 7% slower per step: the 8-bit path has a small fixed cost that a small cache cannot repay.
- Three models decode more slowly under heavy load. On Qwen3-14B, GLM-4.7-Flash and Llama-3.2-3B, whose attention layout our fastest 8-bit kernel does not cover, the 64-stream sweep ran 18%, 14% and 9% slower.
- Our API test suite shows two to three failures in about 2,800 tests on 8-bit builds, where the 16-bit release passed them all: a recall check on Llama-3.2-1B, the smallest model, and occasionally gpt-oss-20b not closing its reasoning. A deployment can keep the 16-bit cache for any model.
- Scope. The 8-bit cache is the default for ARK's 4-bit chat models. Whisper and BGE-M3 keep theirs. The hybrid Qwen3.5, Qwen3.6 and Qwen3.8 models quantize their attention cache but not their recurrent state, so they save less than half.
How we measured
- Capacity, speed and quality. Qwen3-8B and Qwen3.6-35B-A3B quantized to 4 bits, one RTX 5090, the inference engine on its own with a 16-bit and an 8-bit cache, 1 October 2026. Median time per step; quality from teacher-forced comparison over six prompts, with scales calibrated on different text.
- Sweep. 21 models on four RTX 5090: a development build with the 8-bit cache (9 October 2026) against the ARK 2.0.0 release with a 16-bit cache, one run per model at 1 and 64 streams. The 16-bit release ran some multi-GPU servers without our fastest attention kernel, so part of the 64-stream gains comes from that kernel.
- Tests. The ARK API test suite on 21 models, 2,727 to 2,879 tests per run.
Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching