How it works
Reading a prompt leaves behind the model's working state for every token in it. ARK keeps that state on the GPU for prompts it has recently read. When a new request begins with the same text, ARK restores the matching part and reads only what is new. The cache works in whole blocks of the prompt, so the part that matches is rounded down to whole blocks; the rest is read as usual. Nothing has to be enabled: every text request is checked, and the API reports how many prompt tokens came from the cache, in usage.prompt_tokens_details.cached_tokens (Anthropic API: cache_read_input_tokens).
What it saves
The longer the repeated text, the larger the saving. A 64,000-token document asked a second question gets its first token after 0.96 seconds instead of 8.75; a 16,000-token document after 0.63 instead of 2.19. A shared system prompt of 8,000 tokens halves the wait for each new user.
| Repeated text | Not cached | Cached | Faster | From cache |
|---|---|---|---|---|
| 4K document | 0.76 s | 0.22 s | 3.5× | 99% |
| 16K document | 2.19 s | 0.63 s | 3.5× | 88% |
| 64K document | 8.75 s | 0.96 s | 9.1× | 97% |
| 8K system prompt | 1.22 s | 0.59 s | 2.0× | 75% |
Every cached request in the run found its prefix: none had to fall back to a full read. A request that skips most of its prompt also leaves the GPUs free for everyone else sooner.
Getting the most from it
- Keep the beginning identical. Only an exact match from the first token counts. Put what changes, such as the user's question, a timestamp or a request id, at the end, not at the top of the system prompt.
- Repeat soon. The cache has a fixed size and keeps the most recently used prompts; a prompt that has not been seen for a while may have made room for newer ones.
- For long conversations, a stateful session goes further: the conversation stays on the server and each turn sends only the new message.
How we measured
- Where. ARK Cloud's production deployment of Qwen3.5-122B-A10B on four RTX 5090 in two machines, ARK 2.0, on 20 October 2026, through the API from inside our network. Production traffic ran alongside, so single timings carry some noise.
- Documents. A document of 4,000, 16,000 or 64,000 tokens asked one question (not cached), then the same document asked a different question (cached).
- System prompt. An 8,000-token system prompt with a short user question: once with a beginning unique to the request (not cached), and once after one request had primed the cache (cached).
- Measured. Time to first token and the cached tokens the API reported, one request at a time, three rounds with fresh text each round; the table shows medians.
Engineering notes: Inverted processing loops · Decode batching · Fused kernels · Paged attention · Quantized attention · Prefill batching · Prompt caching