DevelopersChangelog

Changelog

The big steps in the ARK platform, newest first. Each one links to its engineering note, with the measurements behind the numbers.

ARK 2.1

Prefill batching

Prompts are read across all the GPUs of a split model at once, like an assembly line, so long prompts reach their first token far sooner, alone or many at a time.

  • A 64,000-token prompt gets its first token in 4.7 seconds instead of 8.4 on Qwen3.5-122B-A10B.
  • 1.6 to 1.8 times the combined prompt-reading rate for prompts of 16,000 tokens and more.
  • 64 prompts of 16,000 tokens at once: 191 of 192 requests answered within 90 seconds, against 50.

Read the engineering note

ARK 2.0

Paged attention

The attention cache is kept in small pages, so short chats and long documents share every decoding step instead of taking turns.

  • 2.6 times the decode speed per stream when request lengths are mixed: 12.2 to 31.4 tokens per second.
  • Memory reserved within 3% of what requests need, down from 15%.

Read the engineering note

ARK 2.0

Quantized attention

The attention cache is kept in 8 bits instead of 16. Each token takes half the memory, so the same GPU holds twice the context.

  • Twice the tokens of context per GPU: 16 conversations of 16,000 tokens fit on one RTX 5090, against 16 of 8,000.
  • Up to 30% less time per decoding step on long contexts.
  • 99.2% of steps pick the same top token as the 16-bit cache.

Read the engineering note

ARK 2.0

Fused kernels

4-bit models decode through GPU kernels that unpack and multiply their weights in one pass, built for many requests at once.

  • 2.6 times the decode throughput with 16 concurrent requests, 2.3 times with 64.
  • Up to 45% faster reading of short prompts.

Read the engineering note

ARK 2.0

Decode batching

Every request in flight shares each decoding step, so total throughput grows with the number of users instead of being divided between them.

  • 9.3 times the total throughput of ARK 1.0 at full load, median of 15 models, with 64 streams instead of 12.
  • 90% less energy per token.

Read the engineering note

ARK 2.0

Inverted processing loops

The processing loops at the heart of the runtime were inverted, so the GPUs spend their time computing instead of waiting. The new structure is what made decode batching possible.

  • 1.51 times the single-stream decode speed of ARK 1.1, median of 16 models, on the same GPUs.
  • 23% less energy per token on a single stream.

Read the engineering note

ARK 0.10

Prompt caching

Recently read prompts stay on the GPU. A request that begins with the same system prompt, document or history restores it instead of reading it again.

  • On ARK Cloud today, a repeated 64,000-token document gets its first token 9.1 times sooner: 0.96 seconds instead of 8.75.
  • A shared 8,000-token system prompt halves the wait for each new user.

Read the engineering note