In production today Qwen3.5-122B-A10B runs on ARK across three machines and two GPU generations: its language layers on four RTX 5090 in two machines, its image encoder on an RTX 4060 Ti in a third.
How ARK splits a model
A language model is a stack of layers, and a token passes through them in order. ARK lets each server hold any contiguous stretch of that stack, on one GPU or several, so the stack can be cut wherever the hardware suggests: a large card takes a long stretch, a small card a short one. Nothing requires the stretches to be equal, the GPUs to match, or each machine to carry the same number of cards. A multimodal model's image encoder can sit on a GPU of its own.
Two servers may also hold the same stretch, or overlapping ones. That gives ARK more than one way through the model, which it uses for balancing and for failover.
How ARK routes each request
- Every server measures itself. At start-up each server times its own layers and announces how fast it is. While it runs, it reports how many sessions it is serving and how much attention memory it has left.
- Every request takes the cheapest path. ARK looks at the servers that hold the model's layers and finds the path through them that costs least: each server's measured time for its stretch, made more expensive the busier it is per GPU. A server without room for the request's full context is not considered at all.
- Near-equal paths share the traffic. Any path within 5% of the best is a candidate, and ARK picks among them at random, so replicas of similar speed split the load, and a faster replica takes more of it until its own queue evens the score.
- Failures are routed around. If a server fails during a request, the request is replayed on another path. A server that joins the running deployment becomes part of the next request's choice, with no restart.
- Each GPU uses what it has. Every server picks the fastest kernels its own GPU supports, such as the 8-bit attention cache and fused 4-bit kernels, and falls back where a generation lacks them.
Exotic mixes
The rules above put no number on what a deployment may look like. These are illustrations of what follows from them, not configurations we have benchmarked:
| Mix | What it shows |
|---|---|
| RTX PRO 6000 + 2 × RTX 3090 + 2 × RTX 4060 Ti + RTX 3060, three machines | Odd GPU counts and an uneven split: each card takes a stretch of layers sized to its memory and speed. |
| H100 + RTX 4090 + RTX 3060, one model | Three generations, Hopper, Ada and Ampere, in one model, each running the kernels it supports. |
| One copy of a model on an L40S, a second on four RTX 3060 | Uneven replicas: ARK sends more requests to the copy that is faster and less busy, and the other copy keeps serving if one fails. |
| Language layers on new cards, the image encoder on an older card | Giving an older or smaller GPU a light, separate job instead of retiring it. |
| Two servers holding overlapping stretches of layers | More than one place to switch servers, so routes can steer around a busy card. |
What to look for
- The slowest stretch sets the pace. Each token passes through every stretch in turn: one stream's speed is the sum of their times, and under load the slowest stretch becomes the bottleneck. Size stretches by speed, not only by memory, and give a slow card fewer layers.
- Memory decides context. A request needs room for its whole context on every server it passes through, so the server with the least free attention memory caps the longest request that path can take.
- Machine boundaries cost time. Every token crosses each boundary between machines on its way through the model. ARK weighs speed and load when it picks a path, not network distance: keep the machines that serve one model on the same local network, and prefer fewer, larger stretches to many small ones.
- Reading and writing stress GPUs differently. Reading a prompt is limited by compute, so newer generations read much faster. Generating is limited by memory bandwidth, so a card's bandwidth, not its age or memory size, sets its decode speed.
- Replicas buy throughput and resilience. A second copy of a stretch, on any GPU, doubles the paths ARK can choose from and keeps the model online if one server goes down.
- Keep the software uniform. Every server of a model runs the same ARK release and the same converted model, and every machine meets the driver requirement.
Planning a mix of your own? Send us the inventory and the models you want to serve, and we reply with a layout.