Two DGX Spark-class boxes, one model: a copy on each beat splitting it
Ops
For seven weeks our local language model ran split across two NVIDIA GB10 machines with tensor parallelism. On 22 October we replaced that with a full copy of the model on each machine and a least-connection proxy in front. Throughput went from about 180 to 210 tokens a second, and either machine can now restart without taking the model away.
Why we had split it
Each machine has 128 GB of memory that the CPU and the GPU share. In September the model fitted on one machine, but its cache barely did: the vLLM build we used could not keep this model's KV cache in FP8, so the cache was in BF16, and one machine had room for 499,110 tokens of it. That is fewer than two requests at the model's full context of 262,144 tokens. Split across both machines with tensor parallelism over their direct network link, the cache grew to more than three million tokens. We had split the model for room, not for speed.
What changed
A newer 4-bit build of the same model (NVFP4, made with quantisation-aware distillation) takes about 71 GiB of memory, and a community vLLM build for GB10 (eugr/spark-vllm-b12x) runs its cache in FP8. Now one machine holds the whole model and a cache of 0.76 to 1.08 million tokens, room for 2.9 to 4.1 requests at full context, depending on how much memory the machine gives it. Both copies use the model's own speculative decoding (MTP, four draft tokens). Before switching we ran our tool-calling test on the new build: it scored 86, the build it replaced 75.
What we measured
| Split across two machines | A copy on each machine | |
|---|---|---|
| KV cache | more than 3 million tokens | 0.76 and 1.08 million tokens |
| Requests at full context | about 12 | 2.9 and 4.1 |
| Throughput under our load | about 180 tokens/s | 210 tokens/s (111 + 99) |
| Start-up | about 20 minutes | 5 to 13 minutes a copy |
| One machine restarts | the model is down | the other copy serves |
One copy on its own, by the number of requests it serves at once: with short prompts, 1 request gave 36 tokens a second, 4 gave 82, 8 gave 109 and 16 gave 150. With our email agent's longer prompts, 4 gave 81, 8 gave 113 and 16 only 87: past eight such requests, more made it slower.
For an afternoon one copy took at most 12 requests at once and the other 8. The first produced 91 to 93 tokens a second and finished 26 to 27 requests a minute; the second 81 to 83 tokens a second and 22 requests a minute. Both now take 12.
What we gave up
Room. The split model had one cache of more than three million tokens; the copies have two smaller ones, and a request can only use the cache of the copy that serves it. Our agents send many short and medium requests, and for them throughput mattered more. If you serve a few very long contexts at once, splitting may still be the better choice. We kept the split setup, and switching back is one script.
What went wrong during the switch
- Eleven minutes without a model, then 30 seconds of 502s. The split model had to stop before a copy could load on the same machines, so our agents waited about eleven minutes. When the first copy came up, the proxy kept answering "502 no live upstreams" for about 30 seconds more: it had marked the backends as failed while they were down (max_fails=3, fail_timeout=30s) and did not try them again until that time ran out.
- A client that skipped the proxy. One of our batch jobs had the first machine's address in its config, so its whole load went to one copy while the other had capacity to spare. Every client now goes through the proxy; the machines' own addresses are only for health checks.
- Metrics that jumped. Our throughput monitor read vLLM's /metrics through the proxy. With two backends, consecutive reads came from different copies, the counters jumped back and forth, and the monitor reported the model as starved. It now reads each copy directly and adds them up.
- An old watchdog that would have stopped the new copy. The split setup had a watchdog that restarted its service when the model stopped answering. The new service declares
Conflicts=with the old one, so that restart would have stopped the new copy. It had to be switched off withsystemctl disable --now, not just stopped, because our ops layer restarts units that are enabled but not running. - Memory on the machine that does other work. One machine also runs our databases and site builds. With vLLM's gpu-memory-utilization at 0.78, only 5 GiB were left for everything else, so we lowered it to 0.74 and restarted only that copy while the other carried the load. On these machines, the share you give vLLM comes out of the same memory as everything else.
If you have two of these machines
- If the model and the cache you need fit on one machine, run a copy on each behind a least-connection proxy. Split the model only when one machine cannot hold the cache you need.
- Point every client at the proxy, including batch jobs and old configs.
- Read metrics from each copy directly.
- Give each machine its own memory share, according to what else runs on it.
- Set how many requests a copy serves at once with your own traffic: for our mix, 12 beat 8.
- Expect the switch to take the model away for as long as a copy takes to start, and plan for it.
Factory Floor ships the proxy and launch configs for one or two machines, the health check and the tool-use test we run before switching models, with the watchdogs and memory guards for the rest of the machine.