Our health check restarted a healthy model server eight times in one day
Ops
On 22 July a watchdog restarted our local model server eight times. Not one of those restarts had a crash behind it. The check that was meant to keep the model up was the thing taking it down.
What happened
The model server ran vLLM on one NVIDIA GB10 machine and produced about 8 tokens a second for a single request. Every five minutes a watchdog sent it a real chat request and waited up to 45 seconds for the answer. Five failed checks in a row meant a hung server, and the watchdog restarted it. A restart took the model away for several minutes while about 19 GB of weights loaded again.
That day a batch job kept two to four requests busy on the server for minutes at a time. The watchdog's request waited in the queue behind them, ran out of time, and counted as a failure. Five such checks later, a healthy server was restarted. We had a "busy" flag file the watchdog respected, but the batch job never set it.
How one false restart led to the next
Every restart failed the requests in progress with a 500 error ("cannot schedule new futures after shutdown"). A second watchdog counted 500 errors and restarted the server when there were too many, and the more errors it saw, the sooner it acted. So each false restart made the next one come sooner. In all, four separate watchdogs could restart that server, on 5, 10, 15 and 30 minute timers.
The rule we took from it
A health check must not use what it guards. A model server's scarce resource is its slots for requests in progress. A check that takes one of those slots, or waits for one, is part of the load it is meant to judge. Check cheaply, and restart only when you can show that the server is stuck.
The check that replaced it
vLLM publishes its own counters at /metrics. The new check reads them first:
- Requests running and the engine making progress: healthy and busy. No test request is sent.
- Requests running and no progress between two reads: stuck. This is the case that still restarts.
- Nothing running: idle. Only then does the check ask for a short answer, which an idle engine gives at once.
The first version counted generated tokens as progress. It passed eight tests, and on its first live run it found the server busy and healthy: 2 requests running, 107 new tokens between the two reads.
What we changed since
- Engine steps, not output tokens. While vLLM reads a long prompt, it produces no output tokens for a while but still takes steps. Counting steps (the count of vllm:iteration_tokens_total) keeps a long prompt from looking like a stuck server.
- Two waits before "stuck", so a short pause does not count as a failure.
- A restart between the two reads is caught by the process start time in the same metrics; the check then asks for an answer instead of comparing counters from two different processes.
- A proxy is checked through /models only. Behind a least-connection proxy, each read of /metrics can come from a different backend, as we found when we ran a copy of the model on each of two machines. The backends are checked directly.
The difference is easy to see. As we wrote this note, both of our model servers were running 12 requests each, their limit. A test request would have waited in the queue. The new check reported both busy and healthy, at 126 and 101 tokens a second, without sending one.
Factory Floor 1.0.2 ships this check as models/llm-health.py, with tests for idle, busy, stuck and restarted servers, a server without metrics, a proxy and a dead backend.