Our GB10 machine kept switching itself off. The log blamed memory; it was heat
Ops
One of our three NVIDIA GB10 machines went dark three times in four days in September: on 10 September, on 12 September (it stayed off for 17 hours) and on 13 September, twelve minutes after it was switched back on. Each time it needed a person to press the power button.
What a heat shutdown looks like
The system journal simply stopped, mid-activity, with no shutdown in it: a hard power-off does not give journald time to write its buffer. In July, after a similar shutdown, the clock even came back with an old date. And the hardware watchdog, which reboots a machine that hangs, did nothing, because the machine had not hung: the power was gone.
The clue that was wrong
The last kernel lines before each death were GPU out-of-memory errors (NV_ERR_NO_MEMORY), so memory looked guilty. It was not. Those lines were simply the most common thing the kernel wrote: 440,580 of them in a single boot that had run for 19 days before it ended. When a line appears that often, it will be the last line before anything. Before blaming a log line, count how often it appears when nothing goes wrong.
The clue that was right
Because the journal loses its last minutes in a power-off, we had added a small sampler that writes temperatures, load and memory to a file every minute and flushes it to disk. It caught all five deaths since July. Every one came at full GPU load (96%), with the CPU at 85 to 97 °C.
The timing explained the rest. Around the start of September we had moved video rendering onto this machine. Its GPU power under full load went from about 24 W to 46 W, its median GPU temperature from 68 to 86 °C, and the minutes per week with the CPU at 95 °C or more from none to 4,283. Our two other GB10 machines, which run the language model at the same 96% load, stayed at 65 to 81 °C. This one simply cooled worse.
The first fix: a GPU clock cap
On GB10, nvidia-smi cannot set a power limit, but it can cap the GPU clock: nvidia-smi -lgc 0,2000. In July that took the hottest zone from 91 °C to 77 to 83 °C within seconds. Two details matter:
- Apply it before the heavy services start. Our cap service first ran about 15 seconds after boot, when the model server had already begun loading at full power, which is the moment that had crashed the machine. It now runs before Docker starts.
- Check it under load. With the cap on, the reported GPU clock still reads about 2,400 MHz when the GPU is idle. Under load it holds at 1,976 MHz.
The second fix: a heat guard in steps
The cap was not enough once rendering moved in, so a guard now reads every sensor once a second and acts in steps: a lower GPU clock from 89 °C and a lower one again from 92 °C; from 94 °C the renderer is paused (SIGSTOP) and the CPU slowed; from 96 °C every GPU job is paused for the seconds it takes to cool. It steps back only after the machine has stayed cooler for a while. At 97 °C for 20 seconds, a last-resort brake stops the video service altogether.
In its first nine minutes under rendering, the hottest zone peaked at 94 °C with a median of 89 °C, against 97 °C plateaus before. It has a price: on one day in mid-September the renderer was paused for 17% of the time, and the GPU ran at 900 MHz or less for most of the day. And it is not a cure: the machine switched itself off once more, on 11 October, at 97 to 98 °C. It has run without a break since then, 20 days as we write this.
Four mistakes we made along the way
- Pausing one job did not cool the machine. The brake fired with the renderer already paused, because an image model was still drawing and a dozen timers started in the same minute. The top step now pauses every heavy GPU job, not just the renderer.
- The guard pauses jobs by name, and the names changed. Twice a new engine would have escaped the pause: an image model with a new process name in September, and in October a renderer that runs in a container instead of as a child of the render service. Every change of engine now includes a look at the guard's list.
- The brake's hold never lifted. It stopped the video service and left a hold file so that nothing restarted it. Nothing alerted anyone either, and output stopped for three days before we noticed. The hold now lifts itself: five minutes after the last brake, with no new brake, at least 30 GB of memory free and a ten-minute average of 93 °C or less. Waiting for the machine to cool down was never going to work: it averaged 88 °C with and without rendering.
- The alarm rang at the machine's normal temperature. The alert threshold was 88 °C, which was this machine's average. Over one day, it spent 32% of the time at 90 °C or more, 8% at 93 °C or more and 3% at 95 °C or more, and never reached 97 °C. We moved the alarm to 97 °C, the plateau where it actually died, and the alarm emails stopped.
The takeaway
Software kept this machine alive, but the cause was physical: its cooling was two to three times worse than its siblings'. If a GB10 machine switches itself off under load, check the airflow first. Then cap the GPU clock before your services start, pause the heavy jobs before the machine reaches its limit, and set alarms from the temperatures it actually runs at.
Factory Floor includes a heat guard of this kind: it reads the hottest of the GPU and thermal-zone sensors, pauses the heavy jobs you name at a set temperature (92 °C by default) and resumes them only below a lower one (84 °C), with memory and disk guards beside it.