When NVENC will not start: a GPU full of models, and a renderer that falls back
Video
Our video renderer runs on the same machine as the language model that writes its scripts. One evening the hardware encoder simply would not open: "CUDA_ERROR_OUT_OF_MEMORY: out of memory", then "No capable devices found".
What happened
The machine is an NVIDIA GB10 desktop, whose CPU and GPU share one pool of memory. It serves our local model through vLLM, and Studio renders on the same box with ffmpeg, using NVENC when it can. During a test run the encoder failed to create its CUDA context. The failure came and went within seconds: a short NVENC probe passed and the render right after it failed; on the next run the probe itself failed; on the run after that, probe and render both went through NVENC.
Why
vLLM reserves a fixed share of the GPU's memory when it starts, for the model and its cache. NVENC needs a little memory of its own for each session's CUDA context. When the model server and other jobs leave nothing over, the encoder cannot start, even though it is a separate engine on the chip and would barely touch the GPU's compute.
The fix: decide at the encode, not at the probe
A probe at start-up tells you that NVENC worked a moment ago, not that it will work for the next file. So Studio tries NVENC for each render and, if ffmpeg fails, runs the same filter graph again with libx264. The live channel runs a half-second test encode with its live settings before every start and uses libx264 if NVENC fails it; an NVENC or CUDA error during the stream switches the next start to libx264. Each finished video records which encoder made it, so a fallback shows up in the job file instead of passing unnoticed.
What the fallback costs
Less than we expected. On the same machine's 20-core Arm CPU, libx264 rendered 12 seconds of 1080p video, with camera moves, burned-in captions and loudness normalisation, in 7.6 seconds. For short videos made once a day, the CPU is a perfectly good encoder. The GPU matters far more for the image model than for the encode.
The takeaway
On a shared machine, treat the GPU encoder as optional: use it when it opens, fall back when it does not, and log which one you used. The same applies to any GPU feature that needs its own memory on a box where a model server has already taken most of it. Studio ships with this behaviour on by default.