Our watchdog restarted the services we had stopped on purpose

Ops2026-10-31

One of our watchdogs starts any of our services that is enabled but not running. Twice in October it started services we had stopped on purpose. The second time, visitors to one of our sites got errors for about 45 minutes.

Why that watchdog exists

A service that exits cleanly when it gets SIGTERM, for example because something killed it by name, is not "failed" as far as systemd is concerned. Restart=on-failure does not bring it back, and systemctl --failed does not list it. It stays down until someone happens to notice. So the watchdog looks for services that are enabled but not running, and starts them.

The trouble is that a service you stopped by hand looks exactly the same.

The first time

On 7 October we stopped a video worker with systemctl stop at 12:47 to edit its job queue. At 12:55 it was running again: the watchdog had started it.

The second time

On 21 October we needed the memory of the machine that also serves two of our sites for a model test. A script stopped the sites' web apps, a sync job and a database container, again with systemctl stop. About five minutes later a timer that watches the sites, and at 19:14 the watchdog, brought the tunnel and the web apps back.

The tunnel took traffic again, but the web apps could not answer: the model under test held 114 GiB of the machine's memory. From about 19:14 to 20:00, some visitors got 502 errors. At 20:00 we disabled the tunnel, the web apps and the sync job, and stopped the watch timer.

The rule we use now

• Pause a service by disabling it: sudo systemctl disable --now <unit>. Bring it back with sudo systemctl enable --now <unit>. Our watchdog only looks at enabled services, so a disabled one is left alone.
• Pause everything that can bring it back: the tunnel in front of it and any timer that watches it.
• A short stop is a race. If you only use stop, finish before the next check, which on our machines comes within minutes.

Two more systemd traps from the same months

• systemctl mask --runtime does nothing for a unit file in /etc/systemd/system. Units in /etc take priority over the runtime mask in /run, so the command succeeds and the service still starts. To hold a service, we use a drop-in with ConditionPathExists=!/etc/<name>.hold: while that file exists, the service will not start, neither at boot nor by hand.
• systemctl is-active calls a running oneshot service inactive. While a Type=oneshot service runs, its state is "activating", so is-active prints that and returns 3. A loop like while systemctl is-active --quiet job.service; do sleep 60; done, meant to wait for a nightly job, exits at once. Read the state instead: active, activating, reloading or deactivating all mean it is still going.

Factory Floor watches for enabled services that are not running. By default it only reports them, and it starts them only if you switch that on. Since version 1.0.3 its alert also says how to pause a service on purpose, and its runbook says the same.