Our wait loop waited 35 hours for itself: pgrep -f and pkill -f over SSH

Ops2026-11-02

On 31 October we started a small loop over SSH: wait until a translation job on another machine finishes, then run a follow-up job. The translation job finished that night. When we looked again two days later, the loop had been waiting for 35 hours, and the follow-up had never run.

What the loop was

ssh build-host 'while pgrep -f "i18n_job.py example.com" >/dev/null; do sleep 30; done; ./follow-up.sh'

pgrep -f matches a pattern against the full command line of every process. SSH runs the quoted command as bash -c 'while pgrep -f "i18n_job.py example.com" …', and that command line contains the pattern. pgrep leaves its own process out of the results, but not the shell that started it. So as long as the loop was alive, it found a match: itself.

Its twin: pkill -f kills the shell it runs in

The same thing happens with pkill -f, with worse results:

ssh build-host 'pkill -f "worker.py --queue a"; nohup python3 worker.py --queue a &'

The shell's own command line contains the pattern, so pkill kills the shell. The worker is stopped and never started again, and SSH exits with code 255 without an error message. It looks like a quoting problem, which is what we took it for the first time.

Between September and the end of October we fell into one or the other at least a dozen times: restarts that never happened, a loop that held a check for three and a half hours, another that held a translation queue for 50 minutes, and this one.

What we do now

• A bracket in the pattern: pgrep -f "[i]18n_job.py example.com". As a regular expression, [i] matches the letter i, so the job still matches; the shell's command line contains the characters [i], which the expression does not match. It only helps if the plain pattern appears nowhere else on that line: a restart command later in the same line matches again.
• Find, stop and start in separate SSH calls. One call lists the processes with pgrep -af, the next kills the PIDs it printed, a third starts the new process.
• Wait on a PID, not a pattern: tail --pid=1234 -f /dev/null, or while kill -0 1234; do sleep 30; done.
• Better still, let systemd own the job. Run it as a unit and ask systemctl is-active, or start it with systemd-run --wait. There is no pattern to get wrong.
• Every wait gets a deadline (timeout 3h) and says when it gives up. A loop with no deadline fails silently.

Factory Floor, the kit that runs our machines, works through systemd units: it never looks for a process by its command line. The loops that bit us were the ones we typed by hand.