LightsOut Factory field noteshttps://lightsoutfactory.ai/field-notes/feed.xml2026-11-02T00:00:00ZLightsOut Factory Our wait loop waited 35 hours for itself: pgrep -f and pkill -f over SSHhttps://lightsoutfactory.ai/field-notes/wait-loop-waited-for-itself/2026-11-02T00:00:00Z2026-11-02T00:00:00ZA loop meant to wait for a job on another machine found its own command line instead, for 35 hours. Why pgrep -f and pkill -f match the shell that runs them, and the patterns we use now. In 14 of our 42 languages, the plural form called "one" is not just for 1https://lightsoutfactory.ai/field-notes/plural-one-is-not-one/2026-11-02T00:00:00Z2026-11-02T00:00:00ZRussian "one" is also 21 and 31, French "one" is also 0. Our translation check got this wrong in both directions on the same day: it refused a right translation and would have passed a wrong one. 1,082 translations had a word in the wrong alphabet. Our checks passed every one.https://lightsoutfactory.ai/field-notes/letters-of-another-script/2026-11-02T00:00:00Z2026-11-02T00:00:00ZA Chinese word inside Hungarian, Cyrillic inside Greek, Thai on Khmer pages: what a local model slipped into our translations, why a script check let it through, and the rule that catches it. Our watchdog restarted the services we had stopped on purposehttps://lightsoutfactory.ai/field-notes/watchdog-restarted-stopped-services/2026-10-31T00:00:00Z2026-10-31T00:00:00ZA watchdog that starts enabled but stopped services cannot tell a crash from a pause. Twice it undid our stops, once with visitors getting 502s. The rule we use now, and two more systemd traps. The view counter that turned our catalogue into a 502https://lightsoutfactory.ai/field-notes/view-counter-broke-the-catalogue/2026-10-31T00:00:00Z2026-10-31T00:00:00ZFor about 41 hours the main catalogue page of our store answered every visitor with a 502, because the code counting its visits crashed on its address. Why none of our checks saw it, and the three changes that would have. Two DGX Spark-class boxes, one model: a copy on each beat splitting ithttps://lightsoutfactory.ai/field-notes/two-sparks-one-model/2026-10-31T00:00:00Z2026-10-31T00:00:00ZWe split one model across two NVIDIA GB10 machines with tensor parallelism, then ran a full copy on each behind a proxy: 180 to 210 tokens a second. What it cost us. We checked every machine translation. The checks were the part that was wrong.https://lightsoutfactory.ai/field-notes/translation-checks-were-wrong/2026-10-31T00:00:00Z2026-10-31T00:00:00ZA local model translated a small app into five languages in six file formats. Four of 350 strings were rejected, and all four rejections were mistakes in our own checks: a loanword, an e-mail address and Japanese plural text. The bug in our secret scanner that canaries could not findhttps://lightsoutfactory.ai/field-notes/secret-gate-quoted-value-bug/2026-10-31T00:00:00Z2026-10-31T00:00:00ZOur release gate plants canaries before every scan, and they always passed. The bug was one step earlier: a quoted secret followed by a comment was read with its quotes. How we found it, why canaries miss this kind of bug, and what tests it now. A scraper flood on Cloudflare's Free plan: what stopped it, and what we left switched offhttps://lightsoutfactory.ai/field-notes/scraper-flood-cloudflare-free-plan/2026-10-31T00:00:00Z2026-10-31T00:00:00ZA scraper behind residential proxies took one of our sites down for about ten hours. What gave it away, the three free WAF rules and the app-level gate that stopped it, and the Cloudflare switches that would have broken something else. Our health check restarted a healthy model server eight times in one dayhttps://lightsoutfactory.ai/field-notes/health-check-restarted-healthy-model/2026-10-31T00:00:00Z2026-10-31T00:00:00ZThe check sent a real request to a busy vLLM server, waited in its queue, timed out and restarted it. Why, and the check that replaced it: the server's own counters. Our GB10 machine kept switching itself off. The log blamed memory; it was heathttps://lightsoutfactory.ai/field-notes/gb10-switching-off-heat/2026-10-31T00:00:00Z2026-10-31T00:00:00ZHard power-offs under GPU load, a kernel error that was only the most common line in the log, and what kept the machine running: a clock cap, a staged heat guard and alarms set from data. YouTube now counts uploads in their own bucket: 100 a day, one unit eachhttps://lightsoutfactory.ai/field-notes/youtube-upload-quota-bucket/2026-10-29T00:00:00Z2026-10-29T00:00:00ZGoogle moved video uploads out of the 10,000-unit daily quota. What changed in the YouTube Data API, what it means for an automated channel, and the limits that still apply. A vision model as a frame checker: what it caught and what it let throughhttps://lightsoutfactory.ai/field-notes/vision-model-frame-check/2026-10-29T00:00:00Z2026-10-29T00:00:00ZBefore a generated picture costs render time, Studio asks a local vision model whether it shows text, a watermark or deformed hands. What our test frames showed. When NVENC will not start: a GPU full of models, and a renderer that falls backhttps://lightsoutfactory.ai/field-notes/nvenc-gpu-full-of-models/2026-10-29T00:00:00Z2026-10-29T00:00:00ZOn a machine that also serves a language model, ffmpeg's hardware encoder can fail to open for lack of GPU memory. How it showed up, why, and why the CPU fallback cost less than expected. Shipping code from a working fleet without shipping its secretshttps://lightsoutfactory.ai/field-notes/keeping-secrets-out-of-kits/2026-10-29T00:00:00Z2026-10-31T00:00:00ZEvery kit we sell passes secret scans on three machines before it is zipped. What they look for, and the false alarms that taught us the most. A 24/7 stream that does not reconnect between videos: one ffmpeg fed through a pipehttps://lightsoutfactory.ai/field-notes/24-7-stream-one-ffmpeg-pipe/2026-10-29T00:00:00Z2026-10-29T00:00:00ZHow Studio's live channel changes videos without dropping the RTMP connection, and the three details that made the timestamps behave.