Factory Floor
The autopilot under everything: local AI, watchdogs and self-healing ops for your own GPU box.
- Linux
- NVIDIA GPU
- ARM64 or x86-64

What it does
Factory Floor is the layer that lets one person run hundreds of scheduled jobs without watching them. It is what keeps our own fleet working at night: it serves the language models every other kit uses, notices the jobs that die quietly, repairs the failure classes it knows, and stays silent unless something truly needs you.
- Local models as a service. Serve an OpenAI-compatible endpoint from one or two GPU boxes behind a least-connection proxy, and run a tool-use test before you switch models, so an upgrade never breaks your agents.
- Watchdogs for the silent failures. Services that are enabled but dead, units that failed, timers that stopped firing, jobs that hang for hours: each has a detector and, where the fix is known, a repair.
- Guards for the machine. Memory, disk and temperature guards pause the heavy jobs before the box falls over, then resume them.
- One email, not a hundred. Alerts are grouped, deduplicated and written for a human: what broke, what was already tried, what you need to do.
- Goals that run in loops. Give a local agent a goal; it works on it round after round in a sandbox, writes a log, checks its own result and stops when you say so.
What is inside
Units and timers
systemd templates for workers, timers with jitter, resource limits and restart rules.
Watchdogs
Detectors for dead, failed and late units, plus a repair rules file you can read and edit.
Model serving
Proxy and launch configs for one or two boxes, and the tool-use evaluation harness.
Machine guards
Memory, disk and thermal guards with pause and resume for heavy jobs.
Alerting
An email sender with grouping and quiet hours, and a daily summary of what ran.
Runbooks
Install, upgrade and recovery guides, written from incidents we actually had.
Requirements
- Ubuntu 22.04 or 24.04 with systemd, x86-64 or ARM64.
- Python 3.10 or newer.
- For local models: an NVIDIA GPU with 24 GB or more (DGX Spark-class machines are ideal), or any OpenAI-compatible endpoint.
- An SMTP mailbox for alerts.
Good to know
- Auto-repair restarts services. Start in report-only mode, read the rules, then switch repairs on.
- Model weights are not included: the setup script downloads the models you choose, under their own licences.
- Your licence covers your own machines and your own business.
Questions
Does it work with one machine?
Yes. The second machine is optional; the proxy simply has one backend.
Which models does it support?
Anything served through an OpenAI-compatible API. The tool-use test tells you whether a model is good enough for agents before you switch.
Do the other kits need it?
No, but they get better with it: every AI-heavy kit can use the endpoint and the watchdogs Factory Floor provides.
Updated: