Night Watch
Keep your sites up through scraper floods: an uptime watch, a Cloudflare Free-plan shield and a bot gate.
- Linux
- Python 3.10+
- Cloudflare Free plan
- ASGI or WSGI apps
What it does
Night Watch opens every site you run the way a visitor does, every five minutes, and tells you the moment one stops opening and the moment it is back. When a site falls over because a scraper is flooding it, the watch switches on a Cloudflare rule that stops the flood at the edge, and your app keeps serving people while it lasts.
- Every site, every five minutes. Every zone in your Cloudflare account, plus any page you list, optionally with text the page must contain. One message when a site goes down, one when it is back with how long it was down, a reminder while it stays down. By mail, a chat webhook, or both.
- A shield on the Free plan. Three WAF custom rules: a Chrome user agent without the headers every real Chrome sends gets a challenge, page requests from scraper clouds get a challenge, and an emergency rule challenges strangers with no cookie and no link during a flood.
- Switched on by itself. When a site goes down from overload, the watch turns on the emergency rule and says so in the alarm; hours after the site is back it turns it off. A shield you switched on yourself stays on.
- A gate inside your app. One file with no dependencies, for FastAPI, Starlette, Django, Flask or any ASGI or WSGI app. It refuses fake browsers before your code runs and keeps strangers within what each worker can serve. People with a cookie or arriving through a link never notice it.
- See what is hitting you. What Cloudflare's firewall challenged in the last minutes, by rule, network, country, user agent and path, read from the raw events the Free plan allows.
- Undo built in. Every rule change saves the zone's rules first, your own rules stay in place ahead of ours, and one command puts the saved set back.
Tested on our own sites
- A real flood, one hour: on a site of ours, the fake-browser rule challenged more than 10,000 requests. The 10,000 that Cloudflare lets us read came from 9,627 home broadband addresses, none sending more than 4: a per-address rate limit would never have fired.
- Every rule, probed live on the Free plan: a stranger was challenged while the emergency rule was on and let through with a cookie or a link; a Chrome user agent without browser headers was challenged; Googlebot passed. The zone's earlier rules were then restored exactly.
- The gate under 1,747 requests a second on uvicorn: every scraper without browser headers refused, strangers kept to what the app could serve, and every returning visitor, link visitor and Googlebot request served in a median 0.20 seconds, the page's own time.
What is inside
The watch
Sites from your Cloudflare account and a list, checks that look like a visitor, messages by mail or webhook, and a test that tells "all down" from "this machine is offline".
The shield
Install, status, on, off, uninstall and restore for one zone or all, with every change backed up first.
The gate
ASGI and WSGI middleware, a 2-second "one moment" page that hands people a cookie, and one log line a minute.
The runbook
What to do when a site is slow or down, how to read a flood, and what not to switch on in Cloudflare's dashboard.
Requirements
- Linux with systemd and Python 3.10 or newer; nothing to install from pip.
- For the shield and the event view, sites on Cloudflare (any plan, the Free plan included) and an API token.
- A machine other than the one serving your sites, so it can tell you when that one is down.
Good to know
- The emergency rule also challenges a person who types your address into a fresh browser, which is why it is off except during a flood.
- Verified bots and robots that name themselves are never challenged, and search engines are never turned away by the gate.
- The shield uses three of the five custom rules the Free plan allows per zone.
Questions
Do I need a paid Cloudflare plan?
No. Everything Night Watch does works on the Free plan: the three rules, the event view and the automatic switching. A paid plan only gives you more rule slots next to ours.
Will it block Google or AI search crawlers?
No. Cloudflare's verified bots and every robot that names itself in its user agent pass the shield, and the gate never turns away a search engine or a link preview. If you want to block AI crawlers, that is a separate choice in Cloudflare's dashboard.
My site is not on Cloudflare. Is it still useful?
The watch and the gate work anywhere: the watch opens any URL, and the gate protects any Python web app. The shield and the event view need Cloudflare.
How is this different from an uptime monitor?
An uptime monitor tells you that a site is down. Night Watch tells you too, then acts on the most common cause we see: it switches on the shield at Cloudflare's edge, and the gate keeps your app serving the people you want while the flood lasts.
Updated: