A scraper flood on Cloudflare's Free plan: what stopped it, and what we left switched off

Ops2026-10-31

One morning a listings site of ours stopped opening. Cloudflare showed visitors a 524, the origin took two to six minutes per page, and nothing told us for about ten hours. The cause was a scraper going through residential proxies at about 225 requests a second.

The shape of the flood

• Thousands of addresses, one to three requests each. Home broadband connections in the US, Spain, France, Brazil, the UK and elsewhere, the kind of traffic a residential proxy network rents out.
• Chrome user agents on a Mac, with version numbers from 103 to 151.
• Almost never a cookie or a Referer. Of 2,697 requests we sampled at the server, 4 carried a cookie and 4 a Referer.
• Two kinds of client. About half sent no Sec-Fetch-* headers and no client hints: a script with a copied user agent string. The other half sent full browser headers: a headless browser.

Every page was rendered for every request, so each worker queued hundreds of renders and hit its CPU limit.

Why per-address limits could not help

A rate limit counts requests per address. With one to three requests from each of thousands of addresses, no single address ever comes near a limit. Blocking countries would have blocked our visitors. What the scraper could not fake cheaply was behaving like a browser that has been to the site before.

At Cloudflare's edge: three free rules

The Free plan allows five WAF custom rules per zone. We used three, all with a managed challenge rather than a block, so a person caught by mistake clicks through:

• Fake browser: a GET whose user agent contains Chrome/ and which has no Sec-Fetch-Mode header. Every Chrome since 2019 sends it.
• Scraper clouds: a page request from the networks of Alibaba, Tencent and Huawei Cloud, where much of what we challenged came from.
• Emergency, off by default: a GET with no cookie at all and no Referer. A person who has been to the site, or who arrives through a link, passes untouched; a scraper that keeps no cookies meets a challenge on every request. We switch it on only during a flood, because it also challenges someone who types the address into a fresh browser.

All three let through Cloudflare's verified bots, user agents that name themselves as robots, our own addresses, the API and webhook paths, and files that are not pages. Two days later the flood was still going: in one hour the fake-browser rule challenged more than 10,000 requests. The 10,000 events Cloudflare returns per query came from 9,627 addresses, none with more than 4.

At the origin: a gate in the app

The headless half has full browser headers, so the edge rules alone do not catch it while the emergency rule is off. In the app itself we put a small middleware in front of the page renderer:

• a Chrome user agent with neither Sec-Fetch-Mode nor client hints gets a 403 before any work is done;
• a stranger, with none of our cookies and no Referer, is served only while the worker has spare capacity, at a few pages a second. Over that, a 400-byte 503 page sets a cookie and reloads after two seconds. A person gets the page on the second try; a scraper that keeps no cookies never does.

In its first minute the gate refused about 6,700 fake browsers and turned away about 6,700 strangers, while serving strangers about 9 pages a second. Slow requests dropped to zero, the workers' CPU fell from their limit to 12 to 15 percent, and their memory from 4 to 5 GB each to 0.65 GB.

What we left switched off

• Bot Fight Mode. It challenges automated traffic and cannot be skipped with a custom rule on the Free plan, and it caught our payment provider's webhooks.
• Under Attack mode for the whole zone. It puts every visitor, API client and webhook through the interstitial. The emergency rule does the same job for strangers' page requests only.
• The Free plan's rate-limiting rule. It counts per address, which this flood never triggers.
• Blocking AI crawlers. We want to be found by AI search assistants, and those crawlers were not the problem.

The takeaway

A flood through residential proxies is easy to spot by what it lacks: browser fetch headers, cookies and Referers. Challenge that at the edge, keep strangers within what your workers can render, and make sure something outside the server notices when a page stops opening. Night Watch packages all three: the watch, the rules with an automatic switch for the emergency one, and the gate for Python web apps.