The view counter that turned our catalogue into a 502
Shipping
The page that lists every kit we sell answered every visitor with a 502 Bad Gateway for about 41 hours after launch. Each kit's own page worked, the home page worked, checkout worked. The broken part was a few lines that counted page views.
The bug
Our store's server counts visits to kit pages. To find which kit a request is for, it took the second part of the path:
self._count_view(path.strip("/").split("/")[1] if path.count("/") >= 2 else "")For /kits/studio/ that is studio. For the catalogue itself, /kits/, the path has two slashes, so the guard let it through, but after stripping the slashes there is only one part, and [1] raised an IndexError. The handler is Python's standard http.server: an exception in do_GET closes the connection without an answer, and Cloudflare showed the visitor a 502.
The fix is one small function that returns an empty string when the part is missing. The view count only ever needed the kit's name; it never needed to be able to break the page it was counting.
Why none of our checks saw it
- The uptime watch opened home pages only. The home page was fine.
- The release step checked each kit it shipped: its page, its checkout redirect, the paid path end to end. It never opened the list of all kits.
- The analytics could not show it. The code that would have counted the catalogue's visitors was the code that crashed. The server log had the answer all along: 17 tracebacks in those 41 hours, each one a request, from a person or a crawler, that got an error instead of the page.
What changed
- Every page, from outside, every hour. A small job reads the sitemap and opens every English page and, in turn, one of the other 42 languages, through Cloudflare like a visitor. Two failed runs in a row send an email; recovery sends another.
- The release step opens every English page after it rebuilds the site, and refuses to finish if one does not answer 200.
- Index pages count as pages. The pages that list other pages are the ones a visitor reaches first, and they were the ones nobody tested.
The takeaway
Side work in a request handler, such as counting, logging or tracking, must never be able to fail the request, and a page check that only covers the pages you just changed misses the ones that link to them. The watch in Night Watch takes any list of pages, each with text it must contain, not just home pages.