The bug in our secret scanner that canaries could not find

Shipping2026-10-31

Before any kit we sell is zipped, a gate scans it for the actual values of our own secrets. Every scan starts by planting canaries, one of every kind, and refuses to run unless it catches all of them. It always caught them. It still had a bug.

How the gate reads secrets

The gate does not only look for things shaped like keys. It reads the files where our secrets live (env files, JSON configs, key files), takes every value whose name says it is secret, and then searches what we are about to ship for those exact values, as written and base64-encoded.

The bug

For a line like this one:

DB_PASSWORD='Tr0ub4dor-and-3'  # rotated in May

the reader stripped the surrounding quotes only when the value both started and ended with a quote character. With a comment after the value, the line ends in a letter, so the quotes stayed. The comment was cut off next, and the gate went on to search for 'Tr0ub4dor-and-3', quotes included. A kit containing the password without those quotes would have passed.

Why the canaries never noticed

A canary tests a net: plant a value the scanner already knows, check that it is found. Our canary values were handed straight to the scanner, so they never went through the code that reads a config line. Every net was tested, and the parser in front of them was not.

How we found it

We were turning the gate into a product of its own, and wrote a test that fed it a config file in the shapes people actually write: export, quotes, trailing comments. The test failed on the quoted line with a comment. The fix reads a quoted value up to its closing quote and ignores whatever follows.

Did it matter?

We checked before changing anything. On our build machine the gate reads 133 secret values from the 59 files it is given, and on a second machine 499; not one had that shape. No earlier verdict changes: every kit we have released was scanned for the same values it would be scanned for now. We fixed the gate on all three machines anyway.

The takeaway

If a scanner extracts what it searches for, test the extraction with realistic input, not only the search with planted values. A canary proves the net works; it says nothing about whether the right fish ever reach it. The tests of Secret Gate now cover both, and before its release we checked that it reads exactly the same 133 values as our build gate.