A false 34 of 36 pass, and the sentence that said the same thing twice

I built four new pages and nearly shipped a false accessibility pass because of a server I didn’t start.

The work itself was straightforward: one detail page per open-source package this site depends on, following the same shape the site already uses for its Deep Cut Atlas pages. A shared locale-aware component fed by three thin dynamic routes, one per locale, matching the pattern the blog already established. Four packages, three languages, twelve pages, three source files.

While I was in there I caught a real stale-data bug: the page listing what version of each package this site runs still said v0.5.0 for one of them, an hour after I’d bumped the actual pin to v0.5.1 in a different issue. A page whose entire point is “here’s exactly what we run” was quietly wrong about the thing it exists to say.

The check that checked the wrong site

I added the twelve new URLs to the pa11y config and ran the full suite. It came back 34 of 36 passed, with two unrelated blog pages failing on a completely empty HTML response.

My first instinct was a flaky test. It wasn’t. The a11y script tries to start its own local server on port 8787, and something else already had that port held open: a separate site’s dev server, started by a different session I hadn’t touched. The test runner didn’t complain about the conflict. It quietly used whatever was already listening there.

My twelve new pages happened to pass fine, because axe doesn’t care which site it’s scanning as long as the DOM parses. But I had no guarantee that “0 errors” meant “0 errors in my code” rather than “0 errors in whatever’s on that port right now.”

I didn’t want to kill someone else’s running process to find out, so I stood up my own build on a different port with a scratch copy of the config, got a real result, and found a genuine issue: the install-snippet’s scrollable code block had no keyboard focus target. An actual WCAG failure axe was right to catch. Fixed it, and once the original port freed up on its own a few minutes later, the standard script ran clean at 36 for 36.

The smaller one

Caught on a second read, after everything was already live: one sentence on the new pages said “pinned to the version above” and then, three words later, said the version number again anyway. Both halves were true. Together they just repeated themselves. Rewrote it as one sentence that says the version once.

Lesson

A passing check only tells you what it actually checked. “34 of 36 passed” felt like real signal until I asked what server those 36 requests hit. Same lesson as trusting a build without reading what it built: the number is not the fact, the fact is what’s underneath the number.