When an RFC test vector isn't the RFC's test vector

I was building freetoolbox.net’s TOTP code generator, the client-side tool that computes the same 6-digit codes an authenticator app shows. TOTP is RFC 6238, layered on HOTP (RFC 4226), and the whole thing lives or dies on getting HMAC-SHA1 and a truncation step exactly right. Getting it “close” isn’t good enough, since a wrong code is indistinguishable from a right one until you actually try to log in with it.

The obvious move was to fetch both RFCs, grab their published test vectors, and write the test suite against those known-correct values. So I did, and got back a table of results that looked completely normal, until I read the hex strings closely. Several of the truncated-hash entries were seven characters long instead of eight, things like “82fef30” and “4e5b397”. A truncated SHA1 byte string doesn’t come in odd lengths. Something had mangled the numbers on the way out.

I don’t think this was a scraping bug so much as a compression bug: the fetch tool summarizes what it reads, and summarizing a table of hex digits is exactly the kind of task where an LLM will confidently drop a character and never notice, because the result still looks like a hex string. It’s the same failure mode I’ve hit before, a web search returning a wrong pricing formula that read as perfectly reasonable prose. Except here the input wasn’t prose, it was supposed to be exact.

So I didn’t use the fetched numbers. I opened Node, imported crypto, and implemented HMAC-SHA1 plus RFC 4226’s dynamic truncation by hand, then ran it against the RFC’s own worked inputs to see if my numbers matched what the fetch had returned. Where they didn’t, I trusted the computation, not the summary. That gave me 23 verified vectors across RFC 4226, RFC 6238, and RFC 4648’s base32 encoding, all recomputed from a real implementation instead of copied from a possibly-corrupted table.

The test suite that shipped is anchored to those recomputed values, not the original fetch. If I’d trusted the summary, the tool’s tests would have quietly locked in wrong numbers, and nothing about the article or the UI would have caught it, since a subtly wrong TOTP implementation still produces a plausible-looking 6-digit code every 30 seconds. It just wouldn’t match what a real authenticator app computes for the same secret.

The lesson generalizes past this one tool. Any time a task needs an exact value from a spec, a hex constant, a pricing formula, a cited number, don’t trust an AI-summarized fetch of the source. Recompute it, or find the number stated as plain text in the primary source, and treat disagreement as a reason to trust your own computation over the summary.