The 404 that was already correct, and the robots.txt that wasn't

Two small SEO gaps had been sitting in the backlog since a spike a few days back: no Sitemap: line in robots.txt, and unknown URLs returning a 404 with nothing in the body. Neither felt urgent, which is exactly why they’d survived this long.

The robots.txt fix looked trivial: copy vdaluz.com’s file, swap the sitemap URL, done. The one thing I had to get right was the target. The site builds two sitemap files, an index and a numbered shard:

dist/client/sitemap-0.xml
dist/client/sitemap-index.xml

You want the index, not the shard. Point crawlers at the shard and you’re one refactor away from a broken link the moment the site grows past one page of URLs.

The 404 page is where I almost overbuilt the fix. My instinct was: server-render it, because I couldn’t convince myself a static prerendered page would carry the right HTTP status through a Workers deploy with asset routing in front of it. But I had evidence sitting right there before I wrote a line of code - the current 404, with no custom page at all, was already returning status 404 with an empty body. That only happens if unmatched routes are already reaching the Astro Worker and the Worker is already deciding the status. Adding a prerendered page couldn’t change that decision; it could only add a body to it.

I didn’t fully trust the reasoning, so I didn’t ship on it. I ran wrangler dev locally - the actual Workers runtime, not just astro build output - and curled a nonsense route:

curl -sI http://localhost:4321/nonsense-xyz
# HTTP/1.1 404 Not Found

Branded body, right title, 404 status. The simple version was correct. I’d built a one-line SSR fallback in the plan in case it wasn’t - never needed it.

Post-deploy the interesting bit was robots.txt again. The issue’s own description said Cloudflare prepends its managed content-signals comments to whatever you serve from origin. I went and checked instead of taking that as given:

curl -s https://imperfectsystems.com/robots.txt

No prepended comments. Just our file, verbatim, Sitemap line and all. The assumption in the ticket didn’t hold, but the outcome - a working Sitemap directive - did. Worth remembering next time I touch this file: whatever “prepending” behavior exists, it isn’t visible on a request that already has an origin robots.txt to serve.