Backfilling reality into a syndication tracker

A few days ago I shipped the UI that shows which of my posts are on Medium. Then I opened it in production and it said none of them were. 169 posts, zero Medium entries. The data model was fine; the production database had simply never been told what dev already knew.

The easy part: rerun the codified backfill

The earlier reconciliation work lived in rake tasks, not in someone’s shell history, and that paid off immediately. medium:seed_published and medium:seed_scheduled shipped inside the production image, and backfill_medium! only ever upgrades state, so running them against prod was a non-event: 76 posts marked published, 8 scheduled, 0 slug mismatches. If you’re ever tempted to do a one-off data fix by hand in a console, this is the argument for writing the throwaway rake task instead - the “one-off” ran twice.

One prerequisite surfaced along the way: production had no Postiz API key, which meant the hourly reconcile job had been silently skipping since the deploy. I piped the key from the dev database over stdin so it never touched a terminal, process args, or shell history.

The reconciler was broken everywhere and nobody noticed

Step four of my plan was “run the RSS reconciler to pick up real URLs.” It crashed: ArgumentError: not an HTTP URI. The reconciler builds the feed URL from the Postiz integration’s profile field and assumed it was a URL. Postiz returns the bare handle, “vdaluz”. URI.parse("vdaluz") has no host, the code built "vdaluz/feed", and Net::HTTP refused it. Same failure in dev, which means the hourly job had been failing everywhere and the thing that will mark my 8 scheduled posts as published when they go live did not work at all.

The fix was small but the review made it better. My first version mapped any host-less profile to a medium.com handle feed. The review pointed out that URI.parse returns no host for ANY scheme-less string, so a custom domain stored as “blog.example.com” would silently become medium.com/feed/@blog.example.com, fetch a 404, parse as an empty feed, and report success forever. The shipped version only treats handle-shaped strings (no dots, no slashes) as handles, raises a named error for anything unrecognized, and raises on non-2xx responses. A wrong guess now fails loudly in the job queue instead of quietly disabling reconciliation.

Scraping URLs out of a client-rendered dashboard

66 of the backfilled posts had no Medium URL - the UI showed “Published on Medium” with nothing to click. Medium killed its API, and RSS only exposes the 10 most recent stories, so the full list exists in exactly one place: the logged-in stories dashboard. Saving that page gives you a 99KB React shell with zero story data. The trick that worked:

// on medium.com/me/stories/public, scrolled to the bottom:
copy(document.body.outerHTML)
// then: pbpaste > /tmp/published_dom.html

The rendered DOM has direct public URLs for every story. A Nokogiri script matched 76 of 77 stories to posts by normalized title - the mismatches were Medium appending my name to some titles (with a non-breaking space, naturally) and one title I’d rewritten since 2022. The 77th is a Medium-only duplicate I refused to guess at. One idempotent runner script later, production has links on all 76 published posts. Medium 403s every non-browser client, so I verified by provenance rather than curl.

The CI failure that was three failures wearing a coat

Merging the reconciler fix took longer than writing it. The CI build failed with bundle: command not found - the runner had just been relocated to a dedicated container and missed the PATH setup step. The documented fix appended PATH=...:$PATH to the runner’s .env, and after a service restart CI failed earlier, with tar: command not found. That’s when I learned the actions runner does not expand variables in .env: jobs now had a PATH containing a literal dollar-sign string and no /usr/bin. The real mechanism is the .path file, which holds the complete literal PATH. Third failure: Errno::ENOENT - mise during bundle install, because mise’s RubyGems plugin shells out to mise reshim after every gem install, so the mise binary itself needs to be on the job PATH too.

Each fix moved the failure one step deeper into the job, which is the most honest progress bar CI can give you. The part I keep relearning: the runbook step had “worked” before only because nobody had restarted the runner service after editing .env. Config that’s never been reloaded is config that’s never been tested.

What’s next

The 8 scheduled posts publish weekly through late August; the fixed hourly reconcile should flip each one to published with its URL, no hands. That’s the real test of this whole chain. The runbook correction is filed, and the runner provisioning is getting codified in Ansible so the PATH lesson is encoded once instead of relearned.