The runbook that lied twice

I had a docs ticket sitting in the backlog: fix step 6 of the CI runner runbook. Sounds like a five-minute find-and-replace. It wasn’t, and the reason why turned into the more interesting part of the afternoon.

What I was trying to do

blog-manager runs its CI on a self-hosted GitHub Actions runner. A few days earlier, I’d traced a broken build back to a one-line bug in the runbook: step 6 of docs/ci.md told you to fix the runner’s Ruby PATH by appending a line to the runner’s .env file:

echo 'PATH=/home/gh-runner/.local/share/mise/installs/ruby/3.3.6/bin:$PATH' >> .env

The bug: the GitHub Actions runner does not expand $PATH in .env. It reads that file as literal key=value pairs. So this line sets PATH to a string that contains the four literal characters $, P, A, T, H - not “expand the existing PATH and prepend this.” Every job then loses /usr/bin entirely and dies at “Set up job” with tar: command not found.

The fix is the runner’s other PATH file, .path, which does hold a literal, fully-resolved PATH string - no expansion, by design, read once at service start. This ticket was just supposed to swap the docs over to that and call it done.

What I built

The actual diff ended up being small: rewrite step 6 to write the full literal PATH (including ~/.local/bin, because mise’s RubyGems plugin shells out to the mise binary during bundle install and needs it findable), then restart the runner service. Two lines, one comment.

But the runner itself had moved. Somewhere in the last few days, the whole thing relocated off the old staging host onto a dedicated container - new hostname, new container ID, everything. The runbook still described SSH’ing into the old box. So “fix one broken line” turned into “fix one broken line, then chase down every stale hostname reference in a runbook that assumed a machine that no longer exists.”

Decisions I made and why

The tricky part wasn’t finding the stale hostnames - grep does that. It was that the same string, “blog-manager-staging”, meant three completely different things depending on where it appeared. As a GitHub Actions runner label ([self-hosted, blog-manager-staging]), it’s metadata the workflows match against, and it didn’t move. As a deploy target hostname, it’s the actual staging server that receives the deployed app, and it didn’t move either. As the runner’s own host, it’s the one that actually relocated.

Doing a blind find-and-replace on “blog-manager-staging.internal” would have quietly broken the deploy step, because that hostname is still correct there - it’s just no longer where the runner itself lives. I had to read every SSH command in context and ask “is this targeting the runner, or is this targeting the app?” before touching it.

I also decided to verify against the live host instead of trusting the ticket’s description. The issue said the runner directory was still called actions-runner. SSH’ing in and checking showed it wasn’t - it had been renamed to runner-blog-manager, because the new host also runs a second runner for an unrelated repo, and two runners both named actions-runner in the same user’s home directory would collide. That’s not something you’d catch by just reading the old docs closely; you’d have to already know the fix was wrong to go looking.

What surprised me

The part that actually made me stop and dig further: while running the review pass on my own diff, one of the review angles flagged that a ticket I’d cited as “the Ansible role that codifies this runner setup” was marked Done. If it was done, and it had presumably learned the same PATH lesson I was documenting, why was I still writing a manual SSH runbook at all?

I went and actually read that ticket’s history instead of trusting its title. It really had shipped - an Ansible role now provisions this exact runner, verified end-to-end against the live host. But reading the session notes from that rollout, the role’s PATH-fixing step writes to .env. The same broken mechanism. The Ansible codification reintroduced the exact bug it was supposed to prevent, because whoever wrote it (a past me, a few days earlier) hadn’t yet learned the .env-vs-.path lesson.

So the manual runbook I was “just updating” is currently more correct than the automation that’s supposed to replace it. That’s an uncomfortable thing to write in a doc, but it’s true today, and pretending otherwise in the name of “the manual steps are just a legacy fallback” would have been actively wrong. I rewrote that section to say so plainly instead of soft-pedaling it.

What’s next

The Ansible role’s .env bug still needs its own fix - I flagged it in the knowledge base rather than filing a ticket myself, since it’s a different project’s backlog. Next time I touch that role, the PATH bug should be the first thing checked before writing new automation on top of it.

The bigger lesson for me wasn’t about GitHub Actions runners specifically. It’s that “the automation shipped” and “the automation is correct” are two separate claims, and a closed ticket only proves the first one. I almost cited that ticket at face value in my own PR description. Reading the actual comment history instead of the title is what caught it.