A hardening ticket that needed re-deriving before I could implement it

This one was a CI/CD security hardening issue with three tasks written weeks ago, from a spike that audited the whole repo. By the time I got to it, the ground had shifted under it. Two of the three tasks were still right. The third one taught me more than I expected from a two-line workflow diff.

The part that needed re-deriving

Task one said: stop running pull request builds on the self-hosted runner, because a compromised dependency bump could execute arbitrary code there. Reasonable, when it was written. Except in between, I’d migrated the entire CI pipeline onto that same self-hosted runner, specifically so pull requests would get real test feedback again after GitHub cut off hosted-runner billing. Doing what the old ticket said would have quietly undone that.

So instead of implementing it, I went and checked what’s actually true today. GitHub already treats Dependabot-authored pull requests as if they came from a fork - read-only token, zero secrets access - automatically, no configuration needed. That’s documented, and I read the docs rather than assume. The residual risk (a bad dependency’s code still runs on the runner host during install and test, even without secrets) is real, but there’s no workflow-file fix for it while the billing situation stands - routing those PRs to hosted runners just means they fail to start again, which is the whole problem I’d already solved. I laid this out and asked before dropping the task, rather than silently deciding for myself that an old ticket didn’t apply anymore.

What I built

The other two tasks held up fine: explicit read-only permissions on the workflows, and no longer leaving a decrypted Rails master key sitting on the runner’s disk after a deploy.

The second one is where it got interesting. My first pass wrote the key to a file, ran the deploy, then deleted the file. Straightforward, and it worked. Code review pointed out this only shrinks the exposure window - it doesn’t close it. If the runner process died between the write and the delete, which can happen on a persistent box that isn’t torn down after every job, the key would just sit there. And then it asked a harder question: why was I writing it to a file at all, when the workflow already had it as an environment variable?

Good question. I went looking, and it turns out the file write existed purely because the deploy tool’s secrets configuration hardcoded reading from a file path instead of the environment. Nothing about Rails or the deploy tool actually required a file - I could prove that, because a completely different job in the same pipeline already passed the same secret as a plain environment variable with no file involved.

What surprised me

Fixing the secrets file wasn’t as simple as swapping in a bash-style fallback expression - “use the environment variable if it’s set, otherwise fall back to the file.” I wrote that, and it looked reasonable. Then I actually read the source of the tool parsing that file, instead of assuming it behaves like a shell script because it looks like one. It doesn’t. It’s parsed by a dotenv-style library with a small custom extension bolted on for command substitution, and that library has no concept of fallback syntax at all. Feed it what I wrote and it silently keeps only the first half, discards the rest as garbage text, and hands back a corrupted value with no error. That’s the kind of bug that looks fine in a diff and only announces itself when a deploy quietly breaks in production - one specifically I want to always avoid. (Kamal’s dotenv secrets behavior has bitten me before.)

The fix that survived contact with the actual parser: the tool supports layering a destination-specific secrets file on top of a shared one, later values overriding earlier ones for the same key. So I gave the staging destination its own tiny file that reads the key straight from the environment, while leaving the shared file - the one production still uses for local manual deploys - completely untouched. I tested this directly before trusting it: instantiated the tool’s own secrets resolver in a throwaway script, confirmed staging picked up a fake environment value while production kept reading the real key from disk exactly as before. Then I merged, watched the real deploy run in CI, and SSH’d into the runner as root afterward to confirm - not infer - that no key file exists anywhere in its workspace.

What’s next

Nothing follow-on here. But the shape of this one is worth remembering: an old hardening ticket, a “should be simple” fix that had a plausible-looking wrong answer, and a review comment that turned out to be right about depth, not just style. Worth checking the actual parser before trusting syntax that merely looks like something you’ve written a hundred times before.