Moving CI to a self-hosted runner after GitHub broke our billing

I’ve had blog-manager’s CI dead for days. Not flaky, not slow. Dead. Every job on ubuntu-latest was failing in about four seconds with a billing annotation before a single step ran. GitHub Actions minutes on hosted runners need a working payment method, and mine wasn’t one right now, so the whole CI workflow just refused to start.

The part that actually worried me: deploys to staging kept working the entire time. The deploy pipeline was gated on the build workflow finishing, not on CI. Build already ran on a self-hosted runner I’d set up months ago for a completely different reason (the Gitea registry only resolves on the internal network, so GitHub-hosted runners can’t reach it). So builds kept happening, staging kept deploying, and nothing was actually testing the code first. That’s the setup you don’t notice is broken until you go looking.

What I built

The fix was to move the rest of CI onto the same self-hosted runner and make the pipeline actually gate on it. Concretely:

• Folded the five old CI jobs (two security scans, lint, tests, system tests) into the same workflow file as the Docker build, as three jobs: checks, test, system-test.
• build now has needs: [checks, test]. If either fails, build gets skipped, and the staging deploy - which triggers off the whole workflow completing successfully - never fires.
• system-test runs headless Chrome on a single-core LXC container. I didn’t trust it to be reliable on day one, so it’s continue-on-error: true - visible if it fails, but not blocking anything yet. I’ll flip that once it’s proven itself over a few runs.

Decisions I made and why

Drop the GitHub-hosted caching machinery entirely rather than trying to preserve it. The old jobs used ruby/setup-ruby with bundler-cache: true and an actions/cache step for RuboCop. Both of those exist to work around GitHub-hosted runners being ephemeral - every job starts from a blank VM, so you need to fetch a cache to avoid reinstalling everything. A self-hosted runner’s disk is already persistent. The gems from the last run are just… still there. So I deleted all of that and let bundle install run against the runner’s already-populated gem directory.

This bit me almost immediately, in a way I didn’t expect. bundler-cache: true doesn’t just cache gems - it also puts Bundler into a mode that fails loudly if Gemfile.lock doesn’t match Gemfile. Drop it in favor of a bare bundle install and that protection disappears silently; Bundler just re-resolves and moves on. I only caught this because I went looking for the equivalent behavior. Fixed it with BUNDLE_FROZEN=true bundle install instead - same effect, no dependency on GitHub-hosted-runner scaffolding.

Only pre-install what the runner actually needs once, not what every job insists on reinstalling. The old test and system-test jobs ran sudo apt-get install libvips / chromium as a step, every single run, because on a fresh VM that’s the only option. I installed both as one-time system packages on the runner host instead and deleted the apt steps from the workflow. Small thing, but it’s the same theme: a lot of what GitHub-hosted CI does is compensate for having no memory between runs.

What surprised me

The first real CI run in weeks turned into a security audit I wasn’t expecting. bundler-audit failed with a stack of nokogiri CVEs I hadn’t seen locally, because my local advisory database was stale and the fresh self-hosted run pulled the current one. Once I updated locally and re-checked, four more showed up behind it: msgpack, faraday (a high-severity one), crass, concurrent-ruby. None of this was caused by the CI migration - the dependencies had just drifted quietly during the weeks CI couldn’t run to catch them. Bumping all five was its own small commit before I could even get back to verifying the pipeline changes. That’s a pretty direct argument for why the whole exercise mattered: the gate had been off, and it had already let real, if boring, vulnerabilities through.

The other surprise was almost a non-event, and I’m glad I checked instead of assuming. While tracing through how deploy-staging gets triggered, a candidate bug came up that looked serious: does GitHub’s workflow_run trigger match a pull request’s base branch or its head branch? If it’s base branch, then every green PR against main would spuriously kick off a staging deploy using a commit that was never actually pushed to the registry. The docs are genuinely ambiguous on this. Rather than trust either my instinct or a search result, I checked the actual run history for the deploy workflow after two green PR runs that day. Nothing fired. It only ever triggers off pushes to main, exactly as intended. Good reminder that for anything with a real blast radius, “the docs say” and “I’m fairly sure” are both weaker than just looking at what actually happened.

What’s next

System tests need a few clean runs before I trust them enough to make them blocking. And the runner itself is still hand-configured - Ruby via mise, the runner binary, now libvips and chromium, all installed by SSHing in and typing commands. That’s fine for a single box I stood up once, but it means rebuilding it from scratch would mean redoing all of that from memory. I filed a follow-up to actually codify it in Ansible.