Making blog scans async with Solid Queue (and a Rails gotcha)

I added a Scan button to blog-manager a few weeks ago. You click it, it hits the GitHub API for every post in the repo, updates the database, and shows you a summary. Simple enough. And it worked fine while the scan stayed small.

Then the scan crossed 300 API calls.

The scan started timing out. Not because Rails was crashing - the job actually finished. But kamal-proxy has a ~30s timeout, and 300 GitHub API calls take about 55 seconds. The browser got a 504. Rails finished anyway and updated the database, completely silently. Not great.

The fix was obvious: move it to a background job. Solid Queue was already in the Gemfile from the Rails 8.1 default. I just hadn’t written my first job yet.

Wiring it up

The first job in any Rails app sets the pattern for everything that follows, so I spent some time thinking about the shape of it before writing a line.

The job needed to do three things:

1. Track state on the Blog record (idle, running, failed) so the UI could reflect what’s happening without polling an endpoint
2. Store the last result summary (last_scan_summary) so users could see what happened after the job finished
3. Handle errors correctly - auth failures and 404s should fail fast, transient GitHub errors should retry

I added a scan_state enum and last_scan_summary text column to blogs. The job sets running at the start, then either idle (with the result summary) or failed (with the error) at the end.

The gotcha

Rails ActiveJob gives you two declarative tools: retry_on for transient failures and discard_on for permanent ones. I wanted auth errors and 404s to discard immediately, and generic GitHub errors to retry 3 times.

I wrote the job with discard_on first, then retry_on. My tests for the discard path failed - the blog stayed in :running state.

The cause: Rails uses rescue_from under the hood for both, and rescue_from uses a LIFO stack. Last registered, first checked. I’d registered discard_on first and retry_on second, so retry_on was at the top of the stack. Since AuthError inherits from Error, retry_on Github::ContentClient::Error matched it before discard_on ever ran.

The fix: swap the order. Define retry_on first, then discard_on. Now discard_on is at the top of the stack and catches auth errors before retry_on sees them.

# CORRECT
retry_on Github::ContentClient::Error, wait: :polynomially_longer, attempts: 3

discard_on Github::ContentClient::AuthError, Github::ContentClient::NotFoundError do |job, error|
  job.arguments.first.update!(scan_state: :failed, last_scan_summary: "...")
end

I hadn’t seen this documented clearly anywhere. It makes sense once you know rescue_from is LIFO, but it’s easy to miss.

Mission Control

With jobs running in the background, I wanted visibility. Solid Queue’s companion gem mission_control-jobs adds a Sidekiq-style dashboard at any path you mount it on. I wired it up at /jobs in about five minutes.

One config note: setting base_controller_class to use the app’s own auth isn’t enough on its own. You also need http_basic_auth_enabled = false, otherwise both auth mechanisms run and HTTP Basic wins.

What’s next

The UI still requires a manual refresh to see job results. That’s the Turbo Streams wiring.