Building self-hosted hero images before I had anything that needed them

I picked this up mostly as plumbing work: blog-manager has a Pexels integration that lets me search for a stock photo and stage it as a post’s hero image, but the URL it stores is a hotlink straight to Pexels’ CDN. That’s fine for Pexels since their terms want hotlinking anyway. It’s not fine for Openverse, the CC-licensed image search I want to add next, because Openverse just proxies searches across Wikimedia, Flickr, and random museum servers, any of which can 404 or start blocking hotlinking whenever they feel like it. CC licenses explicitly allow downloading and rehosting, so the fix is to actually download the image and commit it into the blog repo like every other hero image already there.

A previous session had already resolved the two open scoping questions (storage backend, download pipeline shape) by digging through the actual vdaluz.com repo, so I skipped straight to building.

What I built

A HeroImage::Downloader service: fetch bytes from a URL, reject anything that isn’t image/* or over 10MB, follow up to 3 redirects (a lot of CC image hosts redirect), and normalize to JPEG at quality 82 with ruby-vips:

::Vips::Image.new_from_buffer(body, "").jpegsave_buffer(Q: JPEG_QUALITY)

Then a per-source flag on Post so the self-host behavior is opt-in per provider, not a global switch:

HERO_IMAGE_SOURCE_INFO = {
  "pexels"    => { name: "Pexels", url: "...", self_host: false },
  "unsplash"  => { name: "Unsplash", url: "...", self_host: false },
  "openverse" => { name: "Openverse", url: "...", self_host: true }
}.freeze

And HeroImage::RepoCommitter grew a second commit path. GitHub’s Contents API only accepts one file per PUT, so a self-hosted hero now costs two commits instead of one: first the image bytes land at public/assets/images/<slug>.jpeg, then the existing frontmatter commit runs, except now it writes the self-hosted site-relative path into heroImage instead of the raw provider URL.

Decisions

I skipped the image_processing gem even though it’s already sitting unused in the Gemfile. Its Vips backend wants a file path (it calls Vips::Image.new_from_file), so using it here would mean writing the downloaded bytes to a tempfile just to hand it a path. Raw Vips::Image.new_from_buffer does the same job straight from memory. Since I’m not resizing anything, just re-encoding format, the extra abstraction wasn’t buying me anything.

I also almost used the GitHub content client’s get_file to look up an existing image’s sha for idempotent overwrites, then noticed it force-encodes the response to UTF-8:

decoded = Base64.decode64(encoded).force_encoding("UTF-8")

That’s fine for markdown, not fine for JPEG bytes. list_directory returns file metadata (name, sha, size) without touching content, so I used that instead to find an existing file’s sha before overwriting.

What surprised me

I wanted to verify the download pipeline against something closer to what Openverse will actually proxy, so I tried a real Wikimedia Commons URL. Got a 400, then a 403 even with a descriptive User-Agent. I didn’t chase it further since it’s not something I control, and httpbin’s /image/jpeg and /image/png test endpoints did the same job of proving a real network fetch actually works and actually converts formats.

The bigger surprise was in manual testing, and had nothing to do with my code. The “missing hero” queue in blog-manager is driven by a DB column (live_hero_image_url) that only gets refreshed by scanning the repo, and this blog hadn’t been scanned in over a month. One post had a hero live in its frontmatter that the DB didn’t know about yet, so my new insertion-only guard correctly refused to double-write it, which looked like a bug but was actually the guard doing its job against stale data. Running a fresh scan to fix it took about four minutes and looked hung the whole time. Turned out Solid Queue just doesn’t give you a progress bar, and the real tell that it wasn’t stuck was watching Post#updated_at timestamps tick forward in real time as it worked through 166 posts, 135 of which had drifted.

That same rescan also exposed a real, pre-existing UI gap: the hero assignment card only checks the staged hero_image_url column, not what’s actually live, so any post whose hero was set outside blog-manager’s own Pexels picker (there’s a standalone script that does this directly) shows up looking like it has no hero at all. Filed that separately rather than scope-creeping it into this issue.

What’s next

This capability doesn’t do anything yet on its own, since nothing produces an openverse source. The Openverse provider is what actually exercises it for real. Until then it only has automated coverage: unit tests around the two-commit sequencing and idempotent overwrite behavior, plus that one real network round-trip against httpbin proving the fetch-and-normalize step isn’t just mocks all the way down.