Pulling sanitized prod data into dev when rsync isn't there

blog-manager’s dev environment has always run on next to nothing: one seed user, two fixture blogs. That’s fine for the automated test suite, which uses fixtures anyway, but it makes manual testing (bin/dev, clicking around) mostly useless, since there’s no real post history, no real hero images, nothing that looks like the actual site I’m managing. I wanted a way to pull a real snapshot of production down into dev without hand-rolling it every time.

The app is SQLite, not Postgres, deployed via Kamal with the database file living on a bind-mounted volume (/data/storage on the host, /rails/storage in the container). That actually makes this simpler than a Postgres dump/restore, since the whole database is just a file I can copy.

What I built

A rake task, bin/rails data:pull_from_prod:

1. scp the production SQLite file down to a scratch temp dir
2. Sanitize it in place, before it touches anything local: null out every encrypted token/API key column, reset all user passwords to the same one db/seeds.rb already uses, wipe the sessions table
3. Prompt for confirmation, since it’s about to overwrite storage/development.sqlite3
4. Pull Active Storage files down too, so hero images actually resolve locally
5. Run pending migrations, since prod’s schema can lag what I’ve got checked out

The sanitization part matters more than it looks. This app uses Active Record Encryption on things like GitHub tokens and API keys, and dev doesn’t share prod’s encryption keys, so even if I left those columns alone, they’d just be garbage in dev, unreadable and liable to raise on decrypt the first time anything touched them. Nulling them out is the only sane option, not just the safe one.

What surprised me

First run blew up immediately: no such column: unsplash_access_key. Production was running a schema from before that migration merged locally, which makes sense once I thought about it (prod deploys lag main), but the sanitization SQL had hardcoded column names and assumed local schema matched prod schema. Fixed it by checking PRAGMA table_info before touching each table and only nulling out columns that actually exist in that particular snapshot.

Second surprise: I reached for rsync to pull the Active Storage files and got rsync: command not found, from the remote shell, piped back through the SSH connection, which made the local error (unexpected end of file) pretty confusing until I actually SSHed in and checked. Kamal’s host provisioning installs what Docker/Kamal needs and nothing else; rsync was never part of that. Swapped to a plain tar pipe over SSH instead, tar -cf - | tar -xf -, since tar is about as close to guaranteed-present as it gets, and it worked first try.

Testing it for real

I ran it against the actual production host rather than trusting the code to be right. Pulled the real database, confirmed the sanitized columns were actually nil, booted the dev server, and logged in over HTTP with the reset password against the real prod user, got a 302 and a session cookie, then confirmed the dashboard rendered both real blogs. Nice to have a task that’s boring to run because it just works.

What’s next

This covers the “no realistic data” half of the problem. There’s a separate open question, whether the staging environment is worth keeping at all, given it’s barely used and has this exact same sparse-data problem independently. That’s tracked separately; this sync mechanism works regardless of which way that goes.