Untangling the Colima dependency in Kamal deploys

Every time I needed to push a production deploy for blog-manager, I had to remember one thing: is Colima running? If not, kamal deploy would just hang. My Mac’s local Docker daemon was a load-bearing part of the production deploy path, and I kept forgetting.

The root cause: config/deploy.yml had builder.remote: ssh://kamal@blog-manager.internal. Kamal was SSHing into the production LXC, but it needed a local Docker daemon for the buildx context. No Colima, no build, no deploy.

Staging had the same problem, just hidden better. The staging deploy workflow ran on a self-hosted runner inside the staging LXC and called bin/kamal deploy -d staging, which used builder.remote: ssh://kamal@blog-manager-staging.internal to build right back on itself. It worked, but every merge produced a freshly-built image that was never tested on a GitHub Actions runner, and would be built again on the dev box when deploying to production. “It passed staging” was technically meaningless as a confidence signal.

The fix is the standard Kamal build-once-deploy-many pattern: CI builds and pushes the image once, every environment pulls and deploys that same image.

The plan

Three pieces:

1. A new build.yml workflow that builds and pushes gitea.example.net/vic/blog-manager:<sha> on every push to main. PRs get a build-only run (no push) to catch broken Dockerfiles early.
2. A rewritten deploy-staging.yml that chains off build.yml via workflow_run and deploys with --skip-push --version=<sha> instead of building locally.
3. Remove builder.remote from both deploy configs.

For production: bin/kamal deploy --skip-push --version=<sha> from any machine. No Colima. No local Docker daemon. Just SSH access to the host.

Three things the research got wrong

The original issue description said to push to registry.internal/blog-manager; that’s actually a Docker Hub pull-through cache, not a push target. The real registry is the Gitea instance (container 1009 on Proxmox).

The research also said Gitea was publicly reachable via Cloudflare. It’s not. dig +short gitea.example.net @1.1.1.1 returns 192.168.x.x, an internal RFC1918 address. The GitHub-hosted ubuntu-latest runner can’t reach it. I found this out the hard way when the first build attempt timed out trying to log in to the registry. The fix: run the build job on the [self-hosted, blog-manager-staging] runner, which is inside the homelab.

The third thing: kamal deploy --skip-push validates that the image has a service label matching the service name from config/deploy.yml. kamal build push adds this automatically. docker/build-push-action does not. The deploy pulled the image fine, then rejected it:

Image gitea.../blog-manager:<sha> is missing the 'service' label

One line in the workflow fixed it:

labels: service=blog_manager

The dotenv gotcha, again

There’s one more wrinkle with Kamal secrets in CI, the same one that bit the staging setup. Kamal evaluates .kamal/secrets-common using Dotenv.parse, not a bash subprocess. That means CI environment variables are not available inside ${VAR:-fallback} parameter expansion, only in $(cmd) subprocesses. The workaround: write config/master.key explicitly from the CI secret before running any kamal command.

Layer caching gotcha

The plan called for type=registry,mode=max cache; it’s the “right” answer for caching Docker layers in a self-hosted registry. Except it’s broken against Gitea (gitea#28973, the Gitea registry returns an error on PATCH during cache push). type=gha works correctly and doesn’t require any registry support.

What the pipeline looks like now

push to main
  └─ Build and push image  (self-hosted, blog-manager-staging)
       └─ Deploy staging    (self-hosted, blog-manager-staging, --skip-push)

Production deploy: bin/kamal deploy --skip-push --version=<sha>. Staging and production now run the exact same image.

The remaining piece is auto-deploy to production with a GitHub Environments approval gate. That requires a self-hosted runner on the production LXC (1027), the same setup as staging but on the other host. Planned for a follow-up.