Teaching an agent to click-test my Tauri app

Every UI change in Greenhouse kept hitting the same wall. I’d change a dialog, the backend tests would pass, the type-checker would be happy, and then verification came down to me launching the dev build and clicking through it myself. The agent couldn’t do that last part, so it would stop and hand the click-through back to me. Fine once. Annoying by the tenth time.

The thing tests don’t cover is the seam: does clicking “Capture” actually call the capture_idea command and write a row, or did I only prove the function works when a test calls it directly? That needs a real click and a real IPC round-trip in the same motion. So I set out to give the agent a way to drive the running app.

macOS is the hard part

Tauri has an official WebDriver story, but on the desktop it’s Windows and Linux only. macOS has no WKWebView driver there, so the documented path just doesn’t apply to my machine. For a while the accepted wisdom was “screenshot the web layer with mocked IPC and call it done,” which verifies layout but never touches the real Rust command.

What actually works on macOS now is a community plugin, tauri-plugin-webdriver-automation. It runs a small WebDriver HTTP server inside the app in debug builds, and a companion CLI fronts it with the standard W3C protocol on localhost. The app builds against Tauri v2 without complaint. Good sign.

The security detour

Before wiring it in for real I read the plugin source, and I’m glad I did. Two things stood out. The WebDriver server has no authentication and exposes arbitrary JavaScript and arbitrary IPC execution, which is inherent to any driver but worth stating plainly. And the plugin pulls in Tauri’s dynamic-acl feature.

That second one bit me in a way I didn’t expect. My first cut added the plugin as a plain macOS-target dependency. Cargo’s feature unification then quietly enabled dynamic-acl on my release Tauri too, and compiled the whole WebDriver server into release binaries, even though it never runs there. A testing tool leaking a capability-weakening feature into production is exactly the kind of thing you don’t notice until much later.

The fix was to make it a true opt-in. The plugin is now an optional dependency behind an automation cargo feature, registered only under #[cfg(all(debug_assertions, feature = "automation"))]. A normal build pulls none of it. I verified with cargo tree: without the feature, the crate isn’t in the graph at all.

Two gotchas that ate an afternoon

First: a directly-run debug binary loads its frontend from the Vite dev URL, not from embedded assets. I launched the app under the driver, the window came up blank, and every “find element” call hung forever with no error. The session created fine, then nothing. It took me longer than I’d like to admit to realize the webview was just a blank page waiting on a Vite server that wasn’t running. Start Vite first and it all works.

Second: you can’t drive your own onboarding. Greenhouse’s first-run flow uses a native folder-picker dialog, and a WebDriver can’t touch an OS dialog. So I added a debug-only environment override that boots the app straight past onboarding into a throwaway vault. Small, contained, and it means the driver can start every run from a known state.

It works

The payoff is a short script the agent runs itself. It launches the app against a temp vault, clicks “Capture an idea,” types a name, submits, and then checks the real results: the confirmation dialog shows the new folder path, the topbar flips to “Captured today” off a fresh database read, and on disk there’s a real SQLite row and a markdown file. No mocks anywhere in that chain.

I skipped the natural-language MCP wrapper that some people bolt on top. It wants to live in every session across every project, which is more standing privilege than I want for a tool I only need here. A committed driver script gets me the same result with a much smaller surface.

What I like about where this landed: the agent can now close its own loop on a UI change instead of parking it on me, and the one capability that could have leaked into production is gated behind a feature flag that’s off by default. The lesson I’ll keep is the boring one. Read the dependency’s source before you trust it, and watch what Cargo’s feature unification does behind your back.

Addendum: the cheap tier, and the three-tier picture

The WebDriver driver is the real thing, but it’s heavy: build the app, start Vite, launch a window, drive it. Not what I want running on every push. So I added the cheap tier underneath it - vitest component tests that render the real Svelte component, fire a real click, and mock the one thing that can’t run in CI: the call into Rust.

The trick is to mock at the boundary, not the module. Tauri’s frontend talks to Rust through a single invoke function, and @tauri-apps/api ships a mockIPC helper that intercepts it. So a test looks like: render the touch dialog, type a note, click “Log touch,” then assert two things - that invoke was called with record_touch and the right arguments, and that the DOM reacted. Real click, real component logic, real DOM update, fake backend. It runs headless in about two seconds.

Two things bit me. First, jsdom and the native <dialog> element. Both my dialogs call showModal() the moment they mount, and depending on the jsdom version that method may not exist, so the component throws before a single assertion runs. A small guarded polyfill in the test setup fixes it. Second, my type-checker started failing - it was now checking the test files too and didn’t know what expect or describe were. One line in tsconfig to pull in the vitest and jest-dom globals, and it was happy again.

What I like is the shape it all settled into. I now have three tiers of frontend verification, cheapest first: mocked-IPC component tests that run on every push and catch “the button stopped calling the command”; a screenshot pass with mocked data for “does the layout fill the window”; and the full WebDriver driver for “does clicking this actually write a row.” Each one covers what the cheaper one can’t, and I reach for the expensive one only when I need to. That felt like the right place to stop.