Before any PR merges to Freebo’s main, a thing called Gate A runs. It’s 32 parallel Playwright lanes — admin flows, booking flows, custom checkouts, promo codes, tip charges, API-key handling, calendar holds — each lane a different way a real user might touch the app. A fresh agent that didn’t write the code reads the output and posts a verdict. If the verdict isn’t live-ui-verified, I don’t merge.
Yesterday morning I kicked off Gate A for PR #780 and watched 32/32 come back green. Every lane passed. The verifier posted verifier-failed.
That’s the fun kind of bug.
What the verifier saw
Every lane dumps its API traffic into an api-calls.txt file. The verifier scans those files and applies a blunt rule: any 5xx is a failure. It’s a good rule. 5xx means your server blew up, and your UI covering for it with a nice toast doesn’t change the fact that your server blew up.
The verifier opened custom-checkout’s api-calls.txt and saw this:
POST /public/booking-requests 503Red flag. Posted verifier-failed.
Except nothing on my server ever returned a 503. The lane itself had injected it. Right there in lanes/custom-checkout.mjs around line 385:
The whole point of that lane is to prove the checkout recovers from a backend blip. Playwright intercepts the request at the browser layer and hands back a fake 503 without the request ever leaving the tab. The lane’s own recovery assertions then confirm the UI retried, the user saw the right message, and the charge eventually went through. That lane is supposed to see a 503. It’s the reason it exists.
The verifier didn’t know that. The verifier reads api-calls.txt. api-calls.txt just logs status codes. A 503 is a 503.
The two-layer mistake
The lane knows the 503 is fake because the lane injected it. The verifier knows nothing — it’s a stateless reader of logs. The fix isn’t to loosen the 5xx rule (please no), it’s to make the log say what it actually is:
Now the verifier’s rule gets one carefully scoped exception: a 5xx marked INJECTED that the lane’s own recovery checks then PASS is fine. Anything else — including an injected 5xx that the recovery path didn’t recover from — still fails. The rule stays strict. The log stops lying.
I grepped the entire test suite for route.fulfill with a 4xx or 5xx and labelled every one of them. Took about ten minutes. The verifier is happy. I am happy. The ghosts are gone.
If your test intentionally lies to the browser, your logs have to confess it.
— The rule I'm keeping
The flakes were the same shape
While I was in there, I cleaned up three flakes from the previous Gate A run — runs that passed on retry but not on the first try. Every single one was the same mistake in different clothes:
holds-calendar-shows-block-offwas reading the chips while the calendar still had a loading spinner. The assertion didn’t wait for the events request to finish before it looked. Fix: wait for the network call, not for a hardcoded number of seconds.timed out waiting for pay button enabled after 20shit in two different lanes. The 20 seconds was a bare timeout — it had nothing to do with whether the quote had come back. Fix: bind the wait to the actual quote-or-payment-intent response. If that takes 25 seconds, the test should wait 25 seconds, not fail at 20.last_used_atasserted a value that only gets written asynchronously. Fix: poll briefly for the write. This is roughly the same shape — don’t guess when a side effect has landed, actually watch for it.
Three flakes, one pattern: the test was asserting against time instead of against state.
Why this is interesting past Freebo
I like this story because it’s a tiny version of a problem I keep running into with agent systems in general.
When you have one agent doing the work and another agent checking it, the checker is always working from artifacts — logs, files, API responses, whatever the worker left behind. The checker is only as smart as what’s in those artifacts. If the worker knows something that never makes it into the artifact, the checker will get it wrong, and nobody can tell whether the system is broken or just unobservable.
The tempting fix is to teach the checker more context. Make it smarter. Give it the worker’s state. That’s almost always wrong. The right fix is to make the artifacts carry the thing the checker needs, in a form the checker’s existing rules can read without special-casing.
In this case: don’t teach the verifier about my test harness. Make the test harness mark its own fakes.
The whole trick with running agents in parallel is that none of them can afford to know each other’s internals. The log is the contract. If the log lies, the system can’t be trusted, and it doesn’t matter how many lanes you ran.
32/32 is a nice number. The number I actually care about is whether a fresh agent, reading only what my system wrote down, can reach the same conclusion I would. Yesterday it couldn’t. Today it can.