The Route That Only Fires When Someone Can't Pay

The Slack ping came in at 9:30 PM. One line, in a channel that never has more than a handful of messages a week:

payment.failed — shopper hit an error completing checkout on a four-figure quote.

I was already off my laptop for the night. I read it, decided the shopper had probably retried and given up an hour ago, and went back to whatever I was doing. The real story wasn’t the failed payment. The real story was that the alert was in Slack at all, and what I found when I opened the drawer the next morning.

This post is about the wiring underneath that one line, and the class of bug I ended up writing a PRD to kill instead of the specific one I could have hotfixed.

How the alert got there

Freebo runs on Railway. My ops agent, Hermes, runs on my always-on desktop at home. The two are not on the same network. They talk to each other through a Tailscale Funnel — a public HTTPS URL that terminates on my box, guarded by an HMAC secret both sides share.

The choice I’m proudest of, in retrospect, is that the urgent route has no model in it. Most of the other Hermes routes hand the payload to an agent that summarizes, correlates, files a report. Great for reservation lifecycles and deploy events. Terrible for the one thing you actually need to see fast.

The four routes

Hermes has four routes exposed to Freebo. They are deliberately different shapes.

Railway can’t sign requests, so the random suffix in the Railway route name is the credential. The other three ride the HMAC secret. The whole thing is described in docs/runbooks/hermes-webhooks.md so future-me doesn’t have to reverse-engineer it.

The rate limit is 60 requests per minute per route, and repeats of the same X-Request-ID inside an hour get dropped. That last part matters more than it looks — Stripe retries webhooks on its own schedule, and without idempotency I’d get the same alert three times in ninety seconds.

What I saw when I opened the drawer

The payment failure itself was uninteresting. Card declined, retried a few minutes later on a different card, went through. The shopper is fine. The operator is fine. If I’d stayed asleep another six hours nobody would have noticed anything.

But the trace path from the alert led me to the checkout flow configuration code, and what I found there was a class of bug that had been quietly building for months.

Six settings on the checkout flow — show_availability_hints, show_type_filter, show_group_size_filter, enable_gifts, require_date_selection — were saved to the database, shown in the operator UI, and had toggles that operators could flip. None of them did anything. They were plumbed to nowhere. Operators had been switching them on and off for weeks assuming they changed the shopper experience.

A seventh setting, enable_scarcity, wasn’t even serialized to the public payload. You could toggle it in the editor and it wouldn’t cross the network.

That is not a bug you catch by watching Slack. That is a bug you catch by opening the code and asking: how did we end up here?

Seven places to touch

The answer, when I traced it, was structural. Adding one new setting to the checkout flow today requires touching seven files:

• the SQL migration that adds the column
• the Zod schema for validation
• the PUT handler’s column enumeration
• the public response allowlist
• the frontend TypeScript type
• the editor’s useState initializer
• the editor’s hydration useEffect

No compiler help between them. Miss any one — miss the allowlist, say, and the setting never reaches the client — and the toggle appears in the UI, saves fine, and does nothing. Silent. No error. No test failure. The parity test that was supposed to catch this turned out to be readFileSync + regex over the route source. It asserts the string quoteVersionService appears in a file. That is not a test. That is a smoke detector with the battery pulled.

What I did instead of a hotfix

I could have shipped a one-line fix by 10 AM. The failed payment had a boring cause and a boring workaround.

Instead I spent the morning writing a PRD — PRD-28, checkout flow configuration foundation — that says: one contract, flow identity end-to-end, every setting live or removed. It has a specific “no shortcut rules” section that names the exact traps I want the implementation to avoid. It calls out FM-3, a partial-PUT default re-injection bug I fixed months ago in PR #599, because the fix’s guard registry has to stay green. It says the grid output must remain byte-identical for the existing live operator, and any shared presentation shell must prove that with a rendered-output comparison, not by inspection.

The PRD is 500 lines. The hotfix would have been 15. I’ll take the 500 lines.

The takeaway

If you’re running a small SaaS by yourself, or with a tiny team, the monitoring conversation usually stops at “are we alerting on the right things.” That’s the easy half. The hard half is deciding what to do with the alert.

The instinct is to treat every ping as a fire and stamp it out fast. Sometimes that’s right — the shopper who can’t pay tonight is not going to try again tomorrow. But the alert is also a scent. If you follow it back into the code and only find the one thing that broke, you’ve done half the work. If you follow it back and find a class of failure that’s been sitting there for six months, and you close the class instead of the instance, the same alert doesn’t fire again in three weeks with a different customer holding the bag.

The route that only fires when someone can’t pay is worth having. What matters is whether you read it as a bug report or a map.