The Approval Matrix I Gave My SaaS Ops Agent

The tempting version of an ops agent is: watch production, notice things, react to them. That’s the demo. That’s the version that gets clapped for on X.

The version I actually want running against Freebo is different. It watches production, notices things, drafts responses — and then does absolutely nothing until I click a button in Slack that says approve. It reads five providers. It writes zero of them. It can propose a customer email; it cannot send one. It can open a website PR; it cannot merge one. It knows the exact SHA that shipped to production. It cannot promote main to production.

Before I built the agent I wrote it a PRD. That was the whole trick.

The PRD-first move

I did not open an editor and start prompting my way into a working agent. I opened an editor and wrote a specification for what it was allowed to do. The document lives at ~/Documents/FreeboDecember/docs/prds/PRD-hermes-freebo-operations-agent.md and it’s over a thousand lines. Most of them are constraints.

The reason to write the PRD first is that the default state of an agent is too much authority. If you spin up something that “watches your SaaS,” it will read secrets, dump raw logs into Slack, invent metrics, retry on timeout, and post the same unchanged problem every five minutes because nothing told it not to. The PRD is where I get to say — explicitly, before there’s any code to defend — what the agent is not permitted to do.

Evidence before action

Here is the principle the entire spec rests on. Every release report must identify the production SHA, the compare range, the affected Railway services, the deployment IDs and their statuses, and the specific evidence used to classify the release. No natural-language summary gets to skip past those. If the agent can’t cite what it looked at, it doesn’t get to file a claim.

Same rule for monitoring. Every metric names its PostHog insight and date range. Every alert names its dedupe key and source. “The API is unhealthy” is not a legitimate output; “Railway service api-prod deployment dpl_abc reported CRASHED at 14:12 UTC, source: Railway API” is.

This isn’t paranoia. It’s the antidote to a very common failure mode where an LLM confidently reports a problem, you go check the dashboard, and the problem never existed — the model interpolated it from a scary-looking log line.

The subscription-only constraint

The single hardest rule in the PRD:

The container refuses to start if OPENAI_API_KEY or ANTHROPIC_API_KEY is present in its environment. It refuses to start if the configured provider allows automatic fallback to a metered service. If the ChatGPT included allowance is exhausted, AI-dependent jobs pause instead of silently switching to pay-as-you-go and running up a bill.

I have a specific reason for this. An always-on ops agent that can silently roll over to metered billing is a great way to wake up and find you’ve spent three hundred dollars overnight because a webhook loop retried a classification job 1,200 times. The subscription is the ceiling. The agent can lose the ability to think without losing its ability to collect webhooks, persist state, and fire deterministic critical alerts.

The last check is the one that stops the sneakiest failure mode — a stale successful health state hiding a fresh authentication error. If the last auth error is being hidden by an older green health tick, the agent has to notice, refuse to start, and log the variable name of the offending credential (never its value).

The approval matrix

This is the reveal that felt worth writing a blog post about. Not “the agent can do things” — every AI agent can do things. What matters is the shape of the boundary.

The matrix breaks every action into one of three columns.

Notice what’s in the middle column. Sending an email is not autonomous. Opening a PR is not autonomous. Both are actions I could easily justify letting the agent do “because it’s read-only until a human reviews it.” But the failure mode I care about is not “the agent took too long to send an email.” It’s “the agent sent a broken email to every operator location, from an unverified sender, in the wrong workflow, and I find out on X.” The correct answer to that failure mode is a human eyeball on the drafted content, the audience count, and the send button.

Notice what’s in the third column. These are the actions where the blast radius is largest and reversal is hardest. Restart a Railway service and you take the API down. Rotate a credential and every service that depends on it starts throwing 401s. Force-push production and you may have destroyed the last known-good SHA. I don’t need to model out the harm — I just need to keep those actions out of the agent’s tool belt entirely.

One face, many read-only specialists

The topology is one Slack-facing supervisor and a handful of bounded specialist subprofiles.

• release-sentinel — detects and reconciles production SHA changes
• release-comms — classifies the diff and drafts the announcement
• railway-watch — deploy status, crashes, restart loops, error signatures
• supabase-watch — security and performance advisors, API/Auth/Postgres errors
• posthog-watch — adoption, funnels, checkout conversion, anomaly detection
• github-watch — CI failures, production divergence, risky direct pushes
• activity-summarizer — operator logins, feature usage, consequential actions
• usage-watch — agent tokens, provider cost, plan utilization

Every one of those specialists is read-only by default. Only the notification-operator (which fires the announcement after my Slack approval) gets approval-required — and even then, it invokes the existing Freebo product-update sender rather than calling Resend directly. The agent does not know how to send an email. It knows how to ask an existing, tested code path to send one.

Each specialist returns a normalized result to the supervisor with the same schema: status, summary, evidence, recommendedAction, requiresApproval, dedupeKey. The dedupeKey is doing quiet, load-bearing work — it means the agent cannot post the same unchanged problem every five minutes just because the polling interval ticked over. That is one specific kind of Slack noise I never want to see again.

Three telemetry layers, three purposes

The last constraint that felt worth writing down: don’t blur the layers.

1. Product analytics. PostHog. Useful for adoption trends and journey summaries. Not authoritative for “who changed what.”
2. Business audit trail. Server-written Freebo events, produced only after a mutation succeeds. This is the authoritative answer to “who cancelled reservation X.”
3. Security and infrastructure logs. Supabase Auth, Railway, Sentry, Novu. Explains login failures, delivery failures, operational failures.

An agent that summarizes activity without knowing which layer it’s reading from will confidently tell you that no operator used the checkout page today when the reality is that PostHog was down for two hours. The PRD forces the agent to distinguish “no activity observed” from “provider unavailable” from “instrumentation missing.” Three different statements, three different things to do about them.

Where the value is

The interesting part of an ops agent is not its intelligence. It’s the shape of the constraints you wrap around it. A less-capable model inside a tight, evidence-first, approval-gated harness will do better work in production than a smarter one with the run of the place.

The PRD-first pattern is how you get there. Write the boundary before you write the tool. Decide what “read-only,” “approval-required,” and “prohibited” mean for every category of action. Decide what the definition of “working” is before you ask something to prove it works. Then — and only then — launch the agent.

Freebo’s ops agent boots up with a private Slack channel, a read-only mount of the repo, and a container that refuses to start if I accidentally injected an API key. It watches every production deploy. It will not send a single customer email until I’ve read the draft, checked the audience count, and clicked approve. That’s not a limitation. That’s the point.