An agent writes a blog post here every morning at 5:15am. It reads a context map that another agent wrote at 5:00am, picks an angle, writes the MDX, and hands off to a Node script that privacy-scans the post, generates a cover, commits, pushes to main. Vercel deploys in about a minute. I usually see the desktop notification while pouring coffee.
For seven days in early August, that notification was a lie.
The cron ran clean every morning. Published 1, held 0. Green checkmark. The commit was on main, the push had happened, the log ended with “Done.” Meanwhile the site was frozen on the August 2nd deploy and every URL published after it was returning a 404.
What broke
An earlier post used the <CompareTable> MDX component with the wrong row shape. It passed rows as an array-of-arrays instead of the expected {feature, cells} objects. That threw during Astro’s prerender step. npm run build failed. Vercel’s build failed. Vercel served the last good deploy and 404’d the new URL.
None of that was visible to the cron, because the cron never ran npm run build.
The pipeline was: agent writes MDX → privacy scan → cover attempt → git add / commit / push. That was it. If the MDX had a component prop that would fail at build time, the cron would happily commit it and push it. Every subsequent morning’s post would also be committed and pushed, into a tree that had been broken since the first bad one. Seven mornings in a row.
The signal I actually needed — “the post I just pushed is not on the internet” — was never anywhere in the pipeline. So the pipeline could not report it.
The bug shape underneath the bug
Green in the loop means nothing if the loop doesn’t include the thing you care about.
— The lesson I keep having to relearn
The specific failure was silly — one component being lenient about its input would have absorbed it. The interesting failure is that the cron’s definition of “shipped” ended at git push. Push is the last thing the cron controls, so push is where the log stopped. Everything after that (Vercel building, Vercel deploying, the URL actually serving) was somebody else’s problem, until the somebody else was silently me.
I’ve done this exact shape before with an operator’s monthly calendar in Freebo — a partition function that classified today-and-two-days-forward as the “live compute” branch, and by accident classified every past day into the same branch. The bug was “the predicate didn’t cover what actually flows through it.” Same shape here. The cron’s success criteria didn’t cover what “success” actually meant to me.
The paranoid version
The fix wasn’t clever. The fix was four things I should have had from day one.
1. Real build gate. Run npm run build locally before committing. If it fails with today’s post included, flip the post back to draft and try again. If it still fails, abort without pushing. That way one bad post can’t take the site down, and a pre-existing breakage stops the pipeline instead of stacking on top of it.
if (published.length && !NO_BUILD) {
if (!runBuild()) {
log("BUILD FAILED — reverting today's posts to draft.");
for (const p of published) revertToDraft(p);
if (!runBuild()) {
log('BUILD STILL FAILING — pre-existing breakage; aborting without commit.');
return;
}
}
}2. Post-push deploy verify. After the push, curl the actual URL. If it doesn’t return 200 within about five minutes of retries, alert. This is the part that would have caught the seven-day outage on morning one.
function verifyDeploy(slug, { attempts = 10, waitMs = 30000 } = {}) {
const url = `${SITE}/blog/${slug}`;
for (let i = 1; i <= attempts; i++) {
const code = curlStatus(url);
if (code === '200') { log(` deploy: live at ${url} (attempt ${i})`); return true; }
sleep(waitMs);
}
log(` deploy: NOT live after ${attempts} attempts — ${url}`);
return false;
}3. Component that forgives. <CompareTable> now normalizes any row shape — object, cells-only, bare array — and defaults every missing field. One bad prop from a future writer agent (or from me) can’t take the whole site down again. Robustness on the receiver side, not just the sender side.
4. Fix the covers, too. While in there: cover generation had produced nothing since July 8. generate.py prints pretty-printed multi-line JSON. The parser only ever read the last line, which was }, threw, and logged “no image returned” — a phrase that reads like a normal API failure and got ignored. Parse the whole payload, try every known generator location, and report the coverless-post backlog in the summary so the silence has a number attached.
Where each failure could have been caught
- WriteAgent writes MDX with a bad propA stricter component or an editor check would have caught it here. Neither existed.
- Local buildnpm run build would have thrownThe cron did not build. Every 'shipped' post shipped unbuilt.
- Pushgit push succeededThe last stage the cron cared about. Log ends 'Done.'
- Vercel buildPrerender threw, deploy failedVercel emails on this, but that inbox is muted. Should have been read by the cron.
- ServeNew URL returned 404 for 7 daysNo verifier. First noticed by me clicking a link Sunday morning.
The pipeline had five stages where the bug could have been caught. Zero of them were checking. Adding checks at stages 2 and 5 was six lines of code and about an hour of work. That is roughly a decade less than the seven days production was down.
The rules I’m keeping
The bigger rule, the one I keep re-deriving: autonomous means nobody is watching. A human deploying a blog post sees the 404 the moment they click their own link. An agent doesn’t click. It writes, pushes, and moves on. Whatever check you would have done with your eyeballs, you have to write down and add to the loop. Not because the agent is dumb, but because the agent is exactly as observant as the code you wrote for it.
The reason you’re reading this is that the same cron ran again this morning at 5:15am, wrote this post, built it, pushed it, and curl’d the URL until it returned 200. Then it sent me a notification. Which is what “Published 1, held 0” is now allowed to mean.