Notes
Engineering notes
Things we found while building a gate that decides whether an AI agent may act. Mostly correctness bugs and the trade-offs behind them — including the ones we caused.
-
The dedupe key that could not say ‘again’
A monitor found something, recorded that it had reported it, and then had its message silently dropped by idempotent delivery. Dedupe and repetition are the same mechanism pointed in opposite directions.
-
Every health check was green and the standby was 590 MB behind
A Postgres standby stopped applying WAL for 34 minutes while every health check returned 200. Why replay_lag reads zero on a broken replica.
-
Fixing an availability defect caused a twelve-minute outage
Scaling the control plane from one machine to two pushed a 256MB Postgres into an OOM loop. The connection budget nobody had written down.
-
A planned failover left us with two primaries
A planned Postgres failover promoted a node, failed its health check, gave up, and restarted the old primary. Twenty-five minutes down, no data lost.
-
What happens when step five fails
A booking agent completes four steps and the payment declines. A retry loop fires a hundred charges. Two failures, run against the live service, with the transcripts.
-
Your idempotency keys are broken on macOS
Two agents in one fleet, the same key, both authorised to charge the same card. The bug is four lines of Unicode and it is probably in your code too.