Notes

Fixing an availability defect caused a twelve-minute outage

6 September 2026

An audit found a real defect: with a single control-plane machine, every deploy has a window where nothing serves. For a service sitting in the critical path of side effects, that is not acceptable. So we scaled to two. The database then OOM-killed itself in a loop and took the entire service down for about twelve minutes.

Cause: mine. The fix was correct. The arithmetic underneath it was never done.

The arithmetic

Each application instance opens its own connection pool of DB_POOL_MAX, which was 10. The budget went from 20 connections to 30: 

connections = (app instances + worker instances) x DB_POOL_MAX
            = (2 + 1) x 10 = 30

The database was a 256 MB instance. Each Postgres backend costs memory, and thirty backends was more than it had. It began killing its own: 

Out of memory: Killed process 714 (postgres)
Health check for your postgres role has failed

Then the health checks did their job, and that made it worse

Both application machines failed /readyz — correctly, because the database really was unavailable. The proxy then had no healthy candidate to route to: 

could not find a good candidate within 40 attempts at load balancing

Every path returned nothing at all. Not a 500 — no response. Readiness checks behaving exactly as designed converted a degraded database into a total blackout, which is the correct trade in almost every case and worth understanding before you need it.

Why it was not caught sooner

The health checking was right. What was missing was any notion of a connection budget — nothing anywhere related instance count multiplied by pool size to the database's capacity. So scaling looked like a free operation. You add a machine, you get more availability. Nothing in the system said otherwise.

The 256 MB database was the underlying fragility. It had run fine at 20 connections, which is exactly what made it look adequate. It had no headroom, and the first thing that asked for more took it down.

Recovery, in the order it actually worked

  1. Scaled the app back to one machine to halve connection pressure. Did not recover — the database was already in an OOM loop and could not restart cleanly.
  2. Raised the database to 1 GB. It came back healthy immediately.
  3. Verified integrity, including that the receipt hash chain still audited clean after an unclean shutdown. Postgres crash-safety held; no evidence was corrupted.
  4. Scaled back to two machines, which now holds.

Step one is the instructive one. The intuitive response — undo the change that caused it — did nothing, because by then the cause and the symptom had come apart. The database was not failing because of load any more. It was failing because it was already broken.

The rule, written down so it is not rediscovered

Before changing instance count or DB_POOL_MAX, check that the database can afford the result. Treat scaling the control plane as a change to the database, because it is one.

Horizontal scaling is sold as the safe, boring lever. It is safe for the thing you are scaling and a load increase for everything that thing talks to.

We build Ratchet, a gate that decides whether an AI agent may perform a side effect. This is one of our own incident reports, published because the lesson is not specific to us. The full write-up lives in the repository under docs/handoff/, alongside every other one.