Every health check was green and the standby was 590 MB behind
On 1 September a Postgres standby stopped applying write-ahead log at 17:31 UTC. It was found 34 minutes later, entirely by accident, when a schema migration succeeded on two nodes and failed on the third. At that point it was roughly 590 MB behind the primary and serving stale data — 115 workspaces where the primary had 116.
No customer was affected. Data loss was one failover away.
Every surface was green, and each was telling the truth
This is the part worth sitting with. Nothing was broken in the monitoring. Every check was working correctly and reporting accurately — about something else.
/healthzand/readyzreturned 200. They check the control plane, not the cluster./workerzreturned 200 with every loop running. The worker was fine.- The platform reported the node healthy. The machine was healthy. The Postgres process was up, accepting connections, and answering queries — with data frozen at 17:31.
A replica that has stopped replaying does not look ill from outside. It looks like a database that is not very busy.
The measurement that lies
There was a metric that should have caught it. replay_lag read
37 minutes, which sounds exactly like the alarm you would want. It is
not usable, for two reasons that took an incident to understand:
- It measures how long ago the last applied transaction committed. On a quiet database that is a statement about your traffic, not about your replica.
- It reads near zero on a genuinely broken replica — one that has received nothing new to be late about.
So the metric is loudest when nothing is wrong and quietest when everything is. If you
alert on replay_lag, you have built an alarm that fires on idleness and
stays silent on failure.
Byte distance between the primary's WAL position and the standby's replay position is the honest measure. It does not care how busy you are. It answers the only question that matters: how much of reality is this node missing.
A second signature, worth naming separately
The frozen node sat at exactly 3/B3000000 — a WAL segment boundary — while
the primary moved on. A replay position that does not advance across successive samples
is a wedged receiver, and it is dangerous long before the byte distance
looks alarming.
That is a different fault from "falling behind", and it deserves its own alert. Something slowly losing ground is a capacity problem. Something pinned at one position is stuck, and it will never catch up on its own.
The fix, and how little it was
Restarting the machine cleared it. Replay resumed immediately — 350 MB in the first minute — and the node was fully caught up in about 25 minutes. The replication slot had preserved its position, so nothing needed rebuilding.
That is the uncomfortable part. The remedy was thirty seconds of work. The exposure was 34 minutes of a silent, stale node that any failover would have promoted, and the only reason it was found at all is that somebody happened to deploy a migration.
What we changed
A watcher now samples the byte distance and the replay position directly, alerts on distance and on non-advancement as separate conditions, and reports a node it cannot reach as unobserved rather than healthy. That last distinction matters more than it sounds: a check that could not run has not passed.
One trap for anyone doing the same on a managed Postgres: inspect nodes on their direct port. Port 5432 is usually a pooler and proxies to the primary, so a standby asked through it will cheerfully tell you it is not in recovery.
We build Ratchet, a gate that decides whether an AI agent may perform a
side effect. This is one of our own incident reports, published because the lesson is
not specific to us. The full write-up lives in
the repository under
docs/handoff/, alongside every other one.