A planned failover left us with two primaries
The primary was measured at 89–91% CPU steal — receiving roughly a tenth of the processor it was asking for. The same starvation at 23.5% had already stopped a standby replaying twice that day. Moving the primary onto healthy hardware was correct and necessary. It took production down for twenty-five minutes.
No data was lost. That was not luck.
What was done first, deliberately
flyctl postgres failover takes no target argument. You cannot say which node
to promote — you can only make sure that whichever one it picks is a good choice.
So both standbys were rebuilt beforehand and verified at 0.0% steal and zero bytes of lag, and a full backup was taken and verified by restore immediately before starting. Every one of those steps turned out to matter.
What happened
The failover promoted a node to a new timeline, then that node failed its health check. The tool gave up and restarted the old primary.
Which left two nodes believing they were primary — on divergent timelines. The cluster could not agree on who was in charge, and while it could not agree, it could not serve.
The lesson we would have preferred to learn on paper
A failover command that cannot be told where to go is not a failover command; it is a request. Rebuilding both standbys first was what made the request safe, and it is the only reason this was a twenty-five minute outage rather than a data-loss event.
The verified-by-restore backup did not end up being needed. It should still be taken every time, because the decision to take it has to be made before you know whether you need it — and a backup nobody has restored is a hypothesis, not a backup.
The cause underneath the cause
The failover was necessary because the hardware was starved. 89–91% steal is not a database problem, and no amount of tuning fixes it. The database was fine; it was being given a tenth of a CPU.
That is worth naming because it is easy to spend a day optimising queries on a node that is simply not being scheduled. Measure steal before you measure anything else. If it is high, nothing else you measure means what you think it means.
What we changed
Replication health is now reported alongside worker liveness, so a divergent or wedged node is visible without anyone going to look. Steal is checked as a first-class signal rather than something noticed during an investigation. And the cluster now runs three nodes across separate zones, so a standby that loses the primary still sees two of three and meets quorum.
We build Ratchet, a gate that decides whether an AI agent may perform a
side effect. This is one of our own incident reports, published because the lesson is
not specific to us. The full write-up lives in
the repository under
docs/handoff/, alongside every other one.