What happens when step five fails
At-most-once answers one question: did this action already happen? It does not answer the two that follow. Four of five steps happened and the fifth failed — now what? An agent is looping and each turn of the loop is a real charge — how do you stop it without stopping everything? These are the two features people ask about least and need most.
What follows is not a case study. We have no customers to write one about, and inventing
"Acme cut duplicate charges by 94%" would be fabricating evidence for a product whose
whole pitch is that it tells you the truth about what happened. So instead: two failures, run
against the live service, with the real transcripts. You can run them yourself —
npm run case-study in
the repository does exactly
what is printed below.
One: a booking agent, four steps in
A travel agent books a flight, a hotel, a seat and a car, then charges the card. The charge declines. Four real bookings exist in four vendors' systems and there is no payment.
Without a group, this is where the agent is on its own. It must remember what it did, in what order, and how to undo each one — while being the same process that just crashed or errored. If it retries a cancellation it may double-cancel. If it gives up, a customer is holding a flight nobody paid for.
With a group, each step declares its own undo when it begins. When the unit fails, you ask one question and get the plan:
POST /v1/groups/trip:1788249508284/unwind
{ "reason": "payment declined" }The reply, trimmed to one step of four:
{
"state": "unwinding",
"steps": [
{
"order": 1,
"original_effect_type": "car.book",
"compensation": { "effect_type": "car.cancel" },
"suggested_idempotency_key": "compensate:eff_c4apvw594qdj1m92"
}
],
"irreversible": [],
"unresolved": [],
"next_step": "Perform the 4 compensation(s) in the order given."
}Three things in that reply are the entire feature.
The order is reversed. Step 1 of the plan is car.book — the last
thing done is the first thing undone. The plan came back
car, seat, hotel, flight, which is the
exact reverse of the order they completed in.
The group is now unwinding, and refuses new forward steps. An
agent that has not yet noticed the failure cannot book a sixth thing into a unit of work that
is being torn down.
Each undo comes with its own idempotency key, because a compensation is itself a gated effect. This is the part that matters. Hand-rolled rollback is dangerous precisely because a retried undo double-refunds. Here it structurally cannot. Running the first compensation twice, the way a crashed agent would:
# car.cancel, attempt 1
{ "decision": "execute" }
# car.cancel, attempt 2 — same key, no report yet
{ "decision": "in_flight" }The second caller is told the first one holds the lease. It is not given a second execute, and it is not told the work succeeded either — because nobody knows yet. That is the honest answer.
The empty fields are the interesting ones
irreversible lists effects that succeeded but declared no compensation. They
cannot be rolled back automatically and you need to know that before you start, not halfway
through.
unresolved is the one worth dwelling on. An earlier run of this same script had a
bug — it reported the failed payment with the wrong field name, the API correctly returned a
400, and the script did not check. The payment was therefore still
pending when we asked to unwind. The reply was not the plan:
"unresolved": [
{ "effect_type": "payment.charge", "state": "pending" }
],
"next_step": "STOP. 1 effect(s) in this group have an unknown outcome.
Resolve those first — rolling back around an effect that may or may
not have happened is how a half-undone state is created."We had not written that scenario. The gate refused to hand back a rollback plan while one step in the unit might or might not have reached the outside world, which is exactly right and is the single most dangerous moment in any compensation flow. We left the bug in the script long enough to keep the transcript.
Two: a loop that will not stop
An agent gets stuck. Not a duplicate — a genuine loop producing a hundred distinct charges with a hundred distinct idempotency keys. Idempotency cannot help here. Every one of them is a different piece of work, and every one is correct in isolation.
A spend budget is the usual answer and it is the wrong one twice over. It only catches this if the amounts are large, and by the time a ceiling in dollars trips, the money is already gone. Surge containment asks a different question: not how much is this costing but how much is this happening.
PUT /v1/policies/payment.charge
{ "surge_per_hour": 20, "surge_action": "deny" }Then a hundred concurrent attempts, all distinct:
100 concurrent attempts, settled in 341ms:
80 denied
20 execute
Reached the vendor: 20. Held back: 80.Twenty is the ceiling, and twenty is what got through — not nineteen, not twenty-three. The count is enforced by the database, not by application logic racing itself. And the breaker says why it opened:
{
"effect_type": "payment.charge",
"state": "open",
"action": "deny",
"resets_at": "2026-09-01T08:58:28.396Z",
"observed": 21,
"threshold": 20,
"reason": "21 \"payment.charge\" effects since this breaker last
cleared exceeds the configured ceiling of 20 per hour."
}One effect type stopped. Everything else in the workspace keeps running. That is the difference between containment and an outage.
A number we threw away
The first version of this test fired six hundred attempts and reported that the ceiling had held them to nineteen. The number was true. The explanation was wrong: 495 of those six hundred never reached the ceiling at all, because the request rate limiter — a different defence, several layers earlier — refused them first. Surge containment only ever judged about a hundred of them.
A true number with the wrong cause attached is worse than a wrong number, because nobody can catch it. We dropped the burst below the request limit so that what is measured is the one thing being described. Six hundred to nineteen was the better headline. A hundred to twenty is the one we can defend.
So are these features important?
They answer the questions that come after the first one. At-most-once initiation stops the same action happening twice. It has nothing to say about four actions that happened and one that did not, or about a thousand different actions that should never have been attempted at all. Those are the failures that produce the phone call.
Both are off by default. An unrequested ceiling that starts refusing work is worse than no ceiling, and a group you did not ask for is just extra fields. You turn them on per effect type, for the handful of actions that would hurt.
We build Ratchet, a gate that decides whether an AI agent may perform a side
effect. The script that produced every transcript above is
in the repository as
scripts/case-study.ts. Run it against your own instance and you should get the
same shapes and the same counts.