The dedupe key that could not say ‘again’
We shipped a monitor that watches for payments bunched just under a reporting threshold. It announces a finding once, stays quiet for seven days, and speaks again if the pattern is still there. The last part could never happen. Not rarely — never.
The bug survived code review and a careful read of the module, and was found a few hours later by a test written for an entirely different reason.
What it was supposed to do
Structuring is the practice of splitting one large transaction into several smaller ones to stay under a reporting line. Ratchet holds every amount an agent proposed next to the ceiling it was proposed against, so it can see the shape: twenty-three payouts at $9,800 against a $10,000 threshold, and two in the band below.
The detection is the easy half. The hard half is restraint, because a cap produces the same signature honestly — told they may refund up to $10,000, people refund $9,999. So the sweep never enforces, and it is careful not to repeat itself. From its own header:
A monitor that repeats itself gets muted, and a muted monitor is indistinguishable from one that was never built.
Hence the rule: announce once per workspace and effect type, stay quiet for seven days, then speak again if the finding is still there or has got materially worse.
What it actually did
Webhook events are deduplicated. The key is derived from the event, and the insert is conditional:
const dedupeKey = sha256Hex(`${eventType}:${JSON.stringify(payload)}`).slice(0, 40);
INSERT INTO webhook_deliveries (..., dedupe_key)
VALUES (...)
ON CONFLICT (endpoint_id, dedupe_key) DO NOTHINGThat is correct, and the comment above it says why: enqueuing the same logical event twice — a retried transaction, say — must not produce two deliveries.
Now look at what the sweep puts in the payload. Effect type, threshold, window, how many payments landed in the band, how many in the control band, the ratio, the severity. Every one of those is a function of the same counts.
So when the cooldown expires and the finding has not changed — which is precisely the case the cooldown exists to serve — the payload serialises to the same bytes, hashes to the same key, collides, and is dropped.
The part that makes it worse than a dropped message
The sweep decides to announce, and only then enqueues. The decision writes a row:
UPDATE structuring_notices SET notified_at = now(), ratio = ..., severity = ...That update commits. The delivery does not. So the monitor has now recorded that it told you, and goes quiet for another seven days on the strength of a message nobody received. Seven days later it does the same thing. And again.
A finding that is steady — the exact profile of someone deliberately structuring payments at a consistent rate — is reported once and then never again, while the system believes it is reporting it every week.
The failure is silent at every layer. The sweep's return value says announced: 1,
because it did announce. The database write succeeded. The webhook table has no error row,
because nothing errored — ON CONFLICT DO NOTHING did exactly what it was told.
There is no log line, because a deduplicated delivery is a normal, expected, healthy event.
Dedupe and repetition are the same mechanism, pointed opposite ways
This is the generalisable part, and it is why we did not see it by reading the code.
A dedupe key answers: is this the same thing I already said? A recurring alert needs to say: this is the same thing, and I am telling you again. Those are directly opposed, and a key derived purely from what is true cannot express the second one. It has no room for when I decided to tell you.
Everything about the design is individually right. Idempotent delivery is right. A cooldown is right. Deriving the key from the payload is right. The bug lives in the seam, which is where this kind of bug always lives.
The fix is one field — the sweep's own clock, carried in the payload:
announced_at: now.toISOString(),
It is now, the value passed into the sweep, and not new Date()
evaluated at the point of use. That distinction preserves the original property: a
transaction retried within one sweep reproduces the same timestamp and still
deduplicates. Only a separate sweep is a separate announcement.
How it was found
Not by reading it. Coverage had drifted to 90.03% against a floor of 90 — six statements of margin out of nineteen thousand — and the largest untested file was this one. The tests were written to buy back margin.
One of them asserted that a finding older than the cooldown reaches the subscriber. It expected two deliveries and got one.
That is the entire argument for a coverage floor, and it is not the argument usually made for one. The number is not the point; nobody is safer because a percentage went up. The point is that writing a test forces you to state what the code is supposed to do, out loud, in a form that can be wrong. The docstring had claimed the behaviour for hours. The assertion disagreed with the implementation in about four seconds.
If you have a monitor with a cooldown
Two questions, and they take a minute each:
- When it repeats itself, what in the message is different? If the answer is nothing, and anything downstream deduplicates — your queue, your webhook table, your paging provider, the email client that threads by subject — the repeat does not exist.
- Does it record that it told you before confirming that it told you? If so, a dropped message does not merely go missing. It resets the clock, and the next window of silence is indistinguishable from everything being fine.
The second one is the more dangerous, because its symptom is quiet, and quiet is what a working monitor looks like.
Ratchet is an effect gate for AI agents: an agent asks before it charges a card, ships a deploy, or sends the email, and gets back a durable decision. The code in this note is public, as is the test that found it.