Durable automation guarantees that queued work is not lost. It does so by retrying anything whose completion was never recorded, which means some work runs twice. If the step in question charges a card, posts a ledger entry or emails a customer, the guarantee you bought has created a new problem.
Retry safety is not something a queue can give you. It is a property of the action, and you have to build it.
Idempotency defined, without the maths
An operation is idempotent when performing it twice has the same effect as performing it once. The second attempt is allowed to be a no-op, or to return the first attempt’s result, but it must not add anything new to the world.
The test is concrete. Ask what the state of the system is after two executions, and compare it to the state after one. If those two states differ, the step is not idempotent and a retry will corrupt something.
| Step | Idempotent | Why |
|---|---|---|
Set ticket status to resolved | Yes | Absolute assignment; the second write changes nothing |
Add tag vip to a contact | Yes | Set membership; already present means no change |
| Increment escalation level by 1 | No | Relative arithmetic; two runs produce level 2 |
| Send an email to the customer | No | Each execution creates a new artifact in someone’s inbox |
| Charge a card for $240 | No | Two executions, two charges, one refund request |
| Post a journal entry | No | Two executions, two postings, books out by the amount |
| Create a project from a template | No, unless keyed | Two executions, two projects, both half-owned |
| Write a file at a fixed path | Yes | Last write wins at the same location |
The pattern is visible in the table. Absolute assignment and set membership are naturally idempotent. Anything relative, anything that creates a new record, and anything that touches the outside world is not, and has to be made safe deliberately.
The duplicate-charge failure
A billing automation runs nightly: find invoices due today, charge the saved payment method, mark the invoice paid.
One night the payment provider is slow. The charge request goes out and succeeds on the provider’s side, but the response times out on the way back. The worker treats the timeout as a failure, its lease expires, and the job is retried from the top. The second attempt finds the invoice still unmarked, charges again, and this time gets a clean response. The invoice is marked paid.
The books show one payment. The customer’s statement shows two, for $2,400 each, on the same night. It is found not by monitoring but by a customer email, four days later, worded the way those emails are worded. The refund is easy. The cost is that this customer will now check their statement every month, and they will tell their peers to do the same.
Every element of that failure is normal. A slow provider is normal. A timeout that hides a success is normal. A retry after a failure is correct behaviour. The single missing piece is an idempotency key, which would have let the provider recognise the second request as the same request and return the first result instead of charging again.
Idempotency keys, natural and synthetic
An idempotency key is a stable string that identifies a unit of work, carried on every attempt, and stored by whoever performs the work so a repeat can be recognised.
The critical word is stable. A key generated fresh on each attempt is not a key, it is noise. It must be minted once, when the work is created, and persisted with the job so that every retry sends the identical value.
Natural keys are derived from data that already uniquely describes the work: invoice-4417-charge, or ticket-9021-followup-step-3. They need no extra column because the data already exists, and they survive a worker losing all local state. They fail when the fields can legitimately repeat. A key of customer plus amount plus date collapses two genuine identical charges on the same day into one, which is a silent under-billing that is harder to find than a double charge.
Synthetic keys are generated identifiers, typically a UUID created when the job row is written and carried through every attempt. They never collide and never merge two genuine operations. Their cost is a column and the discipline of never re-generating them, which is exactly the bug that makes teams believe idempotency keys do not work.
Deduplication also needs a window. Keys stored forever grow without bound; keys stored for an hour do not protect a job that dead-letters and is replayed the next morning. Match the retention to your longest realistic replay horizon, and state it, because an undocumented dedup window is a landmine.
Why the queue cannot solve this for you
Teams reach for “exactly-once delivery” and find that the phrase collapses under inspection. A queue can guarantee a message is not lost. It can guarantee a message is not delivered twice. It cannot guarantee that the consumer’s side effect happened exactly once, because the side effect and the acknowledgement are two operations that cannot be made atomic across a network boundary.
Consider the ordering. Acknowledge before doing the work, and a crash between the two loses the work. Do the work before acknowledging, and a crash between the two repeats the work. There is no third ordering. Every system marketed as exactly-once is at-least-once delivery plus deduplication somewhere in the consumer, which is idempotency wearing a better name.
That is why these two articles are separate. Durable automation is about state persistence: the pending action is a row rather than a timer, so a restart cannot erase it. Idempotency is about action safety: the step recognises that it has already run, so a retry cannot duplicate it. Durability without idempotency produces duplicates. Idempotency without durability produces silence. Production systems need both, and they are built in different places.
Designing flows that tolerate retries
Three habits do most of the work.
Put irreversible steps early. In a flow that waits, everything after the wait is in the retry zone. If the irreversible step runs before the first pause, the interruption window never covers it.
Prefer absolute over relative. Write escalation_level = 2 rather than escalation_level = escalation_level + 1. Write status = resolved rather than advance_status(). Absolute assignment is idempotent for free.
Record the effect, not the intent. A row written before the side effect says “we meant to send this”. A row written atomically with it, or immediately after with the provider’s returned identifier, says “this was sent”. Only the second one is a safe thing to check on resume.
What Oneop states, and what it does not
Oneop’s automation is explicit about this boundary rather than quiet about it. A waiting run is claimed before it executes and marked done only after execution returns, so two workers cannot pick up the same run at once. That is a real guarantee. The stated limit, on the same page, is that a resumed run restarts the step it was interrupted in, so a customer reply, a notification, an escalation or a metered AI step after a wait can happen a second time. The product’s own advice is the advice above: after a wait, prefer steps that are safe to repeat, and keep irreversible ones early.
Two places in Oneop are deliberately stricter, because repetition there is unacceptable. In finance, a posted journal entry cannot be edited and the same entry cannot be reversed twice. In the help desk, an SLA breach is claimed with a single guarded update before it can fire, so one breach produces one response.
The general lesson generalises past software design. Any counter, any reservation, any money movement needs a defined answer for “what happens if this runs twice”, and the answer must be written into the operation rather than assumed from the infrastructure. Overselling is the same failure in a different module, argued in full in available to promise, and the discipline of deciding which steps should never be automatic at all is covered in when not to automate. The same question is worth asking of any outbound sequences step you schedule, because a duplicate there lands in a customer’s inbox rather than in a log.