A follow-up scheduled for Tuesday is a promise your system made on Monday. If that promise exists only in the memory of a running process, then a deploy, a crash, an out-of-memory kill or a routine autoscaler decision quietly erases it. Nobody gets an error. The work simply does not happen, and the only evidence is a customer who was never contacted.
Delayed work is the part of automation most likely to be built wrong, and the least likely to announce that it was.
Why an in-memory timer is not a schedule
A timer created with setTimeout is not a schedule. It is a note held in the memory of one process, and it has exactly the lifetime of that process.
This is easy to miss because it works perfectly in development, where the process runs for hours and nothing kills it. It also works in production, right up until the first deploy that lands during a waiting period. There is no error, no retry, no alert. The state that said “do this at 14:00 on Tuesday” was never written anywhere that outlives the process, so after the restart there is nothing to recover and nothing to notice.
The property you actually need has a name: durable workflow execution. A paused automation must be represented by a record in storage, independent of any running process, that a future worker can find and act on.
What durable scheduling stores
Durable scheduling writes the pending action to storage and lets a separate worker claim it later. The record has to carry enough to continue the run without the original process, which is more than just a time.
| In-memory timer | Durable scheduled row | |
|---|---|---|
| Where the pending work lives | Heap of one process | Database row |
| Survives a restart | No | Yes |
| Survives a crash mid-wait | No | Yes |
| Visible to operations | No, invisible by construction | Yes, queryable: what is pending, what is overdue |
| Maximum practical wait | Process lifetime | Unbounded |
| Failure mode | Silent disappearance | Late execution, or a recorded failure |
| Can two workers run it at once | N/A | Only if you do not claim it first |
The minimum useful record is: which run this is, which step it stopped at, the variables gathered so far, a snapshot of the flow definition as it was when the run started, and the time it becomes due. The flow snapshot matters more than it looks. If you re-read the live definition on resume, then editing an automation on Wednesday silently changes what every run already waiting is going to do on Thursday, including runs that already took a branch that no longer exists.
The Friday deploy that cancelled 400 follow-ups
Here is the concrete failure, and it is boringly common.
A support team builds a rule: when a ticket is resolved, wait 48 hours, then send a satisfaction follow-up. It is implemented as a timer in the application process. It passes QA, because QA never restarts the server mid-wait.
The team ships on Friday afternoon. The deploy rolls the process. Every follow-up scheduled in the previous 48 hours is destroyed at the moment of the rollout, because each one was a callback in the memory of a process that no longer exists. About 400 customers are never contacted. No row was written, so no row is overdue, so no dashboard shows a backlog. The queue is empty, which looks identical to the queue being done.
The team discovers it three weeks later, when someone asks why the response rate dropped, and the only way to reconstruct the damage is to query resolved tickets with no follow-up activity and diff them against the deploy timestamps. The lost work cannot be re-sent honestly either, because a satisfaction survey three weeks after the fact is a different message.
The lesson is not “timers are bad”. It is that the absence of persisted state makes the failure undetectable, and an undetectable failure is worse than a loud one.
At-least-once delivery is what durability buys you
Once pending work is a row, a worker has to claim it before running it, or two workers running at the same moment will both pick it up. Claiming usually means one conditional update: set the row’s due time forward, or set an owner and a lease expiry, and proceed only if the update actually changed a row. Losers of that race get zero rows changed and move on.
The lease matters because a worker can die mid-execution. If the row were marked done before the work ran, a crash would lose the work. So the row is marked done after the work returns, and if the worker dies in between, the lease expires and the row becomes claimable again.
That ordering gives you at-least-once delivery: nothing is lost, but a step interrupted after its side effects and before its completion record will run again. This is the correct trade for business automation, because a duplicate is visible and a loss is not. It is also an obligation, and it is the subject of the companion to this article: idempotency in business automation covers how to make the action itself safe to repeat. Durability solves “the work must not vanish”. Idempotency solves “the work must not count twice”. Neither substitutes for the other.
Retries, backoff and the message that never succeeds
Durable execution makes retries possible, which makes retry policy necessary.
| Failure | What it means | Correct response |
|---|---|---|
| Transient (timeout, 503, deadlock) | The action may succeed later | Retry with exponential backoff and jitter |
| Rate limited | The provider is asking you to slow down | Retry after the stated interval, do not add attempts |
| Permanent (404 on a deleted record, validation error) | It will never succeed | Fail the run immediately and record why |
| Ambiguous (connection dropped after sending) | It may have already succeeded | Retry only if the action is idempotent |
| Poison (fails every attempt for unknown reasons) | An attempt loop that never terminates | Cap attempts, then dead-letter for a human |
Exponential backoff exists so a struggling dependency is not hammered by every stuck run simultaneously. Jitter exists because without it, all the retries synchronise and arrive in one spike, which is how a brief outage becomes a long one.
The retry ceiling exists because of poison messages. A queued item with malformed data fails, is requeued, fails, is requeued. With no attempt cap it consumes a worker forever and produces an infinite log. Cap attempts, move the item to a dead-letter store, and treat that store as an operational surface a person actually looks at. A dead-letter queue nobody reads is a slower version of losing the work.
How Oneop handles waiting
Oneop’s ticket automations implement waits as durable state rather than timers. When a run reaches a wait step, the remaining steps, the step index it stopped at, the variables collected, the trace so far and a copy of the flow as it was when the run started are written to a database row with a time to resume. A separate process looks for rows that are due, claims a row before executing it, and marks the row done only after execution returns. A twenty-four hour wait set on Monday still runs on Tuesday even if the server was restarted in between. Full mechanics are on the automation page.
The limit is stated on that page rather than hidden: a resumed run restarts the step it was interrupted in, so a step whose side effect already happened can happen again. Oneop claims durability, not exactly-once execution, and those are different guarantees. Waits, approvals and the dry-run simulator are not behind a plan gate, which you can verify on pricing.
The same durability question applies to anything that waits, including an automation that pauses for a person. Deciding which steps deserve a human at all is a separate design problem, and one worth answering before you automate: when not to automate argues the case for leaving some things manual, and the autonomy levels post covers the suggest-approve-autonomous ladder for actions a model proposes.
The question to ask any automation vendor
Ask one question: “If you restart the server right now, what happens to everything currently waiting?”
A vendor with durable execution answers with a mechanism: a row, a due time, a worker, a claim. A vendor without one answers with reassurance. Then ask to see the pending work as a list. If pending work cannot be listed, it is not stored, and if it is not stored, it is not going to survive Friday’s deploy. Ticket-side automations that wait are described in more detail on the help desk page.