An agent task is not a quick request and response. A coding agent may run for ten minutes or more: reading files, editing them, running tests. In RunStead as we are designing it, that work runs on a Mac that belongs to the person who started it, while a control service keeps track of it so the task can be followed from the web or a phone.
Ten minutes is a long time for a laptop. In that window the lid can close, the Wi-Fi can drop, the worker process can crash and restart, or a message can arrive twice. Each of those events raises the same question: what actually happened to the task?
There are two tempting answers, and both are wrong:
- “No reply in time means it failed.” Then a perfectly healthy ten-minute task gets marked as failed at minute two, because nothing was reported for a while.
- “If in doubt, run it again.” Then a task that did finish runs a second time. For an agent that edits files and runs commands, a duplicate run is not harmless.
This post describes the rules we are designing around instead. None of them is new on its own; the value is in applying all of them together, consistently.
1. Accept each request exactly once
Every request that starts work carries an idempotency key chosen by the client. The service stores that key together with a digest of the request body.
- Same key, same body: return the original result. Do not evaluate the request again.
- Same key, different body: reject it as a conflict.
- Two identical requests racing each other: whichever commits first wins, and the other gets the same answer.
The important part is what goes into a single transaction. Accepting a task writes the task record, the operation that tracks it, the idempotency record, the budget reservation, any lock on the working folder, and the outgoing message to the worker, all in one commit. Either all of it exists or none of it does. There is never a task without its message, or a message for a task that was never accepted.
Idempotency records are kept until the operation reaches a final state, plus a retention window. Expiring them earlier would let a late retry create a second task.
Permission checks get the same care. If someone revokes access to a folder or a machine, there must be no window in which a new task is still accepted. We get that by having acceptance take a shared lock on the rows it depends on, and revocation take an exclusive one. The two serialize: if revocation commits first, acceptance sees it and refuses.
2. Record before you acknowledge
On the worker side, the rule is simple: write it down durably, then say “got it”.
When a message arrives, the worker commits it to a local database first, and only then acknowledges it. Our design uses SQLite in WAL mode with full synchronous writes, which on macOS means asking for a real flush to the storage device rather than trusting the write cache. A message acknowledged before a power loss is still there afterwards.
The local database is also where deduplication happens. Each operation has an identifier, and the worker executes a given identifier at most once. If a duplicate arrives after the original, it is recognized and not run.
3. Receipts with an identity and a generation
The worker reports progress with receipts. Two fields on every receipt do most of the work.
An event ID, generated by the worker when it creates the receipt and reused on every retry. The service stores receipts keyed by that ID, so a receipt that arrives twice is applied once, and the second delivery gets the original answer.
A generation number for the worker’s identity. When a machine is re-paired or its keys change, its generation increases. A receipt from an older generation is kept in the history but can never change the current state of a task.
The service also answers definitively. If a receipt is valid but no longer relevant, the reply says, in effect, “received, not applied”. That is a final answer, so the worker stops resending. Retries are kept for genuinely transient failures, with exponential backoff capped at a minute. Malformed messages and state conflicts are final rejections, not reasons to try again.
4. Liveness comes from a lease, not from silence
Receipts tell you about state changes. They are a poor way to tell whether a long task is still alive, because a healthy task can go quiet for minutes.
So once a task actually starts, liveness comes from an execution lease: a permission to keep running that the worker must renew on a schedule. A lease might last 60 seconds, with renewal every 15. Each renewal checks that the task is still allowed to run: the machine is still paired, the folder is still granted, the budget is still in place.
The worker computes its own deadline conservatively:
local_deadline = time_request_was_sent + lease_duration - safety_margin
A few details matter more than they look:
- Count from when the request was sent, not when the reply arrived. However slow the network, the worker’s deadline then falls before the server’s.
- Use relative durations, never compare wall clocks. The worker and the service do not need to agree on the time of day.
- Watch out for sleep. On macOS, a common monotonic clock source stops while the machine sleeps. A worker that only looks at it would wake up and believe an expired lease is still valid. We take the larger of monotonic and wall-clock elapsed time. A wall clock that jumps backward can never extend the deadline, and one that jumps forward only makes the worker stop early, which is the safe direction.
- Poll the deadline. A check every second is more predictable across sleep than a single long timer.
When the deadline passes, or a renewal is refused, the worker stops the task’s whole process group.
On the service side, a lease that is not renewed expires. The task then moves to a state that means “outcome unknown”. It is not marked as failed, it is not retried, and its budget reservation and folder lock stay in place until the facts are known.
5. Judge by when the process ended, not when the receipt arrived
A late receipt is normal: the laptop was asleep, then it woke up and reported. What matters is whether the process ended while it still had permission to run.
So the final receipt carries facts, not just a verdict: whether the process started, its exit code or signal, which lease it ran under, and whether it ended within that lease. Because the worker always stops a process at its local deadline, it can state that last fact reliably.
- Ended within the lease: accept the reported result, even if the receipt arrives after the server-side lease expired.
- Ended after the lease was lost: record what the worker reported, but settle the task as failed because it lost its lease.
The same facts decide what happens to the budget. A reservation is released only when it is certain the process never started. Otherwise it is settled against the work that ran.
6. A restart never runs anything again
Crashes happen in the worst places, including between “about to start the process” and “started the process”. The local database has to make that window safe.
Before starting anything, the worker commits a starting state and a spawn counter set to one, with a database constraint that the counter can never exceed one. Right after the process starts, it records the process ID, the process group, and the process start time.
When the worker restarts and finds a task that was starting or running:
- It looks for the leftover process using the process ID and its start time, so a reused process ID is never mistaken for the original.
- It stops that process, reports “outcome unknown”, and then reports the final state.
- If the record says
startingbut has no process ID, the task may have started. The worker reports exactly that, and nothing more.
In none of these cases does the worker start the task again.
7. “Unknown” is a real state
The thread running through all of this is that “we don’t know” is an answer the system is allowed to give. A task whose outcome is unknown is not the same as a failed task, and it is never quietly retried.
From there, the service can ask the worker what it knows locally. If the worker has a record, that settles it. If the worker has no record at all, the person who owns the task decides what to do next, because they are the one who can look at the files and tell whether the work happened.
The trade-off
These rules cost something. Occasionally the system stops and asks a person instead of making a confident guess. We think that is the right trade for software that edits real files on real machines: a question you can answer beats a duplicate change you have to find and undo.
We will share how these rules hold up once the pilot is running. If you want to follow along, the RSS feed is the easiest way.