# How do I stop an expired job worker from overwriting a newer worker's result? > Lease expiry is not enough. Use fencing tokens enforced at the write boundary so stale workers cannot overwrite newer results after a pause or partition. Canonical URL: https://www.devobs.io/articles/qa-stale-worker-fencing/ By: Samira Haddad Published: 2023-03-13T08:12:44.019Z Updated: 2026-09-06T08:31:04.426Z Section: Architecture A lease timeout by itself is not enough. A worker can pause, lose its lease, get replaced, and then resume late with enough state to still write. To stop stale overwrites, give each worker attempt a monotonically increasing fencing token and require the final sink to reject writes with an older token than the latest accepted one. The crucial check happens at the write boundary, not as a preflight read. ## Why isn't lease expiry enough? Because expiry only tells the coordinator that ownership should move on; it does not physically stop the old process. If worker A pauses during a long GC stop, VM suspension, or network partition, its lease can expire and worker B can legitimately take over. Systems like etcd document that a lease expires when the server does not receive keepalives within the TTL, and keys attached to that lease are deleted on expiry via the [etcd v3.6 API reference](https://etcd.io/docs/v3.6/dev-guide/api_reference_v3/). ZooKeeper makes the same distinction differently: recipes rely on ephemeral nodes and an overall order on updates, but those are client-side conventions, not magic cancellation of already-running code, as described in the [ZooKeeper recipes](https://zookeeper.apache.org/doc/current/recipes.html). So the failure mode is simple: A starts first, stalls, B takes over, finishes correctly, then A resumes and writes last. If the sink accepts “last writer wins,” the stale worker wins. ## What contract actually prevents stale writes? Use fencing. Each successful claim of the job gets a strictly increasing token: 41, then 42, then 43. The worker must present that token when writing the result. The sink must store the highest accepted token and atomically reject any write with a lower one. That ordering boundary matters more than how you issue the lease. etcd gives you the raw primitives for this pattern: every `Put` increments store revision, and `Txn` executes multiple requests in a single transaction, as documented in the [etcd v3.6 API reference](https://etcd.io/docs/v3.6/dev-guide/api_reference_v3/). ZooKeeper recipes similarly emphasize that it “imposes an overall order on updates” and often use sequential ephemeral nodes to expose that order in coordination flows, in the [ZooKeeper recipes](https://zookeeper.apache.org/doc/current/recipes.html). Worked example: 1. Worker A claims job J with token 41. 2. A pauses before committing output. 3. Lease expires; worker B claims J with token 42. 4. B writes result with token 42. Sink stores `last_token = 42`. 5. A resumes and tries to write token 41. 6. Sink rejects token 41 because `41 < 42`. In SQL, that usually means an update such as “write only if incoming token is greater than stored token.” In an object store or external API, you need an equivalent conditional write. If the destination cannot compare and reject stale tokens atomically, you do not have real fencing. ## Where should I enforce the token check? At the system that would be corrupted by a stale write. If the dangerous step is updating a database row, enforce the token in that row update. If the dangerous step is publishing to a downstream service, that service needs idempotency or conditional update semantics tied to the token. A coordinator-side check alone is insufficient because A can pass the check, pause again, and still write later. A short decision checklist: - Does every new worker attempt get a higher token than the previous one? - Is that token carried all the way to the final write? - Can the sink atomically reject `token <= last_seen_token`? - Is your logic safe if worker A resumes minutes later? ## When does this advice not apply? If the work is naturally idempotent and duplicate or late completion cannot corrupt state, simple retries may be enough. But if one stale completion can overwrite newer state, charge twice, or revert progress, use fencing rather than relying on lease expiry alone. ## Follow-up: Can I just re-check the lease before writing? No. That is still a preflight check. The worker can pass it and then write after becoming stale. The sink must enforce ordering during the write itself. ## Follow-up: Can etcd or ZooKeeper alone protect my database write? Not automatically. They can issue leases, sessions, sequencing, and transactions for coordination. Your database or downstream sink still needs its own conditional acceptance rule for the fencing token. Next step: pick the exact write that must never be reverted, then add a monotonic token field and make that write conditional on accepting only newer tokens. Reviewed: 2026-09-05 ## Source references - -