## Question
How should a queue worker recover an expired lease without two workers applying the same result?
## Why this matters
Recurring public developer task for HTTP and integration errors.
## Environment / product
HTTP and integration errors
## What needs to be determined
Current researched guidance, applicability, limitations, and primary sources for this question.
Researched guidance is proposed, not an execution report.
Problem details
- Observed symptom
- How should a queue worker recover an expired lease without two workers applying the same result?
- Context
- Recurring public developer task; researched guidance is proposed, not an execution report.
- Environment
- Unknown · not established
- Symptom signature
- Literal source
- Not supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 1
Researched guidance: How should a queue worker recover an expired lease without two workers applying the same result?
## Summary
Recover expired work by treating the lease as lost, allowing a new claim only through an atomic ownership transition, and making completion conditional on the current ownership epoch or fencing token. The old worker may continue running, but its completion and side effects must be rejected or made idempotent after it loses ownership.
## Candidate action
Use a durable per-job ownership record with an expiry/heartbeat and a monotonically increasing epoch (fencing token). Claim or reclaim with one atomic compare-and-set/transaction: accept only an unowned or expired record, increment the epoch, and record the new worker. Renew only while the worker still owns that epoch. Before committing completion, atomically compare the job's current owner/epoch with the worker's captured epoch and transition the job to completed only on a match; otherwise discard the stale result and let the current owner continue. Protect any external side effect with the same epoch at the receiving system, or use an idempotency key plus a durable outbox/inbox so a retry cannot apply it twice. If the queue's delivery/ack protocol exposes delivery generations or acknowledgment IDs, acknowledge only the latest valid generation and treat expiry as redelivery, not as proof that the original worker stopped.
## Applicability
- Queue workers where a lease, visibility timeout, or acknowledgment deadline can expire while the original worker is paused, partitioned, or still executing.
- Systems with a durable authoritative job store that can perform an atomic conditional update or transaction.
- Workflows where duplicate delivery is possible and completion or side effects must be applied at most once per logical job, or safely deduplicated.
## Key findings
- SQS documents at-least-once delivery and advises idempotent processing because a message copy can be delivered again. (S1)
- SQS warns that a visibility timeout shorter than processing can expose the same message to another consumer, and recommends extending the timeout while processing continues; its documented maximum is 12 hours from ReceiveMessage. (S2)
- Pub/Sub exactly-once delivery treats an expired acknowledgment ID as invalid because a newer delivery may be in flight; only the latest acknowledgment ID may be used, and subscribers should persist progress to prevent duplicate work after an acknowledgment failure. (S3)
- etcd documents lease-bound lock release and atomic transactions guarded by value, version, creation-revision, or modification-revision comparisons; these primitives can conditionally reject a stale completion in the etcd store, but do not automatically protect external side effects. (S4)
## Known limitations
- A lease expiry does not stop a paused or partitioned worker; it only permits ownership to move. The job store's conditional completion protects the recorded job state, not arbitrary external effects already issued by a stale worker.
- Idempotency or exactly-once delivery features do not remove the need to persist progress and handle failed acknowledgments; guarantees are provider-, subscription-, and region-specific.
- A fencing epoch is effective only when every protected writer or side-effect sink validates it, or when the side effect is routed through a durable idempotent protocol.
## Obsolete approaches
- Do not assume that checking the lease immediately before a remote write is sufficient: the lease can expire between the check and the write unless the check and write share an atomic guard or the sink enforces fencing.
- Do not treat a successful local computation as authoritative after the worker's acknowledgment or lease has expired; the result must pass the current-epoch commit condition.
## Negative results
- The reviewed queue and coordination documentation does not define a universal recovery algorithm for arbitrary external side effects; it provides delivery/acknowledgment rules and conditional-store primitives that must be composed into the worker protocol.
## Evidence boundary
- basis=researched_guidance; executed=false; independent_reproduction=false
- The sources document at-least-once delivery, lease/acknowledgment expiry, acknowledgment generations, lease-bound ownership keys, and atomic conditional transactions. They do not demonstrate this exact worker implementation in a live environment.
## What remains unknown
- The queue backend, storage engine, and external side-effect sinks are unspecified, so the exact claim, heartbeat, fencing, timeout, retry, and dead-letter APIs remain to be selected.
- Whether the workload can be made idempotent or routed through an epoch-validating sink is unknown; without that property, a stale worker can still cause an external duplicate even if the job record's final transition is protected.
- The required lease duration and heartbeat margin depend on workload latency, pause behavior, network limits, and provider-specific maximums; they need measurement and operational tuning.
## Evidence
- basis: researched_guidance
- executed: false
- independent reproduction: false
## Sources
- [S1] Amazon SQS at-least-once delivery — https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html (official_documentation; accessed 2026-09-21)
- [S2] Processing messages in a timely manner in Amazon SQS — https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/best-practices-processing-messages-timely-manner.html (official_documentation; accessed 2026-09-21)
- [S3] Exactly-once delivery | Pub/Sub — https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery (official_documentation; accessed 2026-09-21)
- [S4] etcd API — https://etcd.io/docs/v3.7/learning/api/ (technical_reference; accessed 2026-09-21)
- Problem id
- 593c9f46-660e-4eb7-9ce5-04117a5dd211
- Proposed action
- Use a durable per-job ownership record with an expiry/heartbeat and a monotonically increasing epoch (fencing token). Claim or reclaim with one atomic compare-and-set/transaction: accept only an unowned or expired record, increment the epoch, and record the new worker. Renew only while the worker still owns that epoch. Before committing completion, atomically compare the job's current owner/epoch with the worker's captured epoch and transition the job to completed only on a match; otherwise discard the stale result and let the current owner continue. Protect any external side effect with the same epoch at the receiving system, or use an idempotency key plus a durable outbox/inbox so a retry cannot apply it twice. If the queue's delivery/ack protocol exposes delivery generations or acknowledgment IDs, acknowledge only the latest valid generation and treat expiry as redelivery, not as proof that the original worker stopped.
- Applicability
- State
- partial
- Text
- Queue workers where a lease, visibility timeout, or acknowledgment deadline can expire while the original worker is paused, partitioned, or still executing. Systems with a durable authoritative job store that can perform an atomic conditional update or transaction. Workflows where duplicate delivery is possible and completion or side effects must be applied at most once per logical job, or safely deduplicated.
- Limitations
- State
- partial
- Text
- A lease expiry does not stop a paused or partitioned worker; it only permits ownership to move. The job store's conditional completion protects the recorded job state, not arbitrary external effects already issued by a stale worker. Idempotency or exactly-once delivery features do not remove the need to persist progress and handle failed acknowledgments; guarantees are provider-, subscription-, and region-specific. A fencing epoch is effective only when every protected writer or side-effect sink validates it, or when the side effect is routed through a durable idempotent protocol.
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
- Pack
- Schema version
- 1
- Candidate action
- Use a durable per-job ownership record with an expiry/heartbeat and a monotonically increasing epoch (fencing token). Claim or reclaim with one atomic compare-and-set/transaction: accept only an unowned or expired record, increment the epoch, and record the new worker. Renew only while the worker still owns that epoch. Before committing completion, atomically compare the job's current owner/epoch with the worker's captured epoch and transition the job to completed only on a match; otherwise discard the stale result and let the current owner continue. Protect any external side effect with the same epoch at the receiving system, or use an idempotency key plus a durable outbox/inbox so a retry cannot apply it twice. If the queue's delivery/ack protocol exposes delivery generations or acknowledgment IDs, acknowledge only the latest valid generation and treat expiry as redelivery, not as proof that the original worker stopped.
- Applicability
- Queue workers where a lease, visibility timeout, or acknowledgment deadline can expire while the original worker is paused, partitioned, or still executing.
Systems with a durable authoritative job store that can perform an atomic conditional update or transaction.
Workflows where duplicate delivery is possible and completion or side effects must be applied at most once per logical job, or safely deduplicated. - Limitations
- A lease expiry does not stop a paused or partitioned worker; it only permits ownership to move. The job store's conditional completion protects the recorded job state, not arbitrary external effects already issued by a stale worker.
Idempotency or exactly-once delivery features do not remove the need to persist progress and handle failed acknowledgments; guarantees are provider-, subscription-, and region-specific.
A fencing epoch is effective only when every protected writer or side-effect sink validates it, or when the side effect is routed through a durable idempotent protocol. - Evidence boundary
- basis=researched_guidance; executed=false; independent_reproduction=false
The sources document at-least-once delivery, lease/acknowledgment expiry, acknowledgment generations, lease-bound ownership keys, and atomic conditional transactions. They do not demonstrate this exact worker implementation in a live environment. - What remains unknown
- The queue backend, storage engine, and external side-effect sinks are unspecified, so the exact claim, heartbeat, fencing, timeout, retry, and dead-letter APIs remain to be selected.
Whether the workload can be made idempotent or routed through an epoch-validating sink is unknown; without that property, a stale worker can still cause an external duplicate even if the job record's final transition is protected.
The required lease duration and heartbeat margin depend on workload latency, pause behavior, network limits, and provider-specific maximums; they need measurement and operational tuning. - Summary
- Recover expired work by treating the lease as lost, allowing a new claim only through an atomic ownership transition, and making completion conditional on the current ownership epoch or fencing token. The old worker may continue running, but its completion and side effects must be rejected or made idempotent after it loses ownership.
- Obsolete approaches
- Do not assume that checking the lease immediately before a remote write is sufficient: the lease can expire between the check and the write unless the check and write share an atomic guard or the sink enforces fencing.
Do not treat a successful local computation as authoritative after the worker's acknowledgment or lease has expired; the result must pass the current-epoch commit condition. - Negative results
- The reviewed queue and coordination documentation does not define a universal recovery algorithm for arbitrary external side effects; it provides delivery/acknowledgment rules and conditional-store primitives that must be composed into the worker protocol.
- Key findings
- Text
- SQS documents at-least-once delivery and advises idempotent processing because a message copy can be delivered again.
- Source ids
- S1
- Text
- SQS warns that a visibility timeout shorter than processing can expose the same message to another consumer, and recommends extending the timeout while processing continues; its documented maximum is 12 hours from ReceiveMessage.
- Source ids
- S2
- Text
- Pub/Sub exactly-once delivery treats an expired acknowledgment ID as invalid because a newer delivery may be in flight; only the latest acknowledgment ID may be used, and subscribers should persist progress to prevent duplicate work after an acknowledgment failure.
- Source ids
- S3
- Text
- etcd documents lease-bound lock release and atomic transactions guarded by value, version, creation-revision, or modification-revision comparisons; these primitives can conditionally reject a stale completion in the etcd store, but do not automatically protect external side effects.
- Source ids
- S4
- Research sources
- Id
- S1
- Title
- Amazon SQS at-least-once delivery
- Url
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html
- Source class
- official_documentation
- Accessed at
- 2026-09-21
- Id
- S2
- Title
- Processing messages in a timely manner in Amazon SQS
- Url
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/best-practices-processing-messages-timely-manner.html
- Source class
- official_documentation
- Accessed at
- 2026-09-21
- Id
- S3
- Title
- Exactly-once delivery | Pub/Sub
- Url
- https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery
- Source class
- official_documentation
- Accessed at
- 2026-09-21
- Id
- S4
- Title
- etcd API
- Url
- https://etcd.io/docs/v3.7/learning/api/
- Source class
- technical_reference
- Accessed at
- 2026-09-21
Page 1 · 1 children total
Sources and related records
No source relations recorded.