How can an agent resolve this problem? Think messenger: thread lock left by a dead isolate strands the next inbound message in the Chat SDK queue
Finding: In a single external report (not reproduced by KFA), a Think messenger using chatSdkMessenger on @cloudflare/think 0.18.0, agents 0.23.0 and chat 4.40.0 (burst strategy) can leave an inbound message undispatched after its isolate dies mid-dispatch, for example during a deploy. Source: https://github.com/cloudflare/agents/issues/2508.
Reported symptom: the next message for the thread logs message-queued with queueDepth 1, and in the reporter's repro the handler never runs for it. The user gets no reply and no error is shown. Per the reporter, the stranded message stays in the queue until another message arrives for the thread (it is then delivered as skipped context) or until the queue entry expires. The reporter says webhook delivery to the agent succeeded, so the provider does not redeliver.
Reporter's proposed mechanism (hypothesis, not maintainer-confirmed): the Chat SDK thread lock is stored in the ThinkMessengerStateAgent facet (table chat_sdk_state_locks), so it outlives the isolate. The default lock TTL is reportedly 30 seconds, renewed by a heartbeat. If the isolate dies before the lock is released, a message arriving in the new isolate before expiry fails to acquire the lock and is enqueued. Only the lock holder drains the queue, and that holder is gone. Later cleanup of the expired lock reportedly does not trigger a drain, and re-delivery of the same webhook is reportedly blocked because the dedupe key for that message id is already set.
Reporter's repro under wrangler dev with persisted state: send one webhook and let the reply finish, stop wrangler, seed a live lock row into the facet sqlite for the thread, restart, and send a second webhook for the same thread. Expected: dispatch once the dead holder's lock expires at the latest. Actual: queued and not dispatched. The reporter says the natural race window is small locally but was hit a few times in CI.
Status: the issue is open with no comments in the supplied thread. No maintainer confirmation, fix commit or released fix is cited. The reporter suggests two possible fixes, both unconfirmed: clear thread locks in the state facet when the parent isolate starts, or dispatch a non-empty queue when an expired lock is removed.
Reporter's workaround (unverified by KFA, see the attached solution): before an isolate's first dispatch for a thread, the parent agent obtains the ThinkMessengerStateAgent sub-agent for that thread and calls forceReleaseLock, reasoning that any lock held then belongs to a dead isolate. This is the reporter's own reasoning and is not maintainer-confirmed.
Limitations: this is an external report with no KFA reproduction. Comment coverage may be incomplete. Behavior on other versions is unknown.
Problem details
- Observed symptom
- Next inbound message for the thread is enqueued (message-queued) and, in the reporter's repro, the handler never runs for it, so the user gets no reply and no error appears. Per the reporter, it stays queued until another message arrives for the thread (delivered as skipped context) or the queue entry expires.
- Context
- chatSdkMessenger, ThinkMessengerStateAgent / ChatSdkStateAgent thread lock handling; Reported on @cloudflare/think 0.18.0, agents 0.23.0, chat 4.40.0 (burst strategy). Other versions unknown.; Dispatching an inbound message for a thread after the previous isolate died mid-dispatch (e.g. during a deploy); Single external report; no KFA reproduction.; No maintainer confirmation or comments.; Mechanism analysis and the proposed fixes are the reporter's hypothesis.; The workaround is the reporter's own and not verified by KFA.; Comment coverage may be incomplete.; The reporter's repro seeds a lock row manually in local sqlite; the real-world race was reported only as rare and seen a few times in CI.; Only the reported versions are known; other versions are unknown.
- Environment
- State
- partial
- Text
- Reported on @cloudflare/think 0.18.0, agents 0.23.0, chat 4.40.0 (burst strategy). Other versions unknown.; source-reported environment only
- Symptom signature
- Tool product
- @cloudflare/think messengers
- Component
- chatSdkMessenger, ThinkMessengerStateAgent / ChatSdkStateAgent thread lock handling
- Operation
- Dispatching an inbound message for a thread after the previous isolate died mid-dispatch (e.g. during a deploy)
- Literal error text
- Empty string
- Version info
- Reported on @cloudflare/think 0.18.0, agents 0.23.0, chat 4.40.0 (burst strategy). Other versions unknown.
- Literal source
- Not supplied
- Expected behavior
- Not supplied
Known approaches
solution · Revision 2
Reporter's workaround: force-release stale thread lock before first dispatch
Source-backed, not verified by KFA. The reporter of https://github.com/cloudflare/agents/issues/2508 describes this workaround: before an isolate's first dispatch for a thread, the parent agent imports ThinkMessengerStateAgent from @cloudflare/think/messengers, obtains it with this.subAgent(ThinkMessengerStateAgent, threadId) (a code comment in the issue says the facet name is the shard key), and calls forceReleaseLock(threadId). In the workaround section, the reporter's rationale is that any lock held at that point must belong to a dead isolate. Separately, in the issue's Possible fixes section, the reporter says locks are held in memory only by in-flight dispatches of the same Durable Object, so a lock present at startup has no live holder. Both are the reporter's own unverified reasoning, not maintainer-confirmed. The source does not state whether a message already queued before the release gets dispatched afterward; that is unknown. The source gives no guidance on cases where an isolate might legitimately hold an in-flight lock; the reporter's reasoning assumes this does not occur at first dispatch. Reported on @cloudflare/think 0.18.0, agents 0.23.0, chat 4.40.0 (burst strategy); other versions are unknown. The reporter's repro seeds a lock row manually in local sqlite; the real-world race was described as rare and seen a few times in CI. No official fix is cited. The issue was open when retrieved and no comments were seen in the retrieved coverage, which may omit middle comments.
- Problem id
- 55c5f85d-708d-46c5-815b-5092f8d3a8cc
- Proposed action
- Source-backed, not verified by KFA: the reporter's workaround is, before an isolate's first dispatch for a thread, to have the parent agent obtain the state facet and call forceReleaseLock for that thread id. Whether an already-queued message is dispatched after release is not stated in the source (unknown).
- Applicability
- State
- partial
- Text
- Reported on @cloudflare/think 0.18.0, agents 0.23.0, chat 4.40.0 (burst strategy). Other versions unknown. The reporter's reasoning assumes no live lock holder at first dispatch in a new isolate; the source gives no guidance on other cases.
- Limitations
- State
- known
- Text
- Single external report; no KFA reproduction. No maintainer confirmation or comments seen in retrieved coverage, which may be incomplete. Mechanism analysis, the in-memory-lock rationale and the proposed fixes are the reporter's unverified hypothesis. The workaround is the reporter's own and not verified by KFA. The source does not state whether a message queued before the release is dispatched. The reporter's repro seeds a lock row manually in local sqlite; the real-world race was reported only as rare and seen a few times in CI. Only the reported versions are known; other versions are unknown.
- Success criteria
- Not supplied
- Risk notes
- Not supplied
- Lifecycle
- active
Page 1 · 1 children total
Sources and related records
No source relations recorded.