Repository navigation
Decisions never queue behind other conversations; voice=True waits 1.5 s - #17
Merged
Merged
Conversation
…time runs from its send Sync decision calls shared one process-wide pool of 8 threads, and each call's timeout ran from when it was queued. An agent serving many conversations from one process (32 in our benchmark harness) queued its decisions behind each other whenever the service slowed down: the queued calls ran out of time before they were sent and failed open (30 of 1352 calls in one run, most of them answered quickly once they reached the service). The pool now grows with the calls in flight (up to MAX_DECIDE_THREADS = 256; a thread is started only when none is idle), and a call has its full timeout from when it is sent, the budget the deadline header tells the service. Waiting for a free thread, only past 256 at once, is bounded by the timeout too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ruNgP5DUwEkpSrXnv5R8T
A voice caller hears every second of waiting. wrap(..., voice=True) sets the default decision timeout to 1.5 s (chat stays 5 s); an explicit timeout still wins. Decisions take 0.6 s at the median and 1.5 s at p90 in a 1,156-turn voice benchmark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ruNgP5DUwEkpSrXnv5R8T
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wrap(..., voice=True)sets the default decision timeout to 1.5 s (chat stays 5 s); an explicittimeoutstill wins. In a 1,156-turn voice benchmark, decisions take 0.6 s at the median and 1.5 s for the slowest 10%.Tests:
tests/test_decide_pool.py(4 tests; 2 fail on the old code) and a parametrized voice-timeout test intests/test_wrap.py. Ruff and mypy are clean. Locally, 3 async tests can't run without pytest-asyncio; CI installs the dev extras.🤖 Generated with Claude Code
https://claude.ai/code/session_018ruNgP5DUwEkpSrXnv5R8T
Generated by Claude Code