fix(workspace): the audio bridge owns the live-call lease, arms it at connect and always releases it (#2700)

[trinity-ability]

Fixes #2700

Problem

POST …/voice/start marked the Workspace thread "a call is on" before the audio WebSocket existed, and every path able to clear that marker — the bridge's finally, the REST /stop, and the cap watchdog that would end the session — sits downstream of a socket that may never open. A start whose socket never opened therefore stranded the marker for its full TTL (WORKSPACE_VOICE_MAX_DURATION + 120 = 1920 s, ~32 min at defaults), and for that whole window every typed turn in that thread — from any tab and from the headless /chat surface — was refused 409 voice_call_active with "A voice call is on in this chat — end it, then send.", naming a call the person cannot end. The only recovery, start another call and hang up properly, is undiscoverable.

The change

Three moves. The first is the one the issue asked for (option 2); the other two are why it holds.

1. The party that opens the effect closes it. The audio bridge is now the only writer of the marker. It arms it as the first statement inside the same try whose finally releases it, and that release is unconditional, LAST, synchronous (so it also runs when the finally is entered by cancellation) and keyed on the portal_session_id local captured before the try — never on ended.portal_session_id, where an ended is None return (the exact case the REST /stop clear was added for) would re-strand it: the same defect, moved.
2. The marker is a lease, not a latch (option 3, in timer form — not the issue's literal "re-armed on each turn"). TTL VOICE_MARKER_LEASE_SECONDS = 60, renewed every VOICE_MARKER_TICK_SECONDS = 15.0 by a bridge-owned task, bounded at the call's own max_duration + VOICE_MARKER_SLACK_SECONDS so it can never outlive one call. Without this, move 1 only relocates the harm — a SIGKILL, an OOM or a routine backend deploy mid-call still strands the thread for 32 minutes with this issue's own symptom. Both event-driven variants were rejected on verified grounds: re-arming per spoken turn expires a live-but-quiet call (the person listening to a long answer fires no turn), re-arming per audio frame expires a muted call (useVoiceSession.js skips ws.send while muted). Either resurrects bug(workspace): a voice call sits at the top of the thread instead of where it happened, and the agent's next typed turn does not know what was said #2694. The 4× TTL:tick ratio and its reason are lifted from agent_call_limiter.py, copied not invented; the renew write goes off-loop via asyncio.to_thread for the same reason it does there.
3. The lease has an owner. The value stored is the call's own voice_session_id, and a release deletes only on match (client_portal/service.py::clear_turn_inflight's read-then-delete-on-match precedent). owner is keyword-only and required on both primitives, so a future call site cannot silently write an unowned lease. Without this, a closing bridge frees the thread of a newer call — a reload, a second tab — which is the bug(workspace): a voice call sits at the top of the thread instead of where it happened, and the agent's next typed turn does not know what was said #2694 defect re-entered through this fix.

Per-producer bounds, before → after:

Orphan producer	Today	After
socket never opens (the reported bug)	~1920 s	0 s — nothing is armed
call ends normally	0 s	0 s
crash / OOM / deploy mid-call	~1920 s	≤ 60 s — the lease is simply not renewed
reload / second tab closing over a live call	thread freed under a live call	not freed — the release is owner-matched
live bridge, dead peer (half-open TCP)	≤ 1920 s	≤ cap + 120 + 60 s (the renewer stops at the cap)
Redis outage	thread writable (fail-open read)	unchanged

Also in blast radius, two lines: the close-out is wrapped in except Exception + logger.exception (not BaseException — CancelledError still propagates), so a raising end_session/persist no longer skips the gemini cancel, the saved frame and the socket close. The renew task is cancelled as the first statement of the finally, before anything that can await.

The read (voice_call_active) stays fail-OPEN and _refuse_turn_during_voice_call is byte-unchanged — #2735's voice_call_id exemption is intact (client_portal/service.py does not appear in this diff, and tests/unit/test_ent551_voice_background_tasks.py passes unedited). No new endpoint, field, broadcast, model or migration. No frontend file is touched.

Acceptance criterion 3, restated behaviourally

The issue's third AC is "the existing test_2694_voice_delta_context.py marker tests still pass." Three of them cannot pass unchanged, by construction, because they pin exactly what this fix moves:

Test	Why it cannot hold literally
test_the_live_call_marker_is_set_cleared_and_fails_open	calls the primitives positionally; owner is now keyword-only and required
…marks_the_thread_live_for_the_cap_plus_slack	asserts /start is the set site — the thing being removed
test_the_bridge_clears_the_marker_when_the_call_closes	a 600-char inspect.getsource proximity grep that structurally cannot see the if ended: nesting which is the defect

AC 3 is therefore read as: the marker's behavioural guarantees stay pinned — armed while a call is live, released when it ends, held only by its owner, fail-OPEN on Redis, the TTL as the backstop — and every test that changes gets stronger. The first becomes the owner-token test; the second becomes test_start_workspace_voice_does_not_arm_the_thread_before_a_socket_exists; the third is replaced by test_the_bridge_resolves_the_real_marker_helpers (which asserts what a source read can honestly assert — that the real modules resolve each other, that the marker import is hoisted out of the finally, and that the release is not keyed on ended) plus four behavioural bridge tests. Every rewrite is proved to bite by the mutation battery below. The other 25 tests in test_voice_auth.py and the whole regression fence run unedited.

Decisions recorded

• Option 1 (client stop() on ws.onerror / beforeunload) is rejected. Primary reason: under option 2 there is nothing left for it to clear — on the never-opened path no marker is armed, and the only residue is the voice session blob, already bounded by its own max_duration + 60 Redis TTL. A mechanism with no work to do is not belt-and-braces. Secondary: the naive form fires after a successful open too, where a forced REST /stop costs the call's summary row (end_session + remove_session run first, so the bridge's own end_session returns None and the "Voice call · N min" row is never written) — which is what restStop: false protects.
• VOICE_MARKER_LEASE_SECONDS = 60 and VOICE_MARKER_SLACK_SECONDS = 120 were ratified at review as engineering bounds, not product settings: they expose no surface and no knob, they govern only crash-recovery latency and the half-open ceiling, and the lease obeys the in-house ≥4× TTL:tick rule (agent_call_limiter.py) so one BGSAVE-class stall cannot expire a live bridge's lease. They are pinned by a test, so a future change to either is a visible edit rather than a drift.
• docs/memory/architecture/workspace.md is deliberately NOT edited. Its sentence — "a typed turn is refused (409) while a call is on" — states no set site and stays true; the detail's home is the feature flow (editorial rule: one home per feature). Named here so a reader can tell reviewed from overlooked. Same for architecture/backend.md, api-endpoints.md, frontend.md and background-services.md: no endpoint contract, signature, router registration, frontend file or startup loop changes.
• No product or UX decision is opened. The 409 copy, its category/retryable contract, the client's 'Voice connection error' and VOICE_UNAVAILABLE_FALLBACK are untouched, and the owning tab's composer inertness is client-side modal state the marker never drove. The only user-visible change is subtractive: the 409 stops appearing when there is no call.

Tests

Red-on-base (recorded by the implementer against 2c5cfe0e, both source files restored from a scratch copy — never git checkout --): 10 of the 11 new/rewritten tests fail; the one that passes is exactly the test labelled a guard.

#	Test	On base 2c5cfe0e
1	test_the_live_call_marker_is_set_cleared_and_fails_open (owner cases + real constants)	RED
2	test_the_renewer_holds_the_lease_and_stops_at_the_calls_own_cap	RED
3	test_start_workspace_voice_does_not_arm_the_thread_before_a_socket_exists	RED
4	test_the_bridge_resolves_the_real_marker_helpers	RED
5	test_a_start_whose_socket_never_opens_leaves_the_thread_writable (the AC-2 test)	RED
6	test_the_bridge_arms_a_lease_at_connect_and_releases_it_on_close	RED
7	test_the_marker_is_released_even_when_the_session_already_ended (Trap C)	RED
8	test_the_marker_is_released_when_the_close_path_raises	RED
9	test_the_marker_is_released_when_the_bridge_is_cancelled	RED
10	test_an_agent_detail_call_never_arms_a_thread_marker	GREEN — labelled a guard, not counted as regression coverage
11	test_the_rest_stop_releases_the_marker_for_a_stop_that_beats_the_socket	RED (the owner kwarg)

Mutation battery. The plan's must-do 6 named four mutations; review extended the battery to seven and recorded all seven red. Re-run at ship stage on this branch's source (both files copied to a scratch directory first and restored from it afterwards — verified byte-identical, never git checkout --), each mutation pinned to the named test it must turn red:

#	Mutation	Named test	Result
M1	delete the bridge's mark_voice_call_active(...) arm	test_the_bridge_arms_a_lease_at_connect_and_releases_it_on_close	1 failed
M2	re-nest the release under ended.portal_session_id	test_the_marker_is_released_even_when_the_session_already_ended	1 failed
M3	restore the /start arm	test_start_workspace_voice_does_not_arm_the_thread_before_a_socket_exists + test_a_start_whose_socket_never_opens_leaves_the_thread_writable	2 failed
M4	drop the owner match from the release (unconditional delete)	test_the_live_call_marker_is_set_cleared_and_fails_open	1 failed
M5	unbound the renewer (while elapsed < max_seconds → while True)	test_the_renewer_holds_the_lease_and_stops_at_the_calls_own_cap	1 failed
M6	let the close-out raise again (narrow the except Exception)	test_the_marker_is_released_when_the_close_path_raises	1 failed
M7	stop cancelling the renew task in the finally	test_the_bridge_arms_a_lease_at_connect_and_releases_it_on_close	1 failed

Neighbourhood fence (the six files that own this surface), run before and again after the rebase onto bf7ab512e:

cd tests && python3 -m pytest unit/test_2694_voice_delta_context.py unit/test_2694_voice_thread_window.py \
  unit/test_ent534_workspace_voice.py unit/test_ent551_voice_background_tasks.py \
  unit/test_2320_portal_failed_turn_visibility.py unit/test_voice_auth.py -p no:cacheprovider -q
→ 217 passed, 21 warnings in 12.39s     (209 on base; +8 new/rewritten)

The registry consumer also runs clean after the rebase: unit/test_2338_journey_catalog.py unit/test_2339_testing_docs_consolidated.py → 75 passed.

Full local verification — verify-local --skip-agent on the pre-rebase tip b9b52799, project trinity-verify-0d1de12b: PASS.

RESULT: PASS — image, boot, and tests all green.
Project: trinity-verify-0d1de12b
  preflight OK (1s) · unit OK (685s) · build+import-smoke OK (21s) · boot+health OK (14s) · integration OK (4s)
  agent-build+import-smoke: skipped (--skip-agent) · agent-exercise: skipped (--skip-agent)

unit 15,555 passed / 31 skipped / 0 failed (685 s); integration 70 passed / 13 skipped / 2 deselected / 0 failed; build + import-smoke and boot + health green. That run predates the rebase onto bf7ab512e, and every file this branch changes in src/backend/, tests/unit/, docs/memory/requirements/ and the feature-flow doc is byte-identical across the rebase (verified by hash); the rebase re-resolved only tests/registry.json and docs/memory/learnings.md (both append-at-the-tail collisions, both sides kept) and auto-merged one row into docs/memory/feature-flows.md.

Real-path proof (verbatim from the verify-stage run, .plan/proof-2700/README.md)

Every step below ran against the isolated verify-local stack (backend http://localhost:60250, redis trinity-verify-0d1de12b-redis) left up by the run above — never against a live fleet. The live stack was byte-identical before and after (/trinity-backend, /agent-trinity-system uptimes continuous), and no container was created, started, restarted or deleted.

Step 2a — /voice/start arms nothing (RUN — this is the headline)

### BEFORE /voice/start
$ docker exec trinity-verify-0d1de12b-redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning --scan --pattern 'portal_voice_active:*'
[end of scan output — empty above means no keys]

### POST /voice/start  (the WebSocket is NEVER opened)
$ curl -s -X POST http://localhost:60250/api/enterprise/client-portal/agents/proof2700/voice/start \
    -H 'Authorization: Bearer <admin JWT>' -H 'Content-Type: application/json' \
    -d '{"portal_session_id":"09af6b646fee4cb1b714934f4c15fec6"}'
{"voice_session_id":"vs_G4zwRlNEO3_EOa-kfPaYrw","websocket_url":"/ws/voice/vs_G4zwRlNEO3_EOa-kfPaYrw","portal_session_id":"09af6b646fee4cb1b714934f4c15fec6","max_duration_seconds":1800}
HTTP 200

### AFTER /voice/start — the marker must NOT be armed
$ ... --scan --pattern 'portal_voice_active:*'
[end of scan output]

$ ... --scan --pattern 'portal_voice_active:*' | wc -l
       0

# Non-vacuity: the start DID do its work — the voice session metadata is in the same Redis.
$ ... --scan --pattern '*vs_*'
voice_session:vs_G4zwRlNEO3_EOa-kfPaYrw

$ ... EXISTS portal_voice_active:09af6b646fee4cb1b714934f4c15fec6
0

Read this as a pair. voice_session:vs_G4zwRlNEO3_EOa-kfPaYrw exists in the same Redis instance the scan just came up empty on — so the start really ran, really reached this Redis, and still left portal_voice_active:* empty. Pre-#2700 this same call wrote the marker with ex=1920 (client_portal/voice.py:242, the old arm site). The reported orphan window is 0 s.

Step 2b — a typed turn on that thread (RUN, with an honest limit)

$ curl -s -X POST http://localhost:60250/api/enterprise/client-portal/agents/proof2700/chat \
    -H 'Authorization: Bearer <admin JWT>' -H 'Content-Type: application/json' \
    -d '{"message":"proof 2700 typed turn","session_id":"09af6b646fee4cb1b714934f4c15fec6"}'
{"detail":"This agent isn't available right now — its owner needs to start it."}
HTTP 502

Not 200, and not 409. The turn is refused by the #2196 availability gate (client_portal/service.py:2498–2503), which runs before the #2694 voice gate (:2544), because proof2700 deliberately has no container — creating one is forbidden on this host (the backend ignores TRINITY_AGENT_NETWORK, so an agent created from the isolated backend would land on the operator's shared network).

So the honest statement is: the thread is not held by a voice-call refusal — the 409 the issue is about does not fire. The literal 200 half of plan §8 step 3 needs a running agent container and is reported NOT RUN for that reason. The gate itself is exercised for real in step 3.

Step 3b — the production gate function, in the production image, against this Redis (RUN)

### A) marker ABSENT (the state /voice/start now leaves behind)
voice_call_active(thread) = False
gate: PASSED (no refusal) -> a typed turn would proceed

### B) marker ARMED by hand (negative control)
$ redis-cli SET portal_voice_active:09af6b646fee4cb1b714934f4c15fec6 vs_fake EX 60   -> OK
voice_call_active(thread) = True
gate: REFUSED status=409 category=voice_call_active retryable=True detail='A voice call is on in this chat — end it, then send.'

### C) marker DELETED again
$ redis-cli DEL portal_voice_active:09af6b646fee4cb1b714934f4c15fec6   -> 1
voice_call_active(thread) = False
gate: PASSED (no refusal) -> a typed turn would proceed

The 409 the issue reports is reachable and live on this exact stack, keyed on exactly the key the scan in step 2a found empty. That is what makes the empty scan meaningful rather than an artefact of a gate that was never wired.

Step 4b — the lease primitives, for real (RUN)

Run inside trinity-verify-0d1de12b-backend against the isolated Redis through the same redis_breaker_util.get_breaker_redis() client the bridge uses:

import OK: LEASE=60s TICK=15.0s SLACK=120s
--- mark_voice_call_active: writes the OWNER value, not '1' ---
  after mark(owner=vs_OWNER)         value='vs_OWNER' ttl=60
--- clear_voice_call_active with the WRONG owner: leaves it ---
  after clear(owner=vs_SOMEONE_ELSE) value='vs_OWNER' ttl=60
--- clear_voice_call_active with the RIGHT owner: deletes it ---
  after clear(owner=vs_OWNER)        value=None ttl=-2
--- mixed-version (plan D5/R9): a legacy '1' marker is releasable by anyone ---
  legacy marker as pre-#2700 /start wrote it value='1' ttl=1920
  after clear(owner=vs_UNRELATED)    value=None ttl=-2
--- renew_voice_call_marker: TTL refreshed every tick, never above the lease ---
  t= 16.0s  value='vs_OWNER' ttl=59     <-- tick 1 (15 s) re-armed the lease
  t= 30.1s  value='vs_OWNER' ttl=60     <-- tick 2 (30 s) re-armed the lease
  max ttl observed = 60  (must be <= 60)
  min ttl observed = 46  (never expired while the renewer lived)
  after the owner's release          value=None ttl=-2

Steps reported NOT RUN, with reasons

Step	Why
Typed turn returning literal 200 after the start	Needs a running agent container; POST /api/agents from the isolated backend would place an agent on the operator's shared network (the backend ignores TRINITY_AGENT_NETWORK). The not 409 half was proved — at the route (502 agent_unavailable, not voice_call_active) and at the gate itself (step 3b).
The real-call half: key holds the vs_… id while a call runs, TTL ≤60 s across two ticks, gone ≤1 s after End	Needs a live Gemini Live voice session on a real agent container over /ws/voice/{id}. Both halves are forbidden here. The equivalent was run instead (4b/4c) and proves owner value, two-tick renewal, the ≤60 s ceiling, the cap bound, and immediate owner-matched deletion.
End-to-end bridge arm→release pairing under a real disconnect	Same reason. Covered by the branch's own unit cases (test_the_bridge_arms_a_lease_at_connect_and_releases_it_on_close, …_released_even_when_the_session_already_ended, …_when_the_close_path_raises, …_when_the_bridge_is_cancelled, …_an_agent_detail_call_never_arms_a_thread_marker, …_the_rest_stop_releases_the_marker_for_a_stop_that_beats_the_socket), which ran green in the unit stage above.

One caveat the reviewer should have, stated rather than glossed: the conditional marker import is inside the WebSocket handler under if portal_session_id:, so application boot never executes it and a clean boot log is necessary but not sufficient. Step 4b closes that — its first lines import the same six names from client_portal.voice inside the built image and print them resolved.

Known limits (accepted, documented in the flow)

• The /start → connect gap is unmarked. Between the start returning and the bridge arming its lease, a typed turn from a second tab or the headless /chat can land. It posts and is answered normally — no new state, string or default, and nothing is lost: spoken rows are written as they happen and the post-call delta still carries everything to the agent's next typed turn. What is lost is that the voice model's opening context does not contain that one message — precisely where "a reply between two spoken rows sits after the cursor and hides the call's first half" becomes possible again, which is why the claim is now scoped to "no reply lands mid-call once the audio bridge is up". Not new in kind: /start already leaves such a window from its get_turn_inflight check to the mark, spanning the prompt resolve, the thread read and the provider round trip; this roughly doubles it. Closing it needs a grace mark at /start — a setter whose only release is a TTL, i.e. a smaller copy of the dead end being removed.
• [L1] A release is not durable while the lease's renewer is still alive. The bridge cancels its renewer as the first statement of its finally, but asyncio.to_thread hands the Redis write to a worker thread cancellation does not reach, so a renewal already inside that SET can land after the release's DEL. Verified with a standalone reproduction, not read off the docs. Bounded and self-healing: the thread keeps refusing typed turns for ≤60 s (one lease) after a call ends, with the read still fail-OPEN, and it needs the renewer to be inside its ~1 ms write at exactly that instant and that write to outlast the whole close-out. The REST /stop sees the same fact at a coarser scale — it releases, a bridge that is still up re-arms within a tick, so that path frees the thread only once the socket is gone (an API-only path either way: the Workspace passes restStop: false).
• [L2] A bridge whose peer died half-open holds the lease until the socket is torn down, bounded by the renewer's own cap + 120 s lifetime plus one 60 s lease — today's bound plus one lease in the worst case, ~0 s in every normal one. The cap path is the one to watch: useVoiceSession.js::_onEnded neither sends {"type":"end"} nor closes, so the socket closes only on the client's 5 s saved-frame timeout.
• [L3] The marker is one key per thread, so two deliberate concurrent calls on one thread share it, and the owner match narrows that rather than closing it: two live renewers flip-flop the value between their ids, so a closing call frees the newer one only if its own id happened to be the last write, and the newer call's next tick re-arms within 15 s. The permanent free an unconditional delete would have caused is gone; a ≤15 s window in a deliberately rare shape is not.

Mixed-version window

Backend-only: no migration, no config, no feature flag, no frontend build. Takes effect on the next backend restart. New /start + old bridge: nothing is armed, nothing strands. Old /start + new bridge: the old code armed the literal "1", and the new release treats "1" as unowned and releasable by anyone, so a marker stranded across the deploy is not immortal (proved against a real Redis in step 4b). Markers nobody closes expire on their existing 1920 s TTL. No backfill.

Docs touched

Six deltas, all decided before implementation:

#	File	Delta
D-1	feature-flows/workspace-voice-conversation.md ("No reply lands mid-call")	The sentence this fix falsifies — "written by start_workspace_voice … cleared by the bridge's finally and by the REST /stop" — is rewritten to the bridge-held owned lease, and the claim is retitled "once the audio bridge is up". The rule is named in one clause.
D-2	same file, Known limits	The four bullets above ([L1]–[L3] plus the connect gap).
D-3	requirements/public-access.md §48.3 FR-3b	"…and a call whose audio socket never opens never holds the thread: the live-call marker is an owned lease armed by the bridge at connect, not by the start (#2700)."
D-4	requirements/runtimes.md §29.10 VOICE-010 (Session lifetime)	"this thread is on a call" is a property of the session's connection lifetime — the bullet that owns it.
D-5	same flow, Files table row for routers/voice.py	names the arm, the renew and the unconditional release.
D-6	docs/memory/learnings.md	The durable class: a marker whose only release path lives downstream of a connection that may never be established is an orphan generator; relocating the setter is a third of the fix; an unconditional release needs an owner token; and "cancelling a task does not cancel work it already handed to a worker thread".

Plus tests/registry.json: the unit/test_2694_voice_delta_context.py description now names the #2700 additions, and unit/test_voice_auth.py — which had no entry — gets one covering the #600 ownership gates and the #2700 marker lifecycle.

Follow-ups (listed here, not filed)

1. The /start → connect gap — carried as a documented limit rather than closed; closing it needs a grace mark whose only release is a TTL.
2. Half-open-socket teardown from the client (pagehide + fetch(..., {keepalive: true})), which would shorten [L2] at its source rather than bounding it.
3. src/backend/routers/voice.py carries no first-line # mcp: header (Invariant feat: SMARTS trading pipeline with Telegram notifications and Miro visualization #13) — pre-existing, untouched here.
4. renew_voice_call_marker swallows CancelledError by design (except asyncio.CancelledError: pass), so task.cancelled() reads False after cancellation — documented in the proof, worth a second look as a house pattern.
5. [L1] generalised: the "cancel does not reach a thread already inside to_thread" shape is shared with agent_call_limiter's refresher, and deserves one pattern rather than two local notes.
6. docs/memory/feature-flows.md Recent Updates length — listed by this run when the table stood at ~140 rows against its own ~20 cap; already addressed on dev by docs(feature-flows): sync flows for #2638/#2670 and bring the index back under its own cap #2753, and after the rebase the table is 21 rows including this one. Recorded so the earlier observation is not read as still-open.
7. The test-runner catalog is self-inconsistent about its own run commands (observed while syncing it; not touched here).
8. _claim_save / _save_transcript in routers/voice.py have had no caller on the Workspace path since refactor(agent-detail): retire the chat-panel voice overlay — the Talk button opens voice in the Workspace #2559 — the elif branch they sit behind is Agent-Detail-only. Dead-code question, deliberately not answered inside a P2 bug fix.
9. One owed .claude/agents/test-runner.md catalog row for TestWorkspaceLiveCallMarker (see below).

Uncommitted by design

.claude is a private submodule checked out at a detached HEAD in this worktree, and its agents/test-runner.md carries an uncommitted catalog row for the new TestWorkspaceLiveCallMarker tests. It is not part of this PR and cannot be: the submodule is a separate repository. git status in the worktree therefore shows m .claude and nothing else; no file of this PR is left uncommitted.

🤖 Generated with Claude Code

https://claude.ai/code/session_0176XEqK8PTCAK5K6yZURQLv

[vybe]

merge-train: batch validated on train/20260914-1332

[vybe]

merge-train note — pushed one mechanical commit (3b8a9364e, a dev merge) to this branch before landing: #2777 merged first and both PRs appended to docs/memory/feature-flows.md and tests/registry.json. Resolution is keep-both (theirs first) for the flow index and a stage-based rebuild for the registry (244 base → 246 entries, no duplicate files). Nothing else changed.