feat(voice): a long task runs in the background and the agent says it once (abilityai/trinity-enterprise#551, #576) - #2735
Merged
Conversation
… once (ent#551, ent#576) In a Workspace voice call `run_task` now answers the model at once with an accepted task id (t1, t2, ...) and runs the turn as the agent in the bound thread in the background. The person keeps the floor; when the task lands its outcome re-enters the live call as a system notice at a natural boundary (model not mid-turn, person quiet 1.2 s, no tool call pending; held at most 20 s), or as a failure with its reason. If the model says nothing within 4 s of dispatch the platform nudges it, so the acknowledgement is structural. At most 3 tasks run per call; at the cap the tool refuses in words. Ending the call cancels nothing: background turns are not in `_pending_tool_tasks`, and a task that lands later writes its rows into the thread, attributed to the call by `voice_call_id` (typed rows, so the #2694 delta logic is untouched). A `task` WebSocket frame drives a persistent "N tasks running" badge distinct from the amber working badge and refetches the canvas when a task lands. The container (VoIP) path keeps its synchronous 30 s bound as the documented fallback. One spoken-etiquette block for the whole tool cycle replaces the filler-only `_TOOL_ETIQUETTE_INSTRUCTION`: announce a wait, not an action (none for a canvas write); say it once; never read the canvas aloud; one narration per sequence; report a failure once with its reason; background tasks. Built from the manifest (ent#535 AC 6) and worded for the session's dispatch shape; the `run_task` description and the panel instructions agree with it. Model: the pin stays `models/gemini-3.1-flash-live-preview`, verified against Google's Live model list as the newest general-purpose Live model; the `gemini-3.5-*-live` ids are transcribe-only and translate-only. Fixes abilityai/trinity-enterprise#551 Fixes abilityai/trinity-enterprise#576 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…odel does not invent a result (ent#551) Found in the first live run on the branch. Every `run_task` failed at once with "A voice call is on in this chat — end it, then send": #2694's live-call guard (56fd9e3) landed on dev after ent#535's run-as-the-agent path (23a2b59) and refused the call's own turns along with the typed turns it was written for, so a Workspace call could not run a single task. The guard now lets a turn carrying `voice_call_id` through — only the voice dispatcher writes it, no request can — and the #2694 delta cursor skips a reply carrying it, so a task landing mid-call cannot erase the call's earlier spoken rows from the next typed turn. Also from the transcript: right after the accepted result the model said "there are sixty four files in there right now" — an invented count, seconds before the real one. The acceptance and the etiquette block now say it holds no result yet and must not guess one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…er transcribe a platform notice (ent#551, ent#576) From the second live run on the branch: - The person heard "let me check… / I've started counting… / it's done" back to back with no room to speak. The acceptance asked for "one short line about what you started" from a model that had already said its filler; it now asks only if nothing was said, and says the model holds no result yet. The completion notice waits for 2.5 s of quiet from BOTH sides (the model's own speech counted for nothing before), so a fast result no longer lands on the heels of the acknowledgement. - The orb badge and the header line say WHAT is running — the task's one-line label, or the count then the labels — instead of "1 task running". - The sidebar's unread badge counted up during the call: every spoken turn and task reply landed as an unread agent row in a thread the person was in. The conversation now advances the read cursor while the call is on. - The model read a platform notice aloud verbatim as the call ended and it was recorded as the agent's line. Every platform text now opens "never read this aloud", and `_record_turn` scrubs an assistant row that echoes one. - The task's ask row sat above the spoken request that caused it. The dispatcher flushes the turn in progress first; the receive loop accumulates on the session's partial-turn fields rather than in locals so that flush is the one writer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…ing adopted, and the mic worklet loads from self (ent#551) Found in the ent#551 QA run: every first call started from a brand-new chat ended itself at exactly 5 s with an empty transcript. `startVoiceCall` creates the thread and adopts it before the call starts; the shell's route replace then changes the `sessionId` prop from null to that id a moment later, and the thread-change watcher — right for browser back and deep links — stopped the call it had just started. The 5 s was the `saved`-frame timeout inside that stop, after which the socket was closed. `threadChangeEndsCall` is the rule now: an agent change or a switch to a DIFFERENT thread ends the call; the call's own thread arriving does not. Also from the console: the AudioWorklet processor was inlined into a `blob:` URL, which `script-src 'self'` (dev and production CSP alike) blocks, so every call logged a CSP violation and silently fell back to the deprecated ScriptProcessor. The processor is a static file under `public/`, loaded from the page's own origin; no CSP loosening. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…task with a SILENT acceptance; shorter lines; no invented canvas (ent#551, ent#576) Third live run. The model still spoke twice around every task — "Let me count the files for you." then, on receiving the accepted result, "I'm checking on that now. Anything else you're curious about while we wait?" — no matter how the acceptance was worded, because a blocking call's result IS a turn and the model answers it. The Live API has the shape for this: on a Workspace call `run_task` is declared NON_BLOCKING and the accepted dispatch is returned with SILENT scheduling, so it enters context without a turn to answer. Verified on gemini-3.1-flash-live-preview: one line before the call, nothing after. Refusals and the container path stay blocking and spoken; both are getattr-guarded for an SDK without the enums (the unit stub, now extended). The etiquette caps an acknowledgement at one short sentence and a report at one or two. And the model said "the canvas shows the breakdown of your files" and "I've put the weather up there" without one canvas call — the canvas row was untouched since 2026-09-07. A canvas session is now told never to claim a canvas it did not draw, and the completion notice states the result is in the chat only and invites drawing it with show_markdown before speaking. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…s; tools are called, never spoken (ent#576)
Fourth live run. The canvas DID update (three show_markdown calls in the audit
log) but the chart showed as raw JSON: the model wrote a Chart.js-shaped fence
(`data.labels` / `datasets`) because the panel prompt said ```chart fences
exist and never gave their shape. It now carries the chart/kpi/table payloads
in the platform prompt's own words, pinned to the renderer's keys and
CHART_TYPES the ent#536 way, with the Chart.js shape named as the wrong one.
Same run: the model said `show_markdown('Canvas update test') updated now`
aloud, and narrated "the task failed because of a syntax error" for a call the
backend never received. The etiquette now says tools are called, never spoken,
and a failure is reported only when it actually came back as a result. The
block's size cap moves 2200 -> 2600 for the three QA-driven rules (measured
2,259), on purpose.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…he mic, and the voice model draws the canvas itself (ent#551, ent#576) The orb's status text and its two buttons were separately absolute-positioned (4 rem and 1.25 rem up, with 48px buttons), so on a short stage LISTENING sat on top of the mute and end-call buttons. They are one bottom-anchored column now. M toggles the mic during a call: a shared rule (`isMuteHotkey` — plain M only, not in a field, not a key an overlay already claimed, no modifier so ⌘M stays the window's) dispatched from the same keydown handler as Escape, after the end-call rule and before the turn-cancel rule. The mute button's title says so. Fifth live run, on the canvas: asked to update a chart, the model dispatched a task for the numbers AND cleared the panel, the agent replied "canvas tools aren't available in this environment", and the model told the person it "can't generate the live chart" — 90 s over a blank canvas, for a chart it could have drawn itself. The panel prompt now says: you draw, never ask a task to update the canvas, a task's "no canvas tools" is about the agent not you, and never clear before you have what replaces it; the completion notice repeats the first point where it lands. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…at is on the canvas, and doing means calling (ent#551, ent#576)
Sixth live run, three findings.
A click on the very agent being talked to, in the left rail, ended the call
with no warning: `openAgentPage` pushed a route with no guard, the conversation
remounted, and its unmount hook stopped the call. Rooms had the same gap, and
the other exits refused silently. Every exit from the stage other than the End
button — the agent row, a thread, a room, New chat, ⌘J — now routes through one
guard that holds the action and asks ("End the call? … Stay on the call / End
call and leave"); confirming ends the call through the conversation's own
`endVoiceCall` and then navigates. End call itself never asks.
The model said "I don't have a current table on the canvas" while the table was
on screen: it had never been told what the person was looking at. A Workspace
call now starts with a text rendering of the canvas (one line per block — kind,
title, slot, a short extract), bounded and best-effort, and a task that changed
the canvas while it ran says so in its notice and what the canvas shows now.
Three requests, three spoken promises ("searching for recent news about OpenAI
now"), zero tool calls. A blocking-vs-non-blocking probe on the live model put
both at 4–5 of 6, so the declaration is not the cause; the prompt is. The first
etiquette rule for a task-capable session is now "Doing means calling": the
model has no files, web or skills of its own, and saying it will do something is
not doing it. The run_task description says the same.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…ms; a promise without a tool call is nudged (ent#551) Seventh live run. Two tasks dispatched together: the second failed at once with "This conversation is already handling a message" — the portal admits ONE turn per thread. Background turns now take a per-call lock: the cap still bounds what is in flight, but each task waits for the thread and then runs. The `task` frame carries a `status` (queued | running) and a `running` event marks the moment a queued task takes its turn; the orb shows one pill per task — the label, and "· queued" while another holds the thread — never one bunched line. "creating that bar chart now", "looking into OpenAI's marketing strategy now", "let me start that search now": three promises, no tool call, until the person asked "are you sure you ran the task?". The prompt rule did not hold, so the platform watches: an assistant turn that reads as a promise to act, with no tool call in the turn or within 3 s after it, gets a notice naming what was said, stating that nothing has started, and telling the model to call the tool now or say plainly that it has not. Rate-limited to one nudge per 15 s; a call that lands just before the promise (the filler-then-call order) counts as kept. The run_task description also says it is never for the canvas — the model kept asking the agent (which has no canvas tools) to draw a chart it could draw itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
… request twice is one task, and Escape cools down after a call (ent#551)
Eighth live run. The promise watch did its job once (a promise with no call
became a real dispatch) and then misfired: with two tasks in flight the model
said "both … are running now", the watch saw a promise-shaped line with no
fresh call, nudged, and the model — told "nothing has started" — said "I
misspoke, they weren't running before" and started a duplicate of t1 with the
identical prompt. Three changes: no nudge while any task is in flight (the pill
already tells the person the truth); status lines ("already", "still working",
"are running") are not promises; the nudge states only what the platform knows
(no call arrived), conditions the action on "not already running", and forbids
commenting on the notice. And a `run_task` whose prompt equals a running task's
is refused as a duplicate — one task, its result once.
Also in both calls, a running task was cancelled ~3 s after the call ended,
with "You stopped this message before the agent finished." The thread had just
reattached to that task's turn, where Escape means "stop the turn", and the key
that ended the call landing again cancels it. Escape is not a cancel for 5 s
after a call ends (the Stop button stays live — a click is deliberate). Because
access logging is off and nothing named the requester, `terminate_execution`
now logs who asked on every arm and the portal route logs the client's email.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…tasks were the operator's own Stop (ent#551) The two tasks that failed 3 s after End in the eighth QA run were cancelled on purpose by the person testing, not by a stray second Escape. A guard with no defect behind it is a rule nobody can explain later, so it goes. The terminate-caller logging stays: it is what would have answered this in one line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
…ith its voice_call_id argument (ent#551) `_persist_user_turn` gained `voice_call_id=` for the call's own background turns; the ent#473 source guard located the call by its old exact text and raised instead of asserting order. Same three-point ordering, new spelling. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p
vybe
approved these changes
Sep 13, 2026
vybe
left a comment
vybe
left a comment
Contributor
There was a problem hiding this comment.
merge-train: batch validated on train/20260913-0705 (#2745) — lane B, /validate-pr + /review clean; all four headline behaviours driven by executing tests (275 py + 91 JS); execution-area invariant intact (no new terminal writer). Follow-ups noted, none blocking: docs lag the last three QA commits (/sync-feature-flows), two lines of revert residue (portalVoiceMode.js:254 orphan header, gemini_voice.py:1458 stale _run_task_in_chat name), and a low-impact TOCTOU across the _flush_partial_turn await in _dispatch_task_in_chat (gemini_voice.py:1514-1550: two run_tasks in one function-call batch can both pass the cap/duplicate check — execution stays serialised by the turn lock; fix is insert-before-flush or guard check+insert with the lock).
vybe
pushed a commit
that referenced
this pull request
Sep 14, 2026
… connect and always releases it (#2700) (#2775) * docs(requirements): the live-call marker is a lease the audio bridge holds (#2700) Trinity Rule #1 — the requirements change lands before the code that falsifies them. `POST …/voice/start` arms the thread's live-call marker before the audio WebSocket that is the only thing able to clear it exists, so a start whose socket never opens strands a ~32-minute `409 voice_call_active` on every typed turn in that thread, naming a call the person cannot end. - public-access.md §48.3 FR-3b: "no reply lands mid-call" becomes "…once the audio bridge is up", and a call whose socket never opens never holds the thread. - runtimes.md §29.10 VOICE-010 (Session lifetime): "this thread is on a call" is a property of the session's connection lifetime — an owned lease armed by the bridge at connect, renewed while it lives, never past the cap, released on every exit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176XEqK8PTCAK5K6yZURQLv * fix(workspace): the audio bridge owns the live-call lease, arms it at connect and always releases it (#2700) `POST …/voice/start` armed the thread's live-call marker (TTL = the call cap + slack = 1920 s) before the audio WebSocket that is the only thing able to clear it existed. Both clear paths sit downstream of that socket, and so does the cap watchdog that would end the session — so a start whose socket never opened stranded a ~32-minute `409 voice_call_active` on every typed turn in that thread, from any tab and from the headless `/chat`, naming a call the person cannot end. Three moves; the first is the one the issue asked for, the other two are why it holds. 1. The party that opens the effect closes it. The bridge is now the only writer: it arms the marker as the first statement inside the same `try` whose `finally` releases it, and that release is unconditional, LAST, and keyed on the `portal_session_id` local captured before the `try` — never on `ended.portal_session_id`, where an `ended is None` return (the exact case the REST `/stop` clear was added for) would re-strand it. The reported orphan window goes to zero. 2. The marker is a lease, not a latch. Its TTL is 60 s, renewed every 15 s by a bridge-owned task, bounded at the call's own `max_duration + 120` so it can never outlive one. Without this, move 1 only relocates the harm: a SIGKILL, an OOM or a routine backend deploy mid-call would still strand the thread for 32 minutes with this issue's own symptom. The 4x TTL/tick ratio and its reason are `agent_call_limiter`'s, copied not invented; the renew write goes off-loop via `asyncio.to_thread` for the same reason. 3. The lease has an owner. The value stored is the call's `voice_session_id` and a release deletes only on match (`clear_turn_inflight`'s precedent), so a closing bridge cannot free the thread of a newer call — a reload, a second tab. `owner` is keyword-only and required on both primitives. The legacy `"1"` value is treated as unowned and released, so a marker stranded across the deploy is not immortal. Also, in blast radius: the close-out is wrapped in `except Exception` + `logger.exception` (not `BaseException` — cancellation still propagates), so a raising `end_session`/persist no longer skips the gemini cancel, the `saved` frame and the close. The renew task is cancelled as the first statement of the `finally`, before anything that can await, so no renewal can re-arm after the release. The read stays fail-OPEN and `_refuse_turn_during_voice_call` is byte- unchanged (#2735's `voice_call_id` exemption intact). No new user-visible surface, state, string or default; no frontend file touched. The three `test_2694_voice_delta_context.py` tests that pin the old set site are rewritten in the next commit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176XEqK8PTCAK5K6yZURQLv * test(workspace): pin the live-call lease at the bridge, and the orphan shape end to end (#2700) The issue's third AC — "the existing `test_2694_voice_delta_context.py` marker tests still pass" — cannot hold literally: three of them pin what this fix moves. It is read instead as "the marker's behavioural guarantees stay pinned — armed while a call is live, released when it ends, held only by its owner, fail-OPEN on Redis, the TTL as the backstop — and every test that changes gets stronger". `test_2694_voice_delta_context.py` - the primitives test gains the owner token: `owner` is keyword-only and required on both, a release from a DIFFERENT call does not free a live thread, the legacy `"1"` value is releasable, bytes decode, and the real lease/tick/slack constants plus the >=4x TTL:tick ratio are pinned here (the bridge tests import them from a fake module and cannot). - NEW: the renewer ticks at the lease TTL and STOPS at the call's own cap — a fake `mark` that raises after 10 calls turns a missing bound into a fast red instead of a hang — and a cancelled renewer stops quietly. - `…marks_the_thread_live_for_the_cap_plus_slack` becomes `test_start_workspace_voice_does_not_arm_the_thread_before_a_socket_exists`. The recorder takes `owner=None` by default so a RESTORED two-arg `/start` arm is recorded rather than raising: `marks == []` is what must bite. - NEW, the AC-2 test: start a call, never open a socket, then type into the same thread. The REAL `mark_voice_call_active` runs against a fake Redis on purpose — a recorder stub would swallow the write and pass even with the `/start` arm restored, so `fake.store == {}` is the assertion that bites. The turn genuinely dispatches (`recorder.calls` non-empty), and a positive control arms the lease the way the bridge now does and gets the 409 back. - the 600-char proximity grep around `persist_voice_call_end(` is replaced by `test_the_bridge_resolves_the_real_marker_helpers`: it could never see the `if ended:` nesting that IS this bug. What survives is what a source read can honestly assert — that the real modules resolve each other (every bridge test runs against a fake), that the marker import is hoisted out of the `finally`, and that the release is keyed on the pre-`try` local, not on `ended`. `test_voice_auth.py` — new `TestWorkspaceLiveCallMarker`, here because the behavioural bridge harness (the importlib load of `routers/voice.py`, the stubbed voice service, `_FakeWebSocket`, the #762 restore net) exists in this file and nowhere else. Armed at connect and released on close; released when `end_session` returns None (Trap C); released when the close path raises, with the gemini cancel, the `saved` frame and the socket close all still reached; released under task cancellation; an Agent Detail call never arms a thread marker (labelled a guard — it passes against the old code too); and the REST `/stop` release is owner-matched (an API-only guard, D9). Three harness fixes the tests need: `_FakeVoiceSession` gains `portal_session_id`/`max_duration`/`end_reason`/`end_message` (the close-out reads the last two without getattr defaults), the fake `client_portal.voice` exposes `persist_voice_turn` (imported unconditionally for a portal-bound session), and `_YieldingWebSocket` awaits once in `receive_text` — the stock fake pops its queue without a single `await`, so a bridge driven by it never yields and its `create_task`ed children never start. Red-on-base proof (both source files restored from a scratch copy, never `git checkout --`): 10 of the 11 new/rewritten tests fail against `2c5cfe0e`; the one that passes is exactly the labelled Agent Detail guard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176XEqK8PTCAK5K6yZURQLv * docs(workspace): the live-call marker is a bridge-held lease, and the connect gap is named (#2700) The feature flow said verbatim that the marker is "written by `start_workspace_voice` once the provider session exists (TTL = the cap + slack), cleared by the bridge's `finally` and by the REST `/stop`" — the sentence this fix falsifies. - `workspace-voice-conversation.md`: the claim is retitled "no reply lands mid-call **once the audio bridge is up**", and the mechanism is rewritten — an owned lease armed by the bridge at connect, renewed while it lives, never past the call's cap, released unconditionally and owner-matched on every exit. The rule is named in one clause (a marker whose only closer lives downstream of a connection that may never exist is an orphan generator), and all three load-bearing properties are stated with the failure each prevents. The REST `/stop` is described as what it is: idempotent, owner-matched, and API-only — the Workspace passes `restStop: false` and never calls it. - Known limits gains three bullets: the `/start`→connect gap in #2694's own words (the turn posts and is answered normally; what is lost is the voice model's opening context, which is exactly where "a reply between two spoken rows hides the call's first half" becomes possible again); a half-open peer holding the lease until the renewer's cap+slack lifetime ends, including the cap path's 5 s `saved`-timeout close; and the key being one per thread. - The Files row for `routers/voice.py` names the arm, the renew and the unconditional release. - `feature-flows.md` gains the dated index row. - `learnings.md` gains the durable class entry: arm the effect in the `try` whose `finally` releases it; relocating the setter is only a third of the fix (a marker held by a live process is a lease, and the renewer needs its own bound); an unconditional release needs an owner token; and a test that pins the old write site cannot "still pass" — rewrite it and prove it bites. `architecture/workspace.md` is deliberately NOT edited: its sentence ("a typed turn is refused (409) while a call is on") states no set site and stays true, and the detail's home is the feature flow (one home per feature). Also verified unchanged: the ent#551 `voice_call_id` exemption paragraph. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176XEqK8PTCAK5K6yZURQLv * docs(workspace): the lease release is not durable while its renewer lives, and say so (#2700) Three claims the #2700 branch makes are stronger than the implementation provides. All three are the same fact — the renewer's Redis write is off-loop — and each one is a comment or a Known-limits line a future reader would build on. 1. The bridge's `finally` claimed that cancelling the renewer first means "a renewal already in flight lands before the release, and the delete wins". It does not: `asyncio.to_thread` raises `CancelledError` in the awaiting coroutine immediately and lets the worker thread run to completion, so a renewal inside its `SET` can land after the release's `DEL`. Verified with a standalone `asyncio.run` reproduction, not by reading the docs. The consequence is bounded and benign — one lease (<= 60 s) of extra 409s after a call ends, self-healing, read still fail-OPEN — and it needs the renewer to be inside a ~1 ms write at that instant AND that write to outlast the whole close-out. 2. The REST `/stop` release is not durable while the bridge is up: the renewer re-arms within a tick, so that path frees the thread only once the socket is gone. An API-only path either way (`restStop: false`). 3. "a closing call cannot free a newer one" overstated the owner match for two concurrent calls on one thread: two live renewers flip-flop the value, so a closing call frees the newer one when its own id was the last write, and the newer call's next tick re-arms within 15 s. The permanent free an unconditional delete would have caused is gone; a <= 15 s window in a deliberately rare shape is not. No behaviour change: comments, one Known-limits bullet, and a fifth clause on the issue's learnings entry, since "cancelling a task does not cancel work it already handed to a worker thread" is the durable class here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0176XEqK8PTCAK5K6yZURQLv --------- Co-authored-by: trinity-ability <309458136+trinity-ability@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: sim <sim@example.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_taskis declaredNON_BLOCKING; the platform answers it at once with aSILENTacceptance ("Started t1 … you do not have the result yet") and runs the turn as the agent in the thread. Tasks run one at a time per call (per-call lock;queued→running) and show as separate pills, each carrying a one-line description of the ask. The result (or failure, with its reason) re-enters the live call as a platform notice once both sides have been quiet for 2.5 s (held at most 20 s). Cap of 3 in flight with a spoken refusal; the same request twice while it runs is refused as a duplicate, never a second copy. Ending the call cancels nothing — a task that lands later writes its rows into the thread, attributed to the call. The container (VoIP) path keeps its synchronous 30 s bound.VOICE_MODELstaysmodels/gemini-3.1-flash-live-preview— verified against Google's Live model list and docs as the newest general-purpose Live model.gemini-2.5-flash-native-audio-latestwas probed live and called tools less reliably (3/6 vs 5/6).Fixed during live QA (ten rounds with a Workspace agent)
run_task;portal_chat(voice_call_id=…)bypasses it and the delta cursor skips voice-attributed rows.NON_BLOCKING+SILENTacceptance (probed live: one line, then quiet).[System notice …]aloud and it was persisted as an assistant row → marker + scrub in_record_turn.threadChangeEndsCallrule. CSP blocked the blob mic worklet → static/mic-capture.worklet.js.CHART_TYPES.[Terminate] requested … by …,portal cancel: …). An Escape cooldown added on a wrong hypothesis was reverted once the operator confirmed the cancels were their own.Changes
src/backend/services/gemini_voice.py—BackgroundTask,_dispatch_task_in_chat,_portal_turn(per-call lock),_ack_watch,_watch_promise,_task_landed,_deliver_task_notice,_scrub_platform_notice,canvas_context_section,run_task_tool(background=…),spoken_etiquette_instruction(manifest, background=…), fence-shape panel instructions.src/backend/client_portal/service.py,client_portal/db.py,client_portal/voice.py,client_portal/router.py,services/chat_execution_service.py,routers/voice.py(taskWS frame).useVoiceSession.js,portalVoiceMode.js(applyTaskFrame,taskItemLabel,threadChangeEndsCall,isMuteHotkey,leaveCallCopy),VoiceOverlay.vue(per-task pills, bottom stack),PortalConversation.vue,Portal.vue(leave-callConfirmDialog),utils/audio.js+public/mic-capture.worklet.js.runtimes.md§29.7 + §29.11 (VOICE-011),voice-chat.md,workspace-voice-conversation.md(incl. Known limits),voip-telephony.md.Test Plan
pytestover the voice/canvas/portal suites (test_ent551_voice_background_tasks.py,test_ent576_spoken_etiquette.py,test_ent535_voice_tool_set.py,test_2694_voice_thread_window.py,test_voice_tools.py, …) — 272 passed on the last roundnpx vitest run— 2747 passed, incl. the raw-colour and loading-gate ratchets;npm run check:tokensOKNotes for review
Fixes abilityai/trinity-enterprise#551
Fixes abilityai/trinity-enterprise#576
🤖 Generated with Claude Code
https://claude.ai/code/session_01ApbFsEqLtAWVdedXAUqV9p