diff --git a/AGENTS.md b/AGENTS.md index 581bb0ea..8393c49e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,11 +42,32 @@ then update both the code and the relevant contract in the same cut. detail lens or explicit fallback. - Never revert unfamiliar dirty changes. Isolate responsibilities, verify each coherent cut, and stage only the files that belong to its checkpoint. -- Local implementation turns end in a scoped commit. Never push, merge, or - publish a release unless the operator explicitly asks for that action. +- Local implementation turns end in a scoped commit. Push that work to the + active branch. Do not merge to trunk, tag, or publish a GitHub Release + unless the operator asked. The daily notarized DMG below is the one + release artifact that does not wait for a second ask. - Generated UniFFI Swift bindings must match the Rust bridge. Run `make app-bindings` after bridge API changes. +## Install cadence (Maciej + Monika) + +Operator agreement 2026-08-19. + +- After each coherent cut that changes the app, run `make install-if-idle` + (or `make install-app` after a live-recording check). That is the daily + laptop binary. Do not wait to be asked. +- **Refuse the install** when a Codescribe take is in flight. Authority is + the Transcript Bus: last session has `session_started` and no later + `transcript_sealed`. Never tear down `/Applications/Codescribe.app` mid-take. +- A **notarized slim DMG for Monika is once per calendar day**, not every + commit and not "after a batch of key fixes". When the bus is idle, cut + `make release-standard` (sign + notarize + `verify-dmg`). One artifact + per day is enough; do not recut for later same-day commits unless the + operator asks. Say the path and staple result in the turn. That is a + local release artifact; still not a silent merge to trunk, tag, or + GitHub Release. +- Ad-hoc `/Applications` from `install-app` is never "the Monika DMG". + ## Verification - `make check` — static formatting, Clippy, Semgrep, env registry, gate ledger. diff --git a/CHANGELOG.md b/CHANGELOG.md index 31a2ac22..c443bc8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,94 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- **`make install-app` accepts keys from Get license.** A keyed local + install verifies CSK1 with the same public key the site signs. The + forgeable development verifier is no longer baked into that path. +- **Refused paste does not steal the user's clipboard.** Synthetic Cmd+V + still snapshots and restores after a confirmed paste into a foreign app. + `CopyTargetUnavailable` / target mismatch / Accessibility deny no longer + dump the transcript onto `NSPasteboard`. The text parks in the Paste Here + slot (⌘⌥V when that chord is bound); the overlay keeps it; the user's + previous clipboard stays put. Explicit overlay Copy is unchanged. +- **Paste status lives in the overlay footer.** Insert that cannot reach + the ambulance no longer throws a capsule over the action row. A quiet + chip sits next to `local apple` (`⌘⌥V` / `copied` / `no ax`). +- **Auto-paste may land in the Agent window and in Alacritty.** The overlay + canvas is still never a Cmd+V sink (Swift caret probe). The whole + Codescribe app is not "self": Agent is a legal ambulance, and a floating + overlay that leaves `NSWorkspace` naming Codescribe no longer vetoes a + confirmed Alacritty/Zellij activate. Failures still restore the user + clipboard and park our buffer. +- **CS Voice Lab starts with the take.** A keyed `install-app` bake + spawns `~/.codescribe/voice-lab` when recording prepares, and the + existing Voice Lab buttons ensure `:8765` before opening the + console. Production stays inert. The child stops when the take + ends (and on quit). Loopback STT `:8444` / `:8446` stay up. + `docs/loopback.html` and `~/.codescribe/voice-lab/loopback.html` + point at those URLs. +- **Agent chat shows live capture.** Assistive/Agent hides the overlay, + so the composer now renders the growing transcript above the field. +- **Overlay default stays pinned top-right.** Free motion is only the + explicit toggle. A drag without it is ephemeral. Edge-resize always + persists, independent of the pin. +- **Format/Retranscribe keep the overlay hittable.** One-shot Format + and file retranscribe no longer hide the panel (including when the + tray ticks Assistive), steal focus, or pin MainActor for the LLM / + Whisper round-trip. The status pill is a static `formatting` / + `retranscribing` phase, not a waveform grind. +- **Mid-hold Shift attaches `{selection_N}`.** Shift or Command during an + already-started Fn hold captures the current selection into the context + bucket and overlay marker. It does not open Agent, hide the overlay, or + stop the take. Fn+Shift from idle stays dictation, not Assistive. +- **Fn hold-down attaches a live selection as `{selection_1}`.** A + selection already present when Fn goes down is captured immediately. + Later Shift pulses still add `{selection_2..n}`. Destination stays + dictation. +- **Layer 1 `cloud_session` stays up on Voice Lab `:8446`.** The live + socket opens with `hello` (`stt-ws-v1`); treating that as protocol + dropped the lane at take start (`disconnect`, zero frames). Handshake + and VAD control are ignored, and the start frame is Voice Lab `set`. +- **Compound Apple chops take the joined Layer 1 rewrite.** Five short + fragments share one Whisper window and apply the aligned sentence + swap. Fusion no longer rewrites only the last piece or skips the + joined sentence at the 50% change cap. +- **File retranscribe names the programming domain.** Overlay/Dictionary + `cloud:` on `last_session.wav` (remapped loopback `:8444`) sends + `vocabulary=programming` — test-locked on the multipart body. Official + OpenAI still omits the field. Overlay Format is not HQ authority; + compare Whisper file vs raw, never vs Format. + +### Changed + +- **Supervisor findings own transcript-quality categories.** Engine catalog + `codescribe-supervisor-findings/v1` (`core/quality/supervisor.rs`) names + every issue class the tree already had — contract forbiddens, clock-lie, + speech gaps, Teacher attention, confidence flags, delivery gates, Whisper + residue, and the Voice Lab lies (HQ-as-document, omitted + `vocabulary=programming`, live overlay paired with last_session.wav). + Voice Lab three-judge emits those findings. WER stays a footnote of + proposal agreement, not accuracy. + + +- **Layer 1 applies aligned same-utterance wording.** When live Apple and + the Whisper window share most words, Layer 1 now substitutes those + spans instead of discarding the repair at the 50% change cap. Unrelated + dumps and pause-tail inserts still skip. +- **Layer 1 Whisper windows join about five Apple segments.** Short + fragments wait for a sentence-sized window (or a pause) before the + background swap, instead of each breath becoming its own failed + repair. +- **Cloud STT names the programming domain.** Loopback and Libraxis file + and live requests send `vocabulary=programming`. Official OpenAI file + audio omits the field. The client does not classify audio to pick a + dictionary. Overlay Format is not that compare — HQ is Whisper file vs + raw. +- **Dev-power corner mark.** A keyed local install paints a small + “You use dev power mode” caption in the bottom-right of overlay, Agent + chat, and Settings. Production DMGs stay unmarked. + ## [0.14.1] - 2026-08-18 > Patch: everyday-stable 0.14.x. Same slim public SKU as 0.14.0, plus the two @@ -16,8 +104,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed - **STT Test is a file probe.** Settings Test no longer POSTs to a live - Voice Lab WebSocket (`STT_ENDPOINT`). Known live sockets map to - `/v1/audio/transcriptions`; loopback `:8446` → `:8444`. + socket. Known live sockets map to `/v1/audio/transcriptions`. - **ChatGPT sign-in no longer requires Responses write.** OAuth persists identity after exchange. `api.responses.write` stays a lane Test, so Codex public tokens can sign in. @@ -32,7 +119,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`make release-stable`** is the everyday cut: slim sign + notarize + `verify-dmg`, then install that stapled Developer ID `.app` to `/Applications` without re-signing. `make install-app` remains the - local-release / Lab path. + local-release path. - **`make release-full` is fail-closed.** Whisper embed no longer falls back to a slim dylib when the HF snapshot is weights-only. It uses the composed `~/.codescribe/models` tree from `make download-model`. @@ -45,9 +132,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **Developer Lab on keyed `make install-app`.** `CSDeveloperSurface` bakes - only when Sparkle + production license public keys resolve on the machine. - Public `git clone && make` stays Lab-off. Production DMG refuses the bit. +- **Developer Lab on a keyed local install.** A public `git clone && make` + stays Lab-off. Production DMG refuses the bit. - **Lab mode overlay-off.** Developer veto hides the daily HUD without flipping the tray "Transcription Overlay" toggle. Leftover UserDefaults cannot hide overlay on a production bundle. diff --git a/Cargo.toml b/Cargo.toml index 4c6f6991..c850c3fc 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -161,6 +161,11 @@ strip = true # artifact; scripts/build-dmg.sh and CI use the literal `release` profile. [profile.local-release] inherits = "release" +# rustc `strip = true` leaves LC_SYMTAB.stroff unaligned. Xcode 27's ld +# (ld-27036) then dies: "mis-aligned LINKEDIT string pool". Daily +# `make install-app` uses this profile; keep symbols so the dylib links. +# Dist `release` still strips — `make release-standard` is a separate cut. +strip = false # Diagnostic / crash-reproduction profile. # Inherits the optimized release build but keeps full debug info (no strip) so diff --git a/Makefile b/Makefile index 86ce34e8..7d8c610b 100644 --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ # The user-facing app is built by `make app` (xcodebuild); the Rust side no # longer ships a standalone `codescribe` tray binary. -.PHONY: all build release release-codescribe release-codescribe-embedded release-qube app app-bindings install install-no-embed config install-app \ +.PHONY: all build release release-codescribe release-codescribe-embedded release-qube app app-bindings install install-no-embed config install-app install-if-idle install-voice-lab \ start stop restart status logs logs-follow \ bump bump-patch bump-minor bump-major version \ lint format test test-quick test-e2e test-e2e-real test-sse test-sse-release test-responses-live test-sse-heavy test-formatting test-all \ @@ -12,7 +12,7 @@ demo demo-raw demo-assistive check verify semgrep fix clean help corpus-census test-corpus-parity \ dist-preflight dist-preflight-signed verify-canaries smoke-canaries \ dmg dmg-signed release-standard release-full release-dmgs release-stable install-app-release notarize verify-dmg download-model download-e5 download-embedder ensure-models \ - hooks + hooks site-dev SHELL := /bin/bash VERSION_FILE := Cargo.toml @@ -49,7 +49,7 @@ CODESCRIBE_DIST_CODESIGN_IDENTITY ?= $(if $(strip $(CODESCRIBE_DEVELOPER_ID_IDEN # `env -u CODESCRIBE_LICENSE_PUBLIC_KEY_HEX`, and that must keep working — a # plain `?=` fallback here would silently re-arm the production key in a local # build that deliberately wants the development verifier. -CODESCRIBE_LICENSE_PUBLIC_KEY_FILE ?= $(HOME)/.vibecrafted/secrets/codescribe/license-public.hex +CODESCRIBE_LICENSE_PUBLIC_KEY_FILE ?= $(firstword $(wildcard $(HOME)/.codescribe/config/dev/keys/license-public.hex $(HOME)/.vibecrafted/secrets/codescribe/license-public.hex)) CODESCRIBE_DIST_LICENSE_KEY = $(if $(CODESCRIBE_LICENSE_PUBLIC_KEY_HEX),$(CODESCRIBE_LICENSE_PUBLIC_KEY_HEX),$(shell cat $(CODESCRIBE_LICENSE_PUBLIC_KEY_FILE) 2>/dev/null | tr -d '[:space:]')) # Sparkle's update-verification public key has the same missing-local-source # problem: release.yml supplies SPARKLE_ED_PUBLIC_KEY as a repository variable, @@ -58,7 +58,7 @@ CODESCRIBE_DIST_LICENSE_KEY = $(if $(CODESCRIBE_LICENSE_PUBLIC_KEY_HEX),$(CODESC # would reject every update"). A local `make release-standard` had no way to # supply it, so a locally cut release failed the gate at the very last check — # after codesigning, notarisation and stapling had already been paid for. -CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE ?= $(HOME)/.vibecrafted/secrets/codescribe/sparkle-public.b64 +CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE ?= $(firstword $(wildcard $(HOME)/.codescribe/config/dev/keys/sparkle-public.b64 $(HOME)/.vibecrafted/secrets/codescribe/sparkle-public.b64)) CODESCRIBE_DIST_SPARKLE_KEY = $(if $(SPARKLE_ED_PUBLIC_KEY),$(SPARKLE_ED_PUBLIC_KEY),$(shell cat $(CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE) 2>/dev/null | tr -d '[:space:]')) CODESCRIBE_APP_NAME ?= Codescribe CODESCRIBE_DISPLAY_NAME ?= Codescribe @@ -179,13 +179,36 @@ config: @$(EDITOR) ~/.codescribe/.env -install-app: +# Colon form `make site:dev` is not a target — Make treats `:` as a rule +# separator. The website lives in site/; this is `npm run dev` from there. +site-dev: + cd site && npm run dev + +install-voice-lab: + @./scripts/install-voice-lab.sh + +install-if-idle: + @./scripts/install-if-idle.sh + +install-app: install-voice-lab @echo "Building $(CODESCRIBE_APP_NAME).app (SwiftUI, optimized local profile) via scripts/build-app.sh ..." - @echo "Local install uses the development license verifier; CODESCRIBE_LICENSE_PUBLIC_KEY_HEX is reserved for distribution builds." @BIT=$$(./scripts/developer-surface-gate.sh); \ - echo "Developer surface: $$BIT (1 needs legit Sparkle + license public keys on this machine)."; \ - env -u CODESCRIBE_LICENSE_PUBLIC_KEY_HEX \ - CODESCRIBE_DEVELOPER_SURFACE=$$BIT \ + if [ "$$BIT" != "1" ]; then \ + echo "Developer surface stayed off after the Voice Lab pack — Sparkle/Ed public keys did not resolve."; \ + echo " need: ~/.codescribe/config/dev/keys/{sparkle-public.b64,license-public.hex}"; \ + echo " or: ~/.vibecrafted/secrets/codescribe/{sparkle-public.b64,license-public.hex}"; \ + exit 1; \ + fi; \ + LICENSE=$$(tr -d '[:space:]' < "$(CODESCRIBE_LICENSE_PUBLIC_KEY_FILE)" 2>/dev/null || true); \ + if [ "$${#LICENSE}" -ne 64 ]; then \ + echo "install-app: license-public.hex missing or not 64 hex at $(CODESCRIBE_LICENSE_PUBLIC_KEY_FILE)"; \ + exit 1; \ + fi; \ + echo "Developer surface: 1. License verifier from $(CODESCRIBE_LICENSE_PUBLIC_KEY_FILE) (Get license / CSK1)."; \ + SPARKLE=$$(tr -d '[:space:]' < "$(CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE)" 2>/dev/null || true); \ + CODESCRIBE_LICENSE_PUBLIC_KEY_HEX="$$LICENSE" \ + CODESCRIBE_DEVELOPER_SURFACE=1 \ + SPARKLE_ED_PUBLIC_KEY="$$SPARKLE" \ $(MAKE) --no-print-directory app PROFILE=local-release @APP_SRC="macos/build/Build/Products/Release/Codescribe.app"; \ if [ ! -d "$$APP_SRC" ]; then \ @@ -1187,7 +1210,9 @@ help: @printf '%s\n' ' make install-no-embed DEV/RECOVERY: no optional embeds (runtime paths only)' @printf '%s\n' ' make release-codescribe-embedded Fat dylib with Whisper baked in (not daily)' @printf ' $(HELP_C_GREEN)%-18s$(HELP_C_RESET) %s\n' 'config' 'Edit ~/.codescribe/.env' - @printf ' $(HELP_C_GREEN)%-18s$(HELP_C_RESET) %s\n' 'install-app' 'Local-release install to /Applications (may re-sign; Lab if keys resolve)' + @printf ' $(HELP_C_GREEN)%-18s$(HELP_C_RESET) %s\n' 'install-app' 'Local-release install to /Applications' + @printf ' $(HELP_C_GREEN)%-18s$(HELP_C_RESET) %s\n' 'install-voice-lab' 'Install the private developer console when this machine can reach it' + @printf ' $(HELP_C_GREEN)%-18s$(HELP_C_RESET) %s\n' 'site-dev' 'Astro site at site/ (http://localhost:4321) — not make site:dev' @printf ' $(HELP_C_GREEN)%-18s$(HELP_C_RESET) %s\n' 'release-stable' 'Everyday: notarize slim DMG + install that stapled .app' @printf '\n' @printf ' $(HELP_C_YELLOW)%s$(HELP_C_RESET)\n' 'RELEASE & DISTRIBUTION' diff --git a/README.md b/README.md index e7dda202..b02c90ee 100644 --- a/README.md +++ b/README.md @@ -405,7 +405,7 @@ make format # cargo fmt ``` make app # Debug SwiftUI app build make app PROFILE=local-release # Optimized local SwiftUI app build -make install-app # Local-release install (may re-sign; Lab only if keys resolve) +make install-app # Local-release install to /Applications make release-stable # Everyday: notarize slim DMG + install that stapled .app make release-qube # Build qube CLI tools make install # Install qube CLI tools + repo-local hooks diff --git a/app/controller/delivery_route.rs b/app/controller/delivery_route.rs index 42416c08..271165c0 100644 --- a/app/controller/delivery_route.rs +++ b/app/controller/delivery_route.rs @@ -10,9 +10,10 @@ //! - `resolve_delivery_route` is the only function allowed to pick a //! [`DeliveryRoute`]. Auto-paste, overlay Insert, and To Agent consult it; //! they do not invent a second destination. -//! - Codescribe is never a legal Cmd+V target. A latched self-app (Agent -//! composer / overlay / settings) routes to the Orient canvas or the Agent -//! composer as a first-class message — never as a tagged paste into ourselves. +//! - The overlay canvas is never a legal Cmd+V target (caret in our panel). +//! The Agent window, Alacritty/Zellij, Notes, and every other caret are +//! legal ambulances. Assistive still delivers as a first-class Agent +//! message — that is a different intent, not a paste ban. /// Where a finished transcript is allowed to land. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -90,7 +91,9 @@ pub struct DeliveryFacts { pub overlay_enabled: bool, pub live_stream_session: bool, pub commit_required: bool, - /// Latched pre-overlay target is Codescribe itself (Agent / overlay / settings). + /// True only when the overlay canvas holds the caret. The Codescribe + /// **app** (Agent window, Settings) is not this flag — those are legal + /// Cmd+V sinks. Swift `defer_text_from_overlay` is the constructor. pub latched_target_is_self: bool, } @@ -119,16 +122,19 @@ pub fn delivery_intent_from_session( } } -/// Codescribe (any chrome) is never a legal synthetic-paste target. +/// Localized name of **this process**. Used to skip `NSRunningApplication` +/// activate (we are already running). Not a paste veto — the Agent window +/// is a legal Cmd+V sink. Overlay-canvas veto is the Swift caret probe. pub fn target_is_self_app(name: &str) -> bool { name.trim().eq_ignore_ascii_case("codescribe") } /// Facts an overlay Insert / defer click may feed the throne. /// -/// Focus-at-click is not an input. `latched_target_is_self` is true when the -/// recorded target is Codescribe, or when Swift already knows the caret is -/// still inside our chrome (`defer_text_from_overlay`). +/// Focus-at-click is not an input. `latched_target_is_self` is true only when +/// Swift already knows the overlay canvas holds the caret +/// (`defer_text_from_overlay`). A latched Codescribe **app** name is the +/// Agent window, not this flag. pub fn overlay_insert_facts(has_text: bool, latched_target_is_self: bool) -> DeliveryFacts { DeliveryFacts { has_text, @@ -170,8 +176,9 @@ pub fn resolve_delivery_route(intent: DeliveryIntent, facts: DeliveryFacts) -> D } /// Explicit overlay click. Orient vetoes (live stream, quality commit) do not -/// apply — the user asked to insert *now*. Codescribe as the latched target -/// still refuses Cmd+V into ourselves. +/// apply — the user asked to insert *now*. Overlay **caret** still refuses +/// Cmd+V into the canvas (`latched_target_is_self` from Swift). A latched +/// Codescribe **app** name is the Agent window, not the canvas. fn overlay_insert_route(facts: DeliveryFacts) -> DeliveryDecision { if facts.latched_target_is_self { return DeliveryDecision { @@ -299,7 +306,20 @@ mod tests { } #[test] - fn hold_fn_with_agent_focused_stays_on_canvas() { + fn hold_fn_with_agent_window_latched_auto_pastes() { + let decision = resolve_delivery_route( + DeliveryIntent::OrientDictation, + facts(|f| { + f.auto_paste_enabled = true; + }), + ); + assert_eq!(decision.route, DeliveryRoute::ClipboardPaste); + assert_eq!(decision.reason, "auto_paste_to_latched_target"); + assert!(decision.route.posts_synthetic_paste()); + } + + #[test] + fn hold_fn_with_overlay_caret_stays_on_canvas() { let decision = resolve_delivery_route( DeliveryIntent::OrientDictation, facts(|f| { diff --git a/app/controller/hotkey_policy.rs b/app/controller/hotkey_policy.rs index 669843c9..616c58c9 100644 --- a/app/controller/hotkey_policy.rs +++ b/app/controller/hotkey_policy.rs @@ -63,7 +63,14 @@ pub(super) fn should_use_toggle_adjudicated_stop( /// /// A toggle press that *stops* an in-progress toggle recording is excluded: the /// stop must not retroactively change the mode the recording started in. +/// +/// A hold *Press* (legacy mid-hold `HoldUpdate`) never flips destination. +/// Destination is latched at hold-down; Shift/Command attach `{selection_N}` +/// through `RecordingController::attach_hold_selection` instead. pub(super) fn should_apply_incoming_mode_flags(current_state: State, event: &HotkeyInput) -> bool { + if event.key_type == HotkeyType::Hold && event.action == HotkeyAction::Press { + return false; + } matches!(event.action, HotkeyAction::Down | HotkeyAction::Press) && !(event.key_type == HotkeyType::Toggle && current_state == State::RecToggle) } diff --git a/app/controller/mod.rs b/app/controller/mod.rs index 6d9dca3e..96aead14 100644 --- a/app/controller/mod.rs +++ b/app/controller/mod.rs @@ -73,9 +73,8 @@ use crate::os::clipboard; use crate::os::hold_badge::BadgeMode; use crate::os::hotkeys::{self, HoldMode}; use crate::os::selection::{ - AssistiveContext, activate_app_by_name, capture_assistive_context, + AssistiveContext, capture_assistive_context, capture_assistive_context_with_image_with_prior_frontmost, capture_frontmost_app_only, - wait_for_frontmost_app, }; use crate::os::shortcut_registry; use codescribe_core::asr_session::gateway_session_availability; @@ -101,7 +100,7 @@ use assistive_delivery::{ }; use delivery_route::{ DeliveryFacts, DeliveryIntent, DeliveryRoute, delivery_intent_from_session, - format_delivery_route_line, overlay_insert_facts, resolve_delivery_route, target_is_self_app, + format_delivery_route_line, overlay_insert_facts, resolve_delivery_route, }; #[cfg(test)] pub(crate) use final_pass::FinalPassRoutingMode; @@ -140,8 +139,8 @@ use hotkey_policy::{ #[cfg(test)] use hotkey_policy::{is_assistive_start_event, toggle_stop_adjudicate_timeout}; use overlay_paste::{ - DeferredInsertRegistration, OVERLAY_PASTE_FOCUS_BUDGET, OverlayPasteDisposition, - deferred_insert_registration, overlay_paste_disposition, + DeferredInsertRegistration, OverlayPasteDisposition, confirm_latched_paste_target, + deferred_insert_registration, overlay_paste_disposition, park_refused_paste, }; #[cfg(test)] use quality_delivery::AutomaticDeliverySink; @@ -750,6 +749,23 @@ impl RecordingController { } } + /// Attach the current OS selection as `{selection_N}` during an in-flight + /// hold. Destination, overlay visibility, and Agent UI stay unchanged. + pub async fn attach_hold_selection(&self) -> Result<()> { + let current_state = self.current_state().await; + let pending_hold = self.hold_start_task.lock().await.is_some(); + if !matches!(current_state, State::RecHold | State::RecToggle) && !pending_hold { + debug!("attach_hold_selection ignored: no in-flight hold"); + return Ok(()); + } + + let prior_frontmost_app = self.pre_overlay_frontmost_app.read().await.clone(); + let _ctx = self + .capture_assistive_combo_context(current_state, prior_frontmost_app) + .await; + Ok(()) + } + /// Deliver the overlay's current transcript with the context captured at /// trigger time. Taking the context makes delivery one-shot. pub async fn deliver_pending_assistive_transcript(&self, transcript: String) -> Result { @@ -927,22 +943,16 @@ impl RecordingController { /// Paste user-edited overlay text through the delivery throne, then restore /// the latched target and synthesize Cmd+V via clipboard. /// - /// `resolve_delivery_route(OverlayInsert)` picks the destination. Codescribe - /// as the latched target arms Paste Here instead of pasting into ourselves. - /// Otherwise delivery is fail-closed: Cmd+V is posted only when the runtime - /// frontmost app exactly matches the latched target and Accessibility - /// permits event posting. Every unconfirmed case becomes a tagged copy. + /// `resolve_delivery_route(OverlayInsert)` picks the destination. Overlay + /// **caret** (Swift `defer_text_from_overlay`) arms Paste Here. Agent + /// window, Alacritty, and every other latched caret get Cmd+V. Unconfirmed + /// ambulances park Paste Here and leave the user's clipboard alone. pub async fn paste_text_from_overlay(&self, text: String) -> Result { let trimmed = text.trim(); let target_app = self.pre_overlay_frontmost_app.read().await.clone(); let intent = DeliveryIntent::OverlayInsert; - let decision = resolve_delivery_route( - intent, - overlay_insert_facts( - !trimmed.is_empty(), - target_app.as_deref().is_some_and(target_is_self_app), - ), - ); + let decision = + resolve_delivery_route(intent, overlay_insert_facts(!trimmed.is_empty(), false)); info!( "{}", format_delivery_route_line(intent, decision, target_app.as_deref()) @@ -962,15 +972,12 @@ impl RecordingController { .await; } - if let Some(app_name) = target_app.as_deref() { - let activated = activate_app_by_name(app_name); - let focus_confirmed = - activated && wait_for_frontmost_app(app_name, OVERLAY_PASTE_FOCUS_BUDGET); - debug!( - app_name, - activated, focus_confirmed, "Overlay paste target activation" - ); - } + let focus_confirmed = confirm_latched_paste_target(target_app.as_deref()); + debug!( + target = ?target_app, + focus_confirmed, + "Overlay paste target activation" + ); let config = self.config.read().await.clone(); let paste_text = maybe_wrap_transcript_for_delivery(trimmed, &config, "dictation"); @@ -980,6 +987,7 @@ impl RecordingController { target_app.as_deref(), frontmost.as_deref(), preflight.can_post_events(), + focus_confirmed, ); let mut deferred_insert_shortcut = None; @@ -1029,12 +1037,12 @@ impl RecordingController { }) } - /// Degrade path when a synthetic paste is not safe to post: arm the payload - /// behind the "Paste Here" shortcut, or fall back to a plain clipboard copy - /// when that shortcut cannot be registered. + /// Degrade path when a synthetic paste is not safe to post: park the payload + /// in the process-local Paste Here slot. Never writes the system pasteboard. /// /// The out-params carry back what the UI must tell the user — which - /// shortcut is now armed, or why arming failed. + /// shortcut is now armed, or why the chord is not bound. The transcript + /// still sits in-process either way; the user's clipboard stays put. fn arm_or_copy_deferred_payload( &self, payload: String, @@ -1042,6 +1050,9 @@ impl RecordingController { shortcut_label: &mut Option, registration_failure: &mut Option, ) -> Result { + if !park_refused_paste(payload) { + return Ok(OverlayPasteDelivery::Noop); + } let collision = shortcut_registry::deferred_insert_shortcut_conflict(config.deferred_insert_shortcut); match deferred_insert_registration( @@ -1052,24 +1063,20 @@ impl RecordingController { DeferredInsertRegistration::Available { shortcut_label: label, } => { - if !clipboard::arm_deferred_insert(payload) { - return Ok(OverlayPasteDelivery::Noop); - } *shortcut_label = Some(label); Ok(OverlayPasteDelivery::DeferredInsertArmed) } DeferredInsertRegistration::Unavailable { reason } => { - clipboard::set_clipboard(&payload) - .context("Failed to copy overlay text after Paste Here registration failure")?; *registration_failure = Some(reason); - Ok(OverlayPasteDelivery::CopiedToClipboard) + // Slot is armed. The chord is not bound, so the overlay stays + // the visible buffer. Do not steal the user's clipboard. + Ok(OverlayPasteDelivery::DeferredInsertArmed) } } } - /// Arm tagged overlay text for Paste Here (or copy if that shortcut cannot - /// register). Shared by the throne's `DeferredInsert` verdict and by the - /// explicit defer click. + /// Arm tagged overlay text for Paste Here. Shared by the throne's + /// `DeferredInsert` verdict and by the explicit defer click. async fn arm_overlay_text( &self, trimmed: &str, @@ -1122,10 +1129,9 @@ impl RecordingController { .await } - /// Copy the tagged transcript to the clipboard without any synthetic paste. - /// Degrade path for the overlay Insert action when the caret already sits - /// inside Codescribe (e.g. the overlay's editable FINAL), where a synthetic - /// Cmd+V would paste the transcript back into the overlay itself. + /// Explicit overlay Copy: write the tagged transcript to the system + /// pasteboard. This is the only automatic-adjacent verb allowed to replace + /// the user's clipboard. Insert / stop-path refuse must not call this. pub async fn copy_text_from_overlay(&self, text: String) -> Result<()> { let trimmed = text.trim(); if trimmed.is_empty() { @@ -1657,6 +1663,12 @@ impl RecordingController { let current_state = self.current_state().await; if current_state == State::Idle { self.schedule_hold_start(event.assistive).await?; + // Fn down with a live OS selection attaches `{selection_1}` + // immediately. Mid-hold arm pulses add `{selection_2..n}`. + // Destination stays dictation — do not arm Chat/Agent. + if !event.assistive && matches!(event.hold_mode, HoldMode::Raw) { + self.attach_hold_selection().await?; + } } } HotkeyAction::Up => { @@ -4341,7 +4353,7 @@ impl RecordingController { overlay_enabled: config.transcription_overlay_enabled, live_stream_session, commit_required: commit_trigger.is_some(), - latched_target_is_self: latched_target.as_deref().is_some_and(target_is_self_app), + latched_target_is_self: false, }, ); info!( @@ -4408,24 +4420,22 @@ impl RecordingController { &mode_label, Some(&truth_metadata), ); - // Restore the *latched* target before Cmd+V. Frontmost-at-stop is - // not the destination — that is how tagged raw landed in the Agent - // composer (operator: walka o tron, delivery axis). - if let Some(app_name) = latched_target.as_deref() { - let activated = activate_app_by_name(app_name); - let focus_confirmed = - activated && wait_for_frontmost_app(app_name, OVERLAY_PASTE_FOCUS_BUDGET); - debug!( - app_name, - activated, focus_confirmed, "Stop-path paste target activation" - ); - } + // Restore the *latched* ambulance before Cmd+V. Overlay canvas is + // not a sink; Agent window and Alacritty are. Frontmost-at-stop is + // not the destination. + let focus_confirmed = confirm_latched_paste_target(latched_target.as_deref()); + debug!( + target = ?latched_target, + focus_confirmed, + "Stop-path paste target activation" + ); let frontmost = crate::os::selection::current_frontmost_app_name(); let preflight = clipboard::synthetic_paste_preflight(); let disposition = overlay_paste_disposition( latched_target.as_deref(), frontmost.as_deref(), preflight.can_post_events(), + focus_confirmed, ); if disposition == OverlayPasteDisposition::Paste { if self @@ -4438,13 +4448,22 @@ impl RecordingController { info!("Automatic delivery skipped: recording timestamp already delivered"); } } else { - clipboard::set_clipboard(&paste_text) - .context("Failed to copy transcript after paste target was not confirmed")?; + let mut deferred_insert_shortcut = None; + let mut deferred_insert_failure = None; + let delivery = self.arm_or_copy_deferred_payload( + paste_text, + &config, + &mut deferred_insert_shortcut, + &mut deferred_insert_failure, + )?; info!( ?disposition, + ?delivery, + shortcut = ?deferred_insert_shortcut, + failure = ?deferred_insert_failure, target = ?latched_target, frontmost = ?frontmost, - "delivery_route: synthetic paste refused; clipboard copy only" + "delivery_route: synthetic paste refused; paste-here parked, user clipboard untouched" ); } } else { diff --git a/app/controller/overlay_paste.rs b/app/controller/overlay_paste.rs index 7a3d98c4..59cc2007 100644 --- a/app/controller/overlay_paste.rs +++ b/app/controller/overlay_paste.rs @@ -4,6 +4,8 @@ use std::time::Duration; use crate::config::DeferredInsertShortcut; +use super::delivery_route::target_is_self_app; + /// Focus restore budget after activating the overlay paste target app. pub(super) const OVERLAY_PASTE_FOCUS_BUDGET: Duration = Duration::from_millis(250); @@ -12,7 +14,9 @@ pub(super) const OVERLAY_PASTE_FOCUS_BUDGET: Duration = Duration::from_millis(25 pub enum OverlayPasteDelivery { /// Synthetic Cmd+V was posted at the restored target's caret. Pasted, - /// Focus never left Codescribe; tagged text was copied instead of pasted. + /// Explicit Copy (or a leftover UniFFI outcome). Automatic refuse of + /// Cmd+V must not take this branch — that path parks Paste Here and + /// leaves the user's pasteboard alone. CopiedToClipboard, /// Synthetic event posting is not trusted; tagged text was copied instead. AccessibilityPermissionNeeded, @@ -91,32 +95,69 @@ pub(super) enum OverlayPasteDisposition { CopyAccessibilityDenied, } -/// Decide whether to paste or fall back to the clipboard. +/// Decide whether to paste or refuse the synthetic Cmd+V. /// -/// Pasting requires that focus actually returned to the recorded target: both app -/// names must be present, the frontmost app must not be Codescribe itself, and it -/// must match the target. Only then does missing Accessibility permission become -/// the deciding factor. Any mismatch degrades to a copy rather than firing Cmd+V -/// at whatever window happens to be in front. +/// The overlay **canvas** is never a Cmd+V sink — that guard lives in Swift +/// (`insertCaretInCodescribeProbe` → `defer_text_from_overlay`). This function +/// decides among legal ambulances: +/// - Agent window (localized name `Codescribe`) is a legal sink. +/// - Alacritty / Zellij / Notes / … must match, or activation must have +/// confirmed the latched target (the floating overlay can leave +/// `NSWorkspace` still reporting Codescribe). +/// - A third app in front without activation → mismatch, park Paste Here. pub(super) fn overlay_paste_disposition( target_app: Option<&str>, frontmost_app: Option<&str>, can_post_events: bool, + activation_confirmed: bool, ) -> OverlayPasteDisposition { let Some(target) = target_app.map(str::trim).filter(|name| !name.is_empty()) else { return OverlayPasteDisposition::CopyTargetUnavailable; }; + if !can_post_events { + return OverlayPasteDisposition::CopyAccessibilityDenied; + } + if target_is_self_app(target) { + return OverlayPasteDisposition::Paste; + } let Some(frontmost) = frontmost_app.map(str::trim).filter(|name| !name.is_empty()) else { - return OverlayPasteDisposition::CopyFrontmostUnavailable; + return if activation_confirmed { + OverlayPasteDisposition::Paste + } else { + OverlayPasteDisposition::CopyFrontmostUnavailable + }; }; - if frontmost.eq_ignore_ascii_case("codescribe") { - return OverlayPasteDisposition::CopyTargetMismatch; + if frontmost.eq_ignore_ascii_case(target) { + return OverlayPasteDisposition::Paste; } - if !frontmost.eq_ignore_ascii_case(target) { - return OverlayPasteDisposition::CopyTargetMismatch; + if target_is_self_app(frontmost) && activation_confirmed { + return OverlayPasteDisposition::Paste; } - if !can_post_events { - return OverlayPasteDisposition::CopyAccessibilityDenied; + OverlayPasteDisposition::CopyTargetMismatch +} + +/// Activate the latched ambulance and confirm it owns focus. +/// +/// Codescribe (Agent window) is already this process — no activate. A foreign +/// app must both activate and match `NSWorkspace` within the budget. The +/// floating overlay can leave workspace still naming Codescribe; callers pass +/// this bit into [`overlay_paste_disposition`] so a confirmed Alacritty is +/// not vetoed as "frontmost is Codescribe". +pub(super) fn confirm_latched_paste_target(target_app: Option<&str>) -> bool { + let Some(name) = target_app.map(str::trim).filter(|n| !n.is_empty()) else { + return false; + }; + if target_is_self_app(name) { + return true; } - OverlayPasteDisposition::Paste + crate::os::selection::activate_app_by_name(name) + && crate::os::selection::wait_for_frontmost_app(name, OVERLAY_PASTE_FOCUS_BUDGET) +} + +/// Park a refused synthetic paste in the process-local Paste Here slot. +/// +/// Never writes `NSPasteboard`. The user's existing clipboard stays put until +/// they press the Paste Here chord, which then does snapshot → Cmd+V → restore. +pub(super) fn park_refused_paste(payload: String) -> bool { + crate::os::clipboard::arm_deferred_insert(payload) } diff --git a/app/controller/tests.rs b/app/controller/tests.rs index 4bee1463..552f692b 100644 --- a/app/controller/tests.rs +++ b/app/controller/tests.rs @@ -2297,34 +2297,55 @@ fn test_overlay_paste_disposition_decision_table() { Some("Pensieve"), Some(" PENSIEVE "), true, + false, OverlayPasteDisposition::Paste, "exact target, case-insensitive", ), + ( + Some("Alacritty"), + Some("Alacritty"), + true, + true, + OverlayPasteDisposition::Paste, + "vc-terminal / Zellij host is a legal ambulance", + ), ( Some("Pensieve"), Some("Alacritty"), true, + false, OverlayPasteDisposition::CopyTargetMismatch, "third app", ), ( - Some("Pensieve"), + Some("Alacritty"), + Some("Codescribe"), + true, + true, + OverlayPasteDisposition::Paste, + "overlay float may leave NSWorkspace on Codescribe after Alacritty activate", + ), + ( + Some("Alacritty"), Some("Codescribe"), true, + false, OverlayPasteDisposition::CopyTargetMismatch, - "overlay app", + "Alacritty not activated — do not fire Cmd+V at overlay", ), ( Some("Codescribe"), Some("Codescribe"), true, - OverlayPasteDisposition::CopyTargetMismatch, - "overlay app is never a delivery target", + false, + OverlayPasteDisposition::Paste, + "Agent window is a legal Cmd+V sink; overlay caret is a Swift probe", ), ( None, Some("Pensieve"), true, + false, OverlayPasteDisposition::CopyTargetUnavailable, "target lost", ), @@ -2332,21 +2353,31 @@ fn test_overlay_paste_disposition_decision_table() { Some("Pensieve"), None, true, + false, OverlayPasteDisposition::CopyFrontmostUnavailable, "focus unconfirmed", ), + ( + Some("Pensieve"), + None, + true, + true, + OverlayPasteDisposition::Paste, + "activation confirmed even if workspace name is missing", + ), ( Some("Pensieve"), Some("Pensieve"), false, + true, OverlayPasteDisposition::CopyAccessibilityDenied, "event posting denied", ), ]; - for (target, frontmost, can_post_events, expected, label) in cases { + for (target, frontmost, can_post_events, activation_confirmed, expected, label) in cases { assert_eq!( - overlay_paste_disposition(target, frontmost, can_post_events), + overlay_paste_disposition(target, frontmost, can_post_events, activation_confirmed), expected, "{label}" ); @@ -2379,7 +2410,7 @@ fn test_transcript_delivery_wrap_uses_config_when_enabled() { } #[test] -fn deferred_insert_registration_failure_preserves_copy_fallback() { +fn deferred_insert_registration_reports_unavailable_reasons() { assert_eq!( deferred_insert_registration(DeferredInsertShortcut::CommandOptionV, false, None,), DeferredInsertRegistration::Unavailable { @@ -2404,6 +2435,25 @@ fn deferred_insert_registration_failure_preserves_copy_fallback() { ); } +/// Refused Cmd+V parks the transcript in-process. The user's pasteboard is +/// not a fallback delivery channel. +#[test] +#[serial] +fn refused_paste_parks_paste_here_without_touching_clipboard() { + use crate::os::clipboard::{get_clipboard, set_clipboard}; + + if set_clipboard("user clipboard sentinel").is_err() { + return; + } + assert!(super::overlay_paste::park_refused_paste( + "tagged refused transcript".to_string() + )); + let Ok(current) = get_clipboard() else { + return; + }; + assert_eq!(current, "user clipboard sentinel"); +} + #[test] fn test_transcript_delivery_wrap_uses_truth_quality_placeholders() { let config = Config { @@ -3454,6 +3504,38 @@ fn test_toggle_stop_event_preserves_active_session_identity() { ); } +#[test] +fn test_hold_press_never_applies_mode_flags() { + let press_chat = HotkeyInput { + key_type: HotkeyType::Hold, + action: HotkeyAction::Press, + assistive: true, + hold_mode: HoldMode::Chat, + force_raw: false, + force_ai: false, + }; + let down_raw = HotkeyInput { + key_type: HotkeyType::Hold, + action: HotkeyAction::Down, + assistive: false, + hold_mode: HoldMode::Raw, + force_raw: true, + force_ai: false, + }; + assert!( + !should_apply_incoming_mode_flags(State::RecHold, &press_chat), + "RecHold + Hold Press must not flip destination" + ); + assert!( + !should_apply_incoming_mode_flags(State::Idle, &press_chat), + "Hold Press during the start-delay window must not upgrade to Chat" + ); + assert!( + should_apply_incoming_mode_flags(State::Idle, &down_raw), + "Hold Down still latches destination" + ); +} + #[tokio::test] #[serial] async fn test_agent_send_in_flight_blocks_nonassistive_hotkey_starts() { @@ -4385,12 +4467,12 @@ async fn test_shift_upgrade_mid_hold_overrides_force_raw() { // Verify RAW mode is set assert!(*controller.force_raw_mode.read().await); assert!(!*controller.assistive_mode.read().await); + *controller.state.write().await = State::RecHold; - // Now: User adds Shift mid-hold (upgrade to Assistive) - // This comes as another event with assistive=true + // Leftover HoldUpdate Chat must not flip a live take to Assistive. let upgrade_event = HotkeyInput { key_type: HotkeyType::Hold, - action: HotkeyAction::Press, // Modifier flags changed while holding + action: HotkeyAction::Press, assistive: true, hold_mode: HoldMode::Chat, force_raw: false, @@ -4398,14 +4480,87 @@ async fn test_shift_upgrade_mid_hold_overrides_force_raw() { }; controller.handle_hotkey_event(upgrade_event).await.unwrap(); - // Should upgrade to Assistive, force_raw should be cleared assert!( - *controller.assistive_mode.read().await, - "Shift added mid-hold should upgrade to assistive_mode=true" + !*controller.assistive_mode.read().await, + "mid-hold Press must not upgrade to assistive_mode" ); assert!( - !*controller.force_raw_mode.read().await, - "Shift upgrade should clear force_raw_mode" + *controller.force_raw_mode.read().await, + "mid-hold Press must not clear force_raw_mode" + ); + assert_eq!(*controller.hold_mode.read().await, HoldMode::Raw); + assert_eq!(controller.current_state().await, State::RecHold); +} + +#[tokio::test] +#[serial] +async fn test_attach_hold_selection_does_not_flip_destination() { + let controller = RecordingController::new(); + *controller.state.write().await = State::RecHold; + *controller.force_raw_mode.write().await = true; + *controller.assistive_mode.write().await = false; + *controller.hold_mode.write().await = HoldMode::Raw; + set_assistive_session(false); + + controller + .context_bucket + .lock() + .await + .add_selection(0, "already attached".to_string()) + .expect("seed selection") + .expect("non-empty seed"); + + controller + .attach_hold_selection() + .await + .expect("attach during RecHold must succeed"); + + assert!( + !*controller.assistive_mode.read().await, + "attach must not set assistive_mode" + ); + assert_eq!(*controller.hold_mode.read().await, HoldMode::Raw); + assert!(*controller.force_raw_mode.read().await); + assert_eq!(controller.current_state().await, State::RecHold); + assert!( + !is_assistive_session(), + "attach must not publish BadgeMode::Assistive" + ); + assert!( + controller.context_bucket.lock().await.has_selection_items(), + "attach must not wipe an existing {{selection_N}} bucket" + ); +} + +#[tokio::test] +#[serial] +async fn test_hold_down_raw_attaches_live_selection_without_chat() { + let controller = RecordingController::new(); + crate::os::selection::set_test_captured_selection("already selected"); + + let down_event = HotkeyInput { + key_type: HotkeyType::Hold, + action: HotkeyAction::Down, + assistive: false, + hold_mode: HoldMode::Raw, + force_raw: true, + force_ai: false, + }; + controller.handle_hotkey_event(down_event).await.unwrap(); + + assert!( + !*controller.assistive_mode.read().await, + "Fn hold-down must not set assistive_mode" + ); + assert_eq!(*controller.hold_mode.read().await, HoldMode::Raw); + assert!( + !is_assistive_session(), + "Fn hold-down must not publish BadgeMode::Assistive" + ); + assert_eq!( + controller.context_bucket.lock().await.len(), + 1, + "live OS selection at Fn down must become {{selection_1}}" ); } diff --git a/app/os/clipboard.rs b/app/os/clipboard.rs index 45165bec..8603366f 100644 --- a/app/os/clipboard.rs +++ b/app/os/clipboard.rs @@ -472,7 +472,7 @@ fn simulate_right_arrow() -> Result<()> { /// Hand the clipboard back on a background thread once the paste has settled. /// /// The restore is conditional on `paste_epoch` still being current. Any -/// clipboard write in the meantime — a "degrade to copy" fallback, a second +/// clipboard write in the meantime — an explicit overlay Copy, a second /// dictation — bumps the epoch, and this thread then exits without writing, so /// a delayed restore can never clobber newer content the user is waiting on. /// diff --git a/app/os/hotkeys/detector.rs b/app/os/hotkeys/detector.rs index b12d8390..b80e12df 100644 --- a/app/os/hotkeys/detector.rs +++ b/app/os/hotkeys/detector.rs @@ -36,23 +36,14 @@ pub enum HoldAction { /// High-level hold intent derived from modifier state. /// -/// UX split — the destination is chosen by the MODE, and only `Raw` reaches the -/// frontmost app: -/// - `Raw`: dictation → auto-paste (fast) -/// - `Chat`: voice chat to AI → reply in the **agent chat window** (no auto-paste) -/// - `Selection`: instruction applied to the selected text → reply in the -/// **agent chat window** (no auto-paste) +/// Destination is latched at hold-down. Live detectors start a dictation hold +/// as `Raw` even when Shift/Command is already down; a later arm pulse emits +/// [`HotkeyEvent::AttachSelection`] instead of promoting to `Chat`. /// -/// Both agent modes previously documented "response in overlay". That has been -/// false since the legacy AppKit overlay sink was removed: replies are -/// broadcast to Swift over `CsAgentDeliveryListener` -/// (`bridge/src/agent_delivery.rs`), deliberately kept off the -/// overlay/dictation stream, and `testAgentModesNeverConstructOrOrderOverlayFront` -/// asserts the overlay is never even constructed for them. The comment is -/// corrected rather than deleted because the reconstruction of this contract -/// (`reports/trigger-routing-contract-reconstruction.md`, gap G2) exists -/// precisely to stop the routing rule being re-invented by the next reader who -/// trusts the code's own doc over the doc. +/// - `Raw`: dictation → overlay + auto-paste (the live hold destination) +/// - `Chat` / `Selection`: leftover controller vocabulary for an assistive +/// *start* that a hold arm no longer produces. Replies still go to the +/// Agent window over `CsAgentDeliveryListener`, never the overlay. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub enum HoldMode { #[default] @@ -70,8 +61,16 @@ pub enum HotkeyEvent { InsertHere, /// Hold gesture detected (press/release configured modifier combo) Hold { action: HoldAction, mode: HoldMode }, - /// Modifier change while hold is active (e.g., add/remove Shift/Cmd). + /// Modifier change while hold is active (legacy mid-hold mode flip). + /// + /// Destination is latched at hold-down. Live detectors emit + /// [`HotkeyEvent::AttachSelection`] instead of upgrading Raw → Chat. HoldUpdate { mode: HoldMode }, + /// Rising edge of the configured arm modifier during an active hold. + /// + /// Captures the current OS selection as `{selection_N}` without changing + /// destination, hiding the overlay, or fronting Agent. + AttachSelection, /// Normal toggle gesture (double-tap left Option) ToggleNormal, /// Raw toggle gesture (double-tap Ctrl) @@ -333,6 +332,9 @@ pub struct HotkeyDetector { insert_here_v_down: bool, /// Edge-trigger for `arm_ignored` diagnostics (visibility only). wrong_arm_logged: bool, + /// Last sampled arm-modifier state while a hold is active, so a Shift + /// (or Cmd) pulse can attach another `{selection_N}` without flipping mode. + arm_modifier_down: bool, } impl Default for HotkeyDetector { @@ -354,6 +356,7 @@ impl Default for HotkeyDetector { show_agent_space_down: false, insert_here_v_down: false, wrong_arm_logged: false, + arm_modifier_down: false, } } } @@ -458,6 +461,7 @@ impl HotkeyDetector { self.hold_active = false; self.hold_active_ts = None; self.hold_event_sent = false; + self.arm_modifier_down = false; self.key_pressed_during_modifier = true; emitted = Some(HotkeyEvent::Hold { action: HoldAction::Up, @@ -532,23 +536,31 @@ impl HotkeyDetector { self.wrong_arm_logged = false; } + let arm_now = arm_modifier_is_down(modifiers, config.hold_arm_modifier); + let mut emitted = None; if combo_active && !self.hold_active { self.hold_active = true; self.hold_active_ts = Some(now); self.hold_mode = mode_now; self.hold_event_sent = true; + self.arm_modifier_down = arm_now; emitted = Some(HotkeyEvent::Hold { action: HoldAction::Down, mode: self.hold_mode, }); - } else if combo_active && self.hold_active && mode_now != self.hold_mode { - self.hold_mode = mode_now; - emitted = Some(HotkeyEvent::HoldUpdate { - mode: self.hold_mode, - }); + } else if combo_active && self.hold_active { + // Destination is latched at hold-down. A later Shift/Cmd pulse + // attaches `{selection_N}`; it must not emit HoldUpdate Chat, + // which fronts Agent and drops the live take. + let arm_rising = arm_now && !self.arm_modifier_down; + self.arm_modifier_down = arm_now; + if arm_rising { + emitted = Some(HotkeyEvent::AttachSelection); + } } else if !combo_active && self.hold_active { self.hold_active = false; + self.arm_modifier_down = false; if self.hold_event_sent { emitted = Some(HotkeyEvent::Hold { action: HoldAction::Up, @@ -815,6 +827,16 @@ fn register_blocked_option_double_tap( /// the neighbourhood of a hold", which is what the start-delay cancel and the /// wrong-arm diagnostic need. Double-tap bindings have no base and return /// `false`. +fn arm_modifier_is_down( + modifiers: HotkeyModifierSnapshot, + arm_modifier: crate::config::HoldArmModifier, +) -> bool { + match arm_modifier { + crate::config::HoldArmModifier::Shift => modifiers.shift, + crate::config::HoldArmModifier::Cmd => modifiers.cmd, + } +} + fn hold_base_pressed( modifiers: HotkeyModifierSnapshot, dictation_binding: ShortcutBinding, @@ -885,47 +907,20 @@ fn assistive_hold_binding(binding: ShortcutBinding) -> Option { } } -/// Decide which [`HoldMode`] a live hold is in, from the arm modifiers. +/// Hold destination at key-down. Arm modifiers no longer promote to `Chat`. /// -/// `hold_exclusive` short-circuits to `Raw` — when the user has asked for -/// exclusive holds, no modifier may promote dictation into an agent mode. Only -/// the bindings that leave a modifier free (`HoldCtrlAlt`, `HoldFn`) can reach -/// `Chat`; the rest already spend Shift or Cmd on the combo itself and stay -/// `Raw`. +/// Shift/Command during an already-started hold attach `{selection_N}`. +/// Fn+Shift from idle stays dictation. The arguments are kept so exclusive / +/// binding / arm Settings still flow through this seam; they must not change +/// the latched destination. fn compute_hold_mode( - shift: bool, - cmd: bool, - dictation_binding: ShortcutBinding, - hold_exclusive: bool, - arm_modifier: crate::config::HoldArmModifier, + _shift: bool, + _cmd: bool, + _dictation_binding: ShortcutBinding, + _hold_exclusive: bool, + _arm_modifier: crate::config::HoldArmModifier, ) -> HoldMode { - if hold_exclusive { - return HoldMode::Raw; - } - - // W10-B: only the *configured* arm modifier arms Chat. The other must stay - // dead so Settings copy and detector agree (default Shift; Cmd alternative). - let arm_active = match arm_modifier { - crate::config::HoldArmModifier::Shift => shift, - crate::config::HoldArmModifier::Cmd => cmd, - }; - - match dictation_binding { - ShortcutBinding::Disabled - | ShortcutBinding::HoldCtrl - | ShortcutBinding::HoldCtrlShift - | ShortcutBinding::HoldCtrlCmd - | ShortcutBinding::DoubleCtrl - | ShortcutBinding::DoubleLeftOption - | ShortcutBinding::DoubleRightOption => HoldMode::Raw, - ShortcutBinding::HoldCtrlAlt | ShortcutBinding::HoldFn => { - if arm_active { - HoldMode::Chat - } else { - HoldMode::Raw - } - } - } + HoldMode::Raw } #[cfg(test)] @@ -1108,138 +1103,65 @@ mod tests { } #[test] - /// Hold mode follows arm modifier and optional assistive Shift without exclusive force. + /// Arm modifiers never upgrade hold destination — attach is a later pulse. fn compute_hold_mode_respects_modifiers() { use crate::config::HoldArmModifier; - // Fn base + default Shift arm: Shift arms, Cmd does not. - assert_eq!( - compute_hold_mode( - false, - false, - ShortcutBinding::HoldFn, - false, - HoldArmModifier::Shift - ), - HoldMode::Raw - ); - assert_eq!( - compute_hold_mode( - true, - false, - ShortcutBinding::HoldFn, - false, - HoldArmModifier::Shift - ), - HoldMode::Chat - ); - assert_eq!( - compute_hold_mode( - false, - true, - ShortcutBinding::HoldFn, + let cases = [ + ( false, - HoldArmModifier::Shift - ), - HoldMode::Raw - ); - - // Cmd-selected arm: Cmd arms, Shift does not. - assert_eq!( - compute_hold_mode( false, - true, ShortcutBinding::HoldFn, - false, - HoldArmModifier::Cmd + HoldArmModifier::Shift, ), - HoldMode::Chat - ); - assert_eq!( - compute_hold_mode( + (true, false, ShortcutBinding::HoldFn, HoldArmModifier::Shift), + (false, true, ShortcutBinding::HoldFn, HoldArmModifier::Shift), + (false, true, ShortcutBinding::HoldFn, HoldArmModifier::Cmd), + (true, false, ShortcutBinding::HoldFn, HoldArmModifier::Cmd), + ( true, false, - ShortcutBinding::HoldFn, - false, - HoldArmModifier::Cmd - ), - HoldMode::Raw - ); - - // Ctrl-only ignores Shift/Cmd modifiers - assert_eq!( - compute_hold_mode( - true, - false, - ShortcutBinding::HoldCtrl, - false, - HoldArmModifier::Shift - ), - HoldMode::Raw - ); - assert_eq!( - compute_hold_mode( - false, - true, ShortcutBinding::HoldCtrl, - false, - HoldArmModifier::Cmd + HoldArmModifier::Shift, ), - HoldMode::Raw - ); - - // Ctrl+Option allows the configured arm - assert_eq!( - compute_hold_mode( + (false, true, ShortcutBinding::HoldCtrl, HoldArmModifier::Cmd), + ( true, false, ShortcutBinding::HoldCtrlAlt, - false, - HoldArmModifier::Shift + HoldArmModifier::Shift, ), - HoldMode::Chat - ); - assert_eq!( - compute_hold_mode( + ( false, true, ShortcutBinding::HoldCtrlAlt, - false, - HoldArmModifier::Shift + HoldArmModifier::Shift, ), - HoldMode::Raw - ); - assert_eq!( - compute_hold_mode( + ( false, false, ShortcutBinding::HoldCtrlAlt, - false, - HoldArmModifier::Shift + HoldArmModifier::Shift, ), - HoldMode::Raw - ); - - // Ctrl+Shift/Cmd are fixed to raw - assert_eq!( - compute_hold_mode( + ( true, false, ShortcutBinding::HoldCtrlShift, - false, - HoldArmModifier::Shift + HoldArmModifier::Shift, ), - HoldMode::Raw - ); - assert_eq!( - compute_hold_mode( + ( false, true, ShortcutBinding::HoldCtrlCmd, - false, - HoldArmModifier::Cmd + HoldArmModifier::Cmd, ), - HoldMode::Raw - ); + ]; + for (shift, cmd, binding, arm) in cases { + assert_eq!( + compute_hold_mode(shift, cmd, binding, false, arm), + HoldMode::Raw, + "arm must not promote {binding:?} (shift={shift}, cmd={cmd}, arm={arm:?})" + ); + } } #[test] @@ -1310,6 +1232,186 @@ mod tests { assert!(!detector.is_combo_active()); } + #[test] + /// Fn then Shift attaches selection; release stays Raw dictation. + fn detector_fn_then_shift_attaches_selection_and_up_stays_raw() { + let mut detector = HotkeyDetector::default(); + let config = test_config( + ShortcutBinding::HoldFn, + ShortcutBinding::DoubleLeftOption, + ShortcutBinding::DoubleRightOption, + ); + let base = Instant::now(); + + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base, + key: HotkeyPhysicalKey::Fn, + modifiers: mods(false, false, false, false, true), + }, + config, + ), + Some(HotkeyEvent::Hold { + action: HoldAction::Down, + mode: HoldMode::Raw, + }) + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(10), + key: HotkeyPhysicalKey::Other, + modifiers: mods(false, false, true, false, true), + }, + config, + ), + Some(HotkeyEvent::AttachSelection) + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(20), + key: HotkeyPhysicalKey::Fn, + modifiers: mods(false, false, false, false, false), + }, + config, + ), + Some(HotkeyEvent::Hold { + action: HoldAction::Up, + mode: HoldMode::Raw, + }) + ); + } + + #[test] + /// Fn+Shift from idle is dictation, not Assistive / Chat. + fn detector_fn_shift_from_idle_stays_dictation() { + let mut detector = HotkeyDetector::default(); + let config = test_config( + ShortcutBinding::HoldFn, + ShortcutBinding::DoubleLeftOption, + ShortcutBinding::DoubleRightOption, + ); + let base = Instant::now(); + + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base, + key: HotkeyPhysicalKey::Fn, + modifiers: mods(false, false, true, false, true), + }, + config, + ), + Some(HotkeyEvent::Hold { + action: HoldAction::Down, + mode: HoldMode::Raw, + }) + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(5), + key: HotkeyPhysicalKey::Other, + modifiers: mods(false, false, true, false, true), + }, + config, + ), + None, + "arm already down at start is not a rising-edge attach" + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(15), + key: HotkeyPhysicalKey::Fn, + modifiers: mods(false, false, false, false, false), + }, + config, + ), + Some(HotkeyEvent::Hold { + action: HoldAction::Up, + mode: HoldMode::Raw, + }) + ); + } + + #[test] + /// Two Shift pulses during one Fn hold emit two AttachSelection events. + fn detector_two_shift_pulses_emit_two_attach_selection() { + let mut detector = HotkeyDetector::default(); + let config = test_config( + ShortcutBinding::HoldFn, + ShortcutBinding::DoubleLeftOption, + ShortcutBinding::DoubleRightOption, + ); + let base = Instant::now(); + + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base, + key: HotkeyPhysicalKey::Fn, + modifiers: mods(false, false, false, false, true), + }, + config, + ), + Some(HotkeyEvent::Hold { + action: HoldAction::Down, + mode: HoldMode::Raw, + }) + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(10), + key: HotkeyPhysicalKey::Other, + modifiers: mods(false, false, true, false, true), + }, + config, + ), + Some(HotkeyEvent::AttachSelection) + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(20), + key: HotkeyPhysicalKey::Other, + modifiers: mods(false, false, false, false, true), + }, + config, + ), + None, + "arm release is silent" + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(30), + key: HotkeyPhysicalKey::Other, + modifiers: mods(false, false, true, false, true), + }, + config, + ), + Some(HotkeyEvent::AttachSelection) + ); + assert_eq!( + detector.feed( + HotkeyDetectorInput::FlagsChanged { + now: base + Duration::from_millis(40), + key: HotkeyPhysicalKey::Fn, + modifiers: mods(false, false, false, false, false), + }, + config, + ), + Some(HotkeyEvent::Hold { + action: HoldAction::Up, + mode: HoldMode::Raw, + }) + ); + } + #[test] /// ModifierFlags::ctrl_only marks only the Control bit. fn test_modifier_flags_ctrl_only() { diff --git a/app/os/selection.rs b/app/os/selection.rs index 06c2e334..8e8b6c07 100644 --- a/app/os/selection.rs +++ b/app/os/selection.rs @@ -99,6 +99,30 @@ fn clear_recent_assistive_context_for_tests() { *guard = None; } +/// Injected OS selection for unit tests (production capture is skipped). +#[cfg(test)] +fn test_captured_selection_store() -> &'static Mutex> { + static STORE: OnceLock>> = OnceLock::new(); + STORE.get_or_init(|| Mutex::new(None)) +} + +/// Queue a one-shot selected-text payload for the next test capture. +#[cfg(test)] +pub fn set_test_captured_selection(text: impl Into) { + let mut guard = test_captured_selection_store() + .lock() + .unwrap_or_else(|e| e.into_inner()); + *guard = Some(text.into()); +} + +#[cfg(test)] +fn take_test_captured_selection() -> Option { + test_captured_selection_store() + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take() +} + /// Read a boolean env knob. Any value other than `0`/`false`/`no`/`off` /// (case-insensitive) counts as enabled; an unset key yields `default`. fn env_flag(key: &str, default: bool) -> bool { @@ -153,6 +177,16 @@ pub fn capture_assistive_context_with_image_with_prior_frontmost( prior_frontmost_app: Option, ) -> CapturedAssistiveContext { // Unit tests should not trigger osascript / clipboard / event simulation. + #[cfg(test)] + if let Some(selected_text) = take_test_captured_selection() { + return CapturedAssistiveContext { + context: AssistiveContext { + selected_text: Some(selected_text), + frontmost_app: None, + }, + image_png: None, + }; + } if cfg!(test) { return CapturedAssistiveContext::default(); } diff --git a/bridge/src/hotkeys.rs b/bridge/src/hotkeys.rs index 826edeae..a176eb5c 100644 --- a/bridge/src/hotkeys.rs +++ b/bridge/src/hotkeys.rs @@ -76,15 +76,15 @@ fn event_can_start_capture(event: &HotkeyEvent) -> bool { /// Agent/Assistive recording still fronts the Agent surface, but the UI callback /// is notification only; audio and transcript events continue to the controller. +/// +/// Mid-hold attach (`AttachSelection`) and leftover `HoldUpdate` Chat must not +/// front Agent — they would hide the overlay and look like the take died. fn event_targets_agent_ui(event: &HotkeyEvent) -> bool { matches!( event, HotkeyEvent::ToggleAssistive | HotkeyEvent::Hold { - mode: HoldMode::Chat | HoldMode::Selection, - .. - } - | HotkeyEvent::HoldUpdate { + action: HoldAction::Down, mode: HoldMode::Chat | HoldMode::Selection, } ) @@ -981,6 +981,9 @@ async fn dispatch_recording_hotkey_event( }; controller.handle_hotkey_event(input).await?; } + HotkeyEvent::AttachSelection => { + controller.attach_hold_selection().await?; + } HotkeyEvent::ToggleNormal => { let input = HotkeyInput { key_type: HotkeyType::Toggle, @@ -1200,6 +1203,49 @@ mod app_action_tests { assert_eq!(deferred_calls.load(Ordering::SeqCst), 1); assert_eq!(listener.show_agent_calls.load(Ordering::SeqCst), 2); } + + #[test] + fn mid_hold_attach_does_not_target_agent_or_claim_capture() { + assert!(!event_can_start_capture(&HotkeyEvent::AttachSelection)); + assert!(!event_targets_agent_ui(&HotkeyEvent::AttachSelection)); + assert!(!event_targets_agent_ui(&HotkeyEvent::HoldUpdate { + mode: HoldMode::Chat, + })); + assert!(!event_targets_agent_ui(&HotkeyEvent::Hold { + action: HoldAction::Up, + mode: HoldMode::Chat, + })); + + let listener = Arc::new(CountingAppActionListener { + show_agent_calls: AtomicUsize::new(0), + }); + let recording_calls = Arc::new(AtomicUsize::new(0)); + let recording_calls_for_route = Arc::clone(&recording_calls); + route_hotkey_event( + HotkeyEvent::AttachSelection, + Some(listener.clone()), + move |_| { + recording_calls_for_route.fetch_add(1, Ordering::SeqCst); + }, + || panic!("attach must not dispatch deferred insert"), + ); + assert_eq!(listener.show_agent_calls.load(Ordering::SeqCst), 0); + assert_eq!(recording_calls.load(Ordering::SeqCst), 1); + + let recording_calls_for_route = Arc::clone(&recording_calls); + route_hotkey_event( + HotkeyEvent::HoldUpdate { + mode: HoldMode::Chat, + }, + Some(listener.clone()), + move |_| { + recording_calls_for_route.fetch_add(1, Ordering::SeqCst); + }, + || panic!("hold update must not dispatch deferred insert"), + ); + assert_eq!(listener.show_agent_calls.load(Ordering::SeqCst), 0); + assert_eq!(recording_calls.load(Ordering::SeqCst), 2); + } } // =========================================================================== diff --git a/bridge/src/recording.rs b/bridge/src/recording.rs index 81f4b848..a7c2b705 100644 --- a/bridge/src/recording.rs +++ b/bridge/src/recording.rs @@ -394,6 +394,20 @@ mod retranscribe_tests { "https://api.libraxis.cloud/v1/audio/transcriptions" ); } + + #[test] + fn remapped_loopback_file_url_names_programming_vocabulary() { + let endpoint = codescribe_core::stt::tail_provider::file_probe_endpoint( + "ws://127.0.0.1:8446/v1/audio/transcribe", + ); + assert_eq!(endpoint, "http://127.0.0.1:8444/v1/audio/transcriptions"); + assert_eq!( + codescribe_core::stt::request_vocabulary::codescribe_stt_vocabulary_form_part( + &endpoint + ), + Some(("vocabulary", "programming")) + ); + } } /// Foreign callback trait — dictation events forwarded to Swift. diff --git a/core/asr_session/bootstrap.rs b/core/asr_session/bootstrap.rs index 3aef9bff..447447d3 100644 --- a/core/asr_session/bootstrap.rs +++ b/core/asr_session/bootstrap.rs @@ -76,6 +76,11 @@ fn live_websocket_endpoint(endpoint: &str) -> Option { let path = url.path().trim_end_matches("transcriptions").to_string() + "transcribe"; url.set_path(&path); } + // Inverse of file_probe_endpoint: Voice Lab file worker :8444 is not a + // live socket. The live socket is :8446. + if url.port() == Some(8444) { + url.set_port(Some(8446)).ok()?; + } Some(url.to_string()) } @@ -226,6 +231,14 @@ mod tests { live_websocket_endpoint("ws://127.0.0.1:8446/v1/audio/transcribe").as_deref(), Some("ws://127.0.0.1:8446/v1/audio/transcribe") ); + assert_eq!( + live_websocket_endpoint("http://127.0.0.1:8444/v1/audio/transcriptions").as_deref(), + Some("ws://127.0.0.1:8446/v1/audio/transcribe") + ); + assert_eq!( + live_websocket_endpoint("http://localhost:8444/v1/audio/transcriptions").as_deref(), + Some("ws://localhost:8446/v1/audio/transcribe") + ); } #[test] diff --git a/core/asr_session/cloud.rs b/core/asr_session/cloud.rs index db30bf88..b8129532 100644 --- a/core/asr_session/cloud.rs +++ b/core/asr_session/cloud.rs @@ -160,6 +160,8 @@ pub struct GatewaySessionConfig { protocol_version: u16, session_id: String, locale: Option, + /// Codescribe domain token. The gateway must not classify audio to pick one. + vocabulary: &'static str, audio: GatewayAudioConfig, } @@ -170,6 +172,7 @@ impl GatewaySessionConfig { protocol_version: 1, session_id: input.session_id.as_str().to_string(), locale: input.locale.clone(), + vocabulary: crate::stt::request_vocabulary::CODESCRIBE_STT_VOCABULARY, audio: GatewayAudioConfig { encoding: "pcm_s16le", sample_rate_hz: input.sample_rate, @@ -264,9 +267,14 @@ impl VoiceLabReceiveState { .and_then(serde_json::Value::as_str) .ok_or(AsrErrorKind::Protocol)?; match message_type { - "ack" | "ready" => Ok(None), + // Voice Lab `stt-ws-v1` opens with `hello` and then control/VAD + // frames. Those are not transcript events; treating them as a + // protocol fault used to drop Layer 1 at take start. + "ack" | "ready" | "hello" | "vad.sample" | "speech.start" | "speech.end" => Ok(None), "transcript.partial" | "transcript" => { - let text = voice_lab_text(&value)?; + let Some(text) = voice_lab_text(&value) else { + return Ok(None); + }; self.revision = self.revision.checked_add(1).ok_or(AsrErrorKind::Protocol)?; let event_id = self.event_id()?; Ok(Some(GatewayEvent::Partial { @@ -280,7 +288,9 @@ impl VoiceLabReceiveState { })) } "transcript.final" => { - let text = voice_lab_text(&value)?; + let Some(text) = voice_lab_text(&value) else { + return Ok(None); + }; self.revision = self.revision.checked_add(1).ok_or(AsrErrorKind::Protocol)?; let event_id = self.event_id()?; let event = GatewayEvent::Final { @@ -316,22 +326,34 @@ impl VoiceLabReceiveState { code, })) } - "end" | "session.ended" => Ok(Some(GatewayEvent::SessionEnded { + "end" | "session.ended" | "stream.closed" => Ok(Some(GatewayEvent::SessionEnded { session_id: self.session_id.clone(), })), - _ => Err(AsrErrorKind::Protocol), + _ => Ok(None), } } } -fn voice_lab_text(value: &serde_json::Value) -> Result { +fn voice_lab_text(value: &serde_json::Value) -> Option { value .get("text") .and_then(serde_json::Value::as_str) .map(str::trim) .filter(|text| !text.is_empty()) .map(str::to_string) - .ok_or(AsrErrorKind::Protocol) +} + +/// Voice Lab live start frame. The engine's frozen inbound types are +/// `set` / `chunk` / `flush` / `end` — `config` is rejected as unknown. +fn voice_lab_set_message(config: &GatewaySessionConfig) -> String { + serde_json::json!({ + "type": "set", + "language": config.locale.as_deref().unwrap_or("pl"), + "sample_rate": config.sample_rate_hz(), + "encoding": "pcm16", + "vocabulary": config.vocabulary, + }) + .to_string() } impl fmt::Debug for GatewayPcmFrame { @@ -691,15 +713,10 @@ async fn run_gateway_socket( let (mut socket, _) = connected; // Proven Voice Lab wire: credentials stay in the WebSocket handshake, - // never in the JSON body. - let start = serde_json::json!({ - "type": "config", - "language": config.locale.as_deref().unwrap_or("pl"), - }) - .to_string(); + // never in the JSON body. The engine start type is `set`, not `config`. send_socket_message( &mut socket, - Message::Text(start.into()), + Message::Text(voice_lab_set_message(&config).into()), limits.send_timeout, ) .await?; @@ -1409,6 +1426,7 @@ mod tests { let start_json = serde_json::to_value(&transport.started[0]).expect("serialize start"); assert_eq!(start_json["type"], "session.start"); assert_eq!(start_json["protocol_version"], 1); + assert_eq!(start_json["vocabulary"], "programming"); assert_eq!(start_json["audio"]["encoding"], "pcm_s16le"); assert_eq!(start_json["audio"]["channels"], 1); assert!(start_json.get("provider").is_none()); @@ -1520,6 +1538,63 @@ mod tests { )); } + #[test] + fn voice_lab_hello_and_control_frames_do_not_fault() { + let mut state = VoiceLabReceiveState::new(session_id().to_string()); + assert_eq!( + state.adapt(r#"{"type":"hello","protocol":"stt-ws-v1"}"#), + Ok(None) + ); + assert_eq!( + state.adapt(r#"{"type":"speech.start","energy":0.4}"#), + Ok(None) + ); + assert_eq!( + state.adapt(r#"{"type":"speech.end","energy":0.1}"#), + Ok(None) + ); + assert_eq!( + state.adapt(r#"{"type":"vad.sample","energy":0.2,"is_speech":true}"#), + Ok(None) + ); + assert_eq!( + state.adapt(r#"{"type":"transcript.final","text":""}"#), + Ok(None) + ); + assert_eq!(state.adapt(r#"{"type":"future.control"}"#), Ok(None)); + assert!(matches!( + state.adapt(r#"{"type":"stream.closed"}"#).expect("closed"), + Some(GatewayEvent::SessionEnded { .. }) + )); + assert!(matches!( + state + .adapt(r#"{"type":"transcript.final","text":"zostaje"}"#) + .expect("final after hello") + .expect("text"), + GatewayEvent::Final { + utterance_id: 1, + revision: 1, + ref text, + .. + } if text == "zostaje" + )); + } + + #[test] + fn voice_lab_start_is_set_with_vocabulary_and_no_secret() { + let payload: serde_json::Value = serde_json::from_str(&voice_lab_set_message( + &GatewaySessionConfig::from_input(&input()), + )) + .expect("set json"); + assert_eq!(payload["type"], "set"); + assert_eq!(payload["language"], "pl-PL"); + assert_eq!(payload["sample_rate"], 16_000); + assert_eq!(payload["encoding"], "pcm16"); + assert_eq!(payload["vocabulary"], "programming"); + assert!(payload.get("api_key").is_none()); + assert!(payload.get("type").and_then(|value| value.as_str()) != Some("config")); + } + #[test] fn local_sequence_is_global_across_reordered_utterances_and_duplicates() { let duplicate = partial("u2-r1", 2, 1, "drugi"); @@ -1751,4 +1826,66 @@ mod tests { "loopback live STT must not require a key" ); } + + #[test] + fn voice_lab_loopback_hello_keeps_the_session_open() { + let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind loopback"); + let addr = listener.local_addr().expect("local addr"); + let (first_tx, first_rx) = std::sync::mpsc::channel::(); + std::thread::spawn(move || { + let Ok((stream, _)) = listener.accept() else { + return; + }; + let Ok(mut socket) = tokio_tungstenite::tungstenite::accept(stream) else { + return; + }; + let hello = Message::Text(r#"{"type":"hello","protocol":"stt-ws-v1"}"#.into()); + if socket.send(hello).is_err() { + return; + } + if let Ok(Message::Text(text)) = socket.read() { + let _ = first_tx.send(text.to_string()); + let _ = socket.send(Message::Text(r#"{"type":"ack"}"#.into())); + let _ = socket.send(Message::Text( + r#"{"type":"speech.start","energy":0.5}"#.into(), + )); + } + while socket.read().is_ok() {} + }); + + let endpoint = format!("{}{addr}/v1/audio/transcribe", concat!("ws", "://")); + let limits = CloudSessionLimits { + connect_timeout: Duration::from_secs(2), + send_timeout: Duration::from_secs(1), + close_timeout: Duration::from_millis(200), + ..CloudSessionLimits::default() + }; + let connection = GatewayConnection::new(endpoint, "").expect("loopback connection"); + let transport = GatewayWebSocketTransport::new(connection, limits).expect("transport"); + let mut session = + LiveCloudAsrSession::new(transport, limits, authorization()).expect("session"); + session.open(&input()).expect("open"); + + let first = first_rx + .recv_timeout(Duration::from_secs(2)) + .expect("Voice Lab start frame"); + let start: serde_json::Value = serde_json::from_str(&first).expect("start json"); + assert_eq!(start["type"], "set"); + assert_ne!(start["type"], "config"); + + for _ in 0..20 { + let events = session.drain(); + assert!( + events + .iter() + .all(|event| !matches!(event, AsrSessionEvent::Error(_))), + "hello/control must not degrade the live lane: {events:?}" + ); + std::thread::sleep(Duration::from_millis(10)); + } + session + .push_audio(&[0.0; 4]) + .expect("PCM after hello stays accepted"); + let _ = session.close(); + } } diff --git a/core/build.rs b/core/build.rs index ec730687..21d1c0da 100644 --- a/core/build.rs +++ b/core/build.rs @@ -474,18 +474,16 @@ fn resolve_whisper_embed_model_path( } } } - if embed_model.contains('/') { - if let Some(snapshot) = find_hf_snapshot(embed_model) { - if whisper_dir_complete(&snapshot) { - return snapshot; - } - } + if embed_model.contains('/') + && let Some(snapshot) = find_hf_snapshot(embed_model) + && whisper_dir_complete(&snapshot) + { + return snapshot; } else if embed_model == DEFAULT_MODEL_NAME && let Some(snapshot) = find_hf_snapshot(default_repo) + && whisper_dir_complete(&snapshot) { - if whisper_dir_complete(&snapshot) { - return snapshot; - } + return snapshot; } resolve_embed_model_path(manifest_dir, embed_model) } diff --git a/core/llm/client.rs b/core/llm/client.rs index e03733f1..49171bf5 100644 --- a/core/llm/client.rs +++ b/core/llm/client.rs @@ -113,6 +113,8 @@ struct WsConfig { msg_type: &'static str, language: String, api_key: String, + /// Codescribe domain token. Hosts that do not accept it are not this path. + vocabulary: &'static str, } /// WebSocket end signal (sent after audio) @@ -429,11 +431,12 @@ async fn transcribe_websocket( response.status() ); - // 1. Send config + // 1. Send config. Topic is the product domain, never classified from audio. let config = WsConfig { msg_type: "config", language: language.to_string(), api_key: api_key.to_string(), + vocabulary: crate::stt::request_vocabulary::CODESCRIBE_STT_VOCABULARY, }; ws.send(Message::Text(serde_json::to_string(&config)?.into())) .await @@ -602,6 +605,7 @@ async fn transcribe_ndjson( "sample_rate": sample_rate, "encoding": "pcm16", "language": language, + "request_vocabulary": crate::stt::request_vocabulary::CODESCRIBE_STT_VOCABULARY, "last": true }); @@ -730,11 +734,13 @@ async fn transcribe_multipart( filename: &str, ) -> Result { let start = Instant::now(); + let vocabulary = crate::stt::request_vocabulary::codescribe_stt_vocabulary(url); info!( - "[Multipart STT] POST {} ({} bytes, lang={})", + "[Multipart STT] POST {} ({} bytes, lang={}, vocabulary={})", url, audio_data.len(), - language + language, + vocabulary.unwrap_or("off") ); // Retry loop @@ -751,10 +757,15 @@ async fn transcribe_multipart( let whisper_model = std::env::var("WHISPER_MODEL") .unwrap_or_else(|_| "mlx-community/whisper-large-v3-mlx".to_string()); - let form = Form::new() + let mut form = Form::new() .part("file", file_part) .text("model", whisper_model.clone()) .text("language", language.to_string()); + if let Some((field, value)) = + crate::stt::request_vocabulary::codescribe_stt_vocabulary_form_part(url) + { + form = form.text(field, value.to_string()); + } debug!( "[Multipart STT] attempt {}/{} for {}", @@ -861,6 +872,20 @@ mod tests { format!("{}{}", WS_SCHEME_PREFIX, authority) } + #[test] + fn ws_config_names_programming_domain() { + let encoded = serde_json::to_value(&WsConfig { + msg_type: "config", + language: "pl".to_string(), + api_key: "unused".to_string(), + vocabulary: crate::stt::request_vocabulary::CODESCRIBE_STT_VOCABULARY, + }) + .expect("serialize ws config"); + assert_eq!(encoded["type"], "config"); + assert_eq!(encoded["vocabulary"], "programming"); + assert_ne!(encoded["vocabulary"], "veterinary"); + } + /// Plain WebSocket only on loopback; non-loopback rejected; the secure scheme always ok. #[test] fn ws_plain_rejected_for_non_loopback() { diff --git a/core/llm/key_liveness.rs b/core/llm/key_liveness.rs index 4c12f39b..3858d794 100644 --- a/core/llm/key_liveness.rs +++ b/core/llm/key_liveness.rs @@ -212,11 +212,16 @@ fn probe_stt_key( .with_probed_endpoint(endpoint); } }; - let form = Form::new() + let mut form = Form::new() .part("file", file) .text("model", "whisper-1") .text("language", "pl") .text("response_format", "json"); + if let Some((field, value)) = + crate::stt::request_vocabulary::codescribe_stt_vocabulary_form_part(&endpoint) + { + form = form.text(field, value.to_string()); + } let request = client.post(&endpoint); let auth_mode = crate::stt::tail_provider::stt_auth_mode(&endpoint); let request = match auth_mode { @@ -517,6 +522,11 @@ mod tests { assert!(!request_lower.contains("x-api-key:")); assert!(!request_lower.contains("authorization:")); assert!(request.contains("codescribe-key-probe.wav")); + assert!( + request.contains("name=\"vocabulary\""), + "loopback Codescribe probe must name the programming domain" + ); + assert!(request.contains("programming")); assert_eq!( result.message, "local STT endpoint accepts unauthenticated requests" diff --git a/core/pipeline/streaming/apple_live_session.rs b/core/pipeline/streaming/apple_live_session.rs index f49c9dee..9e7ec2d8 100644 --- a/core/pipeline/streaming/apple_live_session.rs +++ b/core/pipeline/streaming/apple_live_session.rs @@ -59,6 +59,9 @@ use crate::stt::tail_provider::{ TailProviderRequest, TailRequestIdentity, TailSampleRange, TailTimingQuality, TimedTailSegment, }; +use super::layer1_window::{ + CoalesceFlush, CoalescedPiece, ConcatSpan, Layer1Coalesce, split_outcome_for_members, +}; use super::live_audio_buffer::{DEFAULT_RETENTION_SECS, LiveAudioBuffer, ResolvedAudioWindow}; use super::progressive_seal::{ AppleCommit, ProgressiveSealMachine, SealTick, SealedSpan, seal_span_text, @@ -122,6 +125,10 @@ struct TailPatchRequest { /// Exact capture range behind `audio`; this is the window-start authority. provider_request: TailProviderRequest, covered_through_secs: f32, + /// Concat-space map when this job covers more than one Apple seal. + span_map: Vec, + /// Every sealed utterance this job must close (id, covered_through_secs). + member_ids: Vec<(u64, f32)>, } /// Whisper closure returned to the worker that owns Apple + seal state. @@ -130,6 +137,16 @@ struct TailPatchCompletion { covered_through_secs: f32, outcome: TailPatchOutcome, payload: Option, + span_map: Vec, + member_ids: Vec<(u64, f32)>, +} + +/// In-flight Layer 1 job identity, including the coalesce map. +struct TailPatchInFlight { + utterance_id: u64, + covered_through_secs: f32, + span_map: Vec, + member_ids: Vec<(u64, f32)>, } /// Async Layer 1 lane for the Apple progressive path. @@ -213,29 +230,41 @@ impl AppleTailPatchLane { /// after `UtteranceFinal`, preserving event order. fn finish_for_worker( &mut self, - request_id: u64, - req_end_secs: f32, + inflight: Option, result: Result, ) -> TailPatchCompletion { + let (fallback_id, fallback_end, span_map, member_ids) = match inflight { + Some(job) => ( + job.utterance_id, + job.covered_through_secs, + job.span_map, + job.member_ids, + ), + None => (0, 0.0, Vec::new(), Vec::new()), + }; match result { Ok(job) => { let utterance_id = job.utterance_id; let outcome = job.outcome; TailPatchCompletion { utterance_id, - covered_through_secs: req_end_secs, + covered_through_secs: fallback_end, outcome, payload: Some(job.payload), + span_map, + member_ids, } } Err(error) => TailPatchCompletion { - utterance_id: request_id, - covered_through_secs: req_end_secs, + utterance_id: fallback_id, + covered_through_secs: fallback_end, outcome: TailPatchOutcome::skipped( crate::stt::tail_patcher::SkipReasonCode::ProviderError, format!("tail patch failed: {error}"), ), payload: None, + span_map, + member_ids, }, } } @@ -385,7 +414,7 @@ pub(crate) async fn apple_stream_transcription_session( // At-most-one-in-flight gate (F1), tracked outside the lane so the admit // branch's guard does not borrow what the collect branch holds mutably. let mut tail_patch_in_flight = false; - let mut tail_patch_lane_in_flight: Option<(u64, f32)> = None; + let mut tail_patch_lane_in_flight: Option = None; // Bounded: the worker `try_send`s from the PCM-forwarding thread. let (tp_tx, mut tp_rx) = mpsc::channel::(TAIL_PATCH_QUEUE_CAP); let (tp_done_tx, tp_done_rx) = std_mpsc::channel::(); @@ -476,21 +505,26 @@ pub(crate) async fn apple_stream_transcription_session( // only ever schedules and collects — inference never sits on the // event-drain path (F1). Some(req) = tp_rx.recv(), if !tail_patch_in_flight => { - let utterance_id = req.utterance_id; - let covered_through_secs = req.covered_through_secs; + let inflight = TailPatchInFlight { + utterance_id: req.utterance_id, + covered_through_secs: req.covered_through_secs, + span_map: req.span_map.clone(), + member_ids: req.member_ids.clone(), + }; tail_patch_lane.push_request(req); tail_patch_in_flight = true; - // One job is in flight, so one end boundary is enough. - // FuturesOrdered preserves the same request/result order. - tail_patch_lane_in_flight = Some((utterance_id, covered_through_secs)); + // One job is in flight; the coalesce map rides alongside so + // the completion can close every member seal. + tail_patch_lane_in_flight = Some(inflight); } Some(result) = tail_patch_lane.next() => { tail_patch_in_flight = false; - let (id, end) = tail_patch_lane_in_flight.take().unwrap_or_default(); - let completion = tail_patch_lane.finish_for_worker(id, end, result); + let inflight = tail_patch_lane_in_flight.take(); + let completion = tail_patch_lane.finish_for_worker(inflight, result); + let rejected_id = completion.utterance_id; if !tail_patch_lane.forward_completion_to_worker(&tp_done_tx, completion) { warn!( - utterance_id = id, + utterance_id = rejected_id, "Layer 1 completion rejected — Apple seal worker already closed" ); } @@ -656,6 +690,8 @@ struct AppleSealState { under_commit_escalations: u64, /// Layer 1 hand-off, present only when layered transcription is armed. tail_patch: Option>, + /// Sealed fragments waiting to share one Whisper window (~5 segments). + layer1_coalesce: Layer1Coalesce, /// Seals whose tail-patch request found the queue full (F1 backpressure). tail_patch_backpressure_drops: u64, /// Requests accepted by the Layer 1 queue that have not reported back yet. @@ -709,6 +745,7 @@ impl AppleSealState { unresolved_windows: 0, under_commit_escalations: 0, tail_patch: None, + layer1_coalesce: Layer1Coalesce::default(), tail_patch_backpressure_drops: 0, tail_patch_awaiting_completion: 0, sealed_prefix: String::new(), @@ -732,6 +769,77 @@ impl AppleSealState { } } + fn enqueue_layer1_piece(&mut self, piece: CoalescedPiece) -> bool { + if self.tail_patch.is_none() { + return false; + } + if self.layer1_coalesce.is_empty() { + self.layer1_coalesce + .set_neighbour(self.sealed_prefix.clone()); + } + let flushes = self.layer1_coalesce.push(piece, self.sample_rate); + if flushes.is_empty() { + // Held for a larger window. Still counts as queued so the + // no-Whisper fallback does not seal the fragment raw. + return true; + } + let mut sent = false; + for flush in flushes { + sent |= self.queue_layer1_flush(flush); + } + sent + } + + fn flush_layer1_coalesce(&mut self) -> bool { + self.layer1_coalesce + .force_flush() + .is_some_and(|flush| self.queue_layer1_flush(flush)) + } + + fn queue_layer1_flush(&mut self, flush: CoalesceFlush) -> bool { + let Some(tx) = self.tail_patch.as_ref() else { + return false; + }; + let provider_request = TailProviderRequest { + identity: TailRequestIdentity { + request_id: flush.primary_utterance_id, + range: TailSampleRange { + session: self.session_id.clone(), + capture_epoch: self.capture_epoch, + sample_start: flush.sample_start, + sample_end: flush.sample_end, + }, + }, + sample_rate: self.sample_rate, + language: None, + }; + match tx.try_send(TailPatchRequest { + utterance_id: flush.primary_utterance_id, + committed_text: flush.committed_text, + neighbour_context: flush.neighbour_context, + audio: flush.audio, + provider_request, + covered_through_secs: flush.covered_through_secs, + span_map: flush.spans, + member_ids: flush.member_ids, + }) { + Ok(()) => { + self.tail_patch_awaiting_completion = + self.tail_patch_awaiting_completion.saturating_add(1); + true + } + Err(error) => { + self.tail_patch_backpressure_drops = + self.tail_patch_backpressure_drops.saturating_add(1); + warn!( + utterance_id = flush.primary_utterance_id, + "Layer 1 tail-patch request dropped — queue full or lane gone: {error}" + ); + false + } + } + } + fn new_with_tail_patch_for_session( sample_rate: u32, session_id: String, @@ -757,19 +865,37 @@ impl AppleSealState { let (evidence, words) = completion.payload.map_or((None, Vec::new()), |payload| { (Some(payload.evidence), payload.segments) }); - let outcome = if self.fusion.is_some() { + // Coalesced jobs already ran the concat tail-patch. Fusion looks up + // the last piece on the session clock vs concat-PCM Whisper times and + // would return NoChange, dropping the joined rewrite (live 2026-08-19). + let coalesced_window = completion.span_map.len() > 1 || completion.member_ids.len() > 1; + let outcome = if coalesced_window { + completion.outcome + } else if self.fusion.is_some() { apply_conservative_fusion(self, ev_tx, utterance_id, &words, completion.outcome) } else { completion.outcome }; - self.tail_patch_outcomes.insert(utterance_id, outcome); - self.progressive - .note_whisper_window_elapsed_with_provenance( - utterance_id, - completion.covered_through_secs, - evidence, - words, - ); + let member_ids = if completion.member_ids.is_empty() { + vec![(utterance_id, completion.covered_through_secs)] + } else { + completion.member_ids + }; + let split = split_outcome_for_members(outcome, &completion.span_map, &member_ids); + for (index, (id, end, member_outcome)) in split.into_iter().enumerate() { + self.tail_patch_outcomes.insert(id, member_outcome); + if index == 0 { + self.progressive + .note_whisper_window_elapsed_with_provenance( + id, + end, + evidence.clone(), + words.clone(), + ); + } else { + self.progressive.note_whisper_window_elapsed(id, end); + } + } self.emit_ready_progressive_seals(ev_tx, now_secs); // A window that finishes AFTER its span sealed had no reader: the only // drain of `tail_patch_outcomes` runs inside the seal tick, so a patch @@ -780,7 +906,9 @@ impl AppleSealState { // recovered speech was computed, stored, and never delivered. Ordering // is unchanged for the normal case (still emitted after `UtteranceFinal`, // which the seal already sent). - self.deliver_sealed_tail_patch(ev_tx, utterance_id); + for (id, _) in &member_ids { + self.deliver_sealed_tail_patch(ev_tx, *id); + } } /// Deliver a tail-patch outcome whose span is already sealed and emitted. @@ -1522,42 +1650,21 @@ fn seal_sliced_by_silero( let window = state .audio .window_by_samples(request_range.sample_start, request_range.sample_end); - let queued = if let (Some(window), Some(tx)) = (window, state.tail_patch.as_ref()) { - let committed_text = seal_span_text(&text, &state.sealed_prefix, false); - match tx.try_send(TailPatchRequest { - utterance_id, - committed_text, - neighbour_context: state.sealed_prefix.clone(), - audio: window.samples, - provider_request: TailProviderRequest { - identity: TailRequestIdentity { - request_id: utterance_id, - range: TailSampleRange { - session: state.session_id.clone(), - capture_epoch: state.capture_epoch, - sample_start: window.sample_start, - sample_end: window.sample_end, - }, - }, - sample_rate: state.sample_rate, - language: None, - }, - covered_through_secs: span_end, - }) { - Ok(()) => { - state.tail_patch_awaiting_completion = - state.tail_patch_awaiting_completion.saturating_add(1); - true - } - Err(error) => { - state.tail_patch_backpressure_drops = - state.tail_patch_backpressure_drops.saturating_add(1); - warn!( - utterance_id, - "Layer 1 tail-patch request dropped — queue full or lane gone: {error}" - ); - false - } + let queued = if let Some(window) = window { + if state.tail_patch.is_some() { + let committed_text = seal_span_text(&text, &state.sealed_prefix, false); + state.enqueue_layer1_piece(CoalescedPiece { + utterance_id, + committed_text, + audio: window.samples, + sample_start: window.sample_start, + sample_end: window.sample_end, + start_ts: span_start, + covered_through_secs: span_end, + segment_count: disjoint.len().max(1), + }) + } else { + false } } else { false @@ -1814,6 +1921,7 @@ fn seal_utterance_final( }) { return false; } + let segment_count = disjoint.len().max(1); state.pending_events.insert( utterance_id, PendingAppleSeal { @@ -1826,42 +1934,20 @@ fn seal_utterance_final( let window = resolve_sealed_audio_window(state, end_ts); let committed_text = seal_span_text(&after_lexicon, &state.sealed_prefix, false); - let queued = if let (Some(window), Some(tx)) = (window, state.tail_patch.as_ref()) { - let provider_request = TailProviderRequest { - identity: TailRequestIdentity { - request_id: utterance_id, - range: TailSampleRange { - session: state.session_id.clone(), - capture_epoch: state.capture_epoch, - sample_start: window.sample_start, - sample_end: window.sample_end, - }, - }, - sample_rate: state.sample_rate, - language: None, - }; - match tx.try_send(TailPatchRequest { - utterance_id, - committed_text, - neighbour_context: state.sealed_prefix.clone(), - audio: window.samples, - provider_request, - covered_through_secs: end_ts, - }) { - Ok(()) => { - state.tail_patch_awaiting_completion = - state.tail_patch_awaiting_completion.saturating_add(1); - true - } - Err(e) => { - state.tail_patch_backpressure_drops = - state.tail_patch_backpressure_drops.saturating_add(1); - warn!( - utterance_id, - "Layer 1 tail-patch request dropped — queue full or lane gone: {e}" - ); - false - } + let queued = if let Some(window) = window { + if state.tail_patch.is_some() { + state.enqueue_layer1_piece(CoalescedPiece { + utterance_id, + committed_text, + audio: window.samples, + sample_start: window.sample_start, + sample_end: window.sample_end, + start_ts, + covered_through_secs: end_ts, + segment_count, + }) + } else { + false } } else { false @@ -2258,6 +2344,7 @@ fn apple_stream_worker( // left open is sealed here, because no later // callback from this epoch can arrive. seal_open_partial(&mut state, &ev_tx, audio_secs); + let _ = state.flush_layer1_coalesce(); info!( audio_secs, silence_secs, @@ -2301,8 +2388,9 @@ fn apple_stream_worker( // Same seal-time correction as the phrase path — a stop mid-utterance must // not be the one route that commits uncorrected text. seal_open_partial(&mut state, &ev_tx, audio_secs); + let _ = state.flush_layer1_coalesce(); - // Every accepted Layer 1 request must close (success, no-change, or + // Every accepted Layer 1 request must close (success, no-change, or) // explicit skip) before the session task returns. This is bounded by the // queue cap and happens while the async side is still draining jobs. // @@ -2734,6 +2822,8 @@ mod tests { covered_through_secs: 2.0, outcome: TailPatchOutcome::NoChange, payload: Some(synthetic_tail_payload(1, whisper_range, vec![whisper_word])), + span_map: Vec::new(), + member_ids: Vec::new(), }, 5.0, ); @@ -2813,6 +2903,8 @@ mod tests { covered_through_secs: 2.0, outcome, payload: None, + span_map: Vec::new(), + member_ids: Vec::new(), }, 5.0, ); @@ -3163,6 +3255,7 @@ mod tests { &mut state, 3.0, ); + assert!(state.flush_layer1_coalesce()); let initial = tp_rx .try_recv() .expect("the real captured span must reach Layer 1"); @@ -3730,7 +3823,15 @@ mod tests { 1, &TailPatchConfig::default(), ); - let completion = lane.finish_for_worker(1, 2.0, Ok(synthetic_tail_job(1, outcome))); + let completion = lane.finish_for_worker( + Some(TailPatchInFlight { + utterance_id: 1, + covered_through_secs: 2.0, + span_map: Vec::new(), + member_ids: Vec::new(), + }), + Ok(synthetic_tail_job(1, outcome)), + ); assert!( completion .outcome @@ -3757,7 +3858,15 @@ mod tests { 2, &TailPatchConfig::default(), ); - let rejected = lane.finish_for_worker(2, 3.0, Ok(synthetic_tail_job(2, rejected_outcome))); + let rejected = lane.finish_for_worker( + Some(TailPatchInFlight { + utterance_id: 2, + covered_through_secs: 3.0, + span_map: Vec::new(), + member_ids: Vec::new(), + }), + Ok(synthetic_tail_job(2, rejected_outcome)), + ); assert!(!lane.forward_completion_to_worker(&done_tx, rejected)); assert_eq!( lane.replacements(), @@ -3786,6 +3895,10 @@ mod tests { 6.0, ); + assert!( + state.flush_layer1_coalesce(), + "one-seal tests flush the held window so the request is observable" + ); let req = tp_rx .try_recv() .expect("sealed utterance must enqueue a tail-patch request"); @@ -3808,6 +3921,97 @@ mod tests { ); } + /// Five Apple phrase-restarts of one compound sentence must share one + /// Whisper window and take the aligned rewrite, not die at the 0.50 cap. + /// Live 2026-08-19: each chop was its own job (`change_ratio` 0.50–3.00) + /// or fusion rewrote the last fragment and dropped the concat repair. + #[test] + fn five_epoch_apple_chop_rewrites_joined_sentence_not_skip() { + let (tx, mut rx) = mpsc::unbounded_channel(); + let (tp_tx, mut tp_rx) = mpsc::channel::(TAIL_PATCH_QUEUE_CAP); + let mut state = AppleSealState::new_with_tail_patch(TEST_SAMPLE_RATE, tp_tx); + // Live Apple+Layer1 tonight: Silero is loaded for hands-free, so + // `complete_whisper_window` used to run fusion on the last fragment + // and drop the concat repair (`NoChange`). + state.fusion = Some(SileroIngress::new( + TEST_SAMPLE_RATE, + state.session_id.clone(), + 0, + )); + push_capture(&mut state, 8.0); + + let chops = [ + ("ala ma", 0.0, 0.5), + ("czarnego kota", 0.5, 1.1), + ("i białego", 1.1, 1.7), + ("psa dzisiaj", 1.7, 2.3), + ("w domu", 2.3, 2.9), + ]; + let events: Vec<_> = chops + .iter() + .map(|(text, start, end)| LiveStreamEvent::PhraseFinal { + text: (*text).into(), + segments: vec![segment(text, *start, *end)], + }) + .collect(); + emit_stream_events(events, &tx, &mut state, 3.0); + + let req = tp_rx + .try_recv() + .expect("five close chops must flush one coalesced Layer 1 job"); + assert!( + tp_rx.try_recv().is_err(), + "one window, not a job per Apple epoch" + ); + assert_eq!(req.member_ids.len(), 5, "coalesce must keep all five chops"); + assert_eq!(req.span_map.len(), 5); + + let whisper = "ala ma dużego rudego kota oraz małego psa dzisiaj u siebie domu"; + let outcome = compute_tail_patch( + &req.committed_text, + whisper, + req.utterance_id, + &TailPatchConfig::default(), + ); + assert!( + !matches!(outcome, TailPatchOutcome::Skipped { .. }), + "joined window must rewrite, not hit the 0.50 cap, got {outcome:?}" + ); + assert!( + !outcome.events().is_empty(), + "Whisper wording must produce patches, got {outcome:?}" + ); + + while rx.try_recv().is_ok() {} + state.complete_whisper_window( + &tx, + TailPatchCompletion { + utterance_id: req.utterance_id, + covered_through_secs: req.covered_through_secs, + outcome, + payload: None, + span_map: req.span_map, + member_ids: req.member_ids, + }, + 6.0, + ); + + let mut after = Vec::new(); + while let Ok(event) = rx.try_recv() { + after.push(event); + } + assert!( + after.iter().any(|event| matches!( + event, + EngineEvent::ReplaceRange { + source: LayerSource::TailPatch, + .. + } + )), + "coalesced rewrite must reach the canvas, got {after:?}" + ); + } + /// A first final that arrives after the retention horizon must not poison /// the whole session. Measured live 2026-08-14: a 247 s take whose first /// SFSpeech final came at 156 s went 11/11 unresolved — `last_sealed_end` @@ -3894,6 +4098,7 @@ mod tests { &mut state, 6.0, ); + assert!(state.flush_layer1_coalesce()); let req = tp_rx .try_recv() .expect("sealed utterance enqueues a request"); @@ -3911,6 +4116,8 @@ mod tests { "no change", ), payload: None, + span_map: req.span_map, + member_ids: req.member_ids, }, 2.1, ); @@ -3989,24 +4196,24 @@ mod tests { let mut state = AppleSealState::new_with_tail_patch(TEST_SAMPLE_RATE, tp_tx); push_capture(&mut state, 10.0); - emit_stream_events( - vec![ - LiveStreamEvent::PhraseFinal { - text: "pierwsze zdanie".into(), - segments: vec![segment("pierwsze zdanie", 0.5, 2.0)], - }, - LiveStreamEvent::PhraseFinal { - text: "drugie zdanie".into(), - segments: vec![segment("drugie zdanie", 2.5, 4.0)], - }, - ], - &tx, - &mut state, - 10.0, - ); + let mut events = Vec::new(); + for i in 0..10 { + let start = i as f32 * 0.5; + events.push(LiveStreamEvent::PhraseFinal { + text: format!("segment {i}"), + segments: vec![segment(&format!("segment {i}"), start, start + 0.4)], + }); + } + emit_stream_events(events, &tx, &mut state, 10.0); - assert_eq!(state.sealed_count, 2, "seals never wait on the patch queue"); - assert_eq!(state.tail_patch_backpressure_drops, 1); + assert!( + state.tail_patch_backpressure_drops >= 1, + "a second 5-segment flush must drop when the queue already holds one job" + ); + assert!( + state.sealed_count >= 1, + "a dropped flush still seals Apple instead of stalling capture" + ); } /// Acceptance arm: an induced gap (Layer 0 committed a shorter span than diff --git a/core/pipeline/streaming/layer1_window.rs b/core/pipeline/streaming/layer1_window.rs new file mode 100644 index 00000000..c00f72aa --- /dev/null +++ b/core/pipeline/streaming/layer1_window.rs @@ -0,0 +1,469 @@ +//! Layer 1 window: coalesce ~5 Apple segments into one Whisper job. +//! +//! Apple seals short fragments. Diffing each fragment against its own Whisper +//! window hits the change-ratio cap and leaves the chopped canvas standing. +//! This module joins a handful of those fragments — text, PCM, and char +//! offsets — so one decode can rewrite the sentence, then maps +//! `ReplaceRange` events back onto the original utterance ids. + +use crate::pipeline::contracts::{EngineEvent, LayerSource}; +use crate::stt::tail_patcher::{TailPatchOutcome, UnderCommit}; + +/// One utterance's slice inside a concatenated Layer 1 window. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ConcatSpan { + pub utterance_id: u64, + /// Inclusive char start in the concatenated committed string. + pub start: usize, + /// Exclusive char end in the concatenated committed string. + pub end: usize, +} + +/// One sealed Apple fragment waiting to share a Whisper window. +#[derive(Debug, Clone)] +pub struct CoalescedPiece { + pub utterance_id: u64, + pub committed_text: String, + pub audio: Vec, + pub sample_start: u64, + pub sample_end: u64, + pub start_ts: f32, + pub covered_through_secs: f32, + pub segment_count: usize, +} + +/// Ready-to-send Layer 1 job built from one or more coalesced pieces. +#[derive(Debug, Clone)] +pub struct CoalesceFlush { + pub committed_text: String, + pub audio: Vec, + pub spans: Vec, + pub member_ids: Vec<(u64, f32)>, + pub neighbour_context: String, + pub covered_through_secs: f32, + pub sample_start: u64, + pub sample_end: u64, + pub primary_utterance_id: u64, +} + +/// Rolling buffer of sealed Apple fragments for one Layer 1 decode. +#[derive(Debug, Default)] +pub struct Layer1Coalesce { + pieces: Vec, + neighbour_before: String, + segments: usize, +} + +impl Layer1Coalesce { + /// Darek's live window: swap after about five Apple segments. + pub const TARGET_SEGMENTS: usize = 5; + /// Hard cap so a long run-on still gets a decode. + pub const MAX_AUDIO_SECS: f32 = 16.0; + /// A pause this long is a sentence boundary — flush what we have. + pub const PAUSE_SECS: f32 = 1.2; + + pub fn is_empty(&self) -> bool { + self.pieces.is_empty() + } + + /// Remember the canvas already sealed before the next piece. + pub fn set_neighbour(&mut self, neighbour: impl Into) { + if self.pieces.is_empty() { + self.neighbour_before = neighbour.into(); + } + } + + /// Push a sealed fragment. Returns a flush when the window is full, or + /// when `piece` starts after a sentence pause (the previous window first). + pub fn push(&mut self, piece: CoalescedPiece, sample_rate: u32) -> Vec { + let mut out = Vec::new(); + if let Some(last) = self.pieces.last() { + let gap = piece.start_ts - last.covered_through_secs; + if gap >= Self::PAUSE_SECS + && let Some(flush) = self.take_flush() + { + out.push(flush); + } + } + if self.pieces.is_empty() && self.neighbour_before.is_empty() { + // Neighbour is set by the caller before the first push of a window. + } + self.segments = self.segments.saturating_add(piece.segment_count.max(1)); + self.pieces.push(piece); + if self.should_flush(sample_rate) + && let Some(flush) = self.take_flush() + { + out.push(flush); + } + out + } + + /// Drain whatever is held — session end, epoch sleep, or test. + pub fn force_flush(&mut self) -> Option { + self.take_flush() + } + + fn should_flush(&self, sample_rate: u32) -> bool { + if self.pieces.is_empty() { + return false; + } + if self.segments >= Self::TARGET_SEGMENTS { + return true; + } + let samples: u64 = self + .pieces + .iter() + .map(|p| p.sample_end.saturating_sub(p.sample_start)) + .sum(); + let rate = sample_rate.max(1) as f32; + (samples as f32 / rate) >= Self::MAX_AUDIO_SECS + } + + fn take_flush(&mut self) -> Option { + if self.pieces.is_empty() { + return None; + } + let pieces = std::mem::take(&mut self.pieces); + self.segments = 0; + let neighbour_context = std::mem::take(&mut self.neighbour_before); + Some(build_flush(pieces, neighbour_context)) + } +} + +fn build_flush(pieces: Vec, neighbour_context: String) -> CoalesceFlush { + let mut committed_text = String::new(); + let mut audio = Vec::new(); + let mut spans = Vec::with_capacity(pieces.len()); + let mut member_ids = Vec::with_capacity(pieces.len()); + let mut offset = 0usize; + let sample_start = pieces.first().map_or(0, |p| p.sample_start); + let sample_end = pieces.last().map_or(0, |p| p.sample_end); + let covered_through_secs = pieces.last().map_or(0.0, |p| p.covered_through_secs); + let primary_utterance_id = pieces.last().map_or(0, |p| p.utterance_id); + for (i, piece) in pieces.into_iter().enumerate() { + if i > 0 { + committed_text.push(' '); + offset += 1; + } + let start = offset; + committed_text.push_str(&piece.committed_text); + offset += piece.committed_text.chars().count(); + spans.push(ConcatSpan { + utterance_id: piece.utterance_id, + start, + end: offset, + }); + member_ids.push((piece.utterance_id, piece.covered_through_secs)); + audio.extend_from_slice(&piece.audio); + } + CoalesceFlush { + committed_text, + audio, + spans, + member_ids, + neighbour_context, + covered_through_secs, + sample_start, + sample_end, + primary_utterance_id, + } +} + +/// Map concat-space `ReplaceRange` events onto utterance-local offsets. +/// +/// A patch that stays inside one span is remapped 1:1. A patch that crosses +/// a join lands on the first overlapped utterance from the local start to +/// that utterance's end — later fragments in the same cross are left intact +/// so we never wipe a committed span we cannot address cleanly. +pub fn remap_concat_events(events: Vec, spans: &[ConcatSpan]) -> Vec { + if spans.is_empty() { + return events; + } + if spans.len() == 1 { + return events + .into_iter() + .map(|event| remap_single(event, spans[0].utterance_id)) + .collect(); + } + let mut out = Vec::with_capacity(events.len()); + for event in events { + match event { + EngineEvent::ReplaceRange { + start, + end, + text, + source, + .. + } => { + if let Some(mapped) = remap_range(start, end, text, source, spans) { + out.push(mapped); + } + } + other => out.push(other), + } + } + out +} + +fn remap_single(event: EngineEvent, utterance_id: u64) -> EngineEvent { + match event { + EngineEvent::ReplaceRange { + start, + end, + text, + source, + .. + } => EngineEvent::ReplaceRange { + utterance_id, + start, + end, + text, + source, + }, + other => other, + } +} + +fn remap_range( + start: usize, + end: usize, + text: String, + source: LayerSource, + spans: &[ConcatSpan], +) -> Option { + let first = span_owning(start, spans)?; + let last_pos = end.saturating_sub(1).max(start); + let last = span_owning(last_pos, spans).unwrap_or(first); + let local_start = start.saturating_sub(first.start); + let local_end = if first.utterance_id == last.utterance_id { + end.saturating_sub(first.start).min(first.end - first.start) + } else { + first.end - first.start + }; + Some(EngineEvent::ReplaceRange { + utterance_id: first.utterance_id, + start: local_start, + end: local_end, + text, + source, + }) +} + +fn span_owning(pos: usize, spans: &[ConcatSpan]) -> Option<&ConcatSpan> { + spans + .iter() + .find(|span| pos >= span.start && pos < span.end) + .or_else(|| { + // Zero-width insert exactly on a join belongs to the previous span. + spans.iter().rev().find(|span| pos == span.end) + }) +} + +/// Split a remapped outcome so each member utterance can seal independently. +pub fn split_outcome_for_members( + outcome: TailPatchOutcome, + spans: &[ConcatSpan], + member_ids: &[(u64, f32)], +) -> Vec<(u64, f32, TailPatchOutcome)> { + if member_ids.is_empty() { + return Vec::new(); + } + if spans.len() <= 1 { + let (id, end) = member_ids[0]; + return vec![(id, end, outcome)]; + } + match outcome { + TailPatchOutcome::NoChange => member_ids + .iter() + .map(|&(id, end)| (id, end, TailPatchOutcome::NoChange)) + .collect(), + TailPatchOutcome::Skipped { code, reason } => { + let mut out = Vec::with_capacity(member_ids.len()); + out.push(( + member_ids[0].0, + member_ids[0].1, + TailPatchOutcome::Skipped { code, reason }, + )); + for &(id, end) in &member_ids[1..] { + out.push((id, end, TailPatchOutcome::NoChange)); + } + out + } + TailPatchOutcome::Patches(events) => { + let remapped = remap_concat_events(events, spans); + group_events(remapped, member_ids) + } + TailPatchOutcome::UnderCommit(under) => { + let residual = under.residual_required; + let remapped = remap_concat_events(under.appends.clone(), spans); + group_events(remapped, member_ids) + .into_iter() + .map(|(id, end, oc)| { + let appends = oc.into_events(); + ( + id, + end, + TailPatchOutcome::UnderCommit(UnderCommit { + appends, + residual_required: residual && id == member_ids[0].0, + committed_tokens: under.committed_tokens, + retranscribed_tokens: under.retranscribed_tokens, + committed_chars: under.committed_chars, + retranscribed_chars: under.retranscribed_chars, + commit_ratio: under.commit_ratio, + }), + ) + }) + .collect() + } + } +} + +fn group_events( + events: Vec, + member_ids: &[(u64, f32)], +) -> Vec<(u64, f32, TailPatchOutcome)> { + let mut out = Vec::with_capacity(member_ids.len()); + for &(id, end) in member_ids { + let evs: Vec = events + .iter() + .filter(|event| match event { + EngineEvent::ReplaceRange { utterance_id, .. } => *utterance_id == id, + _ => false, + }) + .cloned() + .collect(); + let oc = if evs.is_empty() { + TailPatchOutcome::NoChange + } else { + TailPatchOutcome::Patches(evs) + }; + out.push((id, end, oc)); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + fn piece(id: u64, text: &str, start_ts: f32, end_ts: f32, segs: usize) -> CoalescedPiece { + let rate = 16_000u64; + CoalescedPiece { + utterance_id: id, + committed_text: text.to_string(), + audio: vec![0.0; ((end_ts - start_ts) * rate as f32) as usize], + sample_start: (start_ts * rate as f32) as u64, + sample_end: (end_ts * rate as f32) as u64, + start_ts, + covered_through_secs: end_ts, + segment_count: segs, + } + } + + #[test] + fn flushes_after_five_segments() { + let mut buf = Layer1Coalesce::default(); + buf.set_neighbour("already sealed"); + let mut flushes = Vec::new(); + for i in 0..5 { + flushes.extend(buf.push(piece(i + 1, "słowo", i as f32, i as f32 + 0.4, 1), 16_000)); + } + assert_eq!(flushes.len(), 1); + assert_eq!(flushes[0].spans.len(), 5); + assert_eq!(flushes[0].committed_text, "słowo słowo słowo słowo słowo"); + assert_eq!(flushes[0].neighbour_context, "already sealed"); + assert_eq!(flushes[0].primary_utterance_id, 5); + assert!(buf.is_empty()); + } + + #[test] + fn pause_flushes_the_previous_window() { + let mut buf = Layer1Coalesce::default(); + assert!(buf.push(piece(1, "raz", 0.0, 0.5, 1), 16_000).is_empty()); + let flushes = buf.push(piece(2, "dwa", 3.0, 3.4, 1), 16_000); + assert_eq!(flushes.len(), 1); + assert_eq!(flushes[0].spans.len(), 1); + assert_eq!(flushes[0].committed_text, "raz"); + assert!(!buf.is_empty()); + } + + #[test] + fn remap_stays_inside_the_owning_utterance() { + let spans = vec![ + ConcatSpan { + utterance_id: 1, + start: 0, + end: 4, + }, + ConcatSpan { + utterance_id: 2, + start: 5, + end: 9, + }, + ]; + // "ala ma" — replace "ma" (chars 5..7) on utterance 2. + let events = vec![EngineEvent::ReplaceRange { + utterance_id: 99, + start: 5, + end: 7, + text: "psa".into(), + source: LayerSource::TailPatch, + }]; + let remapped = remap_concat_events(events, &spans); + match &remapped[0] { + EngineEvent::ReplaceRange { + utterance_id, + start, + end, + text, + .. + } => { + assert_eq!(*utterance_id, 2); + assert_eq!(*start, 0); + assert_eq!(*end, 2); + assert_eq!(text, "psa"); + } + other => panic!("expected remap, got {other:?}"), + } + } + + #[test] + fn crossing_patch_lands_on_the_first_span() { + let spans = vec![ + ConcatSpan { + utterance_id: 1, + start: 0, + end: 4, + }, + ConcatSpan { + utterance_id: 2, + start: 5, + end: 9, + }, + ]; + let events = vec![EngineEvent::ReplaceRange { + utterance_id: 99, + start: 2, + end: 8, + text: "pełne zdanie".into(), + source: LayerSource::TailPatch, + }]; + let remapped = remap_concat_events(events, &spans); + match &remapped[0] { + EngineEvent::ReplaceRange { + utterance_id, + start, + end, + text, + .. + } => { + assert_eq!(*utterance_id, 1); + assert_eq!(*start, 2); + assert_eq!(*end, 4); + assert_eq!(text, "pełne zdanie"); + } + other => panic!("expected first-span landing, got {other:?}"), + } + } +} diff --git a/core/pipeline/streaming/mod.rs b/core/pipeline/streaming/mod.rs index cb078180..25b463c0 100644 --- a/core/pipeline/streaming/mod.rs +++ b/core/pipeline/streaming/mod.rs @@ -13,6 +13,8 @@ pub(crate) mod apple_live_session; pub(crate) mod correction; /// Buffered "typing" emission of transcript deltas. pub(crate) mod emitter; +/// Coalesce ~5 Apple segments into one Layer 1 Whisper window. +pub(crate) mod layer1_window; /// Assembly of the live transcript from engine events. pub mod live_assembly; /// Bounded per-session PCM retention, so a sealed utterance can be re-read for tail-patch. diff --git a/core/quality/engine_contract.rs b/core/quality/engine_contract.rs index 80901318..fa8c2273 100644 --- a/core/quality/engine_contract.rs +++ b/core/quality/engine_contract.rs @@ -20,16 +20,34 @@ pub const ENGINE_CONTRACT_DOC: &str = "docs/THE_ENGINE_CONTRACT.md"; /// Path of the HTML-surface contract, relative to the repo root. pub const QUALITY_HTML_CONTRACT_DOC: &str = "docs/quality-reports/CONTRACT.md"; +/// True when a `` tag carries both `name` and `content` on the same tag. +/// Prose elsewhere in the document does not satisfy the handshake. +fn meta_content_is(html: &str, name: &str, content: &str) -> bool { + let name_attr = format!(r#"name="{name}""#); + let content_attr = format!(r#"content="{content}""#); + let mut rest = html; + while let Some(name_at) = rest.find(&name_attr) { + let before = &rest[..name_at]; + let tag_start = before.rfind('<').unwrap_or(0); + let after = &rest[name_at..]; + let tag_end = after.find('>').unwrap_or(after.len()); + let tag = &rest[tag_start..name_at + tag_end]; + if tag.contains(&content_attr) { + return true; + } + rest = &rest[name_at + name_attr.len()..]; + } + false +} + /// Failures if this string is not a Seal Atlas quality report. pub fn validate_quality_html(html: &str) -> Vec { let lowered = html.to_ascii_lowercase(); let mut failures = Vec::new(); - if !html.contains(r#"name="engine-contract""#) || !html.contains(ENGINE_CONTRACT_ID) { + if !meta_content_is(html, "engine-contract", ENGINE_CONTRACT_ID) { failures.push("missing meta engine-contract=the-engine/v1".into()); } - if !html.contains(r#"name="quality-report-surface""#) - || !lowered.contains("seal-atlas") && !lowered.contains("seal atlas") - { + if !meta_content_is(html, "quality-report-surface", QUALITY_REPORT_SURFACE) { failures.push("missing meta quality-report-surface=seal-atlas".into()); } if !lowered.contains("seal atlas") && !lowered.contains("seal-atlas") { @@ -510,6 +528,31 @@ mod tests { assert!(failures.len() >= 3, "{failures:?}"); } + #[test] + fn handshake_rejects_prose_that_is_not_the_meta_content() { + let fake = r#" + + +mentions the-engine/v1 and seal-atlas in prose + + +

the-engine/v1 seal-atlas

+
1word-grain
+

utterance-grain clock-lie SealedSpan.words whisper

+"#; + let failures = validate_quality_html(fake); + assert!( + failures.iter().any(|f| f.contains("engine-contract")), + "{failures:?}" + ); + assert!( + failures + .iter() + .any(|f| f.contains("quality-report-surface")), + "{failures:?}" + ); + } + #[test] fn gold_atlas_html_is_a_pcm_instrument_not_a_wer_table() { let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join(".."); diff --git a/core/quality/mod.rs b/core/quality/mod.rs index 3cf6c7ca..c1bd7947 100644 --- a/core/quality/mod.rs +++ b/core/quality/mod.rs @@ -1,9 +1,11 @@ //! Quality surfaces — where transcription truth is measured, corrected, and learned from. //! -//! Five independent loops share this facade: +//! Six independent loops share this facade: //! //! - `engine_contract` — locked THE ENGINE bars / relay / forbidden ops that //! every quality HTML and `codescribe-corpus` v3 report must carry. +//! - `supervisor` — catalog of every transcript-quality issue kind plus the +//! take classifier Voice Lab embeds as supervisor findings. //! - `overlay_quality` — captures human edits of the overlay FINAL transcript and //! distils them into custom lexicon rules (the live, per-user loop). //! - `qube_report` — batch WAV evaluation: transcribe, format, score, emit artifacts. @@ -12,8 +14,8 @@ //! - `teacher` — the offline learning triangle (Apple live × Whisper × human reference) //! that produces merged deliveries and attention spans. //! -//! Only `teacher` and `engine_contract` are re-exported here; the others are -//! reached through their own module paths. +//! `teacher`, `engine_contract`, and `supervisor` are re-exported here; the +//! others are reached through their own module paths. /// Locked THE ENGINE contract for quality-report HTML and corpus JSON. pub mod engine_contract; @@ -24,6 +26,8 @@ pub mod qube_daemon; pub mod qube_report; /// Seal Atlas HTML renderer — the quality-report surface corpus writes. pub mod seal_atlas_html; +/// Supervisor findings: catalog + take classifier for Voice Lab. +pub mod supervisor; /// Teacher loop: attention flags, lexicon feedback, polygon token helpers. pub mod teacher; @@ -32,6 +36,10 @@ pub use engine_contract::{ render_engine_contract_html, validate_quality_html, }; pub use seal_atlas_html::{SealAtlasPage, SealAtlasStats, render_seal_atlas_html}; +pub use supervisor::{ + SUPERVISOR_FINDINGS_SCHEMA, TakeQualityEvidence, classify_take_findings, quality_issue_catalog, + quality_issue_kind_ids, +}; pub use teacher::{ Layer1MergeMode, Layer1MergedDelivery, MergeMode, MergedDelivery, TeacherInput, TeacherReport, merge_live_layer1, merge_live_whisper, merge_live_whisper_with_terms, report_to_html, teach, diff --git a/core/quality/supervisor.rs b/core/quality/supervisor.rs new file mode 100644 index 00000000..70500b08 --- /dev/null +++ b/core/quality/supervisor.rs @@ -0,0 +1,1329 @@ +//! Supervisor findings — the engine-owned catalog of transcript-quality issues. +//! +//! Voice Lab's three-judge used to score Daily against candle HQ as if HQ were +//! the document. HQ and cloud are [`crate::quality::engine_contract::ReportSurfaceRole::HumanTriggeredProposal`]. +//! This module names every quality issue the engine already knows, and classifies +//! a take into targeted, falsifiable findings a supervisor can act on. +//! +//! Rust is the lock. Voice Lab `judge.py` mirrors the take-evidence subset. + +use serde::{Deserialize, Serialize}; + +use crate::quality::engine_contract::{ReportSurfaceRole, surface_role}; +use crate::quality::teacher::{AlignOp, align_words, tokenize}; + +/// Schema id carried by every supervisor report / Voice Lab `supervisor` object. +pub const SUPERVISOR_FINDINGS_SCHEMA: &str = "codescribe-supervisor-findings/v1"; + +/// Product domain token the file/live loopback lanes must send. +pub const PROGRAMMING_VOCABULARY: &str = "programming"; + +/// Explicit bench opt-out. Omitting the field is not this. +pub const VOCABULARY_OFF: &str = "off"; + +/// Cap on per-lane attention findings so a long take stays readable. +const MAX_ATTENTION_FINDINGS: usize = 8; + +/// Silence-corpus residue Whisper emits on empty audio. Must stay in the +/// same spirit as `pipeline/streaming/quality_gate.rs` (`WHISPER_HALLUCINATIONS_*`). +const SILENCE_CORPUS_RESIDUE: &[&str] = &[ + "thank you", + "thanks for watching", + "thanks for listening", + "dziękuję za uwagę", + "do zobaczenia", + "subscribe", + "like and subscribe", + "napisy stworzone przez społeczność", +]; + +/// Families already named across the engine. Not a second doctrine. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum QualityIssueFamily { + EngineContract, + Clock, + OverlayHighlight, + TeacherAttention, + Confidence, + TranscriptState, + DeliveryGate, + WhisperFilter, + JudgeHygiene, +} + +/// Where a supervisor should cut. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum FindingTarget { + EngineCode, + LabJudge, + LexiconTune, + OperatorReview, +} + +/// Severity the supervisor ranks first. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum FindingSeverity { + P0, + P1, + P2, + Note, +} + +/// How strongly the evidence supports the claim. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum EvidenceGrade { + Strong, + Medium, + Weak, + None, +} + +/// Every transcript-quality issue category the engine already names. +/// +/// Adding a variant without a catalog spec is a compile error (`spec` match). +/// Adding a spec without listing it in [`QualityIssueKind::ALL`] fails the +/// catalog-coverage test. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum QualityIssueKind { + // ── Engine contract forbidden ops ──────────────────────────────────── + RewriteFromZero, + ReorderSpans, + HallucinateIntoSilence, + FullFileInAutomaticPipeline, + AutoReplaceAfterTranscriptSealed, + TreatCommittedAsDocument, + TreatWholeTextMutableUntilSessionSeal, + // ── Clock / Seal Atlas ─────────────────────────────────────────────── + ClockLie, + UtteranceGrainSilenceTail, + LetterTimingAsMeasurement, + ReconstructedTimeline, + // ── Overlay highlights ─────────────────────────────────────────────── + LexiconCorrected, + SpeechGap, + // ── Teacher attention ──────────────────────────────────────────────── + LiveOnly, + WhisperExcess, + Disagreement, + WhisperErrorAtLiveWeakness, + LiveMissWhisperOk, + // ── Confidence flags ───────────────────────────────────────────────── + VeryLowSpeech, + PossibleHallucinationLogprob, + QualityGateDropped, + SileroDroppedTailHallucinations, + LocalFinalPassUnavailable, + CloudFallbackUsed, + StreamingPreviewUsedAsVerdict, + UnverifiedStream, + CloudPrimaryMissing, + AiNoopDetected, + FinalPassLengthRegression, + HighCompression, + // ── Transcript state ───────────────────────────────────────────────── + NoSpeechDetected, + EmptyTranscript, + // ── Delivery gate ──────────────────────────────────────────────────── + RawFinalRewrite, + LossyStreamDrops, + HeavyCorrectionPressure, + SemanticMeaningChange, + // ── Whisper filter ─────────────────────────────────────────────────── + SilenceCorpusResidue, + WordRateAnomaly, + VadDegraded, + ShortUtteranceDrop, + // ── Judge hygiene (the lab lying) ──────────────────────────────────── + HqTreatedAsDocument, + CloudTreatedAsDocument, + WerPromotedToDocumentScore, + OmittedProgrammingVocabulary, + LastSessionPairedWithLiveOverlay, + LeftoverWebsocketPolarity, + ProposalAgreementMisreadAsAccuracy, +} + +/// Static catalog row. Wire id is [`QualityIssueKind`] serde / [`QualityIssueKind::as_str`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +pub struct QualityIssueSpec { + pub kind: QualityIssueKind, + pub family: QualityIssueFamily, + pub default_severity: FindingSeverity, + pub default_target: FindingTarget, + pub what: &'static str, + pub falsifier: &'static str, + pub action: &'static str, +} + +/// One token locus a finding points at. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct FindingSpan { + #[serde(skip_serializing_if = "Option::is_none")] + pub daily_token: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub proposal_token: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub proposal_lane: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub word_index: Option, +} + +/// One targeted finding. Claim + falsifier + action are required. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SupervisorFinding { + pub kind: QualityIssueKind, + pub family: QualityIssueFamily, + pub severity: FindingSeverity, + pub evidence_grade: EvidenceGrade, + pub target: FindingTarget, + pub claim: String, + pub falsifier: String, + pub action: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub span: Option, +} + +/// Surface roles for the three-judge lanes. WER does not change these. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SupervisorRoles { + pub daily: ReportSurfaceRole, + pub candle: ReportSurfaceRole, + pub cloud: ReportSurfaceRole, +} + +/// Supervisor payload Voice Lab embeds next to three-judge WER (the footnote). +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SupervisorReport { + pub schema: String, + pub roles: SupervisorRoles, + pub findings: Vec, + pub catalog_ids: Vec, + pub wer_is_footnote: bool, +} + +/// Evidence a take (or a lying judge) can actually show. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct TakeQualityEvidence { + pub daily_text: String, + pub hq_text: String, + pub cloud_text: String, + pub snapshot_live: bool, + pub audio_from_last_session: bool, + pub vocabulary: Option, + pub cloud_ran: bool, + /// True when the judge scores Daily as if HQ were the document. + pub treats_hq_as_document: bool, + /// True when first_divergence still labels HQ as `websocket` / Daily as `codescribe`. + pub leftover_websocket_polarity: bool, + pub clock_lie_count: usize, + pub speech_gap_count: usize, + pub confidence_flags: Vec, +} + +impl QualityIssueKind { + /// Exhaustive catalog order. Coverage test walks this slice. + pub const ALL: &'static [Self] = &[ + Self::RewriteFromZero, + Self::ReorderSpans, + Self::HallucinateIntoSilence, + Self::FullFileInAutomaticPipeline, + Self::AutoReplaceAfterTranscriptSealed, + Self::TreatCommittedAsDocument, + Self::TreatWholeTextMutableUntilSessionSeal, + Self::ClockLie, + Self::UtteranceGrainSilenceTail, + Self::LetterTimingAsMeasurement, + Self::ReconstructedTimeline, + Self::LexiconCorrected, + Self::SpeechGap, + Self::LiveOnly, + Self::WhisperExcess, + Self::Disagreement, + Self::WhisperErrorAtLiveWeakness, + Self::LiveMissWhisperOk, + Self::VeryLowSpeech, + Self::PossibleHallucinationLogprob, + Self::QualityGateDropped, + Self::SileroDroppedTailHallucinations, + Self::LocalFinalPassUnavailable, + Self::CloudFallbackUsed, + Self::StreamingPreviewUsedAsVerdict, + Self::UnverifiedStream, + Self::CloudPrimaryMissing, + Self::AiNoopDetected, + Self::FinalPassLengthRegression, + Self::HighCompression, + Self::NoSpeechDetected, + Self::EmptyTranscript, + Self::RawFinalRewrite, + Self::LossyStreamDrops, + Self::HeavyCorrectionPressure, + Self::SemanticMeaningChange, + Self::SilenceCorpusResidue, + Self::WordRateAnomaly, + Self::VadDegraded, + Self::ShortUtteranceDrop, + Self::HqTreatedAsDocument, + Self::CloudTreatedAsDocument, + Self::WerPromotedToDocumentScore, + Self::OmittedProgrammingVocabulary, + Self::LastSessionPairedWithLiveOverlay, + Self::LeftoverWebsocketPolarity, + Self::ProposalAgreementMisreadAsAccuracy, + ]; + + /// Wire token. Matches serde `snake_case`. + pub const fn as_str(self) -> &'static str { + match self { + Self::RewriteFromZero => "rewrite_from_zero", + Self::ReorderSpans => "reorder_spans", + Self::HallucinateIntoSilence => "hallucinate_into_silence", + Self::FullFileInAutomaticPipeline => "full_file_in_automatic_pipeline", + Self::AutoReplaceAfterTranscriptSealed => "auto_replace_after_transcript_sealed", + Self::TreatCommittedAsDocument => "treat_committed_as_document", + Self::TreatWholeTextMutableUntilSessionSeal => { + "treat_whole_text_mutable_until_session_seal" + } + Self::ClockLie => "clock_lie", + Self::UtteranceGrainSilenceTail => "utterance_grain_silence_tail", + Self::LetterTimingAsMeasurement => "letter_timing_as_measurement", + Self::ReconstructedTimeline => "reconstructed_timeline", + Self::LexiconCorrected => "lexicon_corrected", + Self::SpeechGap => "speech_gap", + Self::LiveOnly => "live_only", + Self::WhisperExcess => "whisper_excess", + Self::Disagreement => "disagreement", + Self::WhisperErrorAtLiveWeakness => "whisper_error_at_live_weakness", + Self::LiveMissWhisperOk => "live_miss_whisper_ok", + Self::VeryLowSpeech => "very_low_speech", + Self::PossibleHallucinationLogprob => "possible_hallucination_logprob", + Self::QualityGateDropped => "quality_gate_dropped", + Self::SileroDroppedTailHallucinations => "silero_dropped_tail_hallucinations", + Self::LocalFinalPassUnavailable => "local_final_pass_unavailable", + Self::CloudFallbackUsed => "cloud_fallback_used", + Self::StreamingPreviewUsedAsVerdict => "streaming_preview_used_as_verdict", + Self::UnverifiedStream => "unverified_stream", + Self::CloudPrimaryMissing => "cloud_primary_missing", + Self::AiNoopDetected => "ai_noop_detected", + Self::FinalPassLengthRegression => "final_pass_length_regression", + Self::HighCompression => "high_compression", + Self::NoSpeechDetected => "no_speech_detected", + Self::EmptyTranscript => "empty_transcript", + Self::RawFinalRewrite => "raw_final_rewrite", + Self::LossyStreamDrops => "lossy_stream_drops", + Self::HeavyCorrectionPressure => "heavy_correction_pressure", + Self::SemanticMeaningChange => "semantic_meaning_change", + Self::SilenceCorpusResidue => "silence_corpus_residue", + Self::WordRateAnomaly => "word_rate_anomaly", + Self::VadDegraded => "vad_degraded", + Self::ShortUtteranceDrop => "short_utterance_drop", + Self::HqTreatedAsDocument => "hq_treated_as_document", + Self::CloudTreatedAsDocument => "cloud_treated_as_document", + Self::WerPromotedToDocumentScore => "wer_promoted_to_document_score", + Self::OmittedProgrammingVocabulary => "omitted_programming_vocabulary", + Self::LastSessionPairedWithLiveOverlay => "last_session_paired_with_live_overlay", + Self::LeftoverWebsocketPolarity => "leftover_websocket_polarity", + Self::ProposalAgreementMisreadAsAccuracy => "proposal_agreement_misread_as_accuracy", + } + } + + /// Catalog row for this kind. + pub const fn spec(self) -> QualityIssueSpec { + match self { + Self::RewriteFromZero => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P0, + FindingTarget::EngineCode, + "A layer rebuilt the document from tokens instead of appending on pcm_time.", + "Show the span ledger still ordered on the original PCM ranges.", + "Restore append-only ReplaceRange on the sealed span key.", + ), + Self::ReorderSpans => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P0, + FindingTarget::EngineCode, + "Utterances were reordered off the PCM axis.", + "Replay the take: sealed spans stay in capture order.", + "Stop any sort/merge that is not keyed by sample_start.", + ), + Self::HallucinateIntoSilence => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P0, + FindingTarget::EngineCode, + "A later layer invented speech inside Silero silence.", + "Silero p(speech) on that PCM range is below onset, and Apple sealed empty.", + "Ban Whisper/cloud from writing into a silence-classified hole.", + ), + Self::FullFileInAutomaticPipeline => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P0, + FindingTarget::EngineCode, + "A full-file pass ran as if it were the live engine.", + "No automatic stop-path invoked codescribe transcribe / :8444.", + "Keep full-file as button_only_proposal.", + ), + Self::AutoReplaceAfterTranscriptSealed => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P0, + FindingTarget::EngineCode, + "Automation replaced the session document after transcript_sealed.", + "Post-seal HQ/cloud stayed a proposal the human must accept.", + "Fence auto-apply on the seal event.", + ), + Self::TreatCommittedAsDocument => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P1, + FindingTarget::EngineCode, + "A layer-final commit was treated as the session document.", + "utterance_final stayed a per-layer ban, not transcript_sealed.", + "Do not paste or score committed as delivered.", + ), + Self::TreatWholeTextMutableUntilSessionSeal => spec( + self, + QualityIssueFamily::EngineContract, + FindingSeverity::P0, + FindingTarget::EngineCode, + "Closed spans were mutated as if the whole buffer were still open.", + "Sealed [sample_start, sample_end) stayed append-only.", + "Restrict mutation to the open tail.", + ), + Self::ClockLie => spec( + self, + QualityIssueFamily::Clock, + FindingSeverity::P1, + FindingTarget::EngineCode, + "A span claims more characters than speech can produce in that PCM duration.", + "chars/sec over the span range is ≤ CLOCK_LIE_CHARS_PER_SEC (30).", + "Treat the range as an Apple commit window, not the speech outline.", + ), + Self::UtteranceGrainSilenceTail => spec( + self, + QualityIssueFamily::Clock, + FindingSeverity::P2, + FindingTarget::OperatorReview, + "Utterance-grain span includes the silence tail between Apple commits.", + "word-grain pins exist, or Silero edges bound the speech.", + "Do not mint identity from the commit-to-commit window.", + ), + Self::LetterTimingAsMeasurement => spec( + self, + QualityIssueFamily::Clock, + FindingSeverity::P2, + FindingTarget::LabJudge, + "Grapheme ticks were presented as measured times.", + "HTML/report labels letter ticks as interpolation_not_measurement.", + "Strip any forced-aligner claim we do not have.", + ), + Self::ReconstructedTimeline => spec( + self, + QualityIssueFamily::Clock, + FindingSeverity::P1, + FindingTarget::LabJudge, + "A quality surface rebuilt time from the final string instead of PCM.", + "Words come from SealedSpan.words / the live dump on pcm_time.", + "Refuse reports that reconstruct a timeline from tokens.", + ), + Self::LexiconCorrected => spec( + self, + QualityIssueFamily::OverlayHighlight, + FindingSeverity::Note, + FindingTarget::LexiconTune, + "A lexicon rewrite already landed on committed text.", + "The before/after pair is absent from lexicon.custom.jsonl.", + "Keep the rule if the next take still needs it; drop if it over-fires.", + ), + Self::SpeechGap => spec( + self, + QualityIssueFamily::OverlayHighlight, + FindingSeverity::P2, + FindingTarget::EngineCode, + "Silero heard speech and no engine word landed in the span.", + "A word sample range overlaps the speech range, or Silero was wrong.", + "Fill the pustka with Layer 1 ReplaceRange; do not invent into silence.", + ), + Self::LiveOnly => spec( + self, + QualityIssueFamily::TeacherAttention, + FindingSeverity::P2, + FindingTarget::OperatorReview, + "Daily/live kept a token the proposal dropped.", + "Human reference agrees the token was not said, or HQ/cloud both drop it for a reason.", + "Do not delete the live token automatically. Review Apple residue vs Whisper drop.", + ), + Self::WhisperExcess => spec( + self, + QualityIssueFamily::TeacherAttention, + FindingSeverity::P1, + FindingTarget::EngineCode, + "A proposal inserted tokens absent from the daily document.", + "Human said those tokens, or Silero shows a hole Apple left.", + "Allow hole-fill only inside unsealed allowed spans. Tune Layer 1, not Daily WER.", + ), + Self::Disagreement => spec( + self, + QualityIssueFamily::TeacherAttention, + FindingSeverity::P2, + FindingTarget::LexiconTune, + "Daily and a proposal disagree on a token. This is not accuracy.", + "Human reference picks one side, or a lexicon rule already owns the pair.", + "If it is jargon (Rust/raz), teach the custom lexicon. Do not crown HQ.", + ), + Self::WhisperErrorAtLiveWeakness => spec( + self, + QualityIssueFamily::TeacherAttention, + FindingSeverity::P1, + FindingTarget::EngineCode, + "Proposal disagrees with a human at a locus live also missed.", + "Human text is absent, or live actually carried the human form.", + "This is the Teacher thesis site — gap-fill, not a WER hero score.", + ), + Self::LiveMissWhisperOk => spec( + self, + QualityIssueFamily::TeacherAttention, + FindingSeverity::P2, + FindingTarget::EngineCode, + "Live missed a human token the proposal carried.", + "Human text is absent, or live actually had the form.", + "Classic Apple under-gen. Layer 1 may fill; do not replace the floor.", + ), + Self::VeryLowSpeech => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P1, + FindingTarget::OperatorReview, + "VAD speech share is at/below the very-low-speech floor.", + "speech_pct is above the engine threshold on the same take.", + "Do not trust a long transcript on near-silence.", + ), + Self::PossibleHallucinationLogprob => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P1, + FindingTarget::EngineCode, + "avg_logprob crossed the hallucination ceiling.", + "avg_logprob is above -1.0, or the text is short-whitelist speech.", + "Keep the quality gate; inspect the span before teaching lexicon.", + ), + Self::QualityGateDropped => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P1, + FindingTarget::EngineCode, + "A quality gate dropped text that existed.", + "The gate reason is missing, or the text was short-whitelist speech.", + "Attribute the drop. Empty ≠ silence ≠ failure.", + ), + Self::SileroDroppedTailHallucinations => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::Note, + FindingTarget::EngineCode, + "Silero dropped Whisper segments that sat in trailing silence.", + "Those segments overlap speech frames.", + "This is a successful filter, not a Daily error.", + ), + Self::LocalFinalPassUnavailable => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P2, + FindingTarget::OperatorReview, + "Hold path asked for a local file pass and did not get a verdict.", + "codescribe transcribe produced a verdict for the same WAV.", + "Do not pretend HQ ran.", + ), + Self::CloudFallbackUsed => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P2, + FindingTarget::OperatorReview, + "Cloud was committed after the local path failed.", + "Local produced a usable verdict on the same take.", + "Label the lane degraded. Cloud is not a silent upgrade.", + ), + Self::StreamingPreviewUsedAsVerdict => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P1, + FindingTarget::EngineCode, + "Streaming preview was frozen as the verdict.", + "A final-pass disposition exists.", + "Do not score preview as transcript_sealed.", + ), + Self::UnverifiedStream => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P2, + FindingTarget::OperatorReview, + "Stream text was exposed before final-pass adjudication.", + "An explicit final-pass ran.", + "Keep the preview grey until a bar is crossed.", + ), + Self::CloudPrimaryMissing => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P1, + FindingTarget::OperatorReview, + "Cloud was the primary source and returned empty/error.", + "The cloud call returned non-empty text.", + "Refuse to treat a blank cloud lane as a document.", + ), + Self::AiNoopDetected => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::Note, + FindingTarget::OperatorReview, + "Format ran and emitted the raw input.", + "The formatted text actually differs in content.", + "Do not display Format as Applied.", + ), + Self::FinalPassLengthRegression => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P0, + FindingTarget::EngineCode, + "File final collapsed versus the live streaming floor.", + "final kept ≥40% of stream chars, or stream was below the min floor.", + "Keep the stream. Never auto-replace with the collapse.", + ), + Self::HighCompression => spec( + self, + QualityIssueFamily::Confidence, + FindingSeverity::P2, + FindingTarget::EngineCode, + "Whisper compression_ratio crossed the quality-gate threshold.", + "compression_ratio is below the engine threshold.", + "Pair with logprob. Do not teach lexicon from a compressed dump.", + ), + Self::NoSpeechDetected => spec( + self, + QualityIssueFamily::TranscriptState, + FindingSeverity::Note, + FindingTarget::OperatorReview, + "VAD found no speech. This is not an empty-transcript failure.", + "Silero frames show speech, or the operator spoke.", + "Do not score WER against a no-speech take.", + ), + Self::EmptyTranscript => spec( + self, + QualityIssueFamily::TranscriptState, + FindingSeverity::P1, + FindingTarget::EngineCode, + "Daily document is empty with no no-speech reason on record.", + "A no_speech_reason exists, or Daily has text.", + "Investigate attribution. Do not fill with HQ automatically.", + ), + Self::RawFinalRewrite => spec( + self, + QualityIssueFamily::DeliveryGate, + FindingSeverity::P1, + FindingTarget::EngineCode, + "Raw→final character rewrite crossed the delivery gate.", + "shape_only punctuation, or diff_ratio below QUALITY_GATE_DIFF_RATIO.", + "Inspect Format/lexicon. Do not ship a silent rewrite.", + ), + Self::LossyStreamDrops => spec( + self, + QualityIssueFamily::DeliveryGate, + FindingSeverity::P1, + FindingTarget::EngineCode, + "Stream drop_ratio crossed the lossy threshold.", + "dropped_chunks/input_chunks is below QUALITY_GATE_DROP_RATIO.", + "Fix the stream. Do not blame STT wording.", + ), + Self::HeavyCorrectionPressure => spec( + self, + QualityIssueFamily::DeliveryGate, + FindingSeverity::P2, + FindingTarget::OperatorReview, + "Backspace/correction ratio crossed the delivery gate.", + "correction_ratio is below QUALITY_GATE_CORRECTION_RATIO.", + "This is operator pressure, not a WER score.", + ), + Self::SemanticMeaningChange => spec( + self, + QualityIssueFamily::DeliveryGate, + FindingSeverity::P1, + FindingTarget::EngineCode, + "MiniLM cosine says Format changed meaning while length stayed similar.", + "semantic_cosine is None (too short) or above the calibrated floor.", + "Quarantine Format. Daily/raw stays the document.", + ), + Self::SilenceCorpusResidue => spec( + self, + QualityIssueFamily::WhisperFilter, + FindingSeverity::P1, + FindingTarget::EngineCode, + "A proposal matches Whisper silence-corpus residue Daily does not have.", + "Daily also contains the phrase, or Silero shows real speech there.", + "Drop the residue. Do not teach it into the lexicon.", + ), + Self::WordRateAnomaly => spec( + self, + QualityIssueFamily::WhisperFilter, + FindingSeverity::P2, + FindingTarget::EngineCode, + "Words/sec exceeded MAX_WORDS_PER_SEC — clock or dump anomaly.", + "Rate is ≤ 5 w/s over a sample with ≥6 words.", + "Same family as clock-lie. Do not treat as fluent speech.", + ), + Self::VadDegraded => spec( + self, + QualityIssueFamily::WhisperFilter, + FindingSeverity::P2, + FindingTarget::EngineCode, + "VAD predict_errors / unavailable_frames fired on the batch.", + "vad_degraded warning is absent on a clean replay.", + "Do not trust speech_gap / silence calls on a degraded VAD batch.", + ), + Self::ShortUtteranceDrop => spec( + self, + QualityIssueFamily::WhisperFilter, + FindingSeverity::Note, + FindingTarget::OperatorReview, + "A sub-0.5s low-confidence clip was dropped as a click/breath.", + "Duration ≥ 0.5s or Silero speech_prob ≥ 0.55.", + "Whitelist short Polish speech (tak/nie/no) must never hit this.", + ), + Self::HqTreatedAsDocument => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P0, + FindingTarget::LabJudge, + "The judge treated candle HQ as the document / WER reference.", + "roles.candle is human_triggered_proposal and WER is a footnote.", + "Stop scoring Daily as if it must chase HQ. HQ is a button-only proposal.", + ), + Self::CloudTreatedAsDocument => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P0, + FindingTarget::LabJudge, + "The judge treated cloud :8444 as the document.", + "roles.cloud is human_triggered_proposal.", + "Cloud file is a proposal. Daily remains the session document.", + ), + Self::WerPromotedToDocumentScore => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P0, + FindingTarget::LabJudge, + "WER was presented as the quality score of the live engine.", + "WER sits behind wer_is_footnote and a Seal Atlas / findings payload.", + "Demote the hero WER. Findings first.", + ), + Self::OmittedProgrammingVocabulary => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P1, + FindingTarget::LabJudge, + "Cloud :8444 ran without vocabulary=programming (or explicit off).", + "Multipart/live config carries vocabulary=programming, or off for an unbiased bench.", + "Send the product domain token. Omitting it is not a silent default.", + ), + Self::LastSessionPairedWithLiveOverlay => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P0, + FindingTarget::LabJudge, + "Live Daily was scored against last_session.wav from the previous take.", + "Live snapshots use this take's wav_path/audio_path, never last_session.wav.", + "Refuse the compare. Fake 90%+ WER is not a finding against Daily.", + ), + Self::LeftoverWebsocketPolarity => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P1, + FindingTarget::LabJudge, + "first_divergence still labels sides websocket/codescribe regardless of lane.", + "Divergence keys are reference/hypothesis only.", + "Delete the leftover polarity. It inverts HQ vs Daily.", + ), + Self::ProposalAgreementMisreadAsAccuracy => spec( + self, + QualityIssueFamily::JudgeHygiene, + FindingSeverity::P1, + FindingTarget::LabJudge, + "Agreement with a proposal was reported as accuracy against what was said.", + "No human/corpus reference was claimed, and roles stay proposal.", + "Call it proposal_agreement. Accuracy requires a human transcript.", + ), + } + } +} + +const fn spec( + kind: QualityIssueKind, + family: QualityIssueFamily, + default_severity: FindingSeverity, + default_target: FindingTarget, + what: &'static str, + falsifier: &'static str, + action: &'static str, +) -> QualityIssueSpec { + QualityIssueSpec { + kind, + family, + default_severity, + default_target, + what, + falsifier, + action, + } +} + +/// Every catalog row, in [`QualityIssueKind::ALL`] order. +pub fn quality_issue_catalog() -> Vec { + QualityIssueKind::ALL + .iter() + .copied() + .map(QualityIssueKind::spec) + .collect() +} + +/// Wire ids for the Voice Lab lockstep list. +pub fn quality_issue_kind_ids() -> Vec<&'static str> { + QualityIssueKind::ALL + .iter() + .copied() + .map(QualityIssueKind::as_str) + .collect() +} + +/// Classify one take. Missing evidence does not invent a hit. +pub fn classify_take_findings(evidence: &TakeQualityEvidence) -> SupervisorReport { + let mut findings = Vec::new(); + findings.extend(hygiene_findings(evidence)); + findings.extend(count_findings(evidence)); + findings.extend(flag_findings(evidence)); + findings.extend(empty_daily_findings(evidence)); + findings.extend(residue_findings(evidence)); + findings.extend(attention_findings( + &evidence.daily_text, + &evidence.hq_text, + "hq", + )); + findings.extend(attention_findings( + &evidence.daily_text, + &evidence.cloud_text, + "cloud", + )); + + SupervisorReport { + schema: SUPERVISOR_FINDINGS_SCHEMA.to_string(), + roles: SupervisorRoles { + daily: ReportSurfaceRole::SessionDocument, + candle: ReportSurfaceRole::HumanTriggeredProposal, + cloud: ReportSurfaceRole::HumanTriggeredProposal, + }, + findings, + catalog_ids: quality_issue_kind_ids() + .into_iter() + .map(str::to_string) + .collect(), + wer_is_footnote: true, + } +} + +fn hygiene_findings(evidence: &TakeQualityEvidence) -> Vec { + let mut out = Vec::new(); + if evidence.treats_hq_as_document { + out.push(finding( + QualityIssueKind::HqTreatedAsDocument, + EvidenceGrade::Strong, + "Judge scored Daily against candle HQ as if HQ were the document.".into(), + )); + out.push(finding( + QualityIssueKind::WerPromotedToDocumentScore, + EvidenceGrade::Strong, + "Daily-vs-HQ WER was used as the live-engine quality score.".into(), + )); + out.push(finding( + QualityIssueKind::ProposalAgreementMisreadAsAccuracy, + EvidenceGrade::Strong, + "Proposal agreement was reported as accuracy against what was said.".into(), + )); + } + if evidence.leftover_websocket_polarity { + out.push(finding( + QualityIssueKind::LeftoverWebsocketPolarity, + EvidenceGrade::Strong, + "Divergence still carries websocket/codescribe keys that invert lane names.".into(), + )); + } + if evidence.snapshot_live && evidence.audio_from_last_session { + out.push(finding( + QualityIssueKind::LastSessionPairedWithLiveOverlay, + EvidenceGrade::Strong, + "Live overlay text was paired with last_session.wav from another take.".into(), + )); + } + if evidence.cloud_ran && !vocabulary_is_honest(evidence.vocabulary.as_deref()) { + let got = evidence.vocabulary.as_deref().unwrap_or(""); + out.push(finding( + QualityIssueKind::OmittedProgrammingVocabulary, + EvidenceGrade::Strong, + format!( + "Cloud :8444 ran with vocabulary={got}; product default is {PROGRAMMING_VOCABULARY} (or explicit {VOCABULARY_OFF})." + ), + )); + } + out +} + +fn vocabulary_is_honest(value: Option<&str>) -> bool { + matches!( + value.map(str::trim), + Some(PROGRAMMING_VOCABULARY) | Some(VOCABULARY_OFF) + ) +} + +fn count_findings(evidence: &TakeQualityEvidence) -> Vec { + let mut out = Vec::new(); + if evidence.clock_lie_count > 0 { + out.push(finding( + QualityIssueKind::ClockLie, + EvidenceGrade::Strong, + format!( + "{} sealed span(s) exceed CLOCK_LIE_CHARS_PER_SEC.", + evidence.clock_lie_count + ), + )); + } + if evidence.speech_gap_count > 0 { + out.push(finding( + QualityIssueKind::SpeechGap, + EvidenceGrade::Strong, + format!( + "{} Silero speech span(s) have no overlapping engine word.", + evidence.speech_gap_count + ), + )); + } + out +} + +fn flag_findings(evidence: &TakeQualityEvidence) -> Vec { + evidence + .confidence_flags + .iter() + .filter_map(|flag| kind_for_flag(flag).map(|kind| (flag.as_str(), kind))) + .map(|(flag, kind)| { + finding( + kind, + EvidenceGrade::Strong, + format!("Take carries confidence flag `{flag}`."), + ) + }) + .collect() +} + +fn kind_for_flag(flag: &str) -> Option { + let token = flag.trim(); + if token.starts_with("silero_dropped_tail_hallucinations") { + return Some(QualityIssueKind::SileroDroppedTailHallucinations); + } + match token { + "very_low_speech" => Some(QualityIssueKind::VeryLowSpeech), + "possible_hallucination_logprob" => Some(QualityIssueKind::PossibleHallucinationLogprob), + "quality_gate_dropped" => Some(QualityIssueKind::QualityGateDropped), + "local_final_pass_unavailable" => Some(QualityIssueKind::LocalFinalPassUnavailable), + "cloud_fallback_used" => Some(QualityIssueKind::CloudFallbackUsed), + "streaming_preview_used_as_verdict" => { + Some(QualityIssueKind::StreamingPreviewUsedAsVerdict) + } + "unverified_stream" => Some(QualityIssueKind::UnverifiedStream), + "cloud_primary_missing" => Some(QualityIssueKind::CloudPrimaryMissing), + "ai_noop_detected" => Some(QualityIssueKind::AiNoopDetected), + "final_pass_length_regression" => Some(QualityIssueKind::FinalPassLengthRegression), + "high_compression" | "low_logprob" => { + if token == "high_compression" { + Some(QualityIssueKind::HighCompression) + } else { + Some(QualityIssueKind::PossibleHallucinationLogprob) + } + } + _ => None, + } +} + +fn empty_daily_findings(evidence: &TakeQualityEvidence) -> Vec { + if !evidence.daily_text.trim().is_empty() { + return Vec::new(); + } + if evidence + .confidence_flags + .iter() + .any(|flag| flag == "no_speech_detected") + { + return vec![finding( + QualityIssueKind::NoSpeechDetected, + EvidenceGrade::Medium, + "Daily is empty and the take is flagged no_speech_detected.".into(), + )]; + } + vec![finding( + QualityIssueKind::EmptyTranscript, + EvidenceGrade::Medium, + "Daily session document is empty with no no-speech reason.".into(), + )] +} + +fn residue_findings(evidence: &TakeQualityEvidence) -> Vec { + let daily = evidence.daily_text.to_ascii_lowercase(); + let mut out = Vec::new(); + for (lane, text) in [ + ("hq", evidence.hq_text.as_str()), + ("cloud", evidence.cloud_text.as_str()), + ] { + let lowered = text.to_ascii_lowercase(); + if let Some(phrase) = SILENCE_CORPUS_RESIDUE + .iter() + .copied() + .find(|phrase| lowered.contains(phrase) && !daily.contains(phrase)) + { + let kind = if evidence.daily_text.trim().is_empty() { + QualityIssueKind::HallucinateIntoSilence + } else { + QualityIssueKind::SilenceCorpusResidue + }; + let mut row = finding( + kind, + EvidenceGrade::Medium, + format!( + "`{lane}` proposal contains silence-corpus residue `{phrase}` absent from Daily." + ), + ); + row.span = Some(FindingSpan { + daily_token: None, + proposal_token: Some(phrase.to_string()), + proposal_lane: Some(lane.to_string()), + word_index: None, + }); + out.push(row); + } + } + out +} + +fn attention_findings(daily: &str, proposal: &str, lane: &str) -> Vec { + if daily.trim().is_empty() || proposal.trim().is_empty() { + return Vec::new(); + } + let live = tokenize(daily); + let other = tokenize(proposal); + let ops = align_words(&live, &other); + let mut out = Vec::new(); + let mut kept = 0usize; + let mut truncated = 0usize; + for op in ops { + let row = match op { + AlignOp::Equal { .. } => None, + AlignOp::DeleteA { a } => Some(attention_row( + QualityIssueKind::LiveOnly, + lane, + Some(live[a].surface.clone()), + None, + Some(a + 1), + format!( + "Daily kept «{}»; `{lane}` proposal has no counterpart.", + live[a].surface + ), + )), + AlignOp::InsertB { b } => Some(attention_row( + QualityIssueKind::WhisperExcess, + lane, + None, + Some(other[b].surface.clone()), + Some(b + 1), + format!( + "`{lane}` proposal inserted «{}» absent from the Daily document.", + other[b].surface + ), + )), + AlignOp::Substitute { a, b } => Some(attention_row( + QualityIssueKind::Disagreement, + lane, + Some(live[a].surface.clone()), + Some(other[b].surface.clone()), + Some(a + 1), + format!( + "Daily «{}» vs `{lane}` proposal «{}» — agreement, not accuracy.", + live[a].surface, other[b].surface + ), + )), + }; + if let Some(row) = row { + if kept < MAX_ATTENTION_FINDINGS { + out.push(row); + kept += 1; + } else { + truncated += 1; + } + } + } + if truncated > 0 { + out.push(finding( + QualityIssueKind::Disagreement, + EvidenceGrade::Medium, + format!( + "{truncated} further `{lane}` attention loci truncated at {MAX_ATTENTION_FINDINGS}." + ), + )); + } + out +} + +fn attention_row( + kind: QualityIssueKind, + lane: &str, + daily_token: Option, + proposal_token: Option, + word_index: Option, + claim: String, +) -> SupervisorFinding { + let mut row = finding(kind, EvidenceGrade::Medium, claim); + row.span = Some(FindingSpan { + daily_token, + proposal_token, + proposal_lane: Some(lane.to_string()), + word_index, + }); + row +} + +fn finding( + kind: QualityIssueKind, + evidence_grade: EvidenceGrade, + claim: String, +) -> SupervisorFinding { + let spec = kind.spec(); + SupervisorFinding { + kind, + family: spec.family, + severity: spec.default_severity, + evidence_grade, + target: spec.default_target, + claim, + falsifier: spec.falsifier.to_string(), + action: spec.action.to_string(), + span: None, + } +} + +/// Column roles the judge must not invert. Re-export of the contract helper +/// so Voice Lab lockstep tests and this classifier share one function. +pub fn lane_surface_role(column: &str) -> Option { + surface_role(column) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::quality::engine_contract::{ENGINE_CONTRACT_DOC, is_clock_lie}; + use std::collections::HashSet; + + #[test] + fn catalog_covers_every_kind_once() { + let ids = quality_issue_kind_ids(); + let unique: HashSet<_> = ids.iter().copied().collect(); + assert_eq!(ids.len(), QualityIssueKind::ALL.len()); + assert_eq!(unique.len(), QualityIssueKind::ALL.len()); + for kind in QualityIssueKind::ALL { + let spec = kind.spec(); + assert_eq!(spec.kind, *kind); + assert_eq!(kind.as_str(), spec.kind.as_str()); + assert!(!spec.what.is_empty(), "{kind:?} missing what"); + assert!(!spec.falsifier.is_empty(), "{kind:?} missing falsifier"); + assert!(!spec.action.is_empty(), "{kind:?} missing action"); + } + } + + #[test] + fn hq_and_cloud_columns_stay_proposals() { + assert_eq!( + lane_surface_role("hq"), + Some(ReportSurfaceRole::HumanTriggeredProposal) + ); + assert_eq!( + lane_surface_role("cloud"), + Some(ReportSurfaceRole::HumanTriggeredProposal) + ); + assert_eq!( + lane_surface_role("delivered"), + Some(ReportSurfaceRole::SessionDocument) + ); + let report = classify_take_findings(&TakeQualityEvidence::default()); + assert_eq!( + report.roles.candle, + ReportSurfaceRole::HumanTriggeredProposal + ); + assert_eq!( + report.roles.cloud, + ReportSurfaceRole::HumanTriggeredProposal + ); + assert_eq!(report.roles.daily, ReportSurfaceRole::SessionDocument); + assert!(report.wer_is_footnote); + assert_eq!(report.schema, SUPERVISOR_FINDINGS_SCHEMA); + } + + #[test] + fn lying_judge_evidence_emits_hygiene_findings() { + let report = classify_take_findings(&TakeQualityEvidence { + daily_text: "podpinamy websocket na żywo".into(), + hq_text: "podpinamy sok na żywo".into(), + snapshot_live: true, + audio_from_last_session: true, + cloud_ran: true, + vocabulary: None, + treats_hq_as_document: true, + leftover_websocket_polarity: true, + ..TakeQualityEvidence::default() + }); + let kinds: HashSet<_> = report.findings.iter().map(|row| row.kind).collect(); + for required in [ + QualityIssueKind::HqTreatedAsDocument, + QualityIssueKind::WerPromotedToDocumentScore, + QualityIssueKind::ProposalAgreementMisreadAsAccuracy, + QualityIssueKind::LeftoverWebsocketPolarity, + QualityIssueKind::LastSessionPairedWithLiveOverlay, + QualityIssueKind::OmittedProgrammingVocabulary, + QualityIssueKind::Disagreement, + ] { + assert!( + kinds.contains(&required), + "missing {required:?} in {kinds:?}" + ); + } + let disagreement = report + .findings + .iter() + .find(|row| row.kind == QualityIssueKind::Disagreement) + .expect("disagreement"); + assert!( + disagreement.claim.contains("agreement, not accuracy"), + "{}", + disagreement.claim + ); + assert_eq!(disagreement.target, FindingTarget::LexiconTune); + } + + #[test] + fn honest_take_does_not_promote_wer() { + let report = classify_take_findings(&TakeQualityEvidence { + daily_text: "podpinamy websocket na żywo".into(), + hq_text: "podpinamy websocket na żywo".into(), + cloud_text: "podpinamy websocket na żywo".into(), + cloud_ran: true, + vocabulary: Some(PROGRAMMING_VOCABULARY.into()), + treats_hq_as_document: false, + leftover_websocket_polarity: false, + snapshot_live: false, + audio_from_last_session: true, + ..TakeQualityEvidence::default() + }); + let kinds: HashSet<_> = report.findings.iter().map(|row| row.kind).collect(); + assert!(!kinds.contains(&QualityIssueKind::HqTreatedAsDocument)); + assert!(!kinds.contains(&QualityIssueKind::WerPromotedToDocumentScore)); + assert!(!kinds.contains(&QualityIssueKind::OmittedProgrammingVocabulary)); + assert!(!kinds.contains(&QualityIssueKind::LastSessionPairedWithLiveOverlay)); + assert!(!kinds.contains(&QualityIssueKind::Disagreement)); + } + + #[test] + fn vocabulary_off_is_explicit_unbiased_bench() { + let report = classify_take_findings(&TakeQualityEvidence { + cloud_ran: true, + vocabulary: Some(VOCABULARY_OFF.into()), + daily_text: "ok".into(), + hq_text: "ok".into(), + ..TakeQualityEvidence::default() + }); + assert!( + report + .findings + .iter() + .all(|row| row.kind != QualityIssueKind::OmittedProgrammingVocabulary) + ); + } + + #[test] + fn empty_daily_plus_thanks_for_watching_is_silence_hallucination() { + let report = classify_take_findings(&TakeQualityEvidence { + daily_text: String::new(), + hq_text: "Thanks for watching".into(), + ..TakeQualityEvidence::default() + }); + assert!( + report + .findings + .iter() + .any(|row| row.kind == QualityIssueKind::HallucinateIntoSilence) + ); + assert!( + report + .findings + .iter() + .any(|row| row.kind == QualityIssueKind::EmptyTranscript) + ); + } + + #[test] + fn clock_lie_helper_is_the_same_engine_function() { + assert!(is_clock_lie(41, 0.10)); + let report = classify_take_findings(&TakeQualityEvidence { + daily_text: "ten".into(), + clock_lie_count: 1, + ..TakeQualityEvidence::default() + }); + assert!( + report + .findings + .iter() + .any(|row| row.kind == QualityIssueKind::ClockLie) + ); + } + + #[test] + fn confidence_flag_maps_to_typed_kind() { + let report = classify_take_findings(&TakeQualityEvidence { + daily_text: "ok".into(), + confidence_flags: vec![ + "possible_hallucination_logprob".into(), + "silero_dropped_tail_hallucinations:2".into(), + ], + ..TakeQualityEvidence::default() + }); + let kinds: HashSet<_> = report.findings.iter().map(|row| row.kind).collect(); + assert!(kinds.contains(&QualityIssueKind::PossibleHallucinationLogprob)); + assert!(kinds.contains(&QualityIssueKind::SileroDroppedTailHallucinations)); + } + + #[test] + fn contract_doc_names_the_supervisor_lock() { + let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join(".."); + let body = std::fs::read_to_string(root.join(ENGINE_CONTRACT_DOC)) + .unwrap_or_else(|err| panic!("{} must exist: {err}", ENGINE_CONTRACT_DOC)); + for needle in [ + SUPERVISOR_FINDINGS_SCHEMA, + "hq_treated_as_document", + "omitted_programming_vocabulary", + "last_session_paired_with_live_overlay", + ] { + assert!( + body.contains(needle), + "{ENGINE_CONTRACT_DOC} missing {needle:?}" + ); + } + } +} diff --git a/core/stt/mod.rs b/core/stt/mod.rs index cacfdcb3..e54250dc 100644 --- a/core/stt/mod.rs +++ b/core/stt/mod.rs @@ -32,6 +32,8 @@ pub mod apple_stt; pub mod onnx_adapter; /// Whisper sentence shape onto committed Apple words — punctuation only. pub mod punctuation_transplant; +/// Explicit cloud/loopback STT topic token. Client-owned; never from audio. +pub mod request_vocabulary; /// Serialized STT request scheduler: live, commit, and refine lanes with /// supersede semantics for stale requests and thermal-pressure backoff. pub mod scheduler; diff --git a/core/stt/request_vocabulary.rs b/core/stt/request_vocabulary.rs new file mode 100644 index 00000000..e3879620 --- /dev/null +++ b/core/stt/request_vocabulary.rs @@ -0,0 +1,89 @@ +//! Explicit STT topic token. The client names the domain; audio never does. +//! +//! Codescribe takes send `programming`. Official OpenAI file audio does not +//! accept this field, so that host stays omitted. Missing field means no +//! dictionary bias. The client does not classify audio to pick a token. + +use reqwest::Url; + +/// Codescribe product domain for hosts that accept a topic token. +pub const CODESCRIBE_STT_VOCABULARY: &str = "programming"; + +/// Multipart form field the loopback/Libraxis file worker reads. +/// +/// Overlay Retranscribe `cloud:` / `last_session.wav` / `:8444` must attach +/// this field. Official OpenAI file audio must not. +pub const VOCABULARY_FORM_FIELD: &str = "vocabulary"; + +/// Topic token to send on one outbound STT URL, if that host accepts one. +/// +/// `None` for official OpenAI (unknown field) and unparseable URLs. Every +/// other Codescribe take is `programming`. Never inferred from audio. +pub fn codescribe_stt_vocabulary(endpoint: &str) -> Option<&'static str> { + let host = Url::parse(endpoint).ok().and_then(|url| { + url.host_str() + .map(|host| host.trim_matches(['[', ']']).to_owned()) + })?; + if host.eq_ignore_ascii_case("api.openai.com") { + return None; + } + Some(CODESCRIBE_STT_VOCABULARY) +} + +/// Multipart `(field, value)` for one file STT URL, if that host accepts one. +pub fn codescribe_stt_vocabulary_form_part(endpoint: &str) -> Option<(&'static str, &'static str)> { + codescribe_stt_vocabulary(endpoint).map(|value| (VOCABULARY_FORM_FIELD, value)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn codescribe_sends_programming_except_official_openai() { + assert_eq!( + codescribe_stt_vocabulary("http://127.0.0.1:8444/v1/audio/transcriptions"), + Some("programming") + ); + assert_eq!( + codescribe_stt_vocabulary("http://127.0.0.1:8088/v1/audio/transcriptions"), + Some("programming") + ); + assert_eq!( + codescribe_stt_vocabulary("ws://127.0.0.1:8446/v1/audio/transcribe"), + Some("programming") + ); + assert_eq!( + codescribe_stt_vocabulary("https://api.libraxis.cloud/v1/audio/transcriptions"), + Some("programming") + ); + assert_eq!( + codescribe_stt_vocabulary("wss://api.libraxis.cloud/v1/audio/transcribe"), + Some("programming") + ); + assert_eq!( + codescribe_stt_vocabulary("https://stt.example.test/v1/audio/transcriptions"), + Some("programming") + ); + assert_eq!( + codescribe_stt_vocabulary("https://api.openai.com/v1/audio/transcriptions"), + None + ); + assert_eq!(codescribe_stt_vocabulary("not a url"), None); + assert_eq!( + codescribe_stt_vocabulary_form_part("http://127.0.0.1:8444/v1/audio/transcriptions"), + Some((VOCABULARY_FORM_FIELD, "programming")) + ); + assert_eq!( + codescribe_stt_vocabulary_form_part("https://api.openai.com/v1/audio/transcriptions"), + None + ); + } + + #[test] + fn token_is_not_chosen_from_audio() { + assert_eq!(CODESCRIBE_STT_VOCABULARY, "programming"); + assert_ne!(CODESCRIBE_STT_VOCABULARY, "veterinary"); + assert_ne!(CODESCRIBE_STT_VOCABULARY, "off"); + } +} diff --git a/core/stt/tail_patcher/mod.rs b/core/stt/tail_patcher/mod.rs index 8b2c9c22..03c64170 100644 --- a/core/stt/tail_patcher/mod.rs +++ b/core/stt/tail_patcher/mod.rs @@ -52,7 +52,12 @@ //! - **Conservative by default.** If the diff distance exceeds //! [`TailPatchConfig::max_change_ratio`], the whole patch is dropped //! ([`TailPatchOutcome::Skipped`]) and Layer 0 output stands unchanged — -//! "don't patch if uncertain". +//! "don't patch if uncertain". Three measured same-utterance repairs +//! bypass that cap without lifting it: a substitution inside +//! [`TailPatchConfig::small_edit_token_floor`], a high-coverage digit-run +//! / trailing-fill expansion, and an aligned sentence rewrite (coverage +//! ≥ 0.60, similar length, substitutions only). Inserts stay on the +//! under-commit path. A 0-match rewrite is still wholesale divergence. //! //! # Scope of this cut (v1) //! @@ -949,8 +954,32 @@ pub fn compute_tail_patch_with_context( g.committed.len() == 1 && token_is_digit_run(&c_tokens[g.committed.start]) }); let high_coverage_expansion = almost_covered && modest_extra && expansion_groups_ok; + // Same utterance, fuller wording. Operator 2026-08-18: Whisper is already + // fast enough to swap those sentences in the background. The 0.50 cap was + // treating "we still share most tokens" as a dump. Wholesale divergence + // (few or no anchors, or a 3× longer decode) still skips below. + let coverage = matches.len() as f64 / c_tokens.len() as f64; + let similar_length = r_tokens.len() >= c_tokens.len().saturating_sub(2) + && r_tokens.len() <= c_tokens.len().saturating_add(c_tokens.len() / 2 + 3); + let rewrite_groups_ok = groups.iter().all(|g| { + if g.retranscribed.is_empty() { + return true; + } + // Inserts stay on under-commit. This bypass only rewrites spans + // already on the canvas, and never turns one token into a dump. + !g.committed.is_empty() && g.retranscribed.len() <= g.committed.len().saturating_add(2) + }); + let aligned_sentence_rewrite = shares_anchor + && c_tokens.len() >= 4 + && coverage >= 0.60 + && similar_length + && rewrite_groups_ok; let ratio = changed as f64 / c_tokens.len() as f64; - if ratio > cfg.max_change_ratio && !small_substitution_fix && !high_coverage_expansion { + if ratio > cfg.max_change_ratio + && !small_substitution_fix + && !high_coverage_expansion + && !aligned_sentence_rewrite + { // Before the cap discards this: is the canvas starved rather than // wrong? The bounded diff was never an instrument for measuring lost // speech, and using it as one is what threw the recovered 104 s / 107 s @@ -1662,6 +1691,29 @@ mod tests { assert_eq!(outcome, TailPatchOutcome::NoChange); } + /// High-overlap rewrite of the same utterance applies even when more than + /// half the tokens move. Chopped Apple + faster Whisper is the live job. + #[test] + fn aligned_sentence_rewrite_applies_when_most_words_still_match() { + let cfg = TailPatchConfig::default(); + let committed = "ala ma czarnego kota i białego psa dzisiaj w domu"; + let retranscribed = "ala ma dużego rudego kota oraz małego psa dzisiaj u siebie domu"; + let outcome = compute_tail_patch(committed, retranscribed, 31, &cfg); + let applied = apply_all(committed, &outcome); + assert!( + !matches!(outcome, TailPatchOutcome::Skipped { .. }), + "aligned rewrite must not hit the 0.50 cap, got {outcome:?}" + ); + assert!( + applied.contains("dużego") || applied.contains("rudego") || applied.contains("oraz"), + "Whisper wording must land, got {applied:?}" + ); + assert!( + applied.contains("ala") && applied.contains("domu"), + "anchors must stay, got {applied:?}" + ); + } + /// The floor is a small-edit budget, not a hole in the divergence guard: a /// wholesale rewrite of a short utterance still skips. #[test] diff --git a/core/stt/tail_provider.rs b/core/stt/tail_provider.rs index 82bed5e8..07eb80dd 100644 --- a/core/stt/tail_provider.rs +++ b/core/stt/tail_provider.rs @@ -1088,11 +1088,16 @@ impl TailProvider for RemoteTailProvider { let file = Part::bytes(wav) .file_name("tail-window.wav") .mime_str("audio/wav")?; - let form = Form::new() + let mut form = Form::new() .part("file", file) .text("model", model.clone()) .text("language", language.to_string()) .text("response_format", "verbose_json"); + if let Some((field, value)) = + crate::stt::request_vocabulary::codescribe_stt_vocabulary_form_part(&self.endpoint) + { + form = form.text(field, value.to_string()); + } let http_request = Client::builder() .timeout(REMOTE_REQUEST_TIMEOUT) .connect_timeout(SIDECAR_CONNECT_TIMEOUT) @@ -1357,6 +1362,22 @@ mod tests { ); } + #[test] + fn remote_tail_topic_follows_codescribe_product_not_audio() { + assert_eq!( + crate::stt::request_vocabulary::codescribe_stt_vocabulary( + "http://127.0.0.1:8444/v1/audio/transcriptions" + ), + Some("programming") + ); + assert_eq!( + crate::stt::request_vocabulary::codescribe_stt_vocabulary( + "https://api.openai.com/v1/audio/transcriptions" + ), + None + ); + } + #[test] fn file_probe_endpoint_inverts_known_live_sockets() { assert_eq!( diff --git a/docs/DELIVERY_ROUTE.md b/docs/DELIVERY_ROUTE.md index 109707f8..3d688fee 100644 --- a/docs/DELIVERY_ROUTE.md +++ b/docs/DELIVERY_ROUTE.md @@ -11,10 +11,16 @@ 2. **`resolve_delivery_route` is the only function that picks a destination.** Auto-paste, overlay Insert, and To Agent consult it. They do not invent a second king. -3. **Codescribe is never a legal Cmd+V target.** A latched self-app (Agent - composer, overlay, settings) stays on the Orient canvas or goes to the - Agent composer as a first-class message — never as a tagged paste into - ourselves. That is how `` stopped landing in chat. +3. **The overlay canvas is never a legal Cmd+V target.** Caret in our panel + → park Paste Here. The Agent window, Alacritty/Zellij (vc-terminal), + Notes, and every other caret **are** legal ambulances. Assistive still + delivers as a first-class Agent message — that is a different intent, + not a ban on pasting into the Agent window. +4. **Clipboard is borrowed, never stolen.** We may overwrite `NSPasteboard` + for a real Cmd+V. We must restore what the user had. If auto-paste cannot + land, we lose neither: restore the system clipboard, park the transcript + in our buffer (⌘⌥V). Explicit overlay **Copy** is the only verb that + writes the pasteboard on purpose and leaves it. ## Intent → route @@ -22,17 +28,17 @@ | ----------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `AgentVoice` | Double Right Option / assistive hold | `AgentComposer` | | `OverlayToAgent` | overlay **To Agent** | `AgentComposer` | -| `OrientDictation` | Hold Fn / Globe | `ClipboardPaste` if auto-paste + latched foreign app; else `OrientCanvas` | +| `OrientDictation` | Hold Fn / Globe | `ClipboardPaste` if auto-paste; `OrientCanvas` if overlay caret / no auto-paste | | `OrientFormat` | Double Left Option | same as dictation | -| `OverlayInsert` | overlay Insert / defer | `ClipboardPaste` if the latched target is a foreign app; `DeferredInsert` if the latched target (or the caret) is Codescribe | +| `OverlayInsert` | overlay Insert / defer | `ClipboardPaste` into the latched caret (Agent, Alacritty, …); `DeferredInsert` only when the overlay canvas holds the caret | | `NotesOnly` | save-only notes | `ArchiveOnly` | Vetoes that keep Orient off the paste gun: empty / no-speech, live-stream -session, quality-commit pending, latched target is Codescribe. +session, quality-commit pending, overlay canvas holds the caret. Explicit overlay clicks do **not** inherit the live-stream or quality-commit -vetoes. The user asked to insert now. Codescribe as the latched target still -refuses Cmd+V into ourselves and arms Paste Here instead. +vetoes. The user asked to insert now. Overlay caret still refuses Cmd+V into +the canvas and arms Paste Here instead. Agent / Alacritty still get Cmd+V. `paste_text_from_overlay` and `defer_text_from_overlay` consult `resolve_delivery_route`. They do not pick a destination on their own. diff --git a/docs/ENV_REGISTRY.toml b/docs/ENV_REGISTRY.toml index de6dfa87..ad5a3549 100644 --- a/docs/ENV_REGISTRY.toml +++ b/docs/ENV_REGISTRY.toml @@ -1752,3 +1752,24 @@ type = "usize" reload = "restart" category = "assistive" description = "Max characters captured from recent selection for voice chat" + +[vars.VOICE_LAB_REPO_URL] +default = "" +type = "string" +reload = "rebuild" +category = "install" +description = "Optional override of the org Voice Lab git URL. Empty = probe https://github.com/vetcoders/voice-lab.git and git@github.com:vetcoders/voice-lab.git, HTTPS first when gh git_protocol is https (this laptop), SSH first when it is ssh (Monika). Only those two org URLs are accepted." + +[vars.CODESCRIBE_VOICE_LAB_SRC] +default = "" +type = "string" +reload = "rebuild" +category = "install" +description = "Existing Voice Lab checkout used by make install-app instead of cloning (~/.codescribe/src/voice-lab)" + +[vars.VOICE_LAB_INSTALL_SETTINGS] +default = "" +type = "string" +reload = "rebuild" +category = "install" +description = "Voice Lab settings seed: empty=copy pack if missing and fill empty engine keys; 1=overwrite settings.json; 0=never touch Application Support" diff --git a/docs/HOTKEYS_CONTRACT.md b/docs/HOTKEYS_CONTRACT.md index 193dacf2..3831ef7a 100644 --- a/docs/HOTKEYS_CONTRACT.md +++ b/docs/HOTKEYS_CONTRACT.md @@ -72,7 +72,7 @@ flowchart TB end subgraph Events["📨 HotkeyInput"] - HoldEvent["Hold { Down/Up, hold_mode }"] + HoldEvent["Hold { Down/Up, Raw } / AttachSelection"] ToggleEvent["ToggleNormal / ToggleAssistive"] ShowAgent["ShowAgent"] end @@ -116,21 +116,23 @@ flowchart TB | `Dictation=HoldCtrlShift` | Ctrl+Shift | Alternate hold dictation | | `Dictation=HoldCtrlCmd` | Ctrl+Command | macOS power users | -Assistive hold has one arm gesture: add Shift to the active dictation hold. The -legacy second selection-only combo is released. Arming captures any current -selection immediately and attaches it invisibly when the transcript is sent. +Fn hold-down (Raw) captures any live OS selection once as `{selection_1}`. +Mid-hold Shift (or the configured arm modifier, default Shift, optional +Command) attaches further pulses as `{selection_2..n}`. Neither upgrade +`HoldMode` to Chat, fronts Agent, hides the overlay, or stops the Fn take. + +Fn+Shift from idle is dictation, not Assistive. Voice chat is the Assistive +work-mode binding (default: double-tap Right Option), not Hold Fn+Shift. **Events:** ```rust -HotkeyInput { key_type: Hold, action: Down, hold_mode: Raw } // Fn only -HotkeyInput { key_type: Hold, action: Down, hold_mode: Chat } // Fn+Shift -HotkeyInput { key_type: Hold, action: Up, hold_mode: } // Release +HotkeyEvent::Hold { action: Down, mode: Raw } // Fn, or Fn+Shift from idle + // (controller attaches live selection) +HotkeyEvent::AttachSelection // Shift/Command rising edge mid-hold +HotkeyEvent::Hold { action: Up, mode: Raw } // Release — destination stays Raw ``` -**Mode modifier (default Fn):** Shift → Agent arm while holding Fn. Command no -longer creates a second assistive mode. - **Engine and delivery parity:** Hold and toggle both start `StreamingRecorder::start_event_session` and fan the same `EngineEvent` stream through `PresentationEmitter`, IPC, and telemetry sinks. Their intentional @@ -216,7 +218,7 @@ stateDiagram-v2 IDLE --> CONVERSATION : Conversation Down
(custom binding) REC_HOLD --> BUSY : Hold Up
(Fn released) - REC_HOLD --> REC_HOLD : Shift pressed
(upgrade to assistive) + REC_HOLD --> REC_HOLD : Shift pressed
(attach {selection_N}) REC_TOGGLE --> BUSY : Toggle again CONVERSATION --> IDLE : Conversation Up @@ -444,9 +446,10 @@ const DOUBLE_TAP_INTERVAL_MS: u64 = 200; When `HOLD_EXCLUSIVE=false` (default), modifier variants work out of the box: -- Fn+Shift can start voice chat (default arm modifier; configurable to Cmd in Settings) -- Armed hold with a selection acts on the selection (same arm gesture — default Shift, optional Cmd) -- The unconfigured arm modifier does not arm assistive (W10-B detector truth) +- Shift or Command *during* an already-started Fn hold attaches `{selection_N}` + (default arm modifier Shift; configurable to Cmd in Settings) +- Fn+Shift from idle stays dictation — it is not Assistive and does not front Agent +- The unconfigured arm modifier does not attach (W10-B detector truth) Set `HOLD_EXCLUSIVE=true` when you need stricter isolation: diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md index 8a13696a..9a4fd2f4 100644 --- a/docs/INSTALLATION.md +++ b/docs/INSTALLATION.md @@ -37,17 +37,17 @@ make install This matters because macOS TCC permissions are far more stable with a persistent code-signing identity than with ad-hoc signatures. -Local source installs use Cargo's optimized `local-release` profile and the -checked-in development license verifier. Production DMGs use the distinct -`release` profile, which fails closed unless `CODESCRIBE_LICENSE_PUBLIC_KEY_HEX` -is the real 32-byte Ed25519 public key paired with the production signer. A UUID -is not a license public key. - -`make install-app` bakes Lab (`CSDeveloperSurface=1`) only when both the -Sparkle public key and the production-license public key resolve from -`~/.vibecrafted/secrets/codescribe/` (the same files a real release uses). -A public clone without those files still installs the daily app; Lab stays -off. Production DMGs refuse the bit. +`make install-app` bakes the org public keys so Get license CSK1 +verifies. The key files live in the local developer key pack (see +`scripts/developer-surface-gate.sh`). Production DMGs still use the +`release` profile and fail closed without the production signer public +key. A UUID is not a license public key. + +`make install-app` builds the local-release app and copies it to +`/Applications`. Extra developer-console pieces are resolved from a +private sibling checkout when present; they are not part of the public +source path. A machine that already has `settings.json` keeps it. +Production DMGs do not bake the developer surface. ### Method 3: DMG Distribution (For End Users) diff --git a/docs/STT_CONTRACT.md b/docs/STT_CONTRACT.md index ee2e218a..270925f7 100644 --- a/docs/STT_CONTRACT.md +++ b/docs/STT_CONTRACT.md @@ -70,7 +70,9 @@ resolver, so Settings cannot disagree with delivery. Settings → Test is the multipart file probe (`/v1/audio/transcriptions`) for every OpenAI-compatible host. A stored `wss`/`ws` `…/transcribe` URL is remapped to that file path first; loopback Voice Lab `:8446` becomes `:8444`. It is not a WebSocket -handshake. +handshake. The inverse is also explicit: a loopback file URL on `:8444` +(`http(s)://…/v1/audio/transcriptions`) becomes the live socket on `:8446`. +A generic loopback file URL on another port keeps that port. Transport ownership is equally explicit. Live capture uses a stored Voice Lab WebSocket (`config` → bounded PCM `chunk` → periodic `flush` → `end`) and @@ -79,6 +81,28 @@ streams its normalized events into `PresentationEmitter`. A public HTTPS socket. A complete audio-file multipart request is allowed for Settings → Test and for an explicit retranscribe action (Overlay, Dictionary, or Teacher). +**Domain token (client-owned, 2026-08-18).** Codescribe names the take +`vocabulary=programming` on loopback and Libraxis file/live requests +(multipart field `vocabulary`; JSON alias `request_vocabulary`; live +`session.start` / WS `config`). Official OpenAI file audio omits the field. +Absence means no dictionary bias. The client never classifies audio to pick +`programming` vs another domain. A quality bench that must stay unbiased +sends `off` explicitly — omitting the field is not a silent product default. + +**File retranscribe (2026-08-19).** Overlay Retranscribe `cloud:`, Dictionary +`cloud:`, and any `last_session.wav` upload to remapped loopback `:8444` +(`/v1/audio/transcriptions`) are product file takes. They attach +`vocabulary=programming` so Polish+tech speech can prefer `Rust` over +`raz`. Overlay click-Retranscribe without the menu is Full HQ (local +candle) and stays prompt-free — that pass is not the `:8444` worker. +Verify without a live take: POST `~/.codescribe/last_session.wav` to +`http://127.0.0.1:8444/v1/audio/transcriptions` with `vocabulary=programming`. + +**Format is not transcript authority (2026-08-19).** Overlay Format (LLM +cleanup) may guess a language name, invent tokens, and drop the coda. HQ +compare is Whisper file vs raw Apple, never vs Format. Format is a delivery +style pass, not a second STT engine. + **Dictionary helper (everyone, 2026-08-17):** Settings → Dictionary Retranscribe is an explicit file surface on the row's archived `_raw.{m4a,wav,flac}`. Helper engine follows `speech.engine.asr_mode`: `local_power` → `hq:` (same @@ -204,8 +228,8 @@ file-pass belongs only to explicit retranscribe surfaces. | Call site | When | Function / transport | Engine rule | | --------------------- | ---------------- | ---------------------------------------------- | -------------------------------------------------------- | | Live Layer 0 | during recording | Apple progressive | committed canvas floor | -| Live Layer 1 local | during recording | bounded Whisper windows | gap/tail fill only | -| Live Layer 1 cloud | during recording | Voice Lab WSS | normalized gap/tail fill only | +| Live Layer 1 local | during recording | Whisper on ~5 Apple segments | aligned sentence swap on the joined window | +| Live Layer 1 cloud | during recording | Voice Lab WSS | normalized gap/tail fill; same substitution rule | | Explicit Retranscribe | operator action | local completed-file decode or cloud multipart | may replace the selected artifact, never the live canvas | **This split is the MacGyver fracture:** UI can show Whisper readiness while live is Apple-only and fails closed. diff --git a/docs/THE_ENGINE_CONTRACT.md b/docs/THE_ENGINE_CONTRACT.md index 7df290be..0b7e00ec 100644 --- a/docs/THE_ENGINE_CONTRACT.md +++ b/docs/THE_ENGINE_CONTRACT.md @@ -122,3 +122,36 @@ private HTML under: A report that fails this handshake is invisible in the lab, whatever its WER. The gold take 01 HTML already satisfies it. Qube `Codescribe Quality Report` does not — that title is the thing we are retiring. + +## Supervisor findings + +Source of truth: `core/quality/supervisor.rs`. +Schema: `codescribe-supervisor-findings/v1`. + +Voice Lab three-judge emits a `supervisor` object next to the WER footnote. +Daily is the session document. Candle HQ and cloud `:8444` stay +`HumanTriggeredProposal`. WER is agreement with a proposal, not accuracy. + +A finding is only a finding when it names: + +- `kind` from the engine catalog +- `claim` that can be false +- `falsifier` — what would disprove it +- `action` — engine cut, lexicon tune, lab-judge hygiene, or operator review + +Judge hygiene kinds the lab used to commit (these are P0/P1 when they fire): + +- `hq_treated_as_document` +- `cloud_treated_as_document` +- `wer_promoted_to_document_score` +- `omitted_programming_vocabulary` +- `last_session_paired_with_live_overlay` +- `leftover_websocket_polarity` +- `proposal_agreement_misread_as_accuracy` + +The catalog also names every engine-side class already in the tree (contract +forbiddens, clock-lie, speech_gap, Teacher attention, confidence flags, +delivery gates, Whisper silence residue). Missing evidence does not invent a +hit. Relative-zero FP starts here: do not crown HQ, do not omit +`vocabulary=programming` on `:8444`, do not pair a live overlay with the +previous `last_session.wav`. diff --git a/docs/TRANSCRIPT_LANES.md b/docs/TRANSCRIPT_LANES.md index 4c65162d..4c23b3bf 100644 --- a/docs/TRANSCRIPT_LANES.md +++ b/docs/TRANSCRIPT_LANES.md @@ -201,7 +201,7 @@ audio file ▶ `codescribe transcribe` CLI / cloud final pass | ------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------- | | overlay LIVE | LINE A/B via J2 | letters as spoken; live backspace corrections (L1); never a rewrite of committed text | | overlay FINAL | LINE S→F via J7 | formatted draft + buttons (Copy / Insert / Revert / Format / To Agent); Auto Paste when guard allows | -| paste target | J7 | formatted text into the frontmost app; `CopyTargetUnavailable` ⇒ clipboard fallback | +| paste target | J7 | formatted text into the latched foreign app; refuse (`CopyTargetUnavailable` / mismatch) ⇒ Paste Here slot, user clipboard untouched | | thread rail | LINE G | agent conversation, chained turn by turn | | history dir | J6 | `~/.codescribe/transcriptions//` — raw, formatted, m4a, truth receipts | | menu/tray | controller state | recording state; Audio truth section (W13-5: device, level, quality verdict) | diff --git a/docs/loopback.html b/docs/loopback.html new file mode 100644 index 00000000..396e08e1 --- /dev/null +++ b/docs/loopback.html @@ -0,0 +1,16 @@ + + + + + codescribe loopback + + +

codescribe loopback

+

Dev install pointers. Do not rewrite these URLs.

+
    +
  • Voice Lab — http://127.0.0.1:8765/lab
    • +
  • STT file HTTP — http://127.0.0.1:8444/v1/audio/transcriptions
    • +
  • STT live WebSocket — ws://127.0.0.1:8446
    • +
+ + diff --git a/macos/Codescribe/App.swift b/macos/Codescribe/App.swift index 5cdaa697..1d9da126 100644 --- a/macos/Codescribe/App.swift +++ b/macos/Codescribe/App.swift @@ -470,6 +470,7 @@ final class AppDelegate: NSObject, NSApplicationDelegate { // touching the lazy handle here would construct the bridge at teardown // purely to stop something that was never running. guard !Self.isRunningTests else { return } + VoiceLabRuntime.stopOwnedProcess() hotkeys.stop() sleepWakeObserver?.invalidate() sleepWakeObserver = nil diff --git a/macos/Codescribe/Core/AppModel.swift b/macos/Codescribe/Core/AppModel.swift index ad30c4f9..7b001bb5 100644 --- a/macos/Codescribe/Core/AppModel.swift +++ b/macos/Codescribe/Core/AppModel.swift @@ -128,6 +128,9 @@ final class OverlayController: ObservableObject { AppModel.shared.tray.isStartingDictation = true // Block the composer mic while the shared recorder owns the microphone. AppModel.shared.chat.dictationBlocked = true + Task.detached(priority: .utility) { + VoiceLabRuntime.ensureListening() + } } state.onRecordingStarted = { [weak self] in guard let self else { return } @@ -147,6 +150,7 @@ final class OverlayController: ObservableObject { // Unconditional: releases the composer phase, the blocked flag and the // thread-ownership latch in one beat, whatever the lane turned out to be. AppModel.shared.chat.endDictationSession() + VoiceLabRuntime.stopOwnedProcess() } state.onSuccessfulDictation = { Task { @MainActor in @@ -197,18 +201,33 @@ final class OverlayController: ObservableObject { func show() { let panel = panel ?? panelFactory(state, textScale) self.panel = panel + if let floating = panel as? FloatingOverlayPanel { + floating.onUserMove = { [weak self] in + guard let self, !Self.isApplyingFrame, let panel = self.panel else { return } + if self.state.freeMotion { + OverlayPlacement.persistOrigin(panel.frame.origin) + } + self.state.userDraggedOverlay() + } + } // A pending fade-out must not leave a freshly shown panel invisible. panel.alphaValue = 1 applyPlacement(animated: false) orderPanelFront(panel) } + /// True while we `setFrame` from prefs. AppKit still fires `windowDidMove` + /// for those writes; those must not count as a user drag. + static var isApplyingFrame = false + /// Derive and apply the panel's frame from the placement prefs: free motion /// restores the last dragged origin, anchored derives from the anchor — /// in ONE setFrame so there is no transient mismatched frame. Clamping the /// size here covers programmatic sizing, which AppKit's minSize does not. private func applyPlacement(animated: Bool) { guard let panel else { return } + Self.isApplyingFrame = true + defer { Self.isApplyingFrame = false } let screen = NSScreen.main let size = DictationOverlayWindow.clamp(panel.frame.size, to: screen) let origin: NSPoint? @@ -235,7 +254,13 @@ final class OverlayController: ObservableObject { /// Called by the live TrayStatusStore listener. Assistive uses the shared /// controller but keeps the Dictation overlay closed in favor of Agent UI. + /// Format / Retranscribe (and the post-take review they run on) own the + /// panel: an Assistive tray tick must not hide it, steal focus, or arm + /// Agent auto-send. func handleIndicatorModeChange(_ mode: CsIndicatorMode) { + if mode == .assistive, state.blocksAssistiveOverlayHide { + return + } if mode == .assistive { sessionWasAssistive = true hide() diff --git a/macos/Codescribe/Core/DeveloperSurface.swift b/macos/Codescribe/Core/DeveloperSurface.swift index 901d40ab..81e8aa40 100644 --- a/macos/Codescribe/Core/DeveloperSurface.swift +++ b/macos/Codescribe/Core/DeveloperSurface.swift @@ -3,6 +3,9 @@ import OSLog /// Lab extras baked only by keyed `make install-app`. enum DeveloperSurface { + /// Corner caption on overlay, chat, and Settings for an org `install-app` bake. + static let powerModeCaption = "You use dev power mode" + static func parse(_ raw: Any?) -> Bool { if let flag = raw as? Bool { return flag } if let number = raw as? NSNumber { return number.boolValue } diff --git a/macos/Codescribe/Core/VoiceLabRuntime.swift b/macos/Codescribe/Core/VoiceLabRuntime.swift new file mode 100644 index 00000000..8b7dd054 --- /dev/null +++ b/macos/Codescribe/Core/VoiceLabRuntime.swift @@ -0,0 +1,156 @@ +import AppKit +import Foundation +import OSLog + +/// Loopback Voice Lab process owned by a CS-bake Codescribe. +/// +/// Production DMGs never spawn. The XCTest host never spawns. A live +/// `:8765` is left alone. Missing `~/.codescribe/voice-lab/server.py` +/// is a no-op (install-voice-lab never ran). +enum VoiceLabRuntime { + static let consoleURL = URL(string: "http://127.0.0.1:8765/lab")! + + private static let logger = Logger( + subsystem: Bundle.main.bundleIdentifier ?? "com.vetcoders.codescribe", + category: "voice-lab" + ) + private static let lock = NSLock() + private static var child: Process? + + static var labRoot: URL { + FileManager.default.homeDirectoryForCurrentUser + .appendingPathComponent(".codescribe/voice-lab", isDirectory: true) + } + + static var serverScript: URL { + labRoot.appendingPathComponent("server.py") + } + + static func shouldSpawn( + surfaceEnabled: Bool, + runningTests: Bool, + alreadyListening: Bool, + serverExists: Bool + ) -> Bool { + surfaceEnabled && !runningTests && !alreadyListening && serverExists + } + + /// Fire-and-forget: bring `:8765` up if this is a CS bake and Lab is down. + static func ensureListening( + surfaceEnabled: Bool? = nil, + runningTests: Bool = QualityCaptureHost.isRunningTests + ) { + let cs = surfaceEnabled ?? DeveloperSurface.isEnabled() + if cs, !runningTests { writeLoopbackPointer() } + let server = serverScript + lock.lock() + let ownedRunning = child?.isRunning == true + lock.unlock() + let spawn = shouldSpawn( + surfaceEnabled: surfaceEnabled ?? DeveloperSurface.isEnabled(), + runningTests: runningTests, + alreadyListening: ownedRunning || isListening(), + serverExists: FileManager.default.isReadableFile(atPath: server.path) + ) + guard spawn else { return } + startChild(server: server) + } + + /// CS tray/settings: ensure the process, then open the console. + static func openConsole() { + Task.detached(priority: .utility) { + ensureListening() + for _ in 0..<25 where !isListening() { + try? await Task.sleep(nanoseconds: 80_000_000) + } + await MainActor.run { + NSWorkspace.shared.open(consoleURL) + } + } + } + + static func stopOwnedProcess() { + lock.lock() + let process = child + child = nil + lock.unlock() + process?.terminate() + } + + static func isListening(timeout: TimeInterval = 0.2) -> Bool { + var request = URLRequest(url: consoleURL) + request.httpMethod = "GET" + request.timeoutInterval = timeout + let semaphore = DispatchSemaphore(value: 0) + var ok = false + URLSession.shared.dataTask(with: request) { _, response, _ in + ok = (response as? HTTPURLResponse)?.statusCode == 200 + semaphore.signal() + }.resume() + _ = semaphore.wait(timeout: .now() + timeout + 0.05) + return ok + } + + static func writeLoopbackPointer() { + let html = """ + + + + + codescribe loopback + + +

codescribe loopback

+

Dev install pointers. Do not rewrite these URLs.

+
    +
  • Voice Lab — http://127.0.0.1:8765/lab
    • +
  • STT file HTTP — http://127.0.0.1:8444/v1/audio/transcriptions
    • +
  • STT live WebSocket — ws://127.0.0.1:8446
    • +
+ + + """ + do { + try FileManager.default.createDirectory(at: labRoot, withIntermediateDirectories: true) + try html.write( + to: labRoot.appendingPathComponent("loopback.html"), + atomically: true, + encoding: .utf8 + ) + } catch { + logger.error( + "loopback.html write failed: \(error.localizedDescription, privacy: .public)" + ) + } + } + + private static func startChild(server: URL) { + lock.lock() + defer { lock.unlock() } + if let child, child.isRunning { return } + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/python3") + process.arguments = [server.path] + process.currentDirectoryURL = labRoot + var environment = ProcessInfo.processInfo.environment + environment["VOICE_LAB_REMOTE_HOST"] = "off" + process.environment = environment + process.standardInput = FileHandle.nullDevice + process.standardOutput = FileHandle.nullDevice + process.standardError = FileHandle.nullDevice + process.terminationHandler = { _ in + lock.lock() + child = nil + lock.unlock() + } + do { + try process.run() + child = process + logger.info("voice-lab spawned pid=\(process.processIdentifier, privacy: .public)") + } catch { + logger.error( + "voice-lab spawn failed: \(error.localizedDescription, privacy: .public)" + ) + } + } +} diff --git a/macos/Codescribe/DesignSystem/DeveloperPowerMark.swift b/macos/Codescribe/DesignSystem/DeveloperPowerMark.swift new file mode 100644 index 00000000..445daa40 --- /dev/null +++ b/macos/Codescribe/DesignSystem/DeveloperPowerMark.swift @@ -0,0 +1,27 @@ +import SwiftUI + +/// Quiet corner mark for an org `make install-app` bake. +/// Hidden on production DMGs (`CSDeveloperSurface` off). +struct DeveloperPowerMark: View { + var body: some View { + if DeveloperSurface.isEnabled() { + Text(DeveloperSurface.powerModeCaption) + .font(CSFont.mono(10, .medium)) + .tracking(0.2) + .foregroundStyle(CSColor.terracottaLight) + .opacity(0.72) + .allowsHitTesting(false) + .accessibilityIdentifier("developer-power-mark") + } + } +} + +extension View { + func developerPowerCorner(padding: CGFloat = 10) -> some View { + overlay(alignment: .bottomTrailing) { + DeveloperPowerMark() + .padding(.trailing, padding) + .padding(.bottom, padding) + } + } +} diff --git a/macos/Codescribe/Screens/AgentChat/AgentChatView.swift b/macos/Codescribe/Screens/AgentChat/AgentChatView.swift index 22907476..91e96a4b 100644 --- a/macos/Codescribe/Screens/AgentChat/AgentChatView.swift +++ b/macos/Codescribe/Screens/AgentChat/AgentChatView.swift @@ -49,6 +49,7 @@ struct AgentChatView: View { } .navigationSplitViewStyle(.balanced) .csFocusPolicy() + .developerPowerCorner(padding: 12) .background(CSColor.glassBase) .background(AgentWindowCapabilities(isPinned: isPinned)) .frame(minWidth: 760, idealWidth: 960, minHeight: 560, idealHeight: 600) @@ -167,7 +168,7 @@ private struct ThreadDetail: View { .padding(.horizontal, 20) .padding(.bottom, 10) } - Composer(store: store) + Composer(store: store, overlay: AppModel.shared.overlay.state) } .frame(maxWidth: .infinity, maxHeight: .infinity) .background(CSColor.glassBase) diff --git a/macos/Codescribe/Screens/AgentChat/Composer.swift b/macos/Codescribe/Screens/AgentChat/Composer.swift index 27a9ac38..1dee8b3b 100644 --- a/macos/Codescribe/Screens/AgentChat/Composer.swift +++ b/macos/Codescribe/Screens/AgentChat/Composer.swift @@ -17,6 +17,7 @@ private let attachLog = Logger( /// drag & drop, and ⌘V paste — all landing in `store.addAttachments`. struct Composer: View { @ObservedObject var store: AgentChatStore + @ObservedObject var overlay: OverlayState @State private var fieldFocused = false /// Chat text scale (⌘+/-/0) — applied to the message field + placeholder so the /// composer input tracks the message bodies. Chrome (chips, affordance hints, @@ -60,6 +61,8 @@ struct Composer: View { attachmentChips } + liveAgentCapture + HStack(alignment: .bottom, spacing: 10) { // Attach images (NSOpenPanel → staged chips → vision FFI on send). Button(action: pickAttachments) { @@ -338,9 +341,30 @@ struct Composer: View { // MARK: Voice-note mic + /// Live spoken text while Agent/assistive capture owns the mic. Overlay is + /// hidden on that lane, so the chat has to show the growing sentence. + @ViewBuilder + private var liveAgentCapture: some View { + let live = overlay.activeText.trimmingCharacters(in: .whitespacesAndNewlines) + if store.dictationOwnsSelectedThread, + (store.dictationPhase == .preparing || store.dictationPhase == .recording), + !live.isEmpty + { + Text(live) + .font(CSFont.ui(13, .regular)) + .foregroundStyle(CSColor.textHigh) + .frame(maxWidth: .infinity, alignment: .leading) + .padding(.horizontal, 12) + .padding(.vertical, 8) + .background(CSColor.surfaceRaised(0.04)) + .clipShape(RoundedRectangle(cornerRadius: CSRadius.card, style: .continuous)) + .accessibilityIdentifier("agent-live-capture") + } + } + /// The composer mic starts/stops the same Agent route as Right Option. The - /// ripple follows the shared controller lifecycle; no composer recorder or - /// editable transcript copy exists. + /// ripple follows the shared controller lifecycle; live words render above + /// the field from the overlay reducer (the overlay window stays closed). private var micButton: some View { Button(action: { store.toggleDictation() }) { micVisual diff --git a/macos/Codescribe/Screens/Overlay/DictationOverlayView.swift b/macos/Codescribe/Screens/Overlay/DictationOverlayView.swift index e51c7c4e..cf77e360 100644 --- a/macos/Codescribe/Screens/Overlay/DictationOverlayView.swift +++ b/macos/Codescribe/Screens/Overlay/DictationOverlayView.swift @@ -12,7 +12,8 @@ import SwiftUI // All actions are neutral/grey; Close is the ONE red control. // footer ● · meta on the right // -// A transient toast (no-speech / error) floats over the bottom edge. +// Delivery/status whispers in the footer next to the engine chip — never a +// floating pill over the action row. struct DictationOverlayView: View { @ObservedObject var state: OverlayState @@ -80,13 +81,7 @@ struct DictationOverlayView: View { // already falls outside the borderless window (never rendered), so this // clip costs nothing visually. .clipShape(RoundedRectangle(cornerRadius: CSRadius.window, style: .continuous)) - .overlay(alignment: .bottom) { - if let toast = state.toast { - ToastPill(text: toast) - .padding(.bottom, 14) - .transition(.opacity.combined(with: .offset(y: 8))) - } - } + .developerPowerCorner(padding: 10) .animation(CSMotion.floatIn, value: state.toast) .onHover { inside in state.setPointerHovering(inside) @@ -574,8 +569,6 @@ struct DictationOverlayView: View { state.retranscribe(pass: pass) } } - } primaryAction: { - state.retranscribe(pass: .fullHq) } label: { actionButtonLabel( title: state.isRetranscribing @@ -584,6 +577,8 @@ struct DictationOverlayView: View { tone: .neutral, iconOnly: iconOnly ) + } primaryAction: { + state.retranscribe(pass: .fullHq) } .menuStyle(.button) .csFocusRing(cornerRadius: 8) @@ -676,6 +671,14 @@ struct DictationOverlayView: View { // Product truth: never hardcode "local whisper". Chip = last serving // engine when known, else preference (Apple live default). Text(state.footerEngineLabel).foregroundStyle(CSColor.textFaintAlt) + if let toast = state.toast, !toast.isEmpty { + Text("·").foregroundStyle(CSColor.textFaintAlt) + Text(toast) + .foregroundStyle(CSColor.textFaintAlt) + .lineLimit(1) + .truncationMode(.tail) + .accessibilityIdentifier("overlay-footer-notice") + } } Spacer(minLength: 0) Text(state.footerRight) @@ -713,23 +716,6 @@ private struct BlinkingCaret: View { } } -/// Transient notice for no-speech / recoverable engine errors. -private struct ToastPill: View { - let text: String - var body: some View { - Text(text) - .font(CSFont.metaMono) - .foregroundStyle(CSColor.textBody) - .padding(.horizontal, 14) - .padding(.vertical, 8) - .background(CSColor.surfaceRaised(0.06)) - .overlay( - Capsule().strokeBorder(CSColor.hairline(0.14), lineWidth: 1) - ) - .clipShape(Capsule()) - } -} - #if DEBUG #Preview("Listening") { DictationOverlayView(state: .previewListening()) diff --git a/macos/Codescribe/Screens/Overlay/DictationOverlayWindow.swift b/macos/Codescribe/Screens/Overlay/DictationOverlayWindow.swift index c694e7ef..52af482a 100644 --- a/macos/Codescribe/Screens/Overlay/DictationOverlayWindow.swift +++ b/macos/Codescribe/Screens/Overlay/DictationOverlayWindow.swift @@ -127,8 +127,14 @@ enum DictationOverlayWindow { defer: false ) panel.delegate = panel - panel.onUserMove = { [weak state] in state?.userDraggedOverlay() } - panel.onUserResize = { [weak state] in state?.userResizedOverlay() } + panel.onUserMove = { [weak state] in + guard !OverlayController.isApplyingFrame else { return } + state?.userDraggedOverlay() + } + panel.onUserResize = { [weak state] in + guard !OverlayController.isApplyingFrame else { return } + state?.userResizedOverlay() + } panel.contentView = OverlayContentContainer(hosting: hosting) // User-resizable: borderless windows still honour edge-drag resize when diff --git a/macos/Codescribe/Screens/Overlay/OverlayState.swift b/macos/Codescribe/Screens/Overlay/OverlayState.swift index cf33b5ff..3d5f4797 100644 --- a/macos/Codescribe/Screens/Overlay/OverlayState.swift +++ b/macos/Codescribe/Screens/Overlay/OverlayState.swift @@ -414,6 +414,8 @@ final class OverlayState: ObservableObject { // MARK: Derived display (one source of truth for the view) var statusText: String { + if isFormatting { return "formatting" } + if isRetranscribing { return "retranscribing" } if mode == .error { return "failed" } if mode == .formatted { return "done" } if mode == .noSpeech { return "no speech" } @@ -424,6 +426,7 @@ final class OverlayState: ObservableObject { return hasMeasuredAudioLevel ? "recording" : "recording · ambient" } var statusColor: Color { + if isFormatting || isRetranscribing { return CSColor.modeProcessing } switch mode { case .listening: return CSColor.terracotta case .formatted: return CSColor.oliveLight @@ -445,14 +448,22 @@ final class OverlayState: ObservableObject { guard showsLowConfidenceBadge else { return nil } return "low confidence" } - /// Only the live-capture pill ripples. During `transcribing` / `final pass` we swap - /// to the static pill so its repeatForever animation tears down — a second visual - /// cue that capture has ended and post-processing is in flight. + /// Only the live-capture pill ripples. During `transcribing` / `final pass` / + /// one-shot format / retranscribe we swap to the static pill so its + /// repeatForever animation tears down — a second visual cue that capture + /// has ended and post-processing is in flight, not a waveform grind. var statusRippling: Bool { - mode == .listening && !transcribing && !isFinalPass && (audioReady || vadActive) + mode == .listening + && !transcribing + && !isFinalPass + && !isFormatting + && !isRetranscribing + && (audioReady || vadActive) } var tagText: String { + if isFormatting { return "FORMAT" } + if isRetranscribing { return "RETRANSCRIBE" } if isFinalPass || transcribing { return "PROCESSING" } switch mode { case .listening: @@ -463,7 +474,9 @@ final class OverlayState: ObservableObject { } } var tagColor: Color { - if isFinalPass || transcribing { return CSColor.modeProcessing } + if isFormatting || isRetranscribing || isFinalPass || transcribing { + return CSColor.modeProcessing + } switch mode { case .listening: return indicatorMode == .assistive ? CSColor.modeAgent : CSColor.modeRecording @@ -474,6 +487,8 @@ final class OverlayState: ObservableObject { } var metaText: String { + if isFormatting { return "format · one-shot" } + if isRetranscribing { return "retranscribe · file pass" } if isFinalPass { return "final pass · formatting" } switch mode { case .listening: @@ -492,6 +507,7 @@ final class OverlayState: ObservableObject { } var footerRight: String { if isFormatting { return "formatting" } + if isRetranscribing { return "retranscribing" } if isFinalPass { return "final pass" } if mode == .noSpeech { return "no speech" } if mode == .error { return "error" } @@ -663,6 +679,13 @@ final class OverlayState: ObservableObject { preFormatText != nil && !isFormatting && !isRetranscribing } + /// Post-take review owns the floating panel. Format / Retranscribe (and the + /// formatted / no-speech surface they run on) must not yield to an Assistive + /// tray tick — that path calls `hide()` and arms Agent auto-send. + var blocksAssistiveOverlayHide: Bool { + isFormatting || isRetranscribing || mode == .formatted || mode == .noSpeech + } + var insertActionPresentation: OverlayInsertActionPresentation { OverlayInsertActionPresentation(targetAppName: pasteTargetAppName) } @@ -793,17 +816,31 @@ final class OverlayState: ObservableObject { let source = formattedText let sourceLevel = qualityFormattingLevel isFormatting = true + agentAutoSendCancelled = true // Format deliberately suspends passive dismissal. Its result stays until // another user activity explicitly starts a fresh countdown. cancelAutoHide() - Task { @MainActor in - defer { self.isFormatting = false } + Task { [weak self] in + let outcome: Result do { - let formatted = try await engine.formatText( - text: source, - language: nil, - level: level - ) + // UniFFI format is a long LLM round-trip. Pinning it to MainActor + // freezes the panel (operator: overlay "zaczął mielić") for ~17s on + // an 8k-char take. Detach so Close / Copy / resize stay hittable. + let formatted = try await Task.detached(priority: .userInitiated) { + try await engine.formatText( + text: source, + language: nil, + level: level + ) + }.value + outcome = .success(formatted) + } catch { + outcome = .failure(error) + } + guard let self else { return } + defer { self.isFormatting = false } + switch outcome { + case .success(let formatted): let isUsableChange = !formatted .trimmingCharacters(in: .whitespacesAndNewlines) @@ -817,7 +854,7 @@ final class OverlayState: ObservableObject { self.formatFailureStatus = nil self.mode = .formatted self.cancelAutoHide() // User acted (Format); do not auto-hide the result. - } catch { + case .failure(let error): self.formattedText = source self.formatFailureStatus = "raw — formatting failed" self.mode = .formatted @@ -860,12 +897,23 @@ final class OverlayState: ObservableObject { } let source = activeText isRetranscribing = true + agentAutoSendCancelled = true cancelAutoHide() - Task { @MainActor in - defer { self.isRetranscribing = false } + Task { [weak self] in + let outcome: Result do { let prefixed = "\(pass.rawValue):\(audioPath)" - let result = try await engine.transcribeFile(path: prefixed) + let result = try await Task.detached(priority: .userInitiated) { + try await engine.transcribeFile(path: prefixed) + }.value + outcome = .success(result) + } catch { + outcome = .failure(error) + } + guard let self else { return } + defer { self.isRetranscribing = false } + switch outcome { + case .success(let result): let next = result.text.trimmingCharacters(in: .whitespacesAndNewlines) if next.isEmpty { self.formatFailureStatus = "retranscribe — empty" @@ -877,7 +925,7 @@ final class OverlayState: ObservableObject { self.formatFailureStatus = nil self.mode = .formatted self.cancelAutoHide() - } catch { + case .failure(let error): let reason = error.localizedDescription self.formatFailureStatus = "retranscribe — \(pass.visibleName) failed: \(reason)" self.showToast("Couldn't retranscribe — \(reason)") @@ -974,55 +1022,34 @@ final class OverlayState: ObservableObject { } switch result?.outcome { case .deferredInsertArmed: - let target = result?.targetAppName ?? "the target app" let shortcut = result?.deferredInsertShortcut ?? "⌘⌥V" - self.showToast( - "Couldn't reach \(target) — put your cursor where you want the text " - + "and press \(shortcut). Your clipboard is untouched." - ) + self.showFooterNotice(shortcut, persists: true) case .copiedToClipboard: - self.showToast( - self.copiedInsertFallbackToast( - frontmost: result?.frontmostAppName, - target: result?.targetAppName, - failure: result?.deferredInsertFailure - )) + self.showFooterNotice("copied") case .accessibilityPermissionNeeded: - self.showToast( - self.copiedInsertFallbackToast( - frontmost: result?.frontmostAppName, - target: result?.targetAppName, - failure: result?.deferredInsertFailure - )) + self.showFooterNotice("no ax") case .pasted, .noop, nil: break } } catch { self.errorMessage = "Couldn't paste transcript: \(error)" - self.showToast("Couldn't paste transcript") + self.showFooterNotice("no paste") } } } - private func copiedInsertFallbackToast( - frontmost: String?, - target: String?, - failure: String? - ) -> String { - if let failure { - return "\(failure) — copied with tags instead. " - + "Clipboard replaced; press Cmd+V where you want it." - } - if let frontmost, let target { - return "Copied — your cursor is in \(frontmost), not \(target). " - + "Clipboard replaced; press Cmd+V where you want it." - } - if let target { - return "Copied — focus couldn't be confirmed for \(target). " - + "Clipboard replaced; press Cmd+V where you want it." + /// Whisper a short footer chip next to `local apple`. Never a floating pill + /// over the action row. `persists` keeps the chip until the overlay hides + /// (Paste Here chord); otherwise it fades after the usual toast window. + func showFooterNotice(_ message: String, persists: Bool = false) { + toast = message + toastTask?.cancel() + guard !persists else { return } + toastTask = Task { @MainActor [weak self] in + try? await Task.sleep(nanoseconds: 2_600_000_000) + guard !Task.isCancelled else { return } + self?.toast = nil } - return "Copied — the target app was lost. " - + "Clipboard replaced; press Cmd+V where you want it." } /// Persist through C02's single config seam, then immediately replace local @@ -1115,6 +1142,7 @@ final class OverlayState: ObservableObject { } /// AppKit reports window motion separately from SwiftUI content events. + /// Position sticks only in Free motion; anchored mode snaps back on next show. func userDraggedOverlay() { restartAutoHideCountdown() } @@ -1265,6 +1293,7 @@ final class OverlayState: ObservableObject { resetTranscript() formattedText = "" isFormatting = false + isRetranscribing = false errorMessage = nil beginCaptureClock() } @@ -1290,6 +1319,7 @@ final class OverlayState: ObservableObject { } formattedText = "" isFormatting = false + isRetranscribing = false formatFailureStatus = nil errorMessage = nil beginCaptureClock() diff --git a/macos/Codescribe/Screens/Settings/LabPanel.swift b/macos/Codescribe/Screens/Settings/LabPanel.swift index a69b63c6..73432083 100644 --- a/macos/Codescribe/Screens/Settings/LabPanel.swift +++ b/macos/Codescribe/Screens/Settings/LabPanel.swift @@ -29,8 +29,7 @@ struct LabPanel: View { } Button("Open Voice Lab") { - guard let url = URL(string: "http://127.0.0.1:8765/lab") else { return } - NSWorkspace.shared.open(url) + VoiceLabRuntime.openConsole() } .font(CSFont.mono(11, .semibold)) .foregroundStyle(CSColor.chromeAccent) diff --git a/macos/Codescribe/Screens/Settings/SettingsView.swift b/macos/Codescribe/Screens/Settings/SettingsView.swift index 839c5931..73f6ba22 100644 --- a/macos/Codescribe/Screens/Settings/SettingsView.swift +++ b/macos/Codescribe/Screens/Settings/SettingsView.swift @@ -47,6 +47,7 @@ struct SettingsView: View { } } .csFocusPolicy() + .developerPowerCorner(padding: 12) .frame(minWidth: 880, maxWidth: .infinity, minHeight: 620, maxHeight: .infinity) .background(SettingsWindowCapabilities()) // The panels still paint hand-picked dark tokens, so the window stays diff --git a/macos/Codescribe/Screens/Settings/ShortcutsPanel.swift b/macos/Codescribe/Screens/Settings/ShortcutsPanel.swift index 006a4274..b9f52e08 100644 --- a/macos/Codescribe/Screens/Settings/ShortcutsPanel.swift +++ b/macos/Codescribe/Screens/Settings/ShortcutsPanel.swift @@ -88,7 +88,7 @@ struct ShortcutsPanel: View { } if row.mode == .assistive { - assistiveModeSplit(row) + assistiveModeSplit() } } .padding(.horizontal, 16) @@ -132,19 +132,15 @@ struct ShortcutsPanel: View { .fixedSize() } - private func assistiveModeSplit(_ row: CsModeBinding) -> some View { + private func assistiveModeSplit() -> some View { VStack(alignment: .leading, spacing: 7) { assistiveModeVariant( - title: "Voice chat", + title: "Attach selection", gesture: armGestureLabel, - description: "Talk to the agent." + description: + "Shift or Command during an already-started Fn hold attaches {selection_N}. It does not start voice chat, hide the overlay, or stop the take. Fn+Shift from idle is dictation, not Assistive." ) - assistiveModeVariant( - title: "Act on selection", - gesture: selectionAssistiveGesture(row), - description: "Select text, then speak an instruction." - ) - // W10-B: customize arm modifier (default Shift; Cmd alternative). + // Arm modifier is attach-only (default Shift; Cmd alternative). HStack(spacing: 8) { Text("Arm with") .font(CSFont.ui(11, .medium)) @@ -203,15 +199,6 @@ struct ShortcutsPanel: View { ArmGestureCopy.label(for: model.holdArmModifier) } - private func selectionAssistiveGesture(_ row: CsModeBinding) -> String { - // Act-on-selection is the same arm gesture when a selection is present - // (W10-D lane). Copy must match the configured binding, not a dead Cmd. - if row.binding == .disabled { - return armGestureLabel - } - return "\(row.bindingLabel) or \(armGestureLabel)" - } - private var armModifierBinding: Binding { Binding( get: { model.holdArmModifier }, @@ -472,10 +459,10 @@ struct ShortcutsPanel: View { } } -/// Single production owner for assistive-arm gesture copy in Settings. +/// Single production owner for attach-arm gesture copy in Settings. enum ArmGestureCopy { static func label(for modifier: String) -> String { - modifier == "cmd" ? "Hold Fn+Command" : "Hold Fn+Shift" + modifier == "cmd" ? "Command during Fn hold" : "Shift during Fn hold" } } diff --git a/macos/Codescribe/Screens/Tray/TrayMenuView.swift b/macos/Codescribe/Screens/Tray/TrayMenuView.swift index 3323d5d6..1eee7181 100644 --- a/macos/Codescribe/Screens/Tray/TrayMenuView.swift +++ b/macos/Codescribe/Screens/Tray/TrayMenuView.swift @@ -38,9 +38,7 @@ struct TrayMenuView: View { } if DeveloperSurface.isEnabled() { TrayRow(icon: .diagnostics, title: "Voice Lab…") { - if let url = URL(string: "http://127.0.0.1:8765/lab") { - NSWorkspace.shared.open(url) - } + VoiceLabRuntime.openConsole() } } TrayRow(icon: .setupWizard, title: "Setup Wizard…") { viewModel.onOpenSetupWizard() } diff --git a/macos/CodescribeTests/DeveloperSurfaceTests.swift b/macos/CodescribeTests/DeveloperSurfaceTests.swift index 81f921ee..e180f3cd 100644 --- a/macos/CodescribeTests/DeveloperSurfaceTests.swift +++ b/macos/CodescribeTests/DeveloperSurfaceTests.swift @@ -15,6 +15,10 @@ final class DeveloperSurfaceTests: XCTestCase { XCTAssertTrue(DeveloperSurface.parse(NSNumber(value: 1))) } + func testPowerModeCaptionIsTheVisibleInstallTag() { + XCTAssertEqual(DeveloperSurface.powerModeCaption, "You use dev power mode") + } + func testLabSectionIsHiddenOnProductionBundle() { XCTAssertEqual(SettingsSection.lab.availability, .hidden) XCTAssertFalse(SettingsSection.matching(query: "").contains(.lab)) @@ -47,6 +51,49 @@ final class DeveloperSurfaceTests: XCTestCase { ) } + func testVoiceLabSpawnIsCSOnlyAndIdleWhenAlreadyUp() { + XCTAssertTrue( + VoiceLabRuntime.shouldSpawn( + surfaceEnabled: true, + runningTests: false, + alreadyListening: false, + serverExists: true + ) + ) + XCTAssertFalse( + VoiceLabRuntime.shouldSpawn( + surfaceEnabled: false, + runningTests: false, + alreadyListening: false, + serverExists: true + ) + ) + XCTAssertFalse( + VoiceLabRuntime.shouldSpawn( + surfaceEnabled: true, + runningTests: true, + alreadyListening: false, + serverExists: true + ) + ) + XCTAssertFalse( + VoiceLabRuntime.shouldSpawn( + surfaceEnabled: true, + runningTests: false, + alreadyListening: true, + serverExists: true + ) + ) + XCTAssertFalse( + VoiceLabRuntime.shouldSpawn( + surfaceEnabled: true, + runningTests: false, + alreadyListening: false, + serverExists: false + ) + ) + } + func testTrayOffHidesOverlayEvenWhenLabModeIsOff() { let defaults = UserDefaults(suiteName: UUID().uuidString)! XCTAssertFalse( diff --git a/macos/CodescribeTests/FocusPolicyTests.swift b/macos/CodescribeTests/FocusPolicyTests.swift index a80df408..6c15476d 100644 --- a/macos/CodescribeTests/FocusPolicyTests.swift +++ b/macos/CodescribeTests/FocusPolicyTests.swift @@ -106,12 +106,15 @@ final class FocusPolicyTests: XCTestCase { func testArmGestureLabelsDeriveFromHoldArmModifier() { // SettingsViewModel.holdArmModifier normalizes to shift|cmd; ShortcutsPanel - // builds labels from that value (no hardcoded-only "Hold Fn+Command" path). + // builds labels from that value. Mid-hold arm attaches selection — it is + // not "Voice chat = Hold Fn+Shift". let shiftLabel = ArmGestureCopy.label(for: "shift") let cmdLabel = ArmGestureCopy.label(for: "cmd") - XCTAssertEqual(shiftLabel, "Hold Fn+Shift") - XCTAssertEqual(cmdLabel, "Hold Fn+Command") + XCTAssertEqual(shiftLabel, "Shift during Fn hold") + XCTAssertEqual(cmdLabel, "Command during Fn hold") XCTAssertNotEqual(shiftLabel, cmdLabel) + XCTAssertFalse(shiftLabel.contains("Hold Fn+Shift")) + XCTAssertFalse(cmdLabel.contains("Hold Fn+Command")) } } diff --git a/macos/CodescribeTests/OverlayStateTests.swift b/macos/CodescribeTests/OverlayStateTests.swift index 7e80adbf..f87b4971 100644 --- a/macos/CodescribeTests/OverlayStateTests.swift +++ b/macos/CodescribeTests/OverlayStateTests.swift @@ -40,6 +40,9 @@ private final class OverlayStateTestEngine: DictationEngine { var transcribedError: Error? var lastAudioPath: String? = "/tmp/last_session.wav" var onTranscribe: (() -> Void)? + /// When set, format/retranscribe wait here so tests can observe the in-flight + /// phase and fire tray/indicator ticks before the work completes. + var workGate: OverlayStateTestGate? func setListener(_ listener: CsTranscriptionListener) {} func startRecording(language: CsLanguage?) async throws {} @@ -67,6 +70,7 @@ private final class OverlayStateTestEngine: DictationEngine { ) async throws -> String { formattedLevels.append(level) onFormat?(level) + await workGate?.wait() switch formattedResult { case .success(let text): return text case .failure(let error): throw error @@ -111,6 +115,7 @@ private final class OverlayStateTestEngine: DictationEngine { func transcribeFile(path: String) async throws -> CsTranscription { transcribedPaths.append(path) onTranscribe?() + await workGate?.wait() if let transcribedError { throw transcribedError } return CsTranscription(text: transcribedResult, language: "pl") } @@ -121,6 +126,36 @@ private final class OverlayStateTestClock { var now: TimeInterval = 0 } +/// Holds format/retranscribe until the test opens it. Sendable so the detached +/// engine task and the MainActor test can share it. +private final class OverlayStateTestGate: @unchecked Sendable { + private let lock = NSLock() + private var continuation: CheckedContinuation? + private var opened = false + + func wait() async { + await withCheckedContinuation { (c: CheckedContinuation) in + lock.lock() + if opened { + lock.unlock() + c.resume() + } else { + continuation = c + lock.unlock() + } + } + } + + func open() { + lock.lock() + opened = true + let pending = continuation + continuation = nil + lock.unlock() + pending?.resume() + } +} + @MainActor final class OverlayStateTests: XCTestCase { private func makeFinalizedState( @@ -135,6 +170,18 @@ final class OverlayStateTests: XCTestCase { return state } + private func waitUntilOverlayIdle( + _ state: OverlayState, + file: StaticString = #filePath, + line: UInt = #line + ) async { + for _ in 0..<200 { + if !state.isFormatting, !state.isRetranscribing { return } + await Task.yield() + } + XCTFail("overlay format/retranscribe did not finish", file: file, line: line) + } + func testInsertActionPresentationNamesKnownTargetAndFallsBackHonestly() { let known = OverlayInsertActionPresentation(targetAppName: "Ghostty") XCTAssertEqual(known.targetAppName, "Ghostty") @@ -220,7 +267,7 @@ final class OverlayStateTests: XCTestCase { state.retranscribe(pass: .fullHq) await fulfillment(of: [called], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(engine.transcribedPaths, ["hq:/tmp/last_session.wav"]) XCTAssertEqual(state.formattedText, "full hq file pass") XCTAssertTrue(state.canRevert) @@ -256,7 +303,7 @@ final class OverlayStateTests: XCTestCase { state.retranscribe(pass: .fullHq) await fulfillment(of: [called], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(engine.transcribedPaths, ["hq:/tmp/last_session.wav"]) XCTAssertEqual(state.toast, "Couldn't retranscribe — Metal device lost") XCTAssertEqual( @@ -607,7 +654,7 @@ final class OverlayStateTests: XCTestCase { state.formatTranscript(level: .correction) await fulfillment(of: [formatCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(state.formattedText, "raw source transcript") XCTAssertEqual(state.formatFailureStatus, "raw — formatting failed") @@ -627,7 +674,7 @@ final class OverlayStateTests: XCTestCase { engine.onFormat = { _ in firstCalled.fulfill() } state.formatTranscript(level: .smart) await fulfillment(of: [firstCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertTrue(state.canRevert) XCTAssertEqual(state.formattedText, "first formatted result") @@ -637,7 +684,7 @@ final class OverlayStateTests: XCTestCase { engine.onFormat = { _ in secondCalled.fulfill() } state.formatTranscript(level: .max) await fulfillment(of: [secondCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(state.formattedText, "second formatted result") state.revertFormat() @@ -658,7 +705,7 @@ final class OverlayStateTests: XCTestCase { state.formatTranscript(level: .correction) await fulfillment(of: [called], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(state.formattedText, "source transcript") XCTAssertFalse(state.canRevert) @@ -677,14 +724,14 @@ final class OverlayStateTests: XCTestCase { engine.onFormat = { _ in successCalled.fulfill() } state.formatTranscript(level: .smart) await fulfillment(of: [successCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) let failureCalled = expectation(description: "failed second format") engine.formattedResult = .failure(NSError(domain: "OverlayStateTests", code: 2)) engine.onFormat = { _ in failureCalled.fulfill() } state.formatTranscript(level: .max) await fulfillment(of: [failureCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(state.formattedText, "formatted transcript") XCTAssertTrue(state.canRevert) @@ -714,7 +761,7 @@ final class OverlayStateTests: XCTestCase { state.formatTranscript(level: level) await fulfillment(of: [called], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(state.formattedText, "\(level.rawValue) output") XCTAssertEqual(state.autoFormatLevel, .off) @@ -886,11 +933,7 @@ final class OverlayStateTests: XCTestCase { XCTAssertEqual(engine.deferredText, "guarded transcript") XCTAssertNil(engine.copiedTaggedText) XCTAssertNil(engine.pastedText, "guard must not fall through to synthetic paste") - XCTAssertEqual( - state.toast, - "Couldn't reach Pensieve — put your cursor where you want the text " - + "and press ⌘⌥V. Your clipboard is untouched." - ) + XCTAssertEqual(state.toast, "⌘⌥V") } func testInsertFallsBackToTaggedCopyWhenHotkeyRegistrationFails() async { @@ -909,11 +952,7 @@ final class OverlayStateTests: XCTestCase { await fulfillment(of: [deferCalled], timeout: 1) await Task.yield() - XCTAssertEqual( - state.toast, - "Paste Here hotkey registration failed — copied with tags instead. " - + "Clipboard replaced; press Cmd+V where you want it." - ) + XCTAssertEqual(state.toast, "copied") } func testInsertShowsCopiedToastWhenControllerGuardDegrades() async { @@ -934,11 +973,7 @@ final class OverlayStateTests: XCTestCase { await Task.yield() XCTAssertEqual(engine.pastedText, "belt and braces transcript") - XCTAssertEqual( - state.toast, - "Copied — your cursor is in Alacritty, not Pensieve. " - + "Clipboard replaced; press Cmd+V where you want it." - ) + XCTAssertEqual(state.toast, "copied") } func testInsertShowsAccessibilityPermissionToastWhenEventPostingDenied() async { @@ -959,11 +994,7 @@ final class OverlayStateTests: XCTestCase { await Task.yield() XCTAssertEqual(engine.pastedText, "permission transcript") - XCTAssertEqual( - state.toast, - "Paste Here hotkey registration failed — copied with tags instead. " - + "Clipboard replaced; press Cmd+V where you want it." - ) + XCTAssertEqual(state.toast, "no ax") } func testFormatCancelsAutoHideWithoutRearming() async { @@ -980,7 +1011,7 @@ final class OverlayStateTests: XCTestCase { clock.now = 4 state.formatTranscript(level: .correction) await fulfillment(of: [formatCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) XCTAssertEqual(state.formattedText, "formatted result") clock.now = 100 @@ -1003,7 +1034,7 @@ final class OverlayStateTests: XCTestCase { clock.now = 4 state.formatTranscript(level: .max) await fulfillment(of: [formatCalled], timeout: 1) - await Task.yield() + await waitUntilOverlayIdle(state) clock.now = 100 state.revertFormat() @@ -1458,6 +1489,136 @@ final class OverlayStateTests: XCTestCase { XCTAssertFalse(controller.state.autoPasteControlAvailable) } + func testFormatDoesNotHideOverlayOrApplyAssistive() async { + var outCount = 0 + var frontCount = 0 + let state = OverlayState() + let engine = OverlayStateTestEngine() + let gate = OverlayStateTestGate() + engine.workGate = gate + engine.formattedResult = .success("formatted take") + state.engine = engine + state.formattedText = "long take" + state.mode = .formatted + var closeCount = 0 + state.onClose = { closeCount += 1 } + + let controller = OverlayController( + state: state, + engine: engine, + overlayEnabledProvider: { true }, + assistiveStatusProvider: { false }, + panelFactory: { _, _ in NSPanel() }, + orderPanelFront: { _ in frontCount += 1 }, + orderPanelOut: { _ in outCount += 1 } + ) + controller.show() + XCTAssertEqual(frontCount, 1) + + let started = expectation(description: "format in flight") + engine.onFormat = { _ in + XCTAssertFalse(Thread.isMainThread, "format must not pin MainActor") + started.fulfill() + } + state.formatTranscript(level: .correction) + await fulfillment(of: [started], timeout: 1) + + XCTAssertTrue(state.isFormatting) + XCTAssertEqual(state.statusText, "formatting") + XCTAssertEqual(state.tagText, "FORMAT") + XCTAssertEqual(state.metaText, "format · one-shot") + XCTAssertFalse(state.statusRippling) + XCTAssertTrue(state.blocksAssistiveOverlayHide) + + controller.handleIndicatorModeChange(.assistive) + XCTAssertEqual(outCount, 0, "Format must not hide the overlay") + XCTAssertNotEqual(state.indicatorMode, .assistive) + XCTAssertTrue(state.autoPasteControlAvailable) + XCTAssertEqual(closeCount, 0) + + gate.open() + await waitUntilOverlayIdle(state) + XCTAssertEqual(outCount, 0) + XCTAssertEqual(closeCount, 0) + XCTAssertEqual(state.mode, .formatted) + XCTAssertEqual(state.formattedText, "formatted take") + XCTAssertEqual(state.statusText, "done") + XCTAssertEqual(state.indicatorMode, .hold) + } + + func testRetranscribeDoesNotHideOverlayOrApplyAssistive() async { + var outCount = 0 + let clock = OverlayStateTestClock() + let state = makeFinalizedState(clock: clock, text: "live draft") + let engine = OverlayStateTestEngine() + let gate = OverlayStateTestGate() + engine.workGate = gate + engine.transcribedResult = "hq file pass" + state.engine = engine + var closeCount = 0 + state.onClose = { closeCount += 1 } + + let controller = OverlayController( + state: state, + engine: engine, + overlayEnabledProvider: { true }, + assistiveStatusProvider: { false }, + panelFactory: { _, _ in NSPanel() }, + orderPanelFront: { _ in }, + orderPanelOut: { _ in outCount += 1 } + ) + controller.show() + + let started = expectation(description: "retranscribe in flight") + engine.onTranscribe = { + XCTAssertFalse(Thread.isMainThread, "retranscribe must not pin MainActor") + started.fulfill() + } + state.retranscribe(pass: .fullHq) + await fulfillment(of: [started], timeout: 1) + + XCTAssertTrue(state.isRetranscribing) + XCTAssertEqual(state.statusText, "retranscribing") + XCTAssertEqual(state.tagText, "RETRANSCRIBE") + XCTAssertFalse(state.statusRippling) + XCTAssertTrue(state.blocksAssistiveOverlayHide) + + controller.handleIndicatorModeChange(.assistive) + XCTAssertEqual(outCount, 0) + XCTAssertNotEqual(state.indicatorMode, .assistive) + XCTAssertEqual(closeCount, 0) + + gate.open() + await waitUntilOverlayIdle(state) + XCTAssertEqual(outCount, 0) + XCTAssertEqual(closeCount, 0) + XCTAssertEqual(state.formattedText, "hq file pass") + XCTAssertEqual(state.mode, .formatted) + } + + func testFormattedReviewBlocksAssistiveHideWithoutFormatInFlight() { + var outCount = 0 + let state = OverlayState() + state.mode = .formatted + state.formattedText = "review take" + let controller = OverlayController( + state: state, + engine: nil, + overlayEnabledProvider: { true }, + assistiveStatusProvider: { false }, + panelFactory: { _, _ in NSPanel() }, + orderPanelFront: { _ in }, + orderPanelOut: { _ in outCount += 1 } + ) + controller.show() + XCTAssertTrue(state.blocksAssistiveOverlayHide) + + controller.handleIndicatorModeChange(.assistive) + XCTAssertEqual(outCount, 0) + XCTAssertNotEqual(state.indicatorMode, .assistive) + XCTAssertTrue(state.autoPasteControlAvailable) + } + func testOverlayPanelUsesNonActivatingStyle() { let state = OverlayState() let panel = DictationOverlayWindow.make( diff --git a/scripts/build-app.sh b/scripts/build-app.sh index 5ee1aef4..01895c21 100755 --- a/scripts/build-app.sh +++ b/scripts/build-app.sh @@ -191,19 +191,38 @@ DERIVED="$REPO_ROOT/macos/build" # LIBRARY_SEARCH_PATHS must follow the selected Cargo profile. Xcode's Debug / # Release configs cannot distinguish distribution `release` from the optimized # `local-release` Cargo profile because both intentionally use Release Swift. -xcodebuild -project macos/Codescribe.xcodeproj \ - -scheme "$SCHEME" -configuration "$CONFIG" \ - -derivedDataPath "$DERIVED" \ - ONLY_ACTIVE_ARCH=YES \ - LIBRARY_SEARCH_PATHS="$REPO_ROOT/$TARGET_DIR" \ - CODE_SIGNING_ALLOWED="${CODE_SIGNING_ALLOWED:-NO}" \ - MARKETING_VERSION="$STAMP_VERSION" \ - CURRENT_PROJECT_VERSION="$STAMP_BUILD_NUM" \ - CS_BUILD_COMMIT="$STAMP_COMMIT" \ - CS_BUILT_AT="$STAMP_BUILT_AT" \ - SPARKLE_ED_PUBLIC_KEY="${SPARKLE_ED_PUBLIC_KEY:-}" \ - CS_DEVELOPER_SURFACE="${CS_DEVELOPER_SURFACE:-0}" \ - build +# Xcode 27 ld rejects rustc-stripped dylibs (LINKEDIT string pool). Daily +# local-release now ships unstripped. If the selected toolchain still +# refuses the dylib and a stable Xcode.app exists, retry that linker. +run_app_xcodebuild() { + xcodebuild -project macos/Codescribe.xcodeproj \ + -scheme "$SCHEME" -configuration "$CONFIG" \ + -derivedDataPath "$DERIVED" \ + ONLY_ACTIVE_ARCH=YES \ + LIBRARY_SEARCH_PATHS="$REPO_ROOT/$TARGET_DIR" \ + CODE_SIGNING_ALLOWED="${CODE_SIGNING_ALLOWED:-NO}" \ + MARKETING_VERSION="$STAMP_VERSION" \ + CURRENT_PROJECT_VERSION="$STAMP_BUILD_NUM" \ + CS_BUILD_COMMIT="$STAMP_COMMIT" \ + CS_BUILT_AT="$STAMP_BUILT_AT" \ + SPARKLE_ED_PUBLIC_KEY="${SPARKLE_ED_PUBLIC_KEY:-}" \ + CS_DEVELOPER_SURFACE="${CS_DEVELOPER_SURFACE:-0}" \ + build +} + +XCODEBUILD_LOG="$(mktemp)" +if ! run_app_xcodebuild > >(tee "$XCODEBUILD_LOG") 2>&1; then + if grep -q "mis-aligned LINKEDIT" "$XCODEBUILD_LOG" \ + && [[ -d /Applications/Xcode.app ]] \ + && [[ "${DEVELOPER_DIR:-}" != /Applications/Xcode.app* ]]; then + echo "==> beta ld rejected the Rust dylib; retrying with /Applications/Xcode.app" + DEVELOPER_DIR=/Applications/Xcode.app run_app_xcodebuild + else + rm -f "$XCODEBUILD_LOG" + exit 65 + fi +fi +rm -f "$XCODEBUILD_LOG" APP="$DERIVED/Build/Products/$CONFIG/$SCHEME.app" echo "==> [6/7] Embedding runtime artifacts into $SCHEME.app" diff --git a/scripts/developer-surface-gate.sh b/scripts/developer-surface-gate.sh index 1b72f0ed..1fb12463 100755 --- a/scripts/developer-surface-gate.sh +++ b/scripts/developer-surface-gate.sh @@ -10,8 +10,24 @@ # 𝚅𝚒𝚋𝚎𝚌𝚛𝚊𝚏𝚝𝚎𝚍. with AI Agents by Vetcoders (c)2024-2026 LibraxisAI set -euo pipefail -SPARKLE_FILE="${CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE:-$HOME/.vibecrafted/secrets/codescribe/sparkle-public.b64}" -LICENSE_FILE="${CODESCRIBE_LICENSE_PUBLIC_KEY_FILE:-$HOME/.vibecrafted/secrets/codescribe/license-public.hex}" +DEV_PACK="${HOME}/.codescribe/config/dev/keys" +VIBE_SECRETS="${HOME}/.vibecrafted/secrets/codescribe" +if [[ -z "${CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE:-}" ]]; then + if [[ -f "${DEV_PACK}/sparkle-public.b64" ]]; then + CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE="${DEV_PACK}/sparkle-public.b64" + else + CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE="${VIBE_SECRETS}/sparkle-public.b64" + fi +fi +if [[ -z "${CODESCRIBE_LICENSE_PUBLIC_KEY_FILE:-}" ]]; then + if [[ -f "${DEV_PACK}/license-public.hex" ]]; then + CODESCRIBE_LICENSE_PUBLIC_KEY_FILE="${DEV_PACK}/license-public.hex" + else + CODESCRIBE_LICENSE_PUBLIC_KEY_FILE="${VIBE_SECRETS}/license-public.hex" + fi +fi +SPARKLE_FILE="$CODESCRIBE_SPARKLE_PUBLIC_KEY_FILE" +LICENSE_FILE="$CODESCRIBE_LICENSE_PUBLIC_KEY_FILE" read_trimmed() { local path="$1" diff --git a/scripts/install-if-idle.sh b/scripts/install-if-idle.sh new file mode 100755 index 00000000..6fd1cb0c --- /dev/null +++ b/scripts/install-if-idle.sh @@ -0,0 +1,49 @@ +#!/usr/bin/env bash +# Install the local app only when no Codescribe take is in flight. +# Bus authority: session_started without a later transcript_sealed → refuse. +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +BUS="${CODESCRIBE_TRANSCRIPT_BUS:-$HOME/.codescribe/transcript-events.jsonl}" + +recording_live() { + [[ -f "$BUS" ]] || return 1 + python3 - "$BUS" <<'PY' +import json, sys +path = sys.argv[1] +session = None +sealed = True +try: + lines = open(path, encoding="utf-8", errors="replace").read().splitlines() +except OSError: + raise SystemExit(1) +for raw in lines[-4000:]: + raw = raw.strip() + if not raw: + continue + try: + event = json.loads(raw) + except json.JSONDecodeError: + continue + status = event.get("status") + if status == "session_started": + session = event.get("session_id") + sealed = False + continue + if session is None: + continue + if event.get("session_id") != session: + continue + if status == "transcript_sealed": + sealed = True +raise SystemExit(0 if (session is not None and not sealed) else 1) +PY +} + +if recording_live; then + echo "install-if-idle: refuse — Codescribe take is live (Transcript Bus)" >&2 + exit 2 +fi + +echo "install-if-idle: idle — make install-app" +exec make -C "$ROOT" install-app diff --git a/scripts/install-voice-lab.sh b/scripts/install-voice-lab.sh new file mode 100755 index 00000000..00108689 --- /dev/null +++ b/scripts/install-voice-lab.sh @@ -0,0 +1,270 @@ +#!/usr/bin/env bash +# Fail-closed Voice Lab toolbox install for `make install-app`. +# +# Org-only: the operator must be able to read vetcoders/voice-lab (sibling +# checkout or git clone). External contributors cannot walk this path. +# Public Sparkle Ed + license verify keys come from the Monika pack so +# CSDeveloperSurface and agent Lab extras stay armed on this hot path. +# +# Env: +# VOICE_LAB_REPO_URL optional override. Empty = org HTTPS or SSH, +# ordered by `gh config git_protocol` (https on +# this laptop, ssh on Monika). Only vetcoders/ +# voice-lab URLs are accepted. +# CODESCRIBE_VOICE_LAB_SRC existing checkout (skips clone) +# VOICE_LAB_INSTALL_SETTINGS unset = seed missing app settings + empty +# engine keys from examples/monika/settings.json +# 1 = overwrite settings (setup.sh backup) +# 0 = never touch Application Support settings +# HOME runtime dest ~/.codescribe/voice-lab +# +# 𝚅𝚒𝚋𝚎𝚌𝚛𝚊𝚏𝚝𝚎𝚍. with AI Agents by Vetcoders (c)2024-2026 LibraxisAI +set -euo pipefail + +VOICE_LAB_HTTPS="https://github.com/vetcoders/voice-lab.git" +VOICE_LAB_SSH="git@github.com:vetcoders/voice-lab.git" +REPO_URL="${VOICE_LAB_REPO_URL:-}" +CACHE="${HOME}/.codescribe/src/voice-lab" +RUNTIME="${HOME}/.codescribe/voice-lab" +LAUNCHER="${HOME}/.codescribe/bin/voice-lab" +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +CODESCRIBE_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" +SIBLING="$(cd "${CODESCRIBE_ROOT}/.." && pwd)/voice-lab" + +fail() { + echo "install-voice-lab: $*" >&2 + exit 1 +} + +looks_like_voice_lab() { + local root="$1" + [[ -f "${root}/server.py" && -f "${root}/setup.sh" && -d "${root}/examples/monika/keys" ]] +} + +normalize_repo_url() { + local url="${1%%/}" + url="${url%.git}" + printf '%s\n' "$url" +} + +# Org lock: HTTPS (this machine) or SSH (Monika). Not "any URL with voice-lab". +remote_is_voice_lab() { + local url + url="$(normalize_repo_url "$1")" + case "$url" in + https://github.com/vetcoders/voice-lab) return 0 ;; + git@github.com:vetcoders/voice-lab) return 0 ;; + ssh://git@github.com/vetcoders/voice-lab) return 0 ;; + *) return 1 ;; + esac +} + +git_protocol() { + if command -v gh >/dev/null 2>&1; then + gh config get git_protocol 2>/dev/null || echo https + else + echo https + fi +} + +# Preferred transport first, the other as fallback. Explicit override is alone. +candidate_repo_urls() { + if [[ -n "${VOICE_LAB_REPO_URL:-}" ]]; then + printf '%s\n' "$VOICE_LAB_REPO_URL" + return + fi + if [[ "$(git_protocol)" == "ssh" ]]; then + printf '%s\n' "$VOICE_LAB_SSH" "$VOICE_LAB_HTTPS" + else + printf '%s\n' "$VOICE_LAB_HTTPS" "$VOICE_LAB_SSH" + fi +} + +resolve_src() { + if [[ -n "${CODESCRIBE_VOICE_LAB_SRC:-}" ]]; then + echo "${CODESCRIBE_VOICE_LAB_SRC}" + return + fi + if [[ -d "${SIBLING}/.git" ]] && looks_like_voice_lab "$SIBLING"; then + echo "$SIBLING" + return + fi + echo "$CACHE" +} + +need_git() { + command -v git >/dev/null 2>&1 || fail "git is required to fetch the org Voice Lab repo" +} + +ensure_checkout() { + local src="$1" + local cand origin probed="" + + if looks_like_voice_lab "$src"; then + if [[ -d "${src}/.git" && "$src" == "$CACHE" ]]; then + echo "==> updating ${src}" + origin="$(git -C "$src" remote get-url origin 2>/dev/null || true)" + [[ -n "$origin" ]] || fail "${src} has no origin" + remote_is_voice_lab "$origin" || fail "${src} origin is not vetcoders/voice-lab (got ${origin})" + git -C "$src" fetch --tags origin + git -C "$src" checkout --quiet main + git -C "$src" merge --ff-only origin/main + else + echo "==> using existing checkout ${src}" + fi + return + fi + + if [[ -e "$src" ]]; then + fail "${src} exists but is not a Voice Lab checkout" + fi + + need_git + REPO_URL="" + while IFS= read -r cand; do + [[ -n "$cand" ]] || continue + remote_is_voice_lab "$cand" || fail "VOICE_LAB_REPO_URL must point at the org voice-lab repo (got ${cand})" + echo "==> probing ${cand}" + probed="${probed}${probed:+, }${cand}" + if git ls-remote "$cand" HEAD >/dev/null 2>&1; then + REPO_URL="$cand" + break + fi + done < <(candidate_repo_urls) + if [[ -z "$REPO_URL" ]]; then + fail "no access to ${probed:-vetcoders/voice-lab}. Voice Lab is org-closed. Ask for vetcoders/voice-lab, or use make app without install-app." + fi + mkdir -p "$(dirname "$src")" + echo "==> cloning ${REPO_URL} → ${src}" + git clone --branch main --single-branch "$REPO_URL" "$src" + looks_like_voice_lab "$src" || fail "clone succeeded but ${src} is missing server.py / Monika pack" +} + +app_settings_path() { + echo "${HOME}/Library/Application Support/Codescribe/settings.json" +} + +# Codescribe does not invent the org cloud URL on first launch. The Monika +# pack in voice-lab does. Seed it on this hot path: missing file → copy; +# existing file → fill empty asr_mode / cloud_transcription_endpoint only. +# A host that already pointed STT at loopback or Libraxis is left alone. +seed_app_settings() { + local src="$1" + local pack="${src}/examples/monika/settings.json" + local dest + dest="$(app_settings_path)" + local mode="${VOICE_LAB_INSTALL_SETTINGS:-auto}" + + if [[ "$mode" == "0" ]]; then + echo "==> app settings skipped (VOICE_LAB_INSTALL_SETTINGS=0)" + return + fi + [[ -f "$pack" ]] || fail "Monika settings pack missing: ${pack}" + + if [[ "$mode" == "1" ]]; then + return + fi + + command -v python3 >/dev/null 2>&1 || fail "python3 is required to seed Codescribe settings" + python3 - "$pack" "$dest" <<'PY' +import json +import sys +from pathlib import Path + +pack = Path(sys.argv[1]) +dest = Path(sys.argv[2]) +wanted = json.loads(pack.read_text()) +engine = (wanted.get("speech") or {}).get("engine") or {} +want_endpoint = (engine.get("cloud_transcription_endpoint") or "").strip() +want_mode = (engine.get("asr_mode") or "").strip() + +if not dest.is_file(): + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_text(pack.read_text()) + print(f"==> seeded app settings from Monika pack → {dest}") + raise SystemExit(0) + +data = json.loads(dest.read_text()) +speech = data.setdefault("speech", {}) +cur = speech.setdefault("engine", {}) +changed = [] +if not str(cur.get("cloud_transcription_endpoint") or "").strip() and want_endpoint: + cur["cloud_transcription_endpoint"] = want_endpoint + changed.append("cloud_transcription_endpoint") +if not str(cur.get("asr_mode") or "").strip() and want_mode: + cur["asr_mode"] = want_mode + changed.append("asr_mode") +if changed: + dest.write_text(json.dumps(data, indent=2) + "\n") + print("==> filled empty engine keys:", ", ".join(changed)) +else: + print("==> app settings kept (endpoint/mode already set)") +PY +} + +print_settings_guarantee() { + local dest + dest="$(app_settings_path)" + if [[ ! -f "$dest" ]]; then + echo "==> app settings: none at ${dest}" + return + fi + command -v python3 >/dev/null 2>&1 || return + python3 - "$dest" <<'PY' +import json, sys +from pathlib import Path +data = json.loads(Path(sys.argv[1]).read_text()) +engine = (data.get("speech") or {}).get("engine") or {} +mode = engine.get("asr_mode") or "(unset)" +endpoint = engine.get("cloud_transcription_endpoint") or "(unset)" +print(f"==> app settings guarantee asr_mode={mode}") +print(f"==> app settings guarantee endpoint={endpoint}") +PY +} + +run_setup() { + local src="$1" + [[ -x "${src}/setup.sh" ]] || fail "missing ${src}/setup.sh" + local settings_flag="${VOICE_LAB_INSTALL_SETTINGS:-0}" + if [[ -z "${VOICE_LAB_INSTALL_SETTINGS:-}" ]]; then + settings_flag=0 + fi + echo "==> setup.sh → ${RUNTIME}" + INSTALL_PUBLIC_KEYS=1 \ + INSTALL_SETTINGS="$settings_flag" \ + SKIP_CODESCRIBE_CLONE=1 \ + "${src}/setup.sh" +} + +verify_runtime() { + [[ -f "${RUNTIME}/server.py" ]] || fail "runtime missing ${RUNTIME}/server.py" + [[ -x "$LAUNCHER" ]] || fail "launcher missing ${LAUNCHER}" + if [[ -f "${HOME}/.codescribe/config/dev/keys/sparkle-public.b64" ]]; then + : + elif [[ -f "${HOME}/.vibecrafted/secrets/codescribe/sparkle-public.b64" ]]; then + : + else + fail "Sparkle public key missing (~/.codescribe/config/dev/keys or Monika pack)" + fi + if [[ -f "${HOME}/.codescribe/config/dev/keys/license-public.hex" ]]; then + : + elif [[ -f "${HOME}/.vibecrafted/secrets/codescribe/license-public.hex" ]]; then + : + else + fail "license public key missing (~/.codescribe/config/dev/keys or Monika pack)" + fi + echo "==> Voice Lab runtime ${RUNTIME}" + echo "==> launcher ${LAUNCHER}" +} + +main() { + local src + src="$(resolve_src)" + ensure_checkout "$src" + run_setup "$src" + seed_app_settings "$src" + verify_runtime + print_settings_guarantee +} + +main "$@" diff --git a/scripts/tests/install-voice-lab-test.sh b/scripts/tests/install-voice-lab-test.sh new file mode 100755 index 00000000..edd60a8e --- /dev/null +++ b/scripts/tests/install-voice-lab-test.sh @@ -0,0 +1,221 @@ +#!/usr/bin/env bash +set -euo pipefail +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +INSTALL="$ROOT/scripts/install-voice-lab.sh" +WORKDIR="$(mktemp -d)" +trap 'rm -rf "$WORKDIR"' EXIT + +export HOME="$WORKDIR/home" +mkdir -p "$HOME" + +fake_git="$WORKDIR/bin" +mkdir -p "$fake_git" +cat >"$fake_git/git" <<'EOF' +#!/bin/sh +echo "unexpected git $*" >&2 +exit 99 +EOF +chmod +x "$fake_git/git" +export PATH="$fake_git:$PATH" + +ABSENT="$WORKDIR/absent" + +# No checkout, ls-remote fails → fail-closed. Force a clone path so a +# sibling voice-lab checkout on the operator machine cannot satisfy this. +cat >"$fake_git/git" <<'EOF' +#!/bin/sh +if [ "$1" = "ls-remote" ]; then + echo "Permission denied" >&2 + exit 128 +fi +echo "unexpected git $*" >&2 +exit 99 +EOF +set +e +out="$( + CODESCRIBE_VOICE_LAB_SRC="$ABSENT" \ + VOICE_LAB_REPO_URL="git@github.com:vetcoders/voice-lab.git" \ + "$INSTALL" 2>&1 +)" +status=$? +set -e +[[ "$status" -ne 0 ]] || { echo "expected fail without repo access, got: $out" >&2; exit 1; } +[[ "$out" == *org-closed* ]] || { echo "expected org-closed message, got: $out" >&2; exit 1; } + +# Wrong repo URL → fail before clone. +set +e +out="$( + CODESCRIBE_VOICE_LAB_SRC="$ABSENT" \ + VOICE_LAB_REPO_URL="https://github.com/octocat/Hello-World.git" \ + "$INSTALL" 2>&1 +)" +status=$? +set -e +[[ "$status" -ne 0 ]] || { echo "expected fail on non-voice-lab URL, got: $out" >&2; exit 1; } +[[ "$out" == *"voice-lab repo"* ]] || { echo "expected repo-name check, got: $out" >&2; exit 1; } + +# Substring "voice-lab" is not enough — attacker-controlled fork is rejected. +set +e +out="$( + CODESCRIBE_VOICE_LAB_SRC="$ABSENT" \ + VOICE_LAB_REPO_URL="https://github.com/octocat/voice-lab.git" \ + "$INSTALL" 2>&1 +)" +status=$? +set -e +[[ "$status" -ne 0 ]] || { echo "expected fail on non-org voice-lab URL, got: $out" >&2; exit 1; } +[[ "$out" == *"voice-lab repo"* ]] || { echo "expected org lock, got: $out" >&2; exit 1; } + +# Unset URL: HTTPS first when gh git_protocol=https, then SSH. +probe_log="$WORKDIR/probes.log" +: >"$probe_log" +cat >"$fake_git/gh" <<'EOF' +#!/bin/sh +if [ "$1" = "config" ] && [ "$2" = "get" ] && [ "$3" = "git_protocol" ]; then + echo https + exit 0 +fi +exit 1 +EOF +chmod +x "$fake_git/gh" +cat >"$fake_git/git" <> "$probe_log" + echo "Permission denied" >&2 + exit 128 +fi +echo "unexpected git \$*" >&2 +exit 99 +EOF +set +e +out="$( + CODESCRIBE_VOICE_LAB_SRC="$ABSENT" \ + "$INSTALL" 2>&1 +)" +status=$? +set -e +[[ "$status" -ne 0 ]] || { echo "expected fail after both probes, got: $out" >&2; exit 1; } +[[ "$out" == *org-closed* ]] || { echo "expected org-closed after dual probe, got: $out" >&2; exit 1; } +probe_n="$(wc -l <"$probe_log" | tr -d ' ')" +https_probe="$(sed -n '1p' "$probe_log")" +ssh_probe="$(sed -n '2p' "$probe_log")" +[[ "$probe_n" == "2" ]] || { echo "expected 2 probes, got ${probe_n}: $(cat "$probe_log")" >&2; exit 1; } +[[ "$https_probe" == "https://github.com/vetcoders/voice-lab.git" ]] || { + echo "https protocol must probe HTTPS first, got ${https_probe}" >&2 + exit 1 +} +[[ "$ssh_probe" == "git@github.com:vetcoders/voice-lab.git" ]] || { + echo "second probe must be SSH for Monika fallback, got ${ssh_probe}" >&2 + exit 1 +} + +# Monika: gh git_protocol=ssh probes SSH first. +: >"$probe_log" +cat >"$fake_git/gh" <<'EOF' +#!/bin/sh +if [ "$1" = "config" ] && [ "$2" = "get" ] && [ "$3" = "git_protocol" ]; then + echo ssh + exit 0 +fi +exit 1 +EOF +set +e +out="$( + CODESCRIBE_VOICE_LAB_SRC="$ABSENT" \ + "$INSTALL" 2>&1 +)" +status=$? +set -e +[[ "$status" -ne 0 ]] || { echo "expected fail after ssh-first probes, got: $out" >&2; exit 1; } +ssh_first="$(sed -n '1p' "$probe_log")" +https_second="$(sed -n '2p' "$probe_log")" +[[ "$ssh_first" == "git@github.com:vetcoders/voice-lab.git" ]] || { + echo "ssh protocol must probe SSH first, got ${ssh_first}" >&2 + exit 1 +} +[[ "$https_second" == "https://github.com/vetcoders/voice-lab.git" ]] || { + echo "ssh protocol second probe must be HTTPS, got ${https_second}" >&2 + exit 1 +} + +# Restore the fail-closed git stub used by later cases. +cat >"$fake_git/git" <<'EOF' +#!/bin/sh +if [ "$1" = "ls-remote" ]; then + echo "Permission denied" >&2 + exit 128 +fi +echo "unexpected git $*" >&2 +exit 99 +EOF +rm -f "$fake_git/gh" + +# Sibling-shaped checkout via CODESCRIBE_VOICE_LAB_SRC runs setup.sh. +pack="$WORKDIR/pack" +mkdir -p "$pack/examples/monika/keys" +printf 'x' >"$pack/server.py" +cat >"$pack/examples/monika/settings.json" <<'JSON' +{ + "schema_version": 3, + "speech": { + "engine": { + "cloud_transcription_endpoint": "wss://api.libraxis.cloud/v1/audio/transcribe", + "asr_mode": "local_power" + } + } +} +JSON +cat >"$pack/setup.sh" < "\$HOME/.codescribe/voice-lab/server.py" +echo '#!/bin/sh' > "\$HOME/.codescribe/bin/voice-lab" +chmod 755 "\$HOME/.codescribe/bin/voice-lab" +echo sparkle > "\$HOME/.vibecrafted/secrets/codescribe/sparkle-public.b64" +echo license > "\$HOME/.vibecrafted/secrets/codescribe/license-public.hex" +echo ran-setup +EOF +chmod +x "$pack/setup.sh" + +out="$(CODESCRIBE_VOICE_LAB_SRC="$pack" "$INSTALL" 2>&1)" +[[ -f "$HOME/.codescribe/voice-lab/server.py" ]] || { echo "runtime not installed" >&2; exit 1; } +[[ -x "$HOME/.codescribe/bin/voice-lab" ]] || { echo "launcher missing" >&2; exit 1; } +[[ "$out" == *ran-setup* ]] || { echo "setup.sh did not run: $out" >&2; exit 1; } +[[ "$out" == *seeded\ app\ settings* ]] || { echo "expected seed on missing settings: $out" >&2; exit 1; } +[[ "$out" == *endpoint=wss://api.libraxis.cloud/v1/audio/transcribe* ]] || { + echo "expected libraxis guarantee, got: $out" >&2 + exit 1 +} + +# Existing loopback endpoint must not be overwritten. +export HOME="$WORKDIR/home-keep" +mkdir -p "$HOME/Library/Application Support/Codescribe" +cat >"$HOME/Library/Application Support/Codescribe/settings.json" <<'JSON' +{ + "speech": { + "engine": { + "cloud_transcription_endpoint": "ws://127.0.0.1:8446/v1/audio/transcribe", + "asr_mode": "local_power" + } + } +} +JSON +out="$(CODESCRIBE_VOICE_LAB_SRC="$pack" "$INSTALL" 2>&1)" +[[ "$out" == *app\ settings\ kept* ]] || { echo "expected keep existing endpoint: $out" >&2; exit 1; } +[[ "$out" == *endpoint=ws://127.0.0.1:8446/v1/audio/transcribe* ]] || { + echo "loopback must survive seed, got: $out" >&2 + exit 1 +} + +# Empty engine keys get the pack values without replacing the file wholesale. +export HOME="$WORKDIR/home-fill" +mkdir -p "$HOME/Library/Application Support/Codescribe" +printf '%s\n' '{"schema_version":3,"speech":{"engine":{}}}' \ + >"$HOME/Library/Application Support/Codescribe/settings.json" +out="$(CODESCRIBE_VOICE_LAB_SRC="$pack" "$INSTALL" 2>&1)" +[[ "$out" == *filled\ empty\ engine\ keys* ]] || { echo "expected fill: $out" >&2; exit 1; } +[[ "$out" == *asr_mode=local_power* ]] || { echo "expected mode fill: $out" >&2; exit 1; } + +echo "install-voice-lab: ok" diff --git a/site/src/pages/voice/lab.astro b/site/src/pages/voice/lab.astro index 078fd77b..43ce0b37 100644 --- a/site/src/pages/voice/lab.astro +++ b/site/src/pages/voice/lab.astro @@ -1,8 +1,8 @@ --- import Layout from '../../layouts/Layout.astro'; import Nav from '../../components/Nav.astro'; -import Footer from '../../components/Footer.astro'; import { page } from '../../lib/asset'; +import '../../styles/lab-shell.css'; --- -