fix(workspace): the /stt provider error says why, instead of one opaque 422 (#2696)

[dolho]

Stacked on #2699 (base = fix/2695-stt-capability-probe) — both change the same transcribe_portal_audio branch. Retarget to dev once #2699 lands.

Summary

transcribe_portal_audio mapped every non-200 from ElevenLabs onto 422 "Could not transcribe the audio". Missing permission, rejected key, exhausted credits, provider rate limit and a rejected audio container all read identically, while the status word sat in a backend WARNING.

What changes

• classify_stt_failure(status, body) (pure, in stt_capability_service) — one named category per provider condition:

  category	provider	client gets
  permission / auth / quota	401 · 402 · 403, by status word	503 + operator-actionable sentence ("…missing the speech-to-text permission. Ask your operator…")
  rate_limit	429	429, the existing retry wording
  audio	400 · 413 · 415 · 422	422 "That recording could not be read…"
  provider	5xx	502, the existing transport wording
  unknown	anything else	502, still says the speech service failed

  No arm returns the old string; test_there_is_no_arm_that_returns_the_opaque_string sweeps every status 300–599.

• Client never sees the provider body — only the category sentence. record_live_failure keeps status word + category under stt:last_failure:<sha256(key)[:16]> (24h) and still feeds bug(workspace): the mic renders on ElevenLabs key presence, not STT capability — a key without speech_to_text gives a control that fails every time #2695's capability cache on 401/403.

• Operator surface — GET /api/settings/elevenlabs (admin-only route) carries stt_last_failure {category, provider_status, detail, at}; Settings → Voice renders "Last voice-input failure: the key is missing the speech-to-text permission (HTTP 401 missing_permissions) — " under the capability badge (utils/sttCapability.js::describeSttLastFailure, pure + tested).

• Frontend transcriptionErrorMessage() already prefers the server detail, so the new sentences reach the user with no composer change.

Acceptance criteria

• auth/permission failure distinct from audio rejection — test_categories_are_distinguishable_by_their_client_sentence
• quota distinguishable from auth — same test
• provider rate limit → 429 with existing retry wording — test_rate_limit_maps_to_429_with_the_existing_retry_wording
• no raw body to the client; operator detail admin-only — test_client_message_never_carries_the_provider_body, test_the_portal_card_carries_no_operator_detail, test_settings_state_carries_the_last_failure_for_admins_only_by_route
• fail-soft, no 500s — asserted on every endpoint case
• mapping pinned, unrecognised status can't regress — the 300–599 sweep + unknown category

Coverage

Endpoint tests execute transcribe_portal_audio with the provider stubbed at httpx.AsyncClient (6 status cases + success + operator memory); the classifier is driven directly. No source-text assertions.

Verification

pytest test_2696_* test_2695_* test_2212_* test_2163_* test_*portal* test_*settings*  → 730 passed
vitest sttCapability.spec.js (10) + rawColorRatchet.spec.js (8)                     → pass
vite build                                                                          → ok

Fixes #2696

🤖 Generated with Claude Code

https://claude.ai/code/session_015owqMKD5QDjzZrUTF2Joht

[dolho]

Review — /review pass

The shape is right: one named category per provider condition, a pure classifier, the mapping pinned by a 300–599 sweep so an unrecognised status cannot regress to the opaque string, and the client sentence separated from the operator detail. record_live_refusal correctly stays gated on 401/403, so a 402/429/5xx does not poison the #2695 capability verdict — that was the first thing I checked and it is right.

Two findings, the first reproduced.

[C1] A prose 401 is classified as quota, so a rejected key tells the operator to buy credits (confidence 10/10 — executed)

_QUOTA_WORDS is documented as "matched as substrings of the status token", but provider_status_word() falls back to free text:

word = det.get("status") or det.get("code") or det.get("message")

so when the provider sends no token the matcher runs on prose. Driving classify_stt_failure directly:

401 {"detail":{"status":"missing_permissions"}}                       -> permission
401 {"detail":{"status":"invalid_api_key"}}                           -> auth
401 {"detail":"Invalid API key for your plan"}                        -> quota   ← wrong
401 {"detail":{"message":"…not valid for this subscription"}}          -> quota   ← wrong
402 {"detail":{"status":"quota_exceeded"}}                            -> quota

Both wrong rows are ordinary auth failures. The client is told "the speech recognition account is out of credits or not on a plan that allows it" and the operator panel says "out of credits, or the plan does not allow speech-to-text" — pointing at billing for a key that simply needs replacing. That is worse than the opaque 422 this PR replaces, because it is confidently wrong in a direction the operator will act on.

The categories are otherwise well chosen; the defect is only that one matcher cannot tell a token from a sentence. Fix: have provider_status_word report WHICH it found (or return a (token, message) pair) and run _QUOTA_WORDS only against a token; a 401/403 with prose only is auth. Worth a case in the table-driven test for each of the four rows above — the current fixtures all carry a token, which is why it passes.

[I1] The client sentences disclose the operator's account state to an external customer (confidence 8/10)

transcribe_portal_audio serves the Workspace, whose callers include external clients with no users row. The AC drew the line at "no raw body to the client", and the body is indeed withheld — but the category sentences themselves state operator-internal facts:

• "the speech recognition key is missing the speech-to-text permission. Ask your operator to update it"
• "the speech recognition account is out of credits or not on a plan that allows it"

A paying customer of the operator learning that the operator's ElevenLabs account is out of credits is a disclosure the rest of this file is careful about — _scrub_title_detail exists a few hundred lines away for exactly this class, and client_portal/agent_page.py refuses to expose free-form agent JSON for the same reason. Suggest keeping the categories (they drive the status code and the operator panel) but collapsing the client wording for permission/auth/quota to one neutral sentence — "Voice input isn't available right now — you can type instead" — and letting Settings → Voice carry the cause. The operator still gets everything; the customer stops learning about the operator's billing.

[I2] 503 for a condition no retry will fix (confidence 6/10)

permission / auth / quota all answer 503, which is the "try again later" status — clients and proxies may retry, and nothing will change until a human edits a key. Given the portal's own error handling prefers the server detail, the status mostly affects retry behaviour rather than wording, so this is a judgement call rather than a defect — but if the sentence is going to say "ask your operator", the status saying "temporarily unavailable" is the opposite claim.

Checked and clean

• No arm returns the old string, and the 300–599 sweep is the right way to pin that — an unrecognised status lands in unknown with a specific sentence rather than falling through.
• record_live_refusal gating: 401/403 only, so quota/rate-limit/provider failures do not mark a working key refused and hide a mic that works. The fail-soft direction is preserved.
• The client never receives detail — the endpoint raises failure.http_status, failure.client_message and nothing else; detail reaches only the admin-gated settings route.
• Cache keying stays a digest of the key, so the last-failure row rotates with the key by construction, and _local_failures mirrors _local's fail-open shape.
• The frontend half is pure and tested (describeSttLastFailure), consistent with the bug(workspace): the mic renders on ElevenLabs key presence, not STT capability — a key without speech_to_text gives a control that fails every time #2695 module beside it.
• Endpoint tests execute the real function with the provider stubbed at httpx.AsyncClient rather than asserting source text — the right level for this change.

Note

mergeable shows UNKNOWN because this stacks on #2699, which I have just merged dev into and pushed. Once GitHub recomputes, retarget to dev after #2699 lands as the body says.

[dolho]

Re-review — [C1] fixed

1fcc6ed3f. The cause was one function answering two different questions. provider_status_parts(body) -> (token, prose) now keeps them apart:

• token = detail.status / detail.code — a machine value the provider documents;
• prose = detail.message, or a bare string detail — a sentence written for a human, whose words mean nothing in particular.

classify_stt_failure matches only the token. A 401/403 carrying prose alone is auth — the honest reading of "the key was rejected and the provider did not say why". provider_status_word survives as the operator-facing display value (token else prose), so the panel loses nothing: the prose still shows, it just no longer votes.

Re-drove the classifier over the full table — all eleven rows, including the two that were wrong:

401 token missing_permissions   -> permission  detail='missing_permissions'
401 token invalid_api_key       -> auth        detail='invalid_api_key'
401 PROSE mentioning plan       -> auth        detail='Invalid API key for your plan'      <- was quota
401 PROSE via message           -> auth        detail='...not valid for this subscription' <- was quota
401 token quota_exceeded        -> quota       detail='quota_exceeded'
402 (any body) -> quota    429 -> rate_limit    422 -> audio
500 -> provider            418 -> unknown

Note the last column: the operator still gets the sentence, it is just no longer mistaken for a verdict.

Coverage. Five rows added to the mapping table (three prose-shaped auth failures, plus two where a token decides despite quota words in the prose around it), and three focused tests — the token/prose split, the rule itself, and that the operator detail still falls back to prose. Verified in the backend image rather than claimed: 38 passing in this suite, 66 with #2695's alongside.

Checked for the same class elsewhere: provider_status_word(body) now has exactly one remaining caller, classify_response (#2695), where the value is display-only — stored as the refusal detail, never branched on. No other matcher consumes provider text.

No new failures: the *portal* + *settings* sweep has the byte-identical non-passing set (50 items) on this branch and on its base, so the failures there are environmental (no DB or Redis in a bare container), not this change.

[I1] (client sentences disclosing the operator's billing state to an external Workspace customer) and [I2] (503 for a condition no retry fixes) are unchanged and still worth a decision — say the word and I'll take those too.

[vybe]

merge-train: not on today's train, and not because of anything in this PR. It is stacked on #2699, which was held back for the cross-worker cache staleness in stt_capability_service.py. This one lands right behind it on the next train.

Validation came back clean. The leak question in particular: the portal principal receives only failure.client_message, one of seven fixed _MSG_* constants, never interpolated from the provider body or the key. Provider text is confined to SttFailure.detail, truncated to 120 chars, and surfaced solely through the assert_admin-gated settings route. The 300–599 sweep test pins that no status falls through, and the tests execute the handler with a mocked provider rather than matching source text.

Two things to decide before it ships, since both are product calls rather than defects:

• The category sentences tell an external client about the operator's account state — "missing the speech-to-text permission", "out of credits or not on a plan that allows it". Not a secret, but it is the operator's billing posture handed to a third party, and the rest of client_portal/service.py is careful about exactly this. Keeping the categories for the status and admin panel while collapsing the three 503 client strings to one neutral "Voice input isn't available right now — type instead." would cost nothing.
• 503 for permission, auth and quota describes conditions no retry fixes, and proxies may retry them anyway.

One mechanical note for merge time: backend-unit-test has never run on this PR, because the workflow only fires for pull_request against dev or main and your base is a feature branch. Retargeting to dev after #2699 lands triggers it. I ran it locally at 1fcc6ed: 66 backend tests and 52 frontend tests pass. No code change is needed for the retarget — the merge-tree against #2699's head is conflict-free.

[dolho]

/review — head 1fcc6ed3 vs its base PR #2699

Own diff (origin/fix/2695-stt-capability-probe...HEAD): 9 files (+552/−20). Scope: CLEAN — one classifier, one recorder, one reader, the endpoint's raise, and the admin-panel half.
Stacking note: this branch's merge-base with #2699 is 06b86be9; #2699 has since gained four commits (beae0499 cross-worker fix … 2b966b41), which is why mergeable reads UNKNOWN. Merge #2699's head in before the next train — and see I1, which is a direct consequence.

Critical

None.

Informational

[I1] read_last_failure still falls through to the local copy on a Redis MISS — the exact class #2699 just closed for read_cached (Confidence 7/10)
stt_capability_service.py (this branch) — read_last_failure: if not raw: hit = _local_failures.get(k). A Redis miss falls through to _local_failures. That is the pre-beae0499 shape of read_cached, which #2699's 2026-09-14 finding replaced with "a Redis MISS is a miss; _local only when Redis cannot be asked". This branch predates that fix, so after the merge the two readers in one module will disagree about what a miss means. Consequence is bounded (both TTLs are 24 h, and at is shown), but it is the same staleness: the worker that recorded a failure keeps reporting it after the row is gone from Redis.
Fix: mirror read_cached — local only on r is None or a raised read; a miss is None. And invalidate(key) should also delete _failure_row(key) — re-saving a key after fixing it at the provider is the documented recovery, and today the panel keeps showing the pre-fix failure for up to 24 h beside a fresh capable verdict.

[I2] 401/403 → HTTP 503 to the client (Confidence 5/10)
classify_stt_failure: permission/auth/quota all answer 503. Correct for the client (it is unavailable to them, and transcriptionErrorMessage renders the detail sentence — verified at portalUtils.js:1436-1445), and 503 is not one of the statuses the frontend special-cases, so the sentence always wins. Just noting that a 503 from this route will show up in any upstream-error dashboards as a Trinity outage rather than a key problem; the log line one frame up carries the real status, so this is a monitoring nuance, not a defect.

Clean

• Client message never carries the provider body — _MSG_* are constants; the body reaches only the operator-facing detail ([:120]) behind the admin-gated route. Pinned by test_client_message_never_carries_the_provider_body.
• Matcher reads the token, not prose — the 1fcc6ed fix; test_a_quota_word_in_prose_is_not_a_quota_verdict is the regression test and would have caught the original.
• Every status lands in a named category; no arm returns the old opaque string (test_there_is_no_arm_that_returns_the_opaque_string).
• Tests execute the changed path — test_endpoint_answers_with_the_category_message / test_endpoint_remembers_the_failure_for_the_operator drive transcribe_portal_audio past the roster gate with a stubbed provider.
• Capability cache is taught only by 401/403 (test_a_bad_recording_teaches_the_operator_but_not_the_capability_cache).
• Auth: no new route; the last-failure detail is admin-only by virtue of GET /elevenlabs; the portal card gains no operator field (test_the_portal_card_carries_no_operator_detail).

Verdict: READY once rebased on #2699's head; fold I1 into that merge — it is four lines, and it is the finding the base PR was held for.

[dolho]

Rebuilt on dev and retargeted from fix/2695-stt-capability-probe to dev — #2699 squash-merged at 14:50, so every one of this branch's copies of its commits conflicted add/add with the squash. A merge was the wrong tool; the branch is now #2696's own two commits cherry-picked onto origin/dev (105a7024, b40225b7), plus:

• b31a25f2 — the one doc sentence bug(workspace): the /stt 422 collapses every provider failure into "Could not transcribe the audio" while the cause sits in a backend log #2696 adds to the stt_capability_service bullet, re-applied onto dev's final wording of that bullet (the only cherry-pick conflict).
• 32badbee — the /review I1 from yesterday, folded in as promised: read_last_failure now obeys the same cross-worker rule as read_cached (Redis answers ⇒ authoritative, a miss evicts the local copy, local only when Redis cannot be asked), and invalidate drops the key's last-failure row so re-saving a fixed key does not show the pre-fix failure for 24h beside a fresh capable. Three two-worker tests; restoring the fallthrough turns one red.

The only source delta against the old tip outside those is sttCapability.js taking dev's newer hint wording (#2699's final) — correct direction. Backend family (test_2695, test_2696, test_2212, test_2157, test_ent403): 154 passed; sttCapability.spec.js: 10 passed. git merge-tree against dev: clean.

[dolho]

/validate-pr + /review — head 32badbee7 vs dev (first pass against the retargeted base, merge-train pre-validation, lane B)

Note: GitHub was still showing this PR as 260 files from the pre-retarget base.sha; re-PATCHing base=dev made it recompute — 9 files, 4 commits on 0bddbc959.

Scope: CLEAN — named client-facing reason per provider condition, raw body never to the client, operator detail admin-only, 429 keeps retry wording, no 500s, mapping pinned; plus the stt_last_failure admin field. 6/6 ACs DONE.

Rebuild onto dev lost nothing, gained three things. The old stacked head predated a #2699 follow-up and lacked the Redis-is-authority read_cached; the rebuild inherits it. Present in the head: the quota-word rule (lt = (token or "").lower() — the TOKEN only, never prose; mutation token or prose → 4 red), the cross-worker last-failure rule (read_last_failure pops the local copy on a Redis miss; test_last_failure_redis_miss_is_a_miss_even_with_a_local_copy drives two module instances over one fake Redis), the invalidate path (r.delete(k, fk) + local pop). The diff vs dev touches only the #2696 lines (service.py 15, settings.py 6).

Critical: none. Client receives only failure.client_message, one of seven module constants — classify_stt_failure never interpolates body/token/prose into it (test_client_message_never_carries_the_provider_body plants sk_live_abc in prose). New field rides GET/PUT /api/settings/elevenlabs, both assert_admin (settings.py:3043, :3057). Settings.vue renders via mustache, no v-html.

Informational

• I1 (5/10) operator detail is not credential-scrubbed: word = token or prose → Redis 24h → admin panel. ElevenLabs's documented bodies don't echo the key, and the admin owns it; mechanical hardening is scrub_secret(word, api_key) inside record_live_failure.
• I2 (7/10) test_settings_state_carries_the_last_failure_for_admins_only_by_route asserts state shape only; the admin gate is covered by test_117_voice_replies_v2.py:108. Name over-claims, no action.
• I3 no feature-flow doc for the STT path (same gap fix(workspace): the mic renders on speech-to-text capability, not key presence (#2695) #2699 shipped with). Non-blocking.

Coverage: executed — transcribe_portal_audio driven with a monkeypatched httpx.AsyncClient fake provider (6 endpoint cases + operator-memory case), classifier over 21 bodies + a 300–599 sweep, real _elevenlabs_settings_state_with_capability(). Local: 144 passed (PR + voice suites), vitest 30/30.

Verdict: READY. I1 will be applied on this branch before the train (announced here when pushed). Fixes #2696 bare form ✓; CI green incl. e2e (ui label present).

[dolho]

merge-train: pushed to this branch — one commit, mechanical, per the review above (I1).

record_live_failure now runs scrub_secret(detail, api_key) before the operator row is stored, so the Settings panel's "key is never echoed" promise is enforced rather than assumed of the provider. test_operator_detail_never_carries_the_key drives a body carrying the key through the real path; mutation (scrub removed) → red. STT suites: 75 passed. Nothing else touched.

[vybe]

merge-train: batch validated on train/20260916-0919 (train PR #2839)