[feature] phrase dictionary learns from post-dictation corrections - #295
Merged
Conversation
Voice typing pastes and forgets; a word the STT misheard stays wrong in every future dictation. This adds a phrase dictionary with a Typeless-style learning loop: - After auto-paste, watch the pasted-into field via the Accessibility API (already granted for the paste itself) for up to 60 s. When the user fixes a word in place, diff the settled value against what we inserted and offer — in the non-activating overlay bubble — to add the correction (Alt+Enter or click; ignore twice and it stops asking). - dictionary.json (app config dir) is the shared truth: Settings panel, the voice-typing/meeting pipelines, and four new MCP tools (list/add/update/delete_dictionary_phrase) all read and write it. - Entries bias recognition at the source — Soniox context.terms, Deepgram keyterm/keywords, AssemblyAI keyterms_prompt/word_boost, OpenAI transcription prompt, Gemini systemInstruction — and known variants are rewritten in normalizeTranscriptText before display, paste, or import. Voice typing also finally sends real languageHints. - History entries record the app they were pasted into, and gain an inline correct-and-learn editor as the manual fallback for apps whose AX values are unreadable. Observation is deliberately narrow: only the field we just pasted into, only for a minute, diff computed locally and discarded. The hosted batch path carries no vocabulary yet (cloud endpoint has no field for it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Voice typing gains a phrase dictionary with a learn-from-correction loop: after auto-paste, Parley watches the pasted-into field via the Accessibility API (up to 60 s); when the user fixes a misheard word in place, the overlay bubble offers to add the correction (⌥↩ or click, ignore twice and it stops asking). Dictionary phrases bias recognition at every STT provider (Soniox
context.terms, Deepgramkeyterm/keywords, AssemblyAIkeyterms_prompt/word_boost, OpenAI transcription prompt, GeminisystemInstruction) and known variants are rewritten before display/paste/import. Also: a Dictionary panel in Settings, a correct-and-learn editor in voice-typing history (manual fallback for AX-opaque apps), four new MCP tools (list/add/update/delete_dictionary_phrase), history entries now record the app they were pasted into, and voice typing finally sends reallanguageHints.Why
STT keeps mangling proper nouns and voice typing was fire-and-forget — the same word came back wrong in every dictation, with no way to teach the app. A correction made seconds after a paste is a free, high-quality label; catching it needs no keylogging because Accessibility (already granted for the paste itself) can read the focused field. Observation is deliberately narrow: only the field we just pasted into, only for a minute, diff computed locally and discarded.
Closes #294.
How it was verified
bunx tsc --noEmitpassesbunx vitest runpasses (255 tests, incl. 24 new for the correction diff + replacements)cargo test(42) andclippy -D warningscleanzh-TWandeninsrc/i18n/messages.tsNot yet exercised end-to-end: the live mic → in-place correction → bubble flow needs a human at the keyboard; the paste-landing race (baseline snapshotted before the target app processes ⌘V) is guarded by a re-baseline check with unit tests.
Known gap: the hosted-batch path carries no vocabulary yet — the cloud endpoint has no field for it (needs a parley-internal change; hosted live relay is unaffected since it forwards the Soniox config frame verbatim).
Screenshots
The suggestion bubble only exists mid-flow (it is the non-activating dictation overlay re-purposed), so no static capture here; the Settings → Dictionary panel follows the existing Evaluations panel layout.