fix(cartesia): learn word-timestamp support from the response - #6637
fix(cartesia): learn word-timestamp support from the response#6637biztex wants to merge 4 commits into
Conversation
… timestamps word_timestamps=True (the default) set capabilities.aligned_transcript=True unconditionally at construction. For model/language combinations Cartesia documents as unable to return timing data, the plugin warned about the combination but left the capability set - so every turn downstream selected the TTS-aligned transcript path and warned that no agent transcript was returned from tts. Validate at construction instead: when the combination cannot deliver word timestamps, disable word_timestamps with a single clear warning, which both clears the capability and stops requesting timestamps from the API. An explicit word_timestamps=False opt-out stays silent. Fixes livekit#6493
| "word_timestamps disabled: it is only supported for languages en, de, es, and fr" | ||
| " with `sonic` models, or all languages with `preview` models" | ||
| ) | ||
| word_timestamps = False |
There was a problem hiding this comment.
generally I'm not a fan of hardcoding values based on agent-side settings. when the provider adds support for other languages, we would not be able to leverage them.
There was a problem hiding this comment.
Agreed, and the concern is sharper than it looks: that language list is already on main (tts.py:184), but there it only warns. My change promoted it to a behavioural gate, which is exactly where a stale table starts costing users — someone on a newly supported language would silently lose word timestamps, and the only clue would be a warning telling them their language isn't supported when it is.
Reworked in a6a85ed to learn it from the provider instead, and the hardcoded list is gone entirely (both the new gate and the pre-existing warning).
The signal is already in the stream: Cartesia just omits word_timestamps for combinations that can't produce them, so a synthesis that pushed audio but never sent a timestamp payload settles the question. When that happens the plugin clears capabilities.aligned_transcript and stops setting add_timestamps on later requests, so the framework stops selecting the TTS-aligned transcript path and the "no agent transcript was returned from tts" warning stops repeating. Nothing is refused up front, so when Cartesia adds a language or model it works immediately with no release on our side.
The trade-off is that the first synthesis on an unsupported combination still takes the aligned path once before the plugin knows — which seems like the right side to err on, since the alternative is guessing.
One thing I left alone, in case it's worth a separate issue: this only fixes the plugin's half. perform_tts_inference resolves timed_texts_fut to the channel as soon as the node is streaming, before any timed word arrives (generation.py:375-379), and agent_activity.py commits text_source on that truthiness — so any provider whose alignment silently doesn't arrive hits the same warning. #6493 mentions ElevenLabs seeing it too. Happy to take that on if you'd like it fixed centrally.
which model/language combinations return word timestamps is the provider s business and changes over time, so a table in the plugin would block users from combinations Cartesia has since added support for. A synthesis that produced audio but no timestamps settles it instead: the capability is cleared and later requests stop asking, so the transcript stops being read from an alignment that never arrives.
biztex
changed the title
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
update_options() can change the model or the language, which are exactly what decides whether timestamps arrive, so a negative result learned for the previous combination said nothing about the new one and left a supported combination degraded for the rest of the session. The observation is now recorded per combination and the request flag is re-derived when either changes, with the caller original request kept apart from what was learned.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…o the request punctuation, an emoji or the trailing end packet still yield audio while legitimately carrying no words, so treating that as proof the combination cannot align would disable timing for a combination that supports it. The conclusion now requires that word-bearing text was actually sent. It is also keyed to the model and language the finished request was sent with, taken from the stream snapshot: update_options() may have moved the instance on while the synthesis was in flight, which would otherwise record the result against the wrong combination.
2 participants
Addresses the Cartesia half of #6493.
Problem
word_timestamps=True(the default) setscapabilities.aligned_transcript=Trueat construction, but only some model/language combinations actually return timing data. For the others the capability stayed set, so every turn the framework selected the TTS-aligned transcript path, got nothing, and logged:The plugin already knew something was off — it warned about the combination — but the warning didn't change what the capability claimed.
Change
The plugin learns which combinations work from the responses rather than declaring it:
word_timestampsfor combinations that can't produce them, so a synthesis that pushed audio but never sent a timestamp payload settles it.capabilities.aligned_transcriptand stops settingadd_timestampson later requests, so the framework stops selecting the aligned path and the warning stops repeating.The trade-off is that the first synthesis on an unsupported combination still takes the aligned path once, before the plugin has seen a response to learn from.
Not covered here
This is the plugin's half only.
perform_tts_inferenceresolvestimed_texts_futto the channel as soon as the node is streaming, before any timed word has arrived (generation.py:375-379), andagent_activity.pycommitstext_sourceon that truthiness — so any provider whose alignment silently doesn't arrive reaches the same warning. #6493 reports ElevenLabs hitting it too. Happy to follow up centrally if that's wanted.Verification
tests/test_cartesia_tts_capabilities.pycovers construction not prejudging any language, an explicitword_timestamps=Falsestill clearing the capability, and the learning step clearing both the capability and the request flag, naming the model/language it learned about, warning once, and staying a no-op when timestamps were never requested.ruff format --check,ruff check,check_types.py(mypy strict) and the fullpytest --unitsuite pass locally.