diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index 0200cc5..ff85d63 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -6,70 +6,103 @@ The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a given transcription result (`SpeechRecognitionResult`). This limitation creates two major challenges for API clients and end users: -- **Timeline Association:** Developers cannot readily associate transcribed text with specific segments of the audio source, making it difficult to map generated captions to media timelines or audio tracks. +- **Timeline Association:** Developers cannot readily associate transcribed text with specific segments of the audio source, making it difficult to map generated captions to media timelines, audio tracks, or video frames. - **Latency Tracking & Backend Failover:** With the adoption of on-device Automatic Speech Recognition (ASR) to improve privacy and reduce server costs, processing performance becomes heavily dependent on local client hardware resources. The Web Speech API acts as a "black box" regarding local processing delays. Developers cannot programmatically calculate transcription latency or detect when on-device models fall behind real-time. This leads to poor user experiences (e.g. caption lag during live video conferencing) and deprives applications of the signal needed to seamlessly fail over to high-performance cloud backends. ### Proposed Solution -We propose extending the `SpeechRecognitionResult` interface to include optional (nullable) `audioStartTime` and `audioEndTime` attributes. +We propose extending the `SpeechRecognitionResult` interface to include `audioStartTime` and `audioEndTime` attributes. #### Web IDL Definition ```webidl partial interface SpeechRecognitionResult { - // Start timestamp of the audio segment in milliseconds (relative to the start of the audio stream) - readonly attribute DOMHighResTimeStamp? audioStartTime; + // Start timestamp of the audio segment in seconds relative to the start of the audio stream (0.0s). + readonly attribute double audioStartTime; - // End timestamp of the audio segment in milliseconds (relative to the start of the audio stream) - readonly attribute DOMHighResTimeStamp? audioEndTime; + // End timestamp of the audio segment in seconds relative to the start of the audio stream. + readonly attribute double audioEndTime; }; ``` +### Choice of Time Representation: Seconds as `double` + +The timestamps `audioStartTime` and `audioEndTime` are defined as `double` representing **seconds**, rather than `DOMHighResTimeStamp` (milliseconds). This design choice is based on the following considerations: + +1. **Consistency with Adjacent Web Audio & Media APIs:** + * In adjacent W3C media specifications, media-local stream timelines are universally represented in **seconds** as a `double`: + * **Web Audio API:** [`BaseAudioContext.currentTime`](https://webaudio.github.io/web-audio-api/#dom-baseaudiocontext-currenttime) (seconds) + * **HTML Media Elements:** [`HTMLMediaElement.currentTime`](https://html.spec.whatwg.org/multipage/media.html#dom-media-currenttime) (seconds) + * **AudioParam Scheduling:** [`AudioParam.setValueAtTime()`](https://webaudio.github.io/web-audio-api/#dom-audioparam-setvalueattime) (seconds) + * Using seconds ensures seamless interoperability when developers route audio between `