feat: chunk long audio for duration-capped OpenAI models (gpt-4o-transcribe) - #478
Open
Swahjak wants to merge 5 commits into
Open
feat: chunk long audio for duration-capped OpenAI models (gpt-4o-transcribe)#478Swahjak wants to merge 5 commits into
Swahjak wants to merge 5 commits into
Conversation
InitializeModels prepared every registered adapter on startup, so a cloud-only setup still installed the local Python environments and downloaded weights for WhisperX, Parakeet, Canary, Voxtral, PyAnnote and Sortformer. SCRIBERR_ENABLED_MODELS takes a comma separated list of model IDs and limits startup preparation to those models. Unset or empty keeps the current behaviour of preparing everything. Skipped models stay registered and are prepared on demand the first time a job uses them.
AudioFormatPreprocessor.AppliesTo unconditionally returned true, so every upload was transcoded to 16-bit PCM WAV before transcription. Local models need that, but the OpenAI-compatible adapter accepts the original compressed file and rejects requests over 25MB - so a small compressed recording could fail with a 413 purely because of the preprocessing pass. Add ModelCapabilities.SkipAudioNormalization (zero value false, so every existing adapter keeps normalizing), set it on the OpenAI adapter, and gate AppliesTo on it. A separate diarization pass reuses the same preprocessed audio, so the skip is only honoured when the diarization adapter agrees. Closes rishikanthc#475
… file Transcription and diarization consumed the same preprocessed file, so a job combining a cloud transcription model with a local diarization model had to normalize for both — re-enabling the 16kHz mono WAV conversion for the cloud call and reintroducing the request-size failure the skip flag was meant to fix. Each consumer now runs the preprocessing pipeline with its own adapter's capabilities. Identical preprocessing needs still share a single conversion (matched via ProcessingPipeline.PreprocessorSignature), and every temp file produced is tracked for cleanup.
…-transcription' into feat/openai-chunking-for-duration-capped-models # Conflicts: # internal/transcription/unified_service.go
gpt-4o-transcribe and gpt-4o-mini-transcribe reject audio longer than 1400 seconds with an HTTP 400, which makes them unusable for anything past ~23 minutes — the length of a typical meeting recording. whisper-1 has no such limit, so it stayed the only workable choice despite being more expensive and less accurate. Audio longer than a model's known cap is now split with ffmpeg into sequential chunks, transcribed one request at a time, and reassembled into a single transcript: chunk timestamps are shifted onto the original timeline, and the seconds two chunks share are taken from whichever chunk covers them more centrally. Models answering without timestamps get the repeated leading words of a chunk dropped instead. Chunk files are removed when the job finishes. Models without a known cap keep the exact single-request path they had, so whisper-1 gains no ffmpeg work and no behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #477.
Problem
gpt-4o-transcribeandgpt-4o-mini-transcribereject audio longer than 1400 seconds (~23 minutes) with an HTTP 400:whisper-1has no duration cap (only the 25MB request-size limit, addressed separately in #476), so today it is the only OpenAI model usable for meeting-length recordings — even thoughgpt-4o-mini-transcribeis half its price andgpt-4o-transcribeis materially more accurate at the same price. Anything past ~23 minutes simply fails.Fix
Client-side chunking in the OpenAI adapter, transparent to callers.
openAIModelDurationCapsmaps model ID → cap (gpt-4o-transcribe: 1400s,gpt-4o-mini-transcribe: 1400s). The API does not expose these limits, so they are hardcoded with a comment saying so. A model absent from the map is uncapped.input.Durationexceeds it, the file is split with ffmpeg into sequential chunks ofcap - 10s(a small safety margin, since cutting on frame boundaries can make a chunk marginally longer than requested), each overlapping the previous by 3s so a word spoken across a cut point is not lost. ffmpeg stream-copies the chunk (exact for audio-only streams, no re-encode of what may be hours of audio) and falls back to a re-encode if the container cannot be cut that way.transcribeFile, which both the single-shot and the chunked path call — the request is built identically either way, no duplicated code path.json, one segment per chunk) cannot be de-duplicated that way, so the repeated leading words of the next chunk are dropped instead — a bounded word-run comparison ignoring case and punctuation, not a diff algorithm. Segment starts are clamped to stay monotonic across a boundary.deferthere rather than being threaded throughunified_service.go'stempFilesToCleanup.No regression for uncapped models
whisper-1(and any model not in the cap map) takes the exact same single-request path as before: no ffmpeg invocation, no chunk planning, no merge. The only change on that path is thatProcessingTime/ModelUsed/Metadataare now set by the caller instead of inside the request helper, and the whole-text fallback segment falls back to the known audio duration when the response reports none.Why overlap rather than silence/VAD cut points
The issue floats splitting on silence boundaries as the cleaner alternative. The pipeline's
VoiceActivityDetectionPreprocessoris currently a placeholder that returns its input unchanged, so there is no silence detection to wire in — building it is out of scope for this change. Overlap plus de-duplication is the approach that fits what is there today.Base branch
This branch is
mainmerged with #474 (feat/enabled-models-env) and #476 (fix/skip-normalization-for-cloud-transcription), since it builds on the OpenAI adapter changes in #476 and the model-capability plumbing in #474. The one merge conflict (inunified_service.go, where both branches touched the diarization call site) was resolved by keeping both sides:EnsureModelReadyfrom #474 plus per-adapter preprocessing from #476.It should be reviewed/merged after those two land — or rebased if they merge with a different final shape.
Build and tests
go build ./...— cleango vet ./...— cleango test ./internal/... ./tests/...— 230 passed, 1 pre-existing unrelated failure (TestListTranscriptionJobsDeltaSync, fails onmaintoo). No new failures.New tests in
internal/transcription/adapters/openai_chunking_test.gocover the chunking logic in isolation (no network or API key needed):whisper-1and unknown models resolving to uncapped