feat(audio): negotiate stereo opus on the subscriber answer - #1007
feat(audio): negotiate stereo opus on the subscriber answer#1007kirill-jjj wants to merge 3 commits into
Conversation
The server advertises a stereo publisher track via sprop-stereo=1 in its offer, but libwebrtc's Opus decoder only decodes incoming packets as stereo when the local answer's fmtp line also carries stereo=1. Without it, stereo tracks published by other participants are received as mono. - StereoSdpMunging: extract sprop-stereo mids from the server offer and add stereo=1 to the matching fmtp lines of the answer (mirrors client-sdk-js remoteStereoMids handling), applied in onServerOffer - LocalAudioTrackOptions.stereo + TF_STEREO track feature so publishers can declare a stereo capture - AudioTrackPublishOptions.stereo sent as AddTrackRequest.stereo, mirroring client-sdk-js forceStereo Includes unit tests built from real device SDP captures and a changeset.
🦋 Changeset detectedLatest commit: 0212c81 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…ment stereo option Addresses Devin review feedback on livekit#1007: - if the munged answer is rejected by libwebrtc, retry with the original answer and send that instead of leaving the subscriber without a local description (mirrors PeerConnectionTransport.setMungedSdp) - add KDoc to LocalParticipant.AudioTrackPublishOptions.stereo
davidliu
left a comment
davidliu
left a comment
There was a problem hiding this comment.
This really first needs a PR on https://github.com/webrtc-sdk/webrtc to expose the useStereoInput/useStereoOutput, because a lot of it creates potential state mismatches between what the JavaAudioDeviceModule is actually configured to do and what the user thinks these options will do.
Separately, I think this might be better served as two separate PRs, one for the subscriber side and one on the publisher side.
| } | ||
| // Note: stereo is sourced from options, not introspected from | ||
| // JavaAudioDeviceModule; callers that enable stereo input via a module | ||
| // customizer should set LocalAudioTrackOptions.stereo to match. |
There was a problem hiding this comment.
this would need to be surfaced to users, and ideally should be sourced from JavaAudioDeviceModule entirely, to avoid a mismatch in state.
| /** | ||
| * Publish the track as stereo, sent as `stereo` in the AddTrackRequest. | ||
| */ | ||
| val stereo: Boolean = false, |
There was a problem hiding this comment.
This should be in BaseAudioTrackPublishOptions, and the stereo option should be added to the end of the list, so that it won't break existing constructors who use positional arguments.
| return@launch | ||
| } | ||
| } | ||
| client.sendAnswer(answer, offerId) |
There was a problem hiding this comment.
rather than nesting the fallback here, should return from the run block with an indicator that it should try the fallback in a separate same-level block.
| override val dtx: Boolean = true, | ||
| override val red: Boolean = true, | ||
| /** | ||
| * Publish the track as stereo, sent as `stereo` in the AddTrackRequest. |
There was a problem hiding this comment.
AddTrackRequest is an internal class the consumers don't need to know about.
| @@ -0,0 +1,7 @@ | |||
| --- | |||
| 'livekit-android': patch | |||
There was a problem hiding this comment.
There are API changes in here, so it should be marked as a minor upgrade.
3 participants
Problem
When a remote participant publishes a stereo audio track, Android subscribers receive it as mono. libwebrtc's native Opus decoder downmixes incoming stereo packets to mono unless the local answer negotiates
stereo=1in the fmtp line — even when the server offer correctly announces the track withsprop-stereo=1.Observed on real hardware (Android 10, Huawei): stereo capture works (
setUseStereoInput(true), confirmed via AudioRecord), the track is published withstereo=1in AddTrackRequest, the server relayssprop-stereo=1in the subscriber offer — yet the decoded audio plays back mono untilstereo=1is manually added to the answer.Related issues
setUseStereoInput(true), but that only covers the publisher side. The subscriber side remained broken.extractStereoAndNackAudioFromOffercollectsremoteStereoMidsfrom the offer, andensureAudioNackAndStereorewrites the answer's fmtp lines.Changes
Receive path (the main fix):
StereoSdpMunging.kt: extracts mids carryingsprop-stereo=1from the server offer and addsstereo=1to the matching fmtp lines of the answer. Invoked inRTCEngine.onServerOfferbeforesetLocalDescription/sendAnswer. Fully fail-safe on parse failures.Publish path (API parity with client-sdk-js):
LocalAudioTrackOptions.stereo— capture-side flag, surfaced as theTF_STEREOtrack featureAudioTrackPublishOptions.stereo— sent asAddTrackRequest.stereo, mirroring JS'sforceStereooptionTesting
livekit-android-test, using SDP fixtures captured from a real device (the subscriber offer from a live LiveKit session)./gradlew test,spotlessCheck,detektReleasepassAssisted by AI; verified end-to-end on real hardware.