Fix VP9 simulcast freezing on first layer switch - #4718
Open
iamadityaanjana wants to merge 2 commits into
Open
Conversation
VP9 simulcast (ONE_SPATIAL_LAYER_PER_STREAM) subscribers freeze the first time the stream allocator switches layers: the forwarder enters a constant processSourceSwitch loop, output timestamps freeze, and a NACK/PLI storm follows. Three root causes addressed: 1. receiver_base: the in-packet spatial id (from the dependency descriptor) overrode the uptrack/rid index for every packet. Each VP9 simulcast rid is single-spatial, so the spatial id is always 0 and all rids collapsed to spatial layer 0, leaving the forwarder unable to tell the streams apart. Apply the override only for the base uptrack (layer == 0, the SVC case); for simulcast the uptrack index is authoritative. 2. forwarder: per-layer RTCP sender reports for VP9 simulcast do not provide a usable cross-layer timestamp offset, so getRefLayerRTPTimestamp never establishes (tsOffset stays 0) and every layer switch fails with 'switch point too far behind'. Treat VP9 simulcast like ONE_SPATIAL_LAYER_PER_STREAM_INCOMPLETE_RTCP_SR by skipping the reference-timestamp based switch point. skipReferenceTS is now derived deterministically on every DetermineCodec call from the current mime + videoLayerMode (DetermineCodec is re-invoked on upstream codec change and receiver restart, so a forwarder that regresses from VP9 simulcast to another codec re-enables the sender-report based path); the constructor-provided value (forceSkipReferenceTS) still forces it on. 3. dependencydescriptorparser: the frame drop threshold (structureExtFrameNum) advanced on every structure-bearing key frame even when the structure id was unchanged, so late or retransmitted frames were dropped as 'earlier than current structure' with frequent key frames (screen content). Introduce structureChangeExtFrameNum, advanced only on an actual structure id change, and use it as the drop threshold. structureExtFrameNum (used for ExtKeyFrameNum) is unchanged. Fixes livekit#4594 Root cause analysis based on the report and fork by @SpeakNow06. Co-Authored-By: SpeakNow06 <noreply@github.com>
cnderrauber
reviewed
Aug 3, 2026
| @@ -189,6 +195,9 @@ func (r *DependencyDescriptorParser) Parse(pkt *rtp.Packet) (*ExtDependencyDescr | |||
| } | |||
Contributor
There was a problem hiding this comment.
An out-of-order keyframe with the same structure ID may arrive here with the new logic, and confused the downtrack's dependency descriptor selector. Need to add a check to filter out the old keyframe:
if extFN < r.structureExtFrameNum {
r.logger.Debugw("drop out-of-order key frame",
"extFN", extFN, "structureExtFrameNum", r.structureExtFrameNum)
ReleaseExtDependencyDescriptor(extDD)
return nil, videoLayer, ErrFrameEarlierThanKeyFrame
}
Author
There was a problem hiding this comment.
I have resolved these issues and PR is ready for the review and merge.
- dependencydescriptorparser: drop out-of-order key frames carrying an attached structure older than the current structureExtFrameNum. Accepting them would regress structureExtFrameNum (ExtKeyFrameNum) and replay a stale structure update, confusing the downtrack's dependency descriptor selector. - receiver_base: gate the in-packet spatial layer override on videoLayerMode (!sfuutils.IsSimulcastMode) instead of the uptrack index (layer == 0). The index is only forced to 0 for MULTIPLE_SPATIAL_LAYERS_PER_STREAM, so an SVC publisher that does not signal videoLayerMode but sends a rid is registered under a non-zero index and would have all its spatial layers collapsed onto it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #4594 — a VP9 simulcast (
ONE_SPATIAL_LAYER_PER_STREAM) subscriber works on its initial layer, then freezes the instant the allocator first switches layers (e.g. a BWE-driven upgrade): the forwarder enters a constantprocessSourceSwitchloop (~60/s), output timestamps freeze (extNextTS = extLastTS + 1), and a NACK/PLI storm follows. H.264/VP8 simulcast is unaffected.The issue reporter root-caused this on a v1.11.0 fork running in production (write-up). One of their four fixes (Simulcast selector for VP9 simulcast) has already landed upstream; this PR ports the remaining three to current master:
pkg/sfu/receiver_base.go— all rids collapsed to spatial layer 0.if extPkt.Spatial >= 0overrode the rid/uptrack index with the in-packet spatial id. Each VP9 simulcast rid is single-spatial, so the DDSpatialIdis 0 for every rid → all map to spatial layer 0 and the forwarder alternates SSRCs on every packet. (H.264/VP8 are unaffected becauseextPkt.Spatial == -1.) Fix: apply the override only for the base uptrack (layer == 0, the SVC case); for simulcast the uptrack index is authoritative.pkg/sfu/forwarder.go— the cross-layer SR timestamp offset never establishes. Per-layer RTCP sender reports for VP9 simulcast do not provide a usable cross-layer offset, sotsOffsetstays 0 and every switch errorsswitch point too far behind. Fix: treat VP9 simulcast likeONE_SPATIAL_LAYER_PER_STREAM_INCOMPLETE_RTCP_SRby skipping the reference-timestamp based switch point.skipReferenceTSis derived deterministically on everyDetermineCodeccall from the current mime +videoLayerMode, so a forwarder that regresses from VP9 simulcast to another codec (upstream codec change / receiver restart re-invokeDetermineCodec) re-enables the sender-report based path; the constructor-provided value (forceSkipReferenceTS) still forces it on, preserving existing test behavior.pkg/sfu/buffer/dependencydescriptorparser.go— frame drops with frequent key frames. The drop threshold advanced on every structure-bearing key frame even whenStructureIdwas unchanged, so reordered/retransmitted frames were dropped as "earlier than current structure" (severe with screen content). Fix: advance the drop threshold (structureChangeExtFrameNum) only on an actualStructureIdchange;structureExtFrameNum/ExtKeyFrameNumbehavior is unchanged.Note: the client SDK also needs to signal
SimulcastCodec.videoLayerMode = ONE_SPATIAL_LAYER_PER_STREAMplus per-encodingscalabilityModefor VP9 — this is the server-side path only.Test plan
go build ./...passesgo test ./pkg/sfu/ ./pkg/sfu/buffer/passes (including existing forwarder tests that construct withskipReferenceTS=true)Credit for the root-cause analysis and original fixes: @SpeakNow06