Skip to content

Fix VP9 simulcast freezing on first layer switch - #4718

Open
iamadityaanjana wants to merge 2 commits into
livekit:masterfrom
iamadityaanjana:vp9-simulcast-layer-switch-fix
Open

Fix VP9 simulcast freezing on first layer switch#4718
iamadityaanjana wants to merge 2 commits into
livekit:masterfrom
iamadityaanjana:vp9-simulcast-layer-switch-fix

Conversation

@iamadityaanjana

Copy link
Copy Markdown

Summary

Fixes #4594 — a VP9 simulcast (ONE_SPATIAL_LAYER_PER_STREAM) subscriber works on its initial layer, then freezes the instant the allocator first switches layers (e.g. a BWE-driven upgrade): the forwarder enters a constant processSourceSwitch loop (~60/s), output timestamps freeze (extNextTS = extLastTS + 1), and a NACK/PLI storm follows. H.264/VP8 simulcast is unaffected.

The issue reporter root-caused this on a v1.11.0 fork running in production (write-up). One of their four fixes (Simulcast selector for VP9 simulcast) has already landed upstream; this PR ports the remaining three to current master:

  1. pkg/sfu/receiver_base.go — all rids collapsed to spatial layer 0. if extPkt.Spatial >= 0 overrode the rid/uptrack index with the in-packet spatial id. Each VP9 simulcast rid is single-spatial, so the DD SpatialId is 0 for every rid → all map to spatial layer 0 and the forwarder alternates SSRCs on every packet. (H.264/VP8 are unaffected because extPkt.Spatial == -1.) Fix: apply the override only for the base uptrack (layer == 0, the SVC case); for simulcast the uptrack index is authoritative.

  2. pkg/sfu/forwarder.go — the cross-layer SR timestamp offset never establishes. Per-layer RTCP sender reports for VP9 simulcast do not provide a usable cross-layer offset, so tsOffset stays 0 and every switch errors switch point too far behind. Fix: treat VP9 simulcast like ONE_SPATIAL_LAYER_PER_STREAM_INCOMPLETE_RTCP_SR by skipping the reference-timestamp based switch point. skipReferenceTS is derived deterministically on every DetermineCodec call from the current mime + videoLayerMode, so a forwarder that regresses from VP9 simulcast to another codec (upstream codec change / receiver restart re-invoke DetermineCodec) re-enables the sender-report based path; the constructor-provided value (forceSkipReferenceTS) still forces it on, preserving existing test behavior.

  3. pkg/sfu/buffer/dependencydescriptorparser.go — frame drops with frequent key frames. The drop threshold advanced on every structure-bearing key frame even when StructureId was unchanged, so reordered/retransmitted frames were dropped as "earlier than current structure" (severe with screen content). Fix: advance the drop threshold (structureChangeExtFrameNum) only on an actual StructureId change; structureExtFrameNum/ExtKeyFrameNum behavior is unchanged.

Note: the client SDK also needs to signal SimulcastCodec.videoLayerMode = ONE_SPATIAL_LAYER_PER_STREAM plus per-encoding scalabilityMode for VP9 — this is the server-side path only.

Test plan

  • go build ./... passes
  • go test ./pkg/sfu/ ./pkg/sfu/buffer/ passes (including existing forwarder tests that construct with skipReferenceTS=true)
  • The equivalent changes have been running in production on the reporter's v1.11.0 fork (stable layer switches, no NACK/PLI storm)

Credit for the root-cause analysis and original fixes: @SpeakNow06

VP9 simulcast (ONE_SPATIAL_LAYER_PER_STREAM) subscribers freeze the
first time the stream allocator switches layers: the forwarder enters
a constant processSourceSwitch loop, output timestamps freeze, and a
NACK/PLI storm follows.

Three root causes addressed:

1. receiver_base: the in-packet spatial id (from the dependency
   descriptor) overrode the uptrack/rid index for every packet. Each
   VP9 simulcast rid is single-spatial, so the spatial id is always 0
   and all rids collapsed to spatial layer 0, leaving the forwarder
   unable to tell the streams apart. Apply the override only for the
   base uptrack (layer == 0, the SVC case); for simulcast the uptrack
   index is authoritative.

2. forwarder: per-layer RTCP sender reports for VP9 simulcast do not
   provide a usable cross-layer timestamp offset, so
   getRefLayerRTPTimestamp never establishes (tsOffset stays 0) and
   every layer switch fails with 'switch point too far behind'. Treat
   VP9 simulcast like
   ONE_SPATIAL_LAYER_PER_STREAM_INCOMPLETE_RTCP_SR by skipping the
   reference-timestamp based switch point. skipReferenceTS is now
   derived deterministically on every DetermineCodec call from the
   current mime + videoLayerMode (DetermineCodec is re-invoked on
   upstream codec change and receiver restart, so a forwarder that
   regresses from VP9 simulcast to another codec re-enables the
   sender-report based path); the constructor-provided value
   (forceSkipReferenceTS) still forces it on.

3. dependencydescriptorparser: the frame drop threshold
   (structureExtFrameNum) advanced on every structure-bearing key
   frame even when the structure id was unchanged, so late or
   retransmitted frames were dropped as 'earlier than current
   structure' with frequent key frames (screen content). Introduce
   structureChangeExtFrameNum, advanced only on an actual structure
   id change, and use it as the drop threshold. structureExtFrameNum
   (used for ExtKeyFrameNum) is unchanged.

Fixes livekit#4594

Root cause analysis based on the report and fork by @SpeakNow06.

Co-Authored-By: SpeakNow06 <noreply@github.com>
@iamadityaanjana
iamadityaanjana requested a review from a team as a code owner August 2, 2026 10:07
devin-ai-integration[bot]

This comment was marked as resolved.

@@ -189,6 +195,9 @@ func (r *DependencyDescriptorParser) Parse(pkt *rtp.Packet) (*ExtDependencyDescr
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An out-of-order keyframe with the same structure ID may arrive here with the new logic, and confused the downtrack's dependency descriptor selector. Need to add a check to filter out the old keyframe:

    if extFN < r.structureExtFrameNum {
        r.logger.Debugw("drop out-of-order key frame",
            "extFN", extFN, "structureExtFrameNum", r.structureExtFrameNum)
        ReleaseExtDependencyDescriptor(extDD)
        return nil, videoLayer, ErrFrameEarlierThanKeyFrame
    }

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have resolved these issues and PR is ready for the review and merge.

@CLAassistant

CLAassistant commented Aug 3, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

- dependencydescriptorparser: drop out-of-order key frames carrying an
  attached structure older than the current structureExtFrameNum.
  Accepting them would regress structureExtFrameNum (ExtKeyFrameNum) and
  replay a stale structure update, confusing the downtrack's dependency
  descriptor selector.

- receiver_base: gate the in-packet spatial layer override on
  videoLayerMode (!sfuutils.IsSimulcastMode) instead of the uptrack
  index (layer == 0). The index is only forced to 0 for
  MULTIPLE_SPATIAL_LAYERS_PER_STREAM, so an SVC publisher that does not
  signal videoLayerMode but sends a rid is registered under a non-zero
  index and would have all its spatial layers collapsed onto it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VP9 simulcast (ONE_SPATIAL_LAYER_PER_STREAM) freezes on first layer switch — root causes + working fork

4 participants