Skip to content

Surface dropped video frames to the application - #1643

Open
xianshijing-lk wants to merge 3 commits into
mainfrom
sxian/CLTCU-19/surface-dropped-video-frames
Open

xianshijing-lk wants to merge 3 commits into
mainfrom
sxian/CLTCU-19/surface-dropped-video-frames

Conversation

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Part of CLTCU-19. Second of three — see #1642 (log severity) and the buffer-reuse PR stacked on this one.

A decoded video frame can be dropped after framesDecoded counts it and before the application sees it, and no WebRTC stat reports the loss: framesDecoded counts the frame, framesDropped does not. That is the shape of the customer report behind CLTCU-19, and today nobody — them or us — can see it from the SDK.

NativeVideoStream already counted these drops, but the count was write-only: it reached an application as a log line on the first drop and every hundredth after it, at a level no binding shows by default (see #1642).

  • VideoFrameReceived.dropped_frames — new optional field, cumulative and monotonic
  • NativeVideoStream::dropped_frames() — same counter for Rust callers

The counter is per FFI stream, not per NativeVideoStream

Worth a look during review. participant_video_stream_task keeps one stream_handle while the participant swaps tracks, and builds a fresh NativeVideoStream for each one. Reporting that stream's own counter would restart at zero mid-stream, so a client diffing successive events would see the count go backwards.

DropTotal carries finished tracks' losses, and each new track counts on top. A test pins the behaviour across a replacement.

Testing

cargo test -p libwebrtc -p livekit-ffi: 29 + 28 pass, including the counter being readable and cumulative, an unbounded queue never reporting drops, and the carry-over across a track swap.

Verified end to end against the C++ SDK with a live livekit-server, publishing and subscribing a real track.

Follow-ups

Bindings need matching changes to surface this: livekit/python-sdks#927 and livekit/client-sdk-cpp#308 are up and depend on this landing plus a release.

There is also a third, entirely silent drop site upstream of this one — VideoRenderFrames::FrameToRender keeps only the newest due frame and discards older ones with no log at all, reporting frames_dropped_ solely via a UMA histogram at teardown. If the Windows reports survive this, that is where to look next, and this counter is what will tell us.

🤖 Generated with Claude Code

A decoded video frame can be dropped after `framesDecoded` counts it and
before the application sees it, with every WebRTC stat reading zero:
`framesDecoded` counts it and `framesDropped` does not. `NativeVideoStream`
already counted these, but the count was only ever written -- it reached an
application as a log line on the first drop and every hundredth after it.

`VideoFrameReceived` now carries `dropped_frames`, and
`NativeVideoStream::dropped_frames()` exposes the same counter to Rust.

The total is kept per FFI stream rather than per `NativeVideoStream`. A
participant stream holds one handle while the participant swaps tracks and
builds a fresh `NativeVideoStream` for each, so reporting that stream's own
counter would restart at zero mid-stream and a client diffing successive
events would see the count go backwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Changeset ✓

This PR includes a changeset covering all affected packages:

Package Bump
libwebrtc patch
livekit patch
livekit-capture patch
livekit-ffi patch

devin-ai-integration[bot]

This comment was marked as resolved.

xianshijing-lk and others added 2 commits November 6, 2026 17:40
A stream can lose frames after the last one it managed to forward, and
`dropped_frames` only rode on frame events, so those losses had nothing to
arrive on. A track that overflows and then ends -- or a participant that
leaves rather than republishing -- left them invisible, which is exactly the
case the counter exists for.

`VideoStreamEOS` now carries the total too.

Also widens the changeset: bumping libwebrtc requires bumping its dependents.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant