Repository navigation
Surface dropped video frames to the application - #1643
Open
xianshijing-lk wants to merge 3 commits into
Open
xianshijing-lk wants to merge 3 commits into
xianshijing-lk wants to merge 3 commits into
Conversation
A decoded video frame can be dropped after `framesDecoded` counts it and before the application sees it, with every WebRTC stat reading zero: `framesDecoded` counts it and `framesDropped` does not. `NativeVideoStream` already counted these, but the count was only ever written -- it reached an application as a log line on the first drop and every hundredth after it. `VideoFrameReceived` now carries `dropped_frames`, and `NativeVideoStream::dropped_frames()` exposes the same counter to Rust. The total is kept per FFI stream rather than per `NativeVideoStream`. A participant stream holds one handle while the participant swaps tracks and builds a fresh `NativeVideoStream` for each, so reporting that stream's own counter would restart at zero mid-stream and a client diffing successive events would see the count go backwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Changeset ✓This PR includes a changeset covering all affected packages:
|
This was referenced Nov 7, 2026
A stream can lose frames after the last one it managed to forward, and `dropped_frames` only rode on frame events, so those losses had nothing to arrive on. A track that overflows and then ends -- or a participant that leaves rather than republishing -- left them invisible, which is exactly the case the counter exists for. `VideoStreamEOS` now carries the total too. Also widens the changeset: bumping libwebrtc requires bumping its dependents.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of CLTCU-19. Second of three — see #1642 (log severity) and the buffer-reuse PR stacked on this one.
A decoded video frame can be dropped after
framesDecodedcounts it and before the application sees it, and no WebRTC stat reports the loss:framesDecodedcounts the frame,framesDroppeddoes not. That is the shape of the customer report behind CLTCU-19, and today nobody — them or us — can see it from the SDK.NativeVideoStreamalready counted these drops, but the count was write-only: it reached an application as a log line on the first drop and every hundredth after it, at a level no binding shows by default (see #1642).VideoFrameReceived.dropped_frames— new optional field, cumulative and monotonicNativeVideoStream::dropped_frames()— same counter for Rust callersThe counter is per FFI stream, not per
NativeVideoStreamWorth a look during review.
participant_video_stream_taskkeeps onestream_handlewhile the participant swaps tracks, and builds a freshNativeVideoStreamfor each one. Reporting that stream's own counter would restart at zero mid-stream, so a client diffing successive events would see the count go backwards.DropTotalcarries finished tracks' losses, and each new track counts on top. A test pins the behaviour across a replacement.Testing
cargo test -p libwebrtc -p livekit-ffi: 29 + 28 pass, including the counter being readable and cumulative, an unbounded queue never reporting drops, and the carry-over across a track swap.Verified end to end against the C++ SDK with a live
livekit-server, publishing and subscribing a real track.Follow-ups
Bindings need matching changes to surface this: livekit/python-sdks#927 and livekit/client-sdk-cpp#308 are up and depend on this landing plus a release.
There is also a third, entirely silent drop site upstream of this one —
VideoRenderFrames::FrameToRenderkeeps only the newest due frame and discards older ones with no log at all, reportingframes_dropped_solely via a UMA histogram at teardown. If the Windows reports survive this, that is where to look next, and this counter is what will tell us.🤖 Generated with Claude Code