Repository navigation
Reuse video frame conversion buffers - #1644
xianshijing-lk wants to merge 2 commits into
Conversation
| pub fn to_video_buffer_info( | ||
| rtcbuffer: BoxVideoBuffer, | ||
| dst_type: Option<proto::VideoBufferType>, | ||
| _normalize_stride: bool, // always normalize stride for now.. | ||
| ) -> FfiResult<(Box<[u8]>, proto::VideoBufferInfo)> { | ||
| dst: &mut Vec<u8>, | ||
| ) -> FfiResult<proto::VideoBufferInfo> { |
There was a problem hiding this comment.
🟡 Public conversion signatures break callers
Existing callers of to_video_buffer_info lose its owned output and must supply a mutable destination. The public cvtimpl::cvt functions change identically, breaking downstream callers outside this repository.
Learn more
The room-apis feature publicly re-exports server, which publicly exposes colorcvt and cvtimpl. Their previous conversion signatures returned an owned byte buffer alongside the metadata; the new signatures require callers to own a Vec<u8>. This changes the Rust-facing public API even though the internal callers have been updated.
Example: A downstream Rust consumer calling to_video_buffer_info(frame, None, true) and retaining its returned Box<[u8]> can no longer build against this release.
Recommended fix: Keep the existing public entry points as compatibility wrappers. Add pub(crate) conversion-into-buffer entry points for native_video_stream_task and the other internal callers; preserve the old return types and ownership behavior on the wrappers.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // The handle co-owns the buffer. That second reference is | ||
| // what stops the buffer being written again before the | ||
| // binding is done reading it. | ||
| server.store_handle(handle_id, slot.clone()); |
There was a problem hiding this comment.
🟡 Frame handles lack lifecycle regression coverage
Each frame now stores an Arc<Vec<u8>> FFI handle, but tests only clone local Arcs. No repeated stream teardown workload checks handle-map cleanup across normal and failed delivery.
Learn more
The added Arc<Vec<u8>> handle type is stored in the FFI server for every delivered frame. The pool unit tests substitute local clones for the real handle map, so they cannot demonstrate that dropping stream and frame handles releases allocations or terminates stream tasks.
Example: Repeatedly create a stream, receive frames, release their handles, close the stream, and shut down the FFI server; the current tests do not check that handles and pooled allocations disappear afterward.
Recommended fix: Add a repeated initialize → use → disconnect/drop → shutdown regression workload using the real FFI handle map. Include failed delivery and interrupted teardown where applicable; verify handle cleanup and task completion, and use a Weak reference to assert the final frame allocation is released.
Was this helpful? React with 👍 or 👎 to provide feedback.
| unsafe { | ||
| cvtimpl::cvt(info, dst_type.unwrap_or(proto::VideoBufferType::I420), false, dst) | ||
| } |
There was a problem hiding this comment.
🟡 Conversion blocks lack safety explanations
The new unsafe calls to cvtimpl::cvt have no required // SAFETY: comments. Each branch passes raw plane pointers whose validity depends on the live frame buffer.
Learn more
Each new unsafe conversion block passes pointers into a live BoxVideoBuffer to a function that constructs slices from those pointers. The frame buffer remains owned by rtcbuffer until the call finishes, but that invariant is not stated at the unsafe blocks.
Example: The Native and I420 branches create plane pointers from a buffer and pass them to cvtimpl::cvt; both require the source allocation to outlive the conversion.
Recommended fix: Add a precise // SAFETY: comment immediately above each new unsafe block, documenting the pointer validity, plane lengths, and lifetime guaranteed by the source buffer.
Was this helpful? React with 👍 or 👎 to provide feedback.
Converting a frame allocated a fresh buffer the size of the frame and freed it when the binding released the handle. A 1628x1920 I420 frame is 4.5 MB, so a 30 fps stream churned 134 MB/s of short-lived allocations that large, and the Windows heap sends anything over ~512 KB straight to `VirtualAlloc`, decommitting it on free -- which is one reason the same workload can lose frames there and not on Linux. `cvt` now writes into a buffer the caller owns, and each stream keeps a small set to hand back. A buffer may only be written while nothing else holds it, and `Arc` is what knows: the FFI handle map holds a clone for as long as the binding has the frame, so a strong count of 1 means the buffer came back. `Arc::get_mut` turns "is the binding done?" into a question the borrow checker answers rather than one the code has to get right -- the alternative, writing into a buffer the binding is still reading, tears frames rather than dropping them. Nothing depends on the binding releasing promptly: when no buffer is free the stream adds one, up to a small cap, past which it allocates per frame as before. A stream settles at one buffer for a binding that copies and releases immediately, and two for a zero-copy binding holding the current frame while the next arrives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fdd9b56 to
4f55a40
Compare
Two review points. Each arm of to_video_buffer_info builds a VideoBufferInfo of raw plane pointers and calls cvt inside an unsafe block, with no note of what makes that sound. The arms are not alike: the Native arm's pointers address an I420Buffer that to_i420 returned by value, so the call has to stay inside the arm or the buffer is dropped first, while every other arm borrows from rtcbuffer. That asymmetry is now written down, which also rules out the tempting refactor of hoisting the single call out of the match. The FrameBuffers tests stood in for the FFI handle map with a local clone, so nothing covered the real thing. The added test stores a buffer in the handle map, checks the stream will not hand it out while it is there, and checks dropping the handle returns it.
|
On the signature change: The other two are addressed in the latest push:
|
Part of CLTCU-19. Third of three. Stacked on #1643 — review that one first; this PR's base will switch to
mainonce it lands.Converting a video frame allocated a fresh buffer the size of the frame and freed it when the binding released the handle. At 1628x1920 I420 that is 4.5 MB per frame, so a 30 fps stream churned 134 MB/s of short-lived allocations that large.
Allocations that size bypass the allocator's free lists: the Windows heap sends anything over ~512 KB straight to
VirtualAllocand decommits it on free, so each frame faults in a fresh ~1150-page mapping and gives it back. glibc avoids that by raising its mmap threshold as large blocks are freed. That asymmetry is one plausible reason the same workload loses frames on Windows and not on Linux, and it scales with resolution — which matches the customer not reproducing at 720p.Caveat worth stating plainly: that mechanism is reasoned from allocator behaviour, not measured on Windows. The reduction in allocator traffic is real either way; whether it is what fixes the customer's loss is not yet established.
How it works
cvtwrites into a buffer the caller owns, and each stream keeps a small set to hand back.A buffer may only be written while nothing else holds it, and
Arcis what knows: the FFI handle map holds a clone for as long as the binding has the frame, so a strong count of 1 means the buffer came back.Arc::get_mutturns "is the binding done?" into a question the borrow checker answers rather than one the code has to get right. That matters because the failure mode is nasty:send_eventreturns long before the binding reads the pixels — in Python the callback only parses the small protobuf event and defers the copy to the event loop — so writing too early tears frames rather than dropping them.Sizing
Nothing depends on the binding releasing promptly: when no buffer is free the stream adds one, up to
MAX_FRAME_BUFFERS = 3, past which it allocates per frame exactly as before.The cap is small deliberately: the set never shrinks, so a burst would otherwise pin its high-water mark for the life of the stream — 12 MB per 4K frame.
This is not the end state
There is a
TODO(sxian)onto_video_buffer_infospelling this out. For a native-format requestcvtonly repacks the decoder's padded strides into tight ones; we could hand out the decoder's buffer instead, andimpl FfiHandle for BoxVideoBufferplusVideoBufferInfo.componentsalready make that possible. Doing so would delete this pooling entirely, because libwebrtc'sVideoFrameBufferPoolalready pools and refcounts decoder buffers. It needs stride-aware consumers first — the C++ SDK readscomponents[].stridetoday, Python and Unity assume packed.Testing
cargo test -p livekit-ffi: 37 pass, including a released buffer being handed out again, a buffer the binding still holds never being handed out, a renderer settling at exactly two, growth stopping at the cap with all outstanding buffers distinct, and a mid-stream size change (simulcast layer switch) reusing the allocation.Verified end to end against the C++ SDK with a live
livekit-server— reading valid non-black luma across 20 consecutive frames, each of which reuses a buffer a previous frame returned.🤖 Generated with Claude Code