Skip to content

perf(audio): aggregate receive frames before waking Rust consumers - #1645

Draft
theomonnom wants to merge 1 commit into
mainfrom
codex/audio-receive-batching
Draft

theomonnom wants to merge 1 commit into
mainfrom
codex/audio-receive-batching

Conversation

@theomonnom

@theomonnom theomonnom commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

TL;DR: Aggregate requested receive frames at the native sink to reduce Rust wakeups and avoid an extra PCM copy.

FFI/protobuf scope: PCM is passed by pointer with rate/channel/sample-count metadata; protobuf does not serialize the audio samples. For an existing 50 ms stream, this PR reduces native Rust queue deliveries from roughly 100/s to 20/s, while Python already receives 20 frame events/s. It therefore does not provide a 5× reduction in Python protobuf parsing or callbacks. Follow-up live instrumentation of Python and Rust codec calls attributed about 1.14% of total native-patched session CPU to protobuf encode/decode (about 0.71% Python and 0.43% Rust; 1.20% combined before timer-overhead adjustment), across two 16 s profiling runs with four sessions each. This excludes surrounding GIL acquisition, queues, PCM copies, processing, and protobuf object/field work outside the timed calls. It is profiling attribution, not an end-to-end protobuf-removal A/B. Warm message replay estimated only 0.16% including common request construction and understated the live cost; use the live measurement when assessing serialization.

Capacity projections use new_sessions = current_sessions / (1 - measured_CPU_reduction), assuming CPU is the bottleneck, unchanged workload, and unchanged per-session cost at higher load. They apply proportionally per core or machine, are not additive across PRs, and require deployment saturation/tail-latency validation.


When an unbounded audio stream requests 50 ms frames, the native sink currently wakes the Rust consumer for every 10 ms chunk and the FFI layer copies those chunks into the requested frame. Accumulate at the sink observer instead, so the Rust consumer receives one complete frame per requested interval. Move an already correctly sized frame through FFI without a second PCM copy.

The WebRTC 10 ms callback and requested Python frame duration stay the same. Bounded queues retain their existing drop-oldest accounting; streams with native audio filters and requests at or below 10 ms use the existing path. Closing the stream clears the partial accumulation buffer. The additive NativeAudioStream::with_frame_size constructor lets livekit-ffi select this behavior without changing existing option struct literals.

Validation:

  • Six Rust audio stream tests passed, covering PCM order/tails, delivery timing, observer lifetime, bounded overflow, and queue close.
  • Thirteen Python RTC integration cases passed against both stock and patched FFI, including exact PCM/metadata at 8/24/48 kHz, mono/stereo, and 7/10/20/50 ms frames; repeated close checks subscription cleanup.
  • Existing Python RTC lifecycle/queue checks: 31 passed. Focused Agents checks: 122 passed, 8 skipped against both FFI builds.
  • Rust formatting and patch checks passed. No production SFU changes are needed.

Performance context: receive aggregation, the separate Opus AVX2 patch, and optional native log filtering together reduced whole-session CPU by 6.21% in two paired trials (4.97% and 7.47%). This does not isolate this PR's contribution. The workload used four AgentSession processes per core, real RTC/native VAD/AGC, standard asyncio, threaded VAD, and fixture/stub cloud providers, with 16 s warmup and 64 s measurement. Both native-only trials passed the benchmark's receive, loop, playout, and local interrupt gates. These checks are not a production capacity or universal RTC-equivalence proof.

The separate combined experiment including uvloop and inline VAD reduced CPU by 15.14%, but had timing failures in both cohorts. That number must not be attributed to this PR or used as a measured capacity increase.

Related draft PRs: Opus AVX2, Python log filtering, inline VAD, RTC regression tests.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Changeset incomplete

This PR's changeset is missing version bumps for packages that are affected by the change. The following packages still require a bump:

  • livekit
  • livekit-capture

Already covered:

  • libwebrtc (patch)
  • livekit-ffi (patch)

A package must be bumped when its own files change, and whenever a package it depends on is bumped (so downstream consumers get a matching release).

Click here to create a changeset for the missing packages

The link pre-populates a changeset file with patch bumps for the missing packages. You can also add them to your existing changeset. Edit the bump types as needed before committing.

If this change doesn't require a version bump, add the internal label to this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant