Skip to content

Worker panics (FFI Panic: ReadyForRoomEventRequest timeout) when two jobs' ctx.connect() run concurrently, killing all jobs on the worker #7545

Closed
livekit/python-sdks
#918

Description

@coincraft12

Summary

Worker process crashes with a native FFI panic when two jobs' ctx.connect() calls happen within ~1 second of each other in the same worker. This reliably kills every active job on that worker, including ones that were already connected and running fine.

Tested with a non-conversational, JobContext-only entrypoint (no AgentSession, no voice pipeline) — a "programmatic participant" per the docs pattern in Job lifecycle. Repro does not require any real media/webcam — just two lk dispatch create calls to two different rooms a second apart.

Environment

  • livekit-agents 1.8.3
  • livekit (rtc/ffi) 1.1.18
  • OS: Windows 11
  • Python 3.13.3
  • Reproduced with both lk agent dev and lk agent start (i.e. not dev-mode-specific)

Minimal repro entrypoint

from livekit.agents import AgentServer, AutoSubscribe, JobContext, cli

server = AgentServer()

@server.rtc_session(agent_name="my-agent")
async def my_agent(ctx: JobContext) -> None:
    await ctx.connect(auto_subscribe=AutoSubscribe.SUBSCRIBE_ALL)

if __name__ == "__main__":
    cli.run_app(server)

Repro steps

  1. lk agent dev (or lk agent start) to register the worker.
  2. Fire two dispatches to two different, brand-new rooms within ~1 second of each other:
    lk dispatch create --room concurrency-test-a --agent-name my-agent &
    lk dispatch create --room concurrency-test-b --agent-name my-agent &
    wait
    
  3. The first job connects fine. ~10-20s later, the second job's connection attempt panics the whole worker process.

Actual behavior

WARNING livekit.agents - The room connection was not established within 10 seconds after calling job_entry. ... {"room": "concurrency-test-a"}
ERROR livekit - livekit_ffi::server::room:261:livekit_ffi::server::room - timed out waiting for ReadyForRoomEventRequest after ConnectCallback (room_handle=8)
FFI Panic:  invalid request: timed out waiting for ReadyForRoomEventRequest after ConnectCallback (room_handle=8)
[exited with code 15]

The entire worker process exits — this also takes down the other, already-healthy job that was connected to concurrency-test-a.

Reproduced 3/3 times: twice under lk agent dev, once under lk agent start (production mode). Confirmed not specific to our own agent code, since it reproduces with the near-bare entrypoint above and no real participants/media.

Attempted workaround (partial, still broken)

We tried serializing ctx.connect() calls to avoid the race:

  1. asyncio.Lock() at module scope — fails immediately with RuntimeError: <asyncio.locks.Lock object ...> is bound to a different event loop, because each job clearly runs in a separate OS process (confirmed via tasklist/pid), so an in-memory lock can't coordinate across them.
  2. Cross-process file lock (filelock package) — this avoids the panic, but introduces a new failure: the job holding the lock's ctx.connect() call never returns (no exception, no log line at all after Lock ... acquired), even though the participant shows as ACTIVE in lk room participants list. The lock is then held forever, and every subsequent job spins forever retrying acquire() (never timing out, never erroring).

So today there's no way to run multiple concurrent jobs on one worker without either a process-killing panic or a silent permanent hang.

Why this matters

We're building an exam-proctoring agent (candidate video joins a room, agent watches for face-count/etc.) where multiple concurrent exam sessions per worker is the normal case, not an edge case. This bug currently makes that entirely unworkable — one candidate joining while another's session is active can silently take down every other active exam session on the same worker.

Happy to provide the full agent code, logs, or a minimal repo reproducing this if useful.

Activity

  1. HarshXAI commented on Sep 30, 2026

    @HarshXAI

    I can dig into this worker-wide FFI panic.

    Repro shape is clear: two ctx.connect() calls ~1s apart on the same worker → ReadyForRoomEventRequest timeout / FFI panic, and it takes down every job on that process. I’ll reproduce with the minimal JobContext entrypoint, trace whether the FFI client serializes concurrent room connects, and either serialize/connect with a lock or fail the second job cleanly without killing the worker.

    Happy to keep the first PR scoped to a concurrency guard + regression note if the deeper FFI fix belongs in livekit rtc.

  2. self-assigned this
    on Nov 3, 2026
  3. longcw commented on Nov 3, 2026

    @longcw
    Contributor

    created a fix here livekit/python-sdks#918

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions