Skip to content

fix(l0): re-initialize driver in forked child processes#954

Draft
Grynder02 wants to merge 1 commit into
intel:masterfrom
Grynder02:fix/meteor-lake-fork-safety
Draft

fix(l0): re-initialize driver in forked child processes#954
Grynder02 wants to merge 1 commit into
intel:masterfrom
Grynder02:fix/meteor-lake-fork-safety

Conversation

@Grynder02

Copy link
Copy Markdown

Problem

fork() is broken for Level Zero on Linux when the parent has already initialized the driver. Depending on timing, a forked child either deadlocks on its first GPU call or gets 0 drivers back from zeDriverGet.

Root cause

The Level Zero loader (libze_loader, verified on 1.28.6) runs zeInit() inside a process-lifetime std::call_once. A forked child inherits the consumed once_flag and the cached ZE_RESULT_SUCCESS, so the child's zeInit() short-circuits in the loader and never calls back into this driver. The driver's existing pid-check re-init path in initDriver() is therefore unreachable after fork() — it only runs under the loader's init callback, which fires once per process lifetime of the parent.

The loader's zeDriverGet, by contrast, is a direct jump into the driver's entry point with no init logic (verified by disassembly). So zeDriverGetDriver::driverHandleGet() is the first place the driver code actually executes in a forked child.

Evidence gathered while root-causing (strace of failing children shows zero GPU-related syscalls — no /dev/dri opens — proving Driver::initialize() never ran in the child; calling the driver's DDI tables directly, bypassing the loader, made fork work even without this patch).

Fix

  1. pthread_atfork child handler (registered on first successful init): resets the driver pid and marks inherited driver handles as orphaned — they wrap DRM fds that belong to the parent.
  2. Pid-change detection in Driver::driverHandleGet(): re-initializes the driver when called from a new pid, since this is the earliest point the driver can observe the fork given the loader behavior above.
  3. Orphaned handles are dropped from globalDriverHandles but intentionally leaked, never deleted: the child shares the parent's DRM file descriptions, so destructor teardown (GEM close / context destroy ioctls) would destroy the parent's live GPU objects. This mirrors the pid-match policy globalDriverTeardown() already implements, and matches the leak-on-fork convention other GPU runtimes use.

Verification

Tested on Meteor Lake (Core Ultra 5 125H, Arc iGPU, Ubuntu 24.04, kernel 6.17, loader 1.28.6), built from the 26.28 source base with the same change and installed as libze_intel_gpu.so.1.15.0:

  • loader-path fork test (parent inits via loader, forks 3 children, each child calls zeInit + zeDriverGet + creates a context and runs work): 3/3 children pass; before the change all children either deadlocked or saw 0 drivers
  • direct-DDI fork test (bypassing the loader): 3/3 pass
  • grandchild fork test (fork from a forked child): 3/3 pass
  • parent unaffected after children exit (its contexts/allocations stay live — this is what the leak-don't-delete policy protects)

Honest caveats

  • Developed and hardware-verified against the 26.28 release base; rebased onto current master (applies cleanly, and the symbols it relies on — initStatus, DriverHandle::pid, std::erase_if — are all present), but I have not run a full master build or the ULT suite on master.
  • No new ULTs are included yet. If maintainers are interested in this direction I'm happy to add fork-simulation unit tests (e.g. pid-spoofing around driverHandleGet) — guidance on the preferred pattern welcome.
  • An alternative/complementary fix would be a pthread_atfork-based reset of the call_once state in the loader (oneapi-src/level-zero); this PR fixes what the driver can do unilaterally, which is sufficient to make fork work end-to-end.

The Level Zero loader runs zeInit() inside a process-lifetime
std::call_once. A forked child inherits the consumed once_flag and the
cached success result, so the child's zeInit() returns without ever
calling back into the driver: the existing pid-check re-init path in
initDriver() is unreachable after fork(). The child then either hangs
on its first GPU call through inherited state or sees 0 drivers from
zeDriverGet.

Fix, in three parts:

- Register a pthread_atfork child handler on first successful init. It
  resets the driver pid and marks every inherited driver handle as
  orphaned; those handles wrap DRM file descriptors that belong to the
  parent.

- Detect the pid change in Driver::driverHandleGet() and re-initialize
  there. zeDriverGet is the first entry point the loader actually
  forwards to in a forked child (the loader's zeDriverGet is a direct
  jump into the driver with no init logic), so this is the earliest
  point the driver can observe the fork.

- Drop orphaned handles from globalDriverHandles before creating new
  ones, but intentionally leak them rather than delete: the child
  shares the parent's DRM file descriptions, and destructor teardown
  (GEM close, context destroy ioctls) would destroy the parent's live
  GPU objects. This mirrors the pid-match policy globalDriverTeardown()
  already uses.

Verified on Meteor Lake (Core Ultra 5 125H, Ubuntu 24.04, loader
1.28.6): loader-path fork tests, direct-DDI fork tests, and grandchild
fork tests pass 3/3 each; forked children get one functional driver
handle and can run kernels. Before the change, children either
deadlocked or saw 0 drivers.

Signed-off-by: Grynder02 <bbintx33@yahoo.com>
Grynder02 added a commit to Grynder02/intel-arc-vllm-xpu-bootstrap that referenced this pull request Jul 15, 2026
…native

The known-limitations bullet now links the fix branch on
Grynder02/compute-runtime and upstream PR intel/compute-runtime#954,
and states the trade-off explicitly: fixed host driver = bare-metal
path available; this repo stays the zero-host-modification path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant