fix(l0): re-initialize driver in forked child processes#954
Draft
Grynder02 wants to merge 1 commit into
Draft
Conversation
The Level Zero loader runs zeInit() inside a process-lifetime std::call_once. A forked child inherits the consumed once_flag and the cached success result, so the child's zeInit() returns without ever calling back into the driver: the existing pid-check re-init path in initDriver() is unreachable after fork(). The child then either hangs on its first GPU call through inherited state or sees 0 drivers from zeDriverGet. Fix, in three parts: - Register a pthread_atfork child handler on first successful init. It resets the driver pid and marks every inherited driver handle as orphaned; those handles wrap DRM file descriptors that belong to the parent. - Detect the pid change in Driver::driverHandleGet() and re-initialize there. zeDriverGet is the first entry point the loader actually forwards to in a forked child (the loader's zeDriverGet is a direct jump into the driver with no init logic), so this is the earliest point the driver can observe the fork. - Drop orphaned handles from globalDriverHandles before creating new ones, but intentionally leak them rather than delete: the child shares the parent's DRM file descriptions, and destructor teardown (GEM close, context destroy ioctls) would destroy the parent's live GPU objects. This mirrors the pid-match policy globalDriverTeardown() already uses. Verified on Meteor Lake (Core Ultra 5 125H, Ubuntu 24.04, loader 1.28.6): loader-path fork tests, direct-DDI fork tests, and grandchild fork tests pass 3/3 each; forked children get one functional driver handle and can run kernels. Before the change, children either deadlocked or saw 0 drivers. Signed-off-by: Grynder02 <bbintx33@yahoo.com>
Grynder02
added a commit
to Grynder02/intel-arc-vllm-xpu-bootstrap
that referenced
this pull request
Jul 15, 2026
…native The known-limitations bullet now links the fix branch on Grynder02/compute-runtime and upstream PR intel/compute-runtime#954, and states the trade-off explicitly: fixed host driver = bare-metal path available; this repo stays the zero-host-modification path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
fork()is broken for Level Zero on Linux when the parent has already initialized the driver. Depending on timing, a forked child either deadlocks on its first GPU call or gets0drivers back fromzeDriverGet.Root cause
The Level Zero loader (
libze_loader, verified on 1.28.6) runszeInit()inside a process-lifetimestd::call_once. A forked child inherits the consumedonce_flagand the cachedZE_RESULT_SUCCESS, so the child'szeInit()short-circuits in the loader and never calls back into this driver. The driver's existing pid-check re-init path ininitDriver()is therefore unreachable afterfork()— it only runs under the loader's init callback, which fires once per process lifetime of the parent.The loader's
zeDriverGet, by contrast, is a direct jump into the driver's entry point with no init logic (verified by disassembly). SozeDriverGet→Driver::driverHandleGet()is the first place the driver code actually executes in a forked child.Evidence gathered while root-causing (strace of failing children shows zero GPU-related syscalls — no
/dev/driopens — provingDriver::initialize()never ran in the child; calling the driver's DDI tables directly, bypassing the loader, made fork work even without this patch).Fix
pthread_atforkchild handler (registered on first successful init): resets the driver pid and marks inherited driver handles as orphaned — they wrap DRM fds that belong to the parent.Driver::driverHandleGet(): re-initializes the driver when called from a new pid, since this is the earliest point the driver can observe the fork given the loader behavior above.globalDriverHandlesbut intentionally leaked, never deleted: the child shares the parent's DRM file descriptions, so destructor teardown (GEM close / context destroy ioctls) would destroy the parent's live GPU objects. This mirrors the pid-match policyglobalDriverTeardown()already implements, and matches the leak-on-fork convention other GPU runtimes use.Verification
Tested on Meteor Lake (Core Ultra 5 125H, Arc iGPU, Ubuntu 24.04, kernel 6.17, loader 1.28.6), built from the 26.28 source base with the same change and installed as
libze_intel_gpu.so.1.15.0:zeInit+zeDriverGet+ creates a context and runs work): 3/3 children pass; before the change all children either deadlocked or saw 0 driversHonest caveats
initStatus,DriverHandle::pid,std::erase_if— are all present), but I have not run a full master build or the ULT suite on master.driverHandleGet) — guidance on the preferred pattern welcome.pthread_atfork-based reset of thecall_oncestate in the loader (oneapi-src/level-zero); this PR fixes what the driver can do unilaterally, which is sufficient to make fork work end-to-end.