Implements Linux-style execve de_thread semantics and hands non-leader execs to the leader - #306
Merged
Conversation
resolve_clone_stack_range reads the guest region array to find the mapping a clone's child stack falls in. Any concurrent mmap or munmap rewrites that array while holding mmap_lock, and clone never takes it, so the read races g->regions and g->nregions. ThreadSanitizer reports it as soon as one thread allocates while another clones, which no test did until tests/test-threaded-exec.c gave its workers an mmap loop. Neither caller holds a lock at that point and mmap_lock is order 1, the outermost, so taking it there introduces no inversion.
A hang that only reproduces under suite load is otherwise reported as a bare "timeout after Ns" with nothing to diagnose. Both entry points arm a watchdog around every invocation: it samples the live process shortly before timeout(1) kills it, and the caller keeps the output only when the watchdog actually fired. The watchdog polls a sentinel rather than being killed by pid. Lanes that spawn hundreds of short-lived processes can recycle a pid between the watchdog exiting and a kill landing, which would turn this into a random SIGTERM at an unrelated process. The sentinel and the working file are named per arm. Sharing them would force the passing path to block until the watchdog exited, because one still symbolizing would otherwise write into the next test's file, and every passing test would pay the remainder of a collection that is then discarded. With distinct names a straggler can only touch its own, so only the timed-out path waits, and only because the move needs a finished file. The watchdog re-checks its sentinel immediately before writing, and removes its own output when the test it was watching has already been reaped. Its collection ends by appending, which would otherwise recreate a path the caller had just removed and leave it behind whenever no later arm swept it. run_timeout stays out of it: its callers pass their own cap and the coreutils suite expects rc=124 from the guest's own timeout(1), so a 124 there does not mean the harness watchdog fired.
Linux de_thread() destroys every sibling before mapping the new image. elfuse has to do the same before guest_reset zeroes the memory those siblings are still running on, or they resume into a zeroed image. thread_exec_de_thread runs in the post-failure region of sys_execve, before the credential commit and the CLOEXEC sweep, so siblings wind down against the old image. A sibling that outlives its bounded join makes it return non-zero, and sys_execve takes its post-PNR fatal exit rather than resetting guest memory under a live thread. The leader is never a de_thread target, because its run loop returning is what destroys the guest. A non-leader execve is handed to it: the requester publishes the syscall arguments and blocks, the leader runs the whole of sys_execve on its own vCPU, and the requester dies as a sibling. The new image therefore always sees gettid() == getpid() and Threads: 1. For that join to terminate, every blocking wait a guest thread can enter has to be reachable by a teardown wake, which is most of this commit: - poll and ppoll carry the wakeup pipe on every wait rather than only an indefinite one, and a finite wait runs to its deadline in slices that re-check the interrupt conditions. - A blocking O_WRONLY FIFO open polls the non-blocking form instead of parking in the host until a reader arrives. - F_SETLKW polls F_SETLK, decoding the guest's struct flock once before the loop so another thread cannot switch which region is locked underneath the wait. - semop polls with IPC_NOWAIT forced onto a copy. When the set does not apply, repeating the walk the kernel makes before it blocks says which operation stopped it, so a set mixing IPC_NOWAIT with blocking operations keeps its per-operation semantics without ever entering a blocking host call. The walk answers SEMOP_BLOCKER_UNKNOWN when it cannot read the values, which a set granting alter but not read permission does, and the caller waits rather than inventing a refusal it cannot back up. - The io and futex retry waits materialize an expired ITIMER_REAL before sleeping, so an alarm that fires during a contended lock is not delayed until the lock is acquired. Only ITIMER_REAL: Linux charges the other two to CPU time, and a thread parked in a host call spends none. An execve teardown and a fork snapshot must not overlap. A sibling parked in the fork barrier owes the forker its quiet until the copy finishes, because unlike Linux, whose fork() takes a copy-on-write snapshot later writes cannot reach, the copy here is not atomic against a running thread. But a barrier that never releases strands that sibling and the join reports it as one that refused to leave. The two are serialized rather than raced. de_thread waits for an open window to close before it publishes the teardown, and thread_quiesce_siblings refuses to arm a new one once it has, with its callers abandoning the operation the quiet was for. Both sides of that handshake run under thread_lock, so the thread that releases the barrier is never one the teardown is waiting on, and the barrier blocks with no timer and no escape branch. The refusals return through the existing cleanup labels: sys_clone has already spawned its child by then, so a bare return would leak its socketpair and leave the child a zombie nothing reaps. One wait is still unreachable and recorded where it lives: the read side of a blocking FIFO open, which macOS gives nothing to poll on in any state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by cubic
Implements Linux-style execve teardown with leader handoff so new images start single‑threaded and we never reset guest memory under live threads. Also fixes a clone region lookup race and adds per‑test hang stack sampling for timeouts.
Written for commit 5ca06a8. Summary will update on new commits.