feat: prepare shared code for cooperative single-threaded event backends - #953
feat: prepare shared code for cooperative single-threaded event backends#953scottmarchant wants to merge 1 commit into
Conversation
Some platforms run libdispatch on one thread, with no worker threads. An event backend for such a platform cannot wake a worker when work is enqueued. Instead, a poke may run the enqueued work immediately, on the thread that enqueued it. This commit prepares the shared code for that execution model. It makes no functional change on any current platform. Part 1: poke-defer hooks. The macros _dispatch_cooperative_pokes_defer() and _dispatch_cooperative_pokes_undefer() compile to ((void)0) unless a cooperative event backend defines the real versions. The hooks bracket each critical section that can enqueue work while it holds an internal lock: - the dispatch_sync and dispatch_barrier_sync inline funnels - both dispatch_async_and_wait funnels - _dispatch_barrier_trysync_or_async_f - dispatch_once initializers (the once gate is held) - object dispose (destructor batches are enqueued during teardown) - the specifics-hash mutation in dispatch_queue_set_specific On a threaded platform, a poke only wakes another worker, so the hooks change nothing. On a cooperative backend, the hooks defer the inline execution until the outermost section exits. Without them, enqueued work would run under the caller's lock and deadlock. Part 2: a shared main-queue drain. _dispatch_main_queue_drain moves out of DISPATCH_COCOA_COMPAT into its own guard. The two runloop-only steps (the runloop-handle initialization and the thread-QoS override propagation) stay gated under DISPATCH_COCOA_COMPAT. A cooperative backend can then drain the thread-bound main queue through the same code the CFRunLoop callback uses, instead of a divergent copy. The compiled result on Darwin is unchanged. Platforms that define neither macro compile neither version, as before. Co-authored-by: Krzysztof Rodak <krodak.konta@gmail.com>
|
I’m not sure I understand why you’d need to bracket critical sections, I’m vaguely aware from the other pr that you want to run asyncs inline if you don’t hold any locks, but why? Is it a perf thing or a correctness thing? I’m guessing that there are cases where you can wait on future async work, and those go really poorly on a single threaded executor, and this is a way to try to patch around code written that does that? e.g., if you async(q, ^{}) then sync(q, ^{}), that deadlocks if you don’t have worker threads, and inlining the first async removes the deadlock however, if you sync(q1, ^{ async(q2, ^{}); sync(q2, ^{}); });, even under this scheme you’re going to deadlock. my vote would be to crash code that can’t work on single threaded executors early instead of working around a subset of their bugs by doing weird ordering stuff, is that untenable for some reason? Like, replace every slow path futex lock with an abort. |
|
Hi @akmorrison, thanks so much for the quick response! I'll post replies for each question below. Just some quick context, this PR is the first slice of a series. The complete port lives on our fork as a single reviewable branch, PassiveLogic/swift-corelibs-libdispatch#3, with the event backend, a 52-test suite, and the design records. The questions here are about the execution model of that backend, so I'll include link to its tests and docs where they answer a point directly. We weren't sure if the libDispatch reviewer's preferred a single large PR or smaller PRs. We opted for a series of smaller ones. But if seeing the full context in a single PR would make the intentions easier, happy to bring everything into one PR for all of you. |
The brackets protect libdispatch's own locks, not user code. Each hooked site is an internal critical section: the sync funnels' barrier lock, the once gate, object dispose, and the specifics hash under dqsh_lock. With eager pokes and no brackets, a dispatch_once initializer that calls dispatch_async would run client code under an internal lock. That code is correct on every threaded platform. The hooks only defer those pokes until the section exits. The regression test for the sync and dispose sites explains the failure each bracket prevents: sync-nested-async.c#L21-L33. |
Correctness. The main hosts are wasm modules where the embedder calls exports, and dispatch_main() may never run. Nothing can wake a worker, and no later drain point is guaranteed. So a top-level poke runs the pending work on the spot. Work submitted from inside a running item defers to the outer drain. The port pins this as specified behavior, with the ordering rules and the group consequence documented in the test itself: eager-drain.c#L21-L37. |
This case does not deadlock in the port. The inner sync is contended, so it parks. A blocking wait issued from a caller-held section pumps: it drains the queued block on q2 first, then the sync acquires and runs. The suite pins the harder shape sync(qa){ async(qb){ sync(qa) } }: sync-nested-async.c#L56-L61. WAIT-PUMPING-AUDIT.md lists every pumping site and its two invariants: the waiter holds no internal locks while parked, and nested waits never pump. |
The port does crash early where progress is impossible, and the suite enforces it. An indefinite wait issued from inside a drained work item crashes at once: nested-wait.c#L29-L33 runs in crash mode and must print the diagnostic named in CMakeLists.txt#L54-L55. An indefinite wait that nothing can satisfy gets the same treatment, and so do fd-source failure modes that would otherwise hang or spin. The port never hangs and never spins silently. Timed waits are the boundary of that policy. A timed wait keeps exact platform semantics: it consumes its full timeout and then reports timed out, and it must not return early just because there is one thread: blocking-waits.c#L21-L29. That holds even inside a drained work item, where the wait does not pump and does not crash. It sleeps to its own deadline and reports timeout: nested-timed-wait.c#L41-L48. So code with deadline and fallback logic behaves the same as on threaded platforms. We crash only where the requested semantics are impossible. An abort on every slow-path lock would go further and abort your first example too. That sync takes the slow path because the queue is not empty, and on one thread that is normal for correct code. It would also abort the semaphore and group waits that timers and fd sources satisfy. The goal of the series is to run existing dependency code unmodified. The combined branch passes 52 of 52 tests, and 13 of those are unmodified upstream libdispatch tests. This PR only adds the no-op seams. The pump-or-crash policy arrives with the event backend PR, where it is testable. |
|
Hope this helps. Please let me know if I can answer any more questions or help with this. Our company is excited to help move Wasm support forward, and adding libDispatch support for Wasm would unlock compilation for quite a bit of code out there. Feel free to reply here. Or if it is easier or faster, I'm also available as |
2 participants
Summary
This PR prepares shared code for platforms that run libdispatch on one thread, with no worker threads. It adds two things: no-op hook macros around specific critical sections, and one shared main-queue drain function. It makes no functional change on any current platform. Linux builds and test results are identical before and after this change.
Context
WebAssembly (
wasm32-wasip1) runs libdispatch on a single thread. That platform has no way to create a worker thread. A port for it therefore uses a cooperative event backend: when work is enqueued and nothing is running, the backend runs the work immediately, on the thread that enqueued it.The series targets single-threaded WASI first. We validated the multi-threaded triple (
wasm32-wasip1-threads) separately: there libdispatch takes the ordinary threaded shape, and the seams in this PR compile to no-ops, so this preparation serves both modes.We built and tested that port as a series of small, independent changes. This PR is the first change in the series. Later changes add the WASI build system, the cooperative event backend, the Swift overlay, and file-descriptor and signal event sources. The later changes touch shared files only inside wasm-only blocks; they do not change code that other platforms compile.
You do not need the later changes to review this one. The two edits here are self-contained. Each is safe on its own, and each is required before a cooperative backend can exist.
What changes
1. Poke-defer hooks. The macros
_dispatch_cooperative_pokes_defer()and_dispatch_cooperative_pokes_undefer()compile to((void)0)on every current platform. A cooperative event backend defines the real versions and declares its functions when it arrives; this PR pre-stages no declarations. The hooks bracket each critical section that can enqueue work while it holds an internal lock. Each site carries a one-line comment; this list is the long form:dispatch_syncanddispatch_barrier_syncinline funnels: on the inline paths, the client callout runs with the queue's barrier lock helddispatch_async_and_waitfunnels, including the private-data block entry that Swift'sasyncAndWait(execute:)produces: the invoke can run inline with the acquired width or barrier held_dispatch_barrier_trysync_or_async_f: the invoke runs with the barrier lock helddispatch_onceinitializers: the initializer runs with the once gate held, and a drained item that re-enters the samedispatch_oncewould crash where threaded platforms simply waitdispatch_queue_set_specific: the destructor push must not run the client destructor underdqsh_lockOn a threaded platform, a poke only wakes another worker, so the hooks change nothing. On a cooperative backend, the hooks defer the inline execution until the outermost section exits. Without the hooks, the enqueued work would run under the caller's lock. A program that is correct on every threaded platform would then deadlock.
2. A shared main-queue drain.
_dispatch_main_queue_drainmoves out ofDISPATCH_COCOA_COMPATinto its own guard. The two runloop-only steps stay gated underDISPATCH_COCOA_COMPAT: the runloop-handle initialization, and the thread-QoS override propagation. A cooperative backend can then drain the thread-bound main queue through the same code the CFRunLoop callback uses. The alternative is a 63-line divergent copy, which we had, and which drifts. The new guard also namesdefined(__wasi__): that arm compiles nothing in this PR, and it is the seam the cooperative WASI backend in the next change compiles the shared drain through.Why this is safe
_implfunction plus a wrapper) is always-inline. The compiler folds it back together.mainand this branch produce identical results: both pass the same 23 of 23 upstream tests.What we ask from reviewers
The public macOS CMake build of this repository does not currently configure, so we cannot compile the
DISPATCH_COCOA_COMPATpath ourselves. We ask a committer to trigger@swift-ci please test, and we ask for confirmation that the run compiles theDISPATCH_COCOA_COMPATpath. That coverage is the main risk this PR carries, and it is the reason we send this small change first.