Skip to content

[DoNotMerge] Ask whether the Linux arm64 libdispatch crash is a store reordering - #14

Closed
sinoru wants to merge 2 commits into
developfrom
ci/arm64-listless-store-order
Closed

sinoru wants to merge 2 commits into
developfrom
ci/arm64-listless-store-order

Conversation

@sinoru

@sinoru sinoru commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Not to be merged. This exists to run .github/workflows/store-order.yml on the runner the crash in #11 happens on.

The reading under test

When a worker takes Swift's global concurrent queue off a root queue, _dispatch_queue_class_invoke writes DISPATCH_OBJECT_LISTLESS into the queue's do_next with a plain store. If the queue cannot run because its width is used up, _dispatch_queue_drain_try_lock then clears ENQUEUED with an acquire-only compare-and-swap (casa). Nothing orders the store before the swap. Another thread can see ENQUEUED clear, enqueue the queue again — writing NULL into do_next and linking it — and then have the late LISTLESS land on top. The next worker takes LISTLESS for the next item, and the one after it dies at _dispatch_worker_thread + 556: ldr x0, [x25, #0x10]!, next = head->do_next, with head LISTLESS.

The wakeup that enqueues a queue does not check its width, so with hundreds of tasks on a queue four wide, the failing path is routine. x86_64 cannot reorder the store past a locked instruction, which fits 0 of 2546 there.

How

  • Litmus: StoreOrder/litmus.c is the pattern alone, with the same outline-atomic helpers libdispatch uses, and two controls that must stay at zero (acqrel, and dmb ishst before the swap).
  • Reproduce: libdispatch at swift-6.4.0-RELEASE, rebuilt with the toolchain's clang three ways from one patch — as released, with dmb ishst right after the LISTLESS store, and with dmb ishld in the same place as a placebo that costs about as much and orders nothing that matters here — each put in front of the toolchain's own through LD_LIBRARY_PATH, beside the toolchain's library itself. The loader is asked which library it took before any run is counted.
Variant Jobs Expected if the reading is right
stock 2 about three LISTLESS crashes a job
rebuilt 2 as stock
ishst 4 none, where the rate would have given about twelve
ishld 3 as stock

What would say otherwise

  • ishst crashes: the reading is wrong, or not the only way there.
  • ishld drops to zero as well: the barrier's timing, not the order it enforces.
  • rebuilt differs from stock: the rebuild does not stand in for the toolchain's library.

Already seen locally

Apple M4 Pro, Linux VM (swift:6.4-noble), 60 s each:

Order Layout LISTLESS at the end
acq same line 0 of 664,305,664
acq split lines 8 of 486,572,032
acqrel same / split 0 / 0
ishst same / split 0 / 0

So the hardware makes the reordering there too, when the two words are in different lines. The reproducer has never crashed on that machine; why is not known.

Not to be merged. A litmus test for the reordering alone, and the reproducer
from #10 under three rebuilds of the toolchain's libdispatch: as released,
with a store barrier after the LISTLESS store, and with a load barrier in the
same place as a placebo.
…f it

A rebuilt library's RUNPATH names the build job's tree, so swift-backtrace,
inheriting LD_LIBRARY_PATH, found no libBlocksRuntime.so and every crash of
a rebuilt variant went uncounted as LISTLESS. The toolchain's directory now
follows the variant's. The toolchain's library and the plain rebuild get
more copies.
@sinoru

sinoru commented Sep 25, 2026

Copy link
Copy Markdown
Owner Author

Results

The reading holds. Two runs, all on ubuntu-24.04-arm.

36094940006, at fb51953. The rebuilt variants' crashes were counted as other failures: swift-backtrace inherited LD_LIBRARY_PATH, loaded the rebuilt library, and found no libBlocksRuntime.so through its RUNPATH, which names the build job's tree. Every one of them was at the crash's instruction in its own build, ldr x0, [x25, #0x10]! at _dispatch_worker_thread + 556 (0x353d0 in rebuilt, 0x353d8 in ishld), so they are counted as LISTLESS crashes here.

36099069116, at 2d93300, with the toolchain's directory after the variant's in LD_LIBRARY_PATH and more copies of stock and rebuilt. Every crash printed Bad pointer dereference at 0xffffffff89abcdff, _dispatch_worker_thread + 556; nothing else failed.

Variant Run 1 Run 2 Both
stock 7 / 664 13 / 992 20 / 1656
rebuilt 1 / 664 13 / 1327 14 / 1991
ishld 9 / 998 10 / 977 19 / 1975
ishst 0 / 1345 0 / 1337 0 / 2682

ishld's 19 against ishst's none has a chance under 10⁻⁷ of coming from one rate, and rebuilt's low first run was not repeated.

Litmus, x and y in different cache lines: 1205 of 1.60 × 10⁹ and 4668 of 1.37 × 10⁹ with the acquire-only swap; none in one line, and none in either layout with release on the swap or dmb ishst before it.

Reported upstream as swiftlang/swift-corelibs-libdispatch#963. Closing; #11 has the summary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant