Skip to content

Opt the musl legs out of the wall-clock comparisons - #100

Closed
hmsk wants to merge 1 commit into
mainfrom
ci/skip-timing-on-musl
Closed

Opt the musl legs out of the wall-clock comparisons#100
hmsk wants to merge 1 commit into
mainfrom
ci/skip-timing-on-musl

Conversation

@hmsk

@hmsk hmsk commented Aug 27, 2026

Copy link
Copy Markdown
Owner

The musl legs I added in #94 and #96 have taken two runs red on ParallelEval#compiles concurrently with measurable speedup, at ratios of 0.817 and 0.855 against the 0.8 threshold, in two of the first four runs they have had. The second one was on #67, which touches no C at all.

assert_run_in_parallel already documents this phenomenon and the reasoning around it:

different callers have flapped on macOS at 0.811 and 0.831, once taking 4 of 8 jobs down while the same commit passed 8 of 8 in the sibling run, so runner-wide noise rather than the code under test

Widening the 0.8 would be the wrong lever either way. It buys false passes, and a GVL-held case can measure as low as 0.83, so there is little room above 0.8 before these stop detecting a lost release at all.

Both of my values sit in that band, and 0.855 is above where widening could reach without giving up the assertion, so widening is out for the same reason it already was. attempts: 3 is already in play; the failures report "3 of 3 attempts".

Not musl

Measured locally, both libcs pinned to two cores with --cpuset-cpus 0,1:

image result
ruby:3.4-alpine 3 of 3 pass
ruby:3.4-slim 3 of 3 pass

So the variable is the GitHub runner and the container it runs the job in, not the libc.

What this does

The comparisons opt out on the musl legs under QUICKJS_SKIP_PARALLELISM_TIMING, which the job sets. Five tests skip there.

Those legs exist for stack and allocator behaviour, which is what they have actually caught: musl's pthread_attr_getstack reporting only mapped stack, which broke every eval on Alpine before #94. The GVL release is asserted on eight native legs, so a lost release still fails the run.

Worth saying plainly, given the pend_on_ubuntu skips this repository carried for months under "unresolved stack overflow on Ubuntu of GitHub Actions" and which turned out to be #82: this is a skip with a stated cause and a measurement behind it, on assertions that still run elsewhere, not a platform being quietly excluded because something went wrong on it.

assert_run_in_parallel already carries a paragraph about these flapping on
runner noise, at 0.811 and 0.831 on macOS, and about why widening the 0.8
is the wrong lever: a GVL-held workload can measure 0.83, so there is
almost no room above the threshold before the assertion stops detecting a
lost release.

The musl legs added in #94 and #96 landed on the same phenomenon, at 0.817
and 0.855, in two of the first four runs. The second one took an unrelated
PR red. Locally at two cores, alpine and glibc both pass three of three, so
this is the runner and the container rather than the libc.

Those legs are there for stack and allocator behaviour. The GVL release is
asserted on eight native legs, which is where losing it would show, so the
comparisons opt out under an env var the musl job sets. Five tests skip
there; everything else still runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@hmsk

hmsk commented Aug 27, 2026

Copy link
Copy Markdown
Owner Author

Closing in favour of fixing this at the source. The skip was the wrong instrument twice over: it is the same reflex as the pend_on_ubuntu skips that hid #82 for months, and it is asymmetric, since macOS has flapped at 0.811 and 0.831 on the same assertion and was not being excluded.

The deeper problem is that assert_run_in_parallel gates merges on a wall-clock ratio while the property it is actually after is named in its own failure message: "work may not be releasing the GVL". Speedup is a proxy for that, and the proxy is more fragile than the thing it stands for, because it depends on the core count and scheduling of a shared runner rather than on anything the gem promises.

This repository already contains the robust idiom. run_threads in the Blocking tests asserts interleaving order rather than elapsed time, so it answers "was the GVL released" as a yes or no. Replacing the ratio with that shape removes the flakiness on every platform at once, adds no skips, and needs no per-leg exceptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant