Skip to content

Functional parity with MRI: the ordered plan, and what each phase is actually blocked on #941

Open

Description

@tannevaled

Functional parity with MRI: the ordered plan, and what each phase is actually blocked on

This is a roadmap, not a wish. Every figure in it is either a measurement with a
date or marked as unmeasured, and the ordering is derived from the measurements
rather than from how interesting the work looks.

⛔ First, a definition of done, because "parity" is otherwise unfalsifiable

  1. Every ruby/spec tree that can be measured under the shim is swept and
    gated per file, not just language/ + core/.
  2. For every spec file, rbgo's passing count ≥ MRI's, or the shortfall is a
    named open issue.
  3. No silent divergence. Where rbgo cannot do something, it raises. A
    plausible wrong answer is worse than an absence, because an absence is
    honest and a wrong answer is relied upon.
  4. The trees that cannot be measured are made measurable or declared out of
    scope in writing — never left as a silent zero.

Criterion 3 is the one that reorders this list. Criterion 4 is the one that
today's library/socket discovery (#937) exists to prevent repeating.

The ordering principle

By harm per unit of work, where harm ranks:

a hang (nothing downstream is measurable through it) > a silent wrong
answer
(the embedder believes a guarantee that does not hold) > a
measurement defect
(every number below is read through it) > absent
surface
(honest, merely incomplete).

What I verified today, with cmd/oracle against MRI 4.0.7

Five probes, run as the same program under both engines. Three of them
corrected the text of their own issue, and one refuted it outright
— which is
why this plan is built on probes rather than on issue titles.

issue its claim measured 2026-11-15
#803 Timeout.timeout never times out silent, unprotected CONFIRMED — and for CPU-bound blocks too, not only sleeps
#768 no call-depth limit, the host dies fatal Go stack overflow ⭐ REFUTED — rescuable SystemStackError, byte-identical to MRI. Candidate for closing
#909 Enumerable#to_set silent no deprecation warning CONFIRMED — MRI writes 206 bytes to stderr, rbgo 0
#774 Kernel#fork runs in-process the child mutates the parent CONFIRMED — x=99 vs MRI's x=1; and Process.fork is absent while Kernel#fork exists
#817 Object.constants is empty 0 vs MRI's 120 CONFIRMED but INVERTED — rbgo answers 1620, MRI 120. Over-reporting, not emptiness

⚠ My first #774 probe asked Process.respond_to?(:fork), got false, and took
an "absent" branch without ever calling fork — a negative that came from
the wrong question. Worth repeating because it is the cheapest mistake to make
here: a probe that does not execute the thing cannot report on it.


Phase 1 — liveness. Nothing downstream is measurable through a hang.

item state
#940 Thread.new never returns for a block that neither blocks nor finishes filed today, root cause located
#933 Thread#kill cannot interrupt a native socket call — 12 ioBlock sites, 8 hanging files open; #935 landed the primitive on 1 of 13
#895 TestUDPSocketLoopback can hang open; candidate consequence of #940

#940 is the lead item of the whole plan. The safepoint machinery is complete —
serviceSafepoint is called from 19 sites — and the interpreter's four jump
sites call none of them, so the one door a CPU-bound thread can reach is the one
that was never wired. It is the CPU half of #803 and of #933; neither can be
finished before it.

⛔ Blocked on a measurement window, not on design. The change is in the
interpreter's hottest loop, and this repository's rule is that performance is
part of every change. This machine's 1-minute load has been 36–90 all morning,
where a throughput figure cannot tell one design from another. Two designs to
measure against each other: a counter, versus a flag set by a timer (MRI's
rb_thread_schedule).

Phase 2 — measurement integrity. Every number below is read through these.

item what it costs us
#843 the harness reads its tally from the spec's own stdout a spec file can dictate its recorded pass count
#837 two platform sources the comment says are one on the ubuntu lane 48 linux examples never run and 23 darwin ones do
#865 a before(:all)/after(:all) failure counted in no tally at all
#937's unexplained 103 absent examples in one file a third of addrinfo/initialize_spec.rb produces no result, cause unknown
#857 the corpus cache defaults to /tmp a reboot empties it

This phase adds zero parity and makes parity measurable. Today's history
argues for it: library/socket read as a flat zero on both sides until #936,
and the same sweep that now reports −1566 reported "+217 in rbgo's favour"
twenty minutes earlier.

Phase 3 — silent wrong answers (criterion 3)

item the fix's shape
#774 Kernel#fork ⛔ this row was wrong and is corrected — see below
#803 Timeout.timeout after #940
#817 Object.constants 1620 → MRI's curated 120; both receivers return one table in both engines, so this is one defect, not two
#909 to_set deprecation one warning
#676 rbgo -e 'exit 3' exits 0 / #773 Run() returns nil an exit status that never reaches the process
#686 IO.read accepts a positional Hash MRI rejects ~22 examples counted with no judge
#643 the VM closes no IO at shutdown MRI closes every one
#768 ✅ closed as refuted, with the probe, 2026-11-15

⛔ The #774 row above was wrong within the hour, and the correction is the
useful part of this plan.
I had written: raise NotImplementedError, because
MRI does that where fork is unavailable, so it is faithful and an honest absence
beats a silent wrong answer.
Then I read the consumer.

The in-process emulation is load-bearing for the headline puppet apply
path: spawn.go:20 states that Puppet::Util::Execution.execute is built on it,
spawn.go:203 rests the STDOUT.reopen(pipe) on it, io.go:184 and
io_descriptors.go:645 handle descriptors specifically around a forked block,
and socket_dialerr_test.go:101 asserts that a child's exit is
runForkBlock returning its status. Raising would trade a shipped, advertised
capability for a conformance criterion — a maintainer's decision, not something
to slip in under "parity".

The three ways out, and what each costs, are written up on #774. My reading is
that making the emulation opt-in is the only one that satisfies criterion 3
without removing a feature, but it is a scope decision and I am not taking it
unilaterally.

The general lesson, since this plan will be read as instructions: a parity
criterion is not a licence to remove a capability.
Read what depends on a
behaviour before prescribing its replacement — criterion 3 says "raise rather
than answer wrongly", and it cannot mean "raise rather than serve a consumer who
needs the answer".

Phase 4 — the measured surface, best ratio first

item examples
#669 parser: do…end in an arrow-lambda body after a paren-less call 160 from one defect — the best ratio in the tracker
#937 library/socket −1560: Addrinfo#initialize −151, BasicSocket#recvmsg(+_nonblock) −100, the address/option accessors −188, the _nonblock family −79, AncillaryData#initialize −40
matrix 233
bigdecimal 180 (+ #825 #sqrt, promised on the docs site)
date + datetime 179
stringscanner 154 (+ 27 newer byte methods, absent)
zlib 130
stringio 94
openssl 93
small, named Coverage 53, four CGI escape files 19, WeakRef 9, #670 Tempfile 8, #633 32 Errno constants on darwin

#937's first slice is not the 19 predicate methods it first looked like:
measured, 40 of the 48 non-passing examples in addrinfo/initialize_spec.rb are
three missing Socket::SOCK_* constants (15) and one service-name port
path
(25). The 103 absent examples in Phase 2 must be explained first, or a fix
aimed at the visible 48 leaves the larger half untouched.

Phase 5 — what no amount of interpreter work can move

78 files, where MRI itself scores 0 under this shim: 32 command_line/
specs built on ruby_exe (a subprocess), and 46 optional/capi specs needing a
compiled C extension — which CGO=0 makes a declared non-goal rather than a
gap.

This phase is a decision to write down, per criterion 4: either the harness
grows a real ruby_exe, or these are excluded from the denominator in writing.
Leaving them as a silent zero is what criterion 4 exists to forbid.


What the gap total is, honestly

2,341 across the 970 files where the two engines differ (2026-11-10, MRI
4.0.7) — a floor, not a total, because library/socket contributed zero to
it on both sides. Socket alone adds ≈1,560 (2026-11-14, MRI 4.0.6).

⛔ The two are deliberately not summed here: different runs, different MRI
patch releases, and a sum across two measurements is how two errors cancel. One
re-measurement under a quiet machine replaces both numbers with one; until then
the floor is the honest claim.

Refs #940, #933, #895, #843, #837, #865, #857, #937, #774, #803, #817, #909, #676, #773, #686, #643, #768, #669, #825, #670, #633

🤖 Generated with Claude Code

Activity

  1. tannevaled commented on Nov 15, 2026

    @tannevaled
    ContributorAuthor

    Phase 2 gains a concrete item, and it is a pattern this repository already has twice

    Phase 2 is "measurement integrity". The entry it was missing is the one that
    would have prevented the four corrections of #938: the conformance figures are
    retyped into prose, and nothing can tell when the prose and the record
    disagree.
    Seven README drift gates stayed green through all four, because none
    of them reads a figure from a dated run.

    ⛔ I had designed a weaker gate for this and was about to build it in
    go-fleettools: markers in the prose (measured=, code=) checked against
    git log over the declared paths. Two readings stopped it before the first line
    of code, and the second one matters more than the first.

    A sibling org proves the stronger shape

    go-scikit/refbench v0.3.0 does not compare a published table to a record —
    it generates it. refbench table <perf.json> renders perf-table.md;
    refbench check <perf.json> refuses a table the record did not produce.
    Generated prose cannot drift at all, where a comparison always leaves the
    question "with what tolerance".

    And this repository already has that pattern, twice

    Which is the better citation, because it needs no new dependency:

    gate published file derived from
    TestBenchmarkedModulesSayWhereTheirCodeLives (bench_backing_test.go) bench/modules/BACKING.md go.mod + bench/modules/*.rb
    prelude_frozen_test.go internal/vm/prelude_frozen_gen.go the prelude sources

    The first one already carries the two details that make such a gate honest, and
    they should be copied rather than re-invented:

    • a regeneration flag (-update-backing), so the record is rewritten by a
      command and never by hand;
    • a population guard — if len(scripts) == 0 { t.Fatal("no benchmark scripts found; this test would pass by measuring nothing") }. A gate whose
      subject went missing must fail, not pass.

    What applying it to the conformance table takes

    Every piece exists:

    piece state
    the record specdelta writes per-file reports; a sweep is parity-*.txt
    the rollup specroll -depth 2 already renders exactly the module table the README carries
    committing a record scripts/conformance/rubyspec/BASELINE is committed and gated, one row per spec file

    So: commit the sweep's report as BASELINE is committed, generate the README's
    module table from it with specroll, and add the drift test with its own
    population guard.

    ⛔ What it does NOT fix, said here so nobody assumes otherwise

    It makes prose-versus-record drift impossible. It does not make the
    record fresh: a committed report is a dated measurement, and a code change that
    would move it leaves the record and the generated prose stale together.

    That is a strictly smaller problem — one record to re-run instead of three
    surfaces to re-read — and it is the honest limit. The generated rows should
    therefore carry the corpus SHA and the MRI patch release per row, which is also
    what today's prose does inconsistently: #938 had to label the published gap a
    floor precisely because two figures from different runs (MRI 4.0.7 and
    4.0.6) must not be summed, and provenance in a paragraph is provenance nobody
    re-reads.

    ⚠ Filed as a comment here rather than as its own issue on purpose: priorart
    exited 2 on this subject — "repo tracker NOT READ … stopped at the budget
    reserve: 905 left, 1000 reserved"
    — and an unread tracker is not an absence of
    prior art. A peer also pointed at #913/#765 as this same shape already built
    twice, which the two gates above appear to be. When the budget allows a real
    read, this belongs in its own issue if it is not already one.

    🤖 Generated with Claude Code

  2. tannevaled commented on Nov 15, 2026

    @tannevaled
    ContributorAuthor

    Correction to this plan: Phase 2 is NOT load-independent, and I claimed it was

    The plan says Phase 1's safepoint work is "blocked on a measurement window, not
    on design", and implies Phase 2 (measurement integrity) is free to proceed. I
    tried to proceed with the generated-conformance-table gate from the comment
    above, and it is blocked by the same thing.

    A generated table needs a seed record, and the seed has to be a sweep anyone
    would trust.
    Committing a stale one is worse than committing none: the
    README's current table is already labelled a floor (#938) and library/socket
    alone understates it by ~1560 (#937), so generating today's prose from the
    2026-11-10 report would enshrine a figure known to be wrong in a known
    direction — and make it look derived.

    A trustworthy sweep is ~48 minutes with a per-file timeout, which is exactly the
    measurement a loaded host cannot give: a timeout cannot tell a hang from a slow
    file. So Phase 2's first item sits behind the same window as Phase 1's.

    ⛔ And a near-miss worth writing down, because it would have read as success

    Looking for the report behind the README's table, I replayed a 2026-11-10 report
    on disk:

    $ specroll -depth 2 -top 8 parity-lib-b.txt
    TOTAL   339   1563   228   1335   73 (479)
    

    against what the README publishes:

    files where the two engines differ   970
    MRI / rbgo                           3,676 / 1,335
    the gap                              2,341
    

    1335 appears in both — as the gap in the report and as the rbgo score in
    the README.
    Different quantities, same digits. Anyone grepping for the
    published figure to identify its source report would have found it and stopped.
    It is not the source: that file is a library/-only -quiet report covering 339
    moved files, where the README's table covers 970 across four trees.

    ⭐ specroll behaved correctly on the other candidate, which is the part worth
    keeping: given a file that was not a specdelta report at all it exited 2 with
    "the report format and this parser have drifted apart" rather than parsing
    something and returning a plausible table. A tool that refuses is what let the
    first comparison be trusted.

    What this changes in the ordering

    Nothing about the phases, but one honest line about the dependency: Phases 1
    and 2 share a prerequisite that is neither design nor code
    — one quiet host,
    once, long enough for a 48-minute sweep and an 8-round alternating benchmark.
    Until then both phases can be written and neither can be finished, and the
    plan should not read as though Phase 2 were available meanwhile.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions