Skip to content

docs: what the unswept 42% of ruby/spec holds, and cmd/specroll - #917

Merged
tannevaled merged 1 commit into
mainfrom
parity-measurement-docs
Nov 10, 2026
Merged

tannevaled merged 1 commit into
mainfrom
parity-measurement-docs

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The README has always said that library/, optional/, command_line/ and security/ are "not run at all", and that the figure is therefore not a percentage of Ruby. True — and it left the reader with no idea whether the unmeasured half was a rounding error or the bulk of the work.

It has now been measured once, against MRI 4.0.7 through the same shim.

spec files in the corpus 3,814 — 2,206 swept, 1,608 not
files where the two engines differ 970
examples MRI passes / rbgo passes 3,676 / 1,335
the gap 2,341
files rbgo produces no result for 152 — 75 of them hang

Three readings matter more than the total, and each one changes what somebody would do next:

  • 78 of the unswept files cannot be measured under this shim at all. Every command_line/ file is built on ruby_exe, and optional/capi needs a compiled C extension: MRI itself scores 0 in them. That part is harness work, not interpreter work — and nothing said so before.
  • The largest single debt is a hang, not a missing feature. Net::FTP is 442 examples behind, and 39 of its 56 spec files are killed by the clock on rbgo and none on MRI (#916).
  • The rest is concentrated in eight modules, and the outright absences (Coverage 53, StringScanner's byte methods 27, CGI escapes 19, WeakRef 9) are small beside them.

It is marked as one dated run and not a gate, in the same words the table above it already uses: nothing re-measures it, so it is a map of where the work is rather than a figure to quote. The gated language/ + core/ number is untouched, and all four README gates still pass.

The conformance README gains what #914 and #915 added

  • KILLED on timeout, and why it is reported before DID NOT RUN. A killed engine prints no RBGO_RESULT line either, so the absence arms would otherwise claim every killed row — publishing a file the clock stopped as one the engine could not load. With the note that reports written before that commit cannot tell the two apart.
  • How to pick the timeout for the question being asked. ~60× a file's work to detect a regression; a short budget to tell a hang from a gap. net-ftp at 300 s was still running after 77 minutes with nothing printed; at 20 s it answered in 13.
  • cmd/specroll, its reconciliation against each report's own footer, and why the invariant differs between a full and a -quiet report.
  • printable(): a report holding a spec's raw binary was called "RAGE Package Format (RPF)" by file, and grep then skipped it silently — which is how three checks of one report were passed on nothing.

Refs #916 — it stays open; this only stops the README leaving the unswept half
unquantified, and stops those 75 files being described as unimplemented.

🤖 Generated with Claude Code

The README has always said `library/`, `optional/`, `command_line/` and
`security/` are "not run at all" and that the figure is therefore not a
percentage of Ruby. True, and it left the reader with no idea whether the
unmeasured half was a rounding error or the bulk of the work.

It has now been measured once, against MRI 4.0.7 through the same shim:
970 files differ, MRI passes 3,676 examples in them and rbgo 1,335, and
152 files produce no result on rbgo. Three readings matter more than the
total, and all three change what someone would do next:

  - 78 of the unswept files CANNOT be measured under this shim at all.
    Every `command_line/` file is built on `ruby_exe`, and `optional/capi`
    needs a compiled C extension: MRI itself scores 0 in them. That part
    is harness work, not interpreter work.
  - The largest single debt is a HANG, not a missing feature: Net::FTP is
    442 examples behind and 39 of its 56 files are killed by the clock on
    rbgo and none on MRI (#916).
  - The rest is concentrated in eight modules, and the outright absences
    (Coverage 53, StringScanner's byte methods 27, CGI escapes 19,
    WeakRef 9) are small beside them.

Marked as one dated run and NOT a gate, in the same words the table above
it uses: nothing re-measures it, so it is a map of where the work is and
not a figure to quote.

The conformance README gains the two things #915 and #914 added and
nothing documented:

  - `KILLED on timeout` and why it is reported BEFORE `DID NOT RUN` --
    with the note that reports written before that commit cannot tell a
    hang from an absence, and that the timeout should be picked for the
    question (60x the file's work to detect a regression; a SHORT budget
    to tell a hang from a gap -- net-ftp at 300s was still running after
    77 minutes, at 20s it answered in 13).
  - `cmd/specroll`, its reconciliation against each report's own footer,
    and why the invariant differs between a full and a -quiet report.
  - `printable()`: a report holding a spec's raw binary was called "RAGE
    Package Format (RPF)" by `file`, and grep then skipped it silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit e49909e into main Nov 10, 2026
26 checks passed
@tannevaled
tannevaled deleted the parity-measurement-docs branch November 10, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant