Skip to content

conformance: cmd/specroll — roll a specdelta report up to the module - #914

Merged
tannevaled merged 1 commit into
mainfrom
specroll-module-rollup
Nov 10, 2026
Merged

tannevaled merged 1 commit into
mainfrom
specroll-module-rollup

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

specdelta answers per file. "What is left to reach parity with MRI?" is a question about modules, and the two are far apart: one run of library/ produced 497 rows with 189 moved. The shape of the debt is not visible by reading 497 lines, and summarising them by eye is a count done by eye.

Measured with it, on three chunks run today against MRI 4.0.7 on the same corpus:

module                              files      MRI    rbgo     gap  unloadable (examples)
library/bigdecimal                     41      298     118     180  2 (8)
library/stringscanner                  33      215      61     154  3 (27)
library/date                           38      241     109     132
library/zlib                           39      145      15     130
library/stringio                       19      352     258      94
library/uri                            21       93      32      61
library/coverage                        5       53       0      53  5 (53)
library/syslog                         11       49       0      49
…
TOTAL                                 631     2113    1107    1006  79 (141)

The two controls, and what they caught

It checks its arithmetic against the report's own footer. specdelta prints 497 file(s): control 1370 passing, treatment 736 passing, -634; specroll recomputes that from the rows it parsed and refuses when they disagree. A rollup is exactly where a dropped row is invisible — the table still looks tidy, the modules are still sorted, and the only symptom is a total nobody re-derives.

The invariant differs between the two report shapes. A full report must match on all three numbers. A -quiet report prints only the files that moved, so its rows cannot match a footer counting every file walked — what must match is the delta, since an omitted file contributes 0 to it. Every report this tool was built to read is a quiet one, so judging them by the full-report rule would have called each of them corrupt.

The first run of the test suite then found a real sign error of the same family: specdelta's delta is treatment − control, the gap column here is the other way round on purpose, and reconciling with the wrong one reported every correct quiet report as off by twice the gap.

Two distinctions it refuses to blur

  • "Behind" is not "absent." A file the treatment cannot load is counted in its own column with the examples it costs. Implement Coverage and fix 29 Date examples are different work, and summing them hides which is which.
  • It states what its columns cover. With a quiet report in the mix, the MRI and rbgo columns count only the moved files; the gap column is exact either way, which is why the table sorts by the gap and that sentence is printed rather than left as a footnote. A full rollup does not carry it — a caveat printed always is a caveat nobody reads.

Not a second measurement

It reads what specdelta measured and says nothing specdelta did not. The population is the reports named on the command line, and the footer names each one, so a rollup of half a corpus cannot be read as a whole.

Style

stdlib flag, like its four neighbours in this directory (specdelta, ratchet, phantom, oracle). These are repository scripts, not released CLIs, and cobra for one of five would be the odd one out in the directory where they are read side by side.

🤖 Generated with Claude Code

specdelta answers per FILE. "What is left to reach parity with MRI?" is a
question about MODULES, and the two are far apart: one run of library/
produced 497 rows with 189 moved, and the shape of the debt -- 41
bigdecimal files carrying 180 of the missing examples, 5 coverage files
carrying 53 more and not loading at all -- is not visible by reading 497
lines. Summarising them by eye is a count done by eye.

It carries two controls, and both earned their place during this change.

⛔ IT CHECKS ITS ARITHMETIC AGAINST THE REPORT'S OWN FOOTER. specdelta
prints "497 file(s): control 1370 passing, treatment 736 passing, -634";
specroll recomputes that from the rows it parsed and REFUSES when they
disagree. A rollup is exactly where a dropped row is invisible: the table
still looks tidy, the modules are still sorted, and the only symptom is a
total nobody re-derives.

⛔ THE INVARIANT DIFFERS BETWEEN THE TWO REPORT SHAPES. A full report must
match on all three numbers. A -quiet report prints only the files that
MOVED, so its rows cannot match a footer counting every file walked --
what must match is the DELTA, since an omitted file contributes 0 to it.
Every report this tool was built to read is a quiet one, so judging them
by the full-report rule would have called each of them corrupt. The test
for that case is TestAQuietReportIsReconciledByItsDelta.

The first run of the test suite found a real sign error from the same
family: specdelta's delta is TREATMENT MINUS CONTROL, the gap column here
is the other way round on purpose, and reconciling with the wrong one
reported every correct quiet report as off by twice the gap.

⛔ IT SEPARATES "BEHIND" FROM "ABSENT". A file the treatment cannot load
is counted in its own column with the examples it costs, because
"implement Coverage" and "fix 29 Date examples" are different work and
summing them hides which is which.

⛔ IT STATES WHAT ITS COLUMNS COVER. With a quiet report in the mix the
MRI and rbgo columns count only the moved files; the gap column is exact
either way, which is why the table sorts by the gap and the sentence is
printed rather than left as a footnote. A full rollup does not carry it
(TestAFullRollupDoesNotCarryTheQuietCaveat): a caveat printed always is a
caveat nobody reads.

Style: stdlib flag, like its four neighbours in this directory
(specdelta, ratchet, phantom, oracle). These are repository scripts, not
released CLIs, and cobra for one of five would be the odd one out in the
directory where they are read side by side.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 548e25e into main Nov 10, 2026
26 checks passed
@tannevaled
tannevaled deleted the specroll-module-rollup branch November 10, 2026 01:29
tannevaled added a commit that referenced this pull request Nov 10, 2026
The README has always said `library/`, `optional/`, `command_line/` and
`security/` are "not run at all" and that the figure is therefore not a
percentage of Ruby. True, and it left the reader with no idea whether the
unmeasured half was a rounding error or the bulk of the work.

It has now been measured once, against MRI 4.0.7 through the same shim:
970 files differ, MRI passes 3,676 examples in them and rbgo 1,335, and
152 files produce no result on rbgo. Three readings matter more than the
total, and all three change what someone would do next:

  - 78 of the unswept files CANNOT be measured under this shim at all.
    Every `command_line/` file is built on `ruby_exe`, and `optional/capi`
    needs a compiled C extension: MRI itself scores 0 in them. That part
    is harness work, not interpreter work.
  - The largest single debt is a HANG, not a missing feature: Net::FTP is
    442 examples behind and 39 of its 56 files are killed by the clock on
    rbgo and none on MRI (#916).
  - The rest is concentrated in eight modules, and the outright absences
    (Coverage 53, StringScanner's byte methods 27, CGI escapes 19,
    WeakRef 9) are small beside them.

Marked as one dated run and NOT a gate, in the same words the table above
it uses: nothing re-measures it, so it is a map of where the work is and
not a figure to quote.

The conformance README gains the two things #915 and #914 added and
nothing documented:

  - `KILLED on timeout` and why it is reported BEFORE `DID NOT RUN` --
    with the note that reports written before that commit cannot tell a
    hang from an absence, and that the timeout should be picked for the
    question (60x the file's work to detect a regression; a SHORT budget
    to tell a hang from a gap -- net-ftp at 300s was still running after
    77 minutes, at 20s it answered in 13).
  - `cmd/specroll`, its reconciliation against each report's own footer,
    and why the invariant differs between a full and a -quiet report.
  - `printable()`: a report holding a spec's raw binary was called "RAGE
    Package Format (RPF)" by `file`, and grep then skipped it silently.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant