Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,37 @@ there is no honest way to turn it into a percentage of the language:
- **An "example" is not a feature.** Examples are not weighted, so a heavily
specified method contributes far more than a rarely used one.

**What the unswept 42 % holds — measured once, 2026-11-10.** The trees above are
not gated, but they have now been measured a single time, against MRI 4.0.7
through the same shim, so the gap is a number rather than a guess
(`cmd/specdelta` + `cmd/specroll`, corpus at the pinned `SPEC_SHA`, `c604cce1`):

| | |
| --- | --- |
| spec files in the corpus | 3,814 — of which 2,206 are swept and 1,608 are not |
| files where the two engines differ | 970 |
| examples MRI passes in them / rbgo passes | 3,676 / 1,335 |
| the gap | 2,341 |
| files rbgo produces no result for | 152 — **75 of them hang**, the rest are genuine absences |

Three readings matter more than the total:

- **78 of the unswept files cannot be measured at all under this shim.** All 32
`command_line/` files are built on `ruby_exe`, and the 46 `optional/capi/`
files need a compiled C extension: **MRI itself scores 0** in them. That part
of the corpus needs harness work, not interpreter work.
- **The largest single debt is a hang, not a missing feature.** `Net::FTP` alone
is 442 examples behind, and 39 of its 56 spec files have to be killed by the
clock on rbgo and none on MRI ([#916](https://github.com/go-embedded-ruby/ruby/issues/916)).
- **The rest is concentrated:** `matrix` 233, `bigdecimal` 180, `stringscanner`
154, `date`+`datetime` 179, `zlib` 130, `stringio` 94, `openssl` 93. The
outright absences are small by comparison — `Coverage` 53, `StringScanner`'s
newer byte methods 27, four `CGI` escape files 19, `WeakRef` 9.

⛔ This is **one dated run, and not a gate**: nothing re-measures it, so treat it
as a map of where the work is rather than as a figure to quote. The swept
`language/` + `core/` number above is the one CI enforces.

**What CI guarantees is per file, not a total.** `BASELINE` records the passing
count of each of the 2,206 spec files, and the run fails if any file passes
fewer than its record or stops loading. There is no margin and no single number
Expand Down
69 changes: 69 additions & 0 deletions scripts/conformance/rubyspec/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,75 @@ unreadable result line, two result lines in one run, a missing or malformed
population** — a mistyped `-dir` would otherwise print `0 files, no regressions`
and exit 0.

### ⛔ A hang is not an absence, and the report says which

A row whose engine was stopped by `-timeout` reads

```console
library/net-ftp/put_spec.rb 39 0 -39 KILLED on timeout, the treatment
```

and the kill is reported **before** `DID NOT RUN`. The order is the whole point:
a killed engine prints no `RBGO_RESULT` line either, so `ran` is false and the
`DID NOT RUN` arms would otherwise claim every killed row — publishing a file
the **clock** stopped as a file the engine could not **load**. Those call for
opposite work: *the timeout is too short, or it hangs* against *this is
unimplemented*.

It did exactly that until 2026-11-10 (#915): `run()` returned a freshly parsed
result, so `killed` was false on every row ever printed and this note was dead
code. The documented way to judge a whole report — *grep it for `KILLED`; trust
the table when the count is 0* — could only ever answer 0. The first sweep after
the fix turned **75 files** from "unimplemented" into "hangs" (`Net::FTP` 39,
`Net::HTTP` 27, `Socket` 8, `Timeout` 1 — issue #916).

⛔ **Reports written before that commit cannot tell the two apart.**

**Pick the timeout for the question.** For *did this regress*, give a file
roughly 60× its usual work. For *does it hang or is it missing*, a SHORT budget
is the better instrument: 20 s is already ~100× a normal spec file, anything
past it is a hang by any definition, and the answer arrives in minutes. The
first re-measurement of `net-ftp` at 300 s was still running after 77 minutes
with nothing printed; at 20 s it answered in 13.

### Control bytes never reach the report

ruby/spec prints raw binary on purpose — zlib round-trips, `Net::HTTP` bodies,
`String#unpack` — and the shim quotes an example's description verbatim into
`RBGO_DETAIL`. Those bytes used to land in the report, and `file` then called it
*"RAGE Package Format (RPF)"*: **grep skips a file it decides is binary**, so
every search over it returned no output and exit 1, which reads exactly like
"no matches". Three checks of one `library/` report were passed on that silence.
`printable()` strips control bytes before parsing, keeping tab and newline
because the format is built on them — the same guard `run.sh` has always had one
layer out (`tr -d '\000'`).

## Rolling a report up to the module — `cmd/specroll`

specdelta answers per **file**. *"What is left to reach parity with MRI?"* is a
question about **modules**, and one run of `library/` is 497 rows with 189
moved — a shape no one reads by eye, and summarising it by eye is a count done
by eye.

```console
$ go run ./scripts/conformance/rubyspec/cmd/specroll -depth 2 -top 5 parity-*.txt
module files MRI rbgo gap unloadable (examples)
library/net-ftp 47 448 6 442 39 (388)
library/net-http 124 354 14 340 30 (75)
library/matrix 80 334 101 233
library/bigdecimal 41 298 118 180 2 (8)
library/stringscanner 33 215 61 154 3 (27)
TOTAL 970 3676 1335 2341 152 (620)
```

It **checks its arithmetic against each report's own footer** and refuses when
they disagree — a rollup is exactly where a dropped row is invisible, since the
table still looks tidy and the only symptom is a total nobody re-derives. The
invariant differs by report shape: a full report must match on all three
numbers, a `-quiet` one only on the **delta**, since a file it omits contributes
0 to it. It also keeps *behind* and *absent* in separate columns, and says so
when a `-quiet` input makes the two count columns cover only the moved files.

## Caveats

The shim is not full MSpec. What is genuinely stubbed raises `SpecSkip`, so
Expand Down
Loading