diff --git a/README.md b/README.md index 6cd7f788..732fedf4 100644 --- a/README.md +++ b/README.md @@ -196,6 +196,37 @@ there is no honest way to turn it into a percentage of the language: - **An "example" is not a feature.** Examples are not weighted, so a heavily specified method contributes far more than a rarely used one. +**What the unswept 42 % holds — measured once, 2026-11-10.** The trees above are +not gated, but they have now been measured a single time, against MRI 4.0.7 +through the same shim, so the gap is a number rather than a guess +(`cmd/specdelta` + `cmd/specroll`, corpus at the pinned `SPEC_SHA`, `c604cce1`): + +| | | +| --- | --- | +| spec files in the corpus | 3,814 — of which 2,206 are swept and 1,608 are not | +| files where the two engines differ | 970 | +| examples MRI passes in them / rbgo passes | 3,676 / 1,335 | +| the gap | 2,341 | +| files rbgo produces no result for | 152 — **75 of them hang**, the rest are genuine absences | + +Three readings matter more than the total: + +- **78 of the unswept files cannot be measured at all under this shim.** All 32 + `command_line/` files are built on `ruby_exe`, and the 46 `optional/capi/` + files need a compiled C extension: **MRI itself scores 0** in them. That part + of the corpus needs harness work, not interpreter work. +- **The largest single debt is a hang, not a missing feature.** `Net::FTP` alone + is 442 examples behind, and 39 of its 56 spec files have to be killed by the + clock on rbgo and none on MRI ([#916](https://github.com/go-embedded-ruby/ruby/issues/916)). +- **The rest is concentrated:** `matrix` 233, `bigdecimal` 180, `stringscanner` + 154, `date`+`datetime` 179, `zlib` 130, `stringio` 94, `openssl` 93. The + outright absences are small by comparison — `Coverage` 53, `StringScanner`'s + newer byte methods 27, four `CGI` escape files 19, `WeakRef` 9. + +⛔ This is **one dated run, and not a gate**: nothing re-measures it, so treat it +as a map of where the work is rather than as a figure to quote. The swept +`language/` + `core/` number above is the one CI enforces. + **What CI guarantees is per file, not a total.** `BASELINE` records the passing count of each of the 2,206 spec files, and the run fails if any file passes fewer than its record or stops loading. There is no margin and no single number diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index 838295cf..740a2776 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -509,6 +509,75 @@ unreadable result line, two result lines in one run, a missing or malformed population** — a mistyped `-dir` would otherwise print `0 files, no regressions` and exit 0. +### ⛔ A hang is not an absence, and the report says which + +A row whose engine was stopped by `-timeout` reads + +```console +library/net-ftp/put_spec.rb 39 0 -39 KILLED on timeout, the treatment +``` + +and the kill is reported **before** `DID NOT RUN`. The order is the whole point: +a killed engine prints no `RBGO_RESULT` line either, so `ran` is false and the +`DID NOT RUN` arms would otherwise claim every killed row — publishing a file +the **clock** stopped as a file the engine could not **load**. Those call for +opposite work: *the timeout is too short, or it hangs* against *this is +unimplemented*. + +It did exactly that until 2026-11-10 (#915): `run()` returned a freshly parsed +result, so `killed` was false on every row ever printed and this note was dead +code. The documented way to judge a whole report — *grep it for `KILLED`; trust +the table when the count is 0* — could only ever answer 0. The first sweep after +the fix turned **75 files** from "unimplemented" into "hangs" (`Net::FTP` 39, +`Net::HTTP` 27, `Socket` 8, `Timeout` 1 — issue #916). + +⛔ **Reports written before that commit cannot tell the two apart.** + +**Pick the timeout for the question.** For *did this regress*, give a file +roughly 60× its usual work. For *does it hang or is it missing*, a SHORT budget +is the better instrument: 20 s is already ~100× a normal spec file, anything +past it is a hang by any definition, and the answer arrives in minutes. The +first re-measurement of `net-ftp` at 300 s was still running after 77 minutes +with nothing printed; at 20 s it answered in 13. + +### Control bytes never reach the report + +ruby/spec prints raw binary on purpose — zlib round-trips, `Net::HTTP` bodies, +`String#unpack` — and the shim quotes an example's description verbatim into +`RBGO_DETAIL`. Those bytes used to land in the report, and `file` then called it +*"RAGE Package Format (RPF)"*: **grep skips a file it decides is binary**, so +every search over it returned no output and exit 1, which reads exactly like +"no matches". Three checks of one `library/` report were passed on that silence. +`printable()` strips control bytes before parsing, keeping tab and newline +because the format is built on them — the same guard `run.sh` has always had one +layer out (`tr -d '\000'`). + +## Rolling a report up to the module — `cmd/specroll` + +specdelta answers per **file**. *"What is left to reach parity with MRI?"* is a +question about **modules**, and one run of `library/` is 497 rows with 189 +moved — a shape no one reads by eye, and summarising it by eye is a count done +by eye. + +```console +$ go run ./scripts/conformance/rubyspec/cmd/specroll -depth 2 -top 5 parity-*.txt +module files MRI rbgo gap unloadable (examples) +library/net-ftp 47 448 6 442 39 (388) +library/net-http 124 354 14 340 30 (75) +library/matrix 80 334 101 233 +library/bigdecimal 41 298 118 180 2 (8) +library/stringscanner 33 215 61 154 3 (27) +TOTAL 970 3676 1335 2341 152 (620) +``` + +It **checks its arithmetic against each report's own footer** and refuses when +they disagree — a rollup is exactly where a dropped row is invisible, since the +table still looks tidy and the only symptom is a total nobody re-derives. The +invariant differs by report shape: a full report must match on all three +numbers, a `-quiet` one only on the **delta**, since a file it omits contributes +0 to it. It also keeps *behind* and *absent* in separate columns, and says so +when a `-quiet` input makes the two count columns cover only the moved files. + ## Caveats The shim is not full MSpec. What is genuinely stubbed raises `SpecSkip`, so