Skip to content

conformance: define MSpec.enable_feature, without which the oracle scores zero - #936

Merged
tannevaled merged 2 commits into
mainfrom
conformance-mspec
Nov 14, 2026
Merged

tannevaled merged 2 commits into
mainfrom
conformance-mspec

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

conformance: define MSpec.enable_feature, without which the oracle scores zero

library/socket/spec_helper.rb opens with nine MSpec.enable_feature calls.
The shim defines no MSpec, so that file raises NameError at load and
every file under library/socket contributes 0.

⛔ Why this matters most when the engine scored is the reference

MRI then contributes 0 to all of them, and cmd/specdelta prints:

190 file(s): control 7 passing, treatment 224 passing, +217

which reads as rbgo beating MRI by 217 examples. An oracle that cannot load is
not a low score, it is no measurement at all
— the same lesson as #930, one
layer down. I nearly published that +217 before reading the interpreter's
stderr.

With the constant defined, the same sweep says the opposite:

passing
MRI 4.0.6 1790
rbgo 224
−1566

That gap is not a regression: it is a gap that was outside measurement
until now. library/socket has 190 files and was contributing a flat zero to
the control on every sweep this repository has ever run against it.

What the change is

enable_feature is the only MSpec method ruby/spec calls — measured, not
assumed: grepping library/, core/ and language/ for MSpec.<method>
returns that one name and nothing else.

It is a no-op, because with_feature in this shim already runs every
feature block unconditionally. That is also what the corpus asks for, in that
file's own comment: "we want a constant number of spec examples across Ruby
implementations, even if they don't define these constants."

Checked

MRI now scores 13 on library/socket/tcpserver/accept_spec.rb, where it
previously stopped at load.

Refs #916, #930, #933

(#916 and #933 stay open — this measures the socket column rather than fixing
it; #930 is the merged oracle repair this continues.)

🤖 Generated with Claude Code

…ores zero

library/socket/spec_helper.rb opens with nine MSpec.enable_feature calls. The
shim defines no MSpec, so that file raises NameError at LOAD and every file
under library/socket contributes 0.

⛔ That is worst when the engine being scored is the REFERENCE. MRI then
contributes 0 to all of them, and cmd/specdelta prints

    190 file(s): control 7 passing, treatment 224 passing, +217

which reads as rbgo beating MRI by 217 examples. An oracle that cannot load is
not a low score, it is no measurement at all — the same lesson as #930, one
layer down, and I nearly published that +217 before reading the interpreter's
stderr.

enable_feature is the ONLY MSpec method ruby/spec calls. Measured, not assumed:
grepping library/, core/ and language/ for `MSpec.<method>` returns that one
name and nothing else.

It is a no-op because with_feature already runs every feature block
unconditionally in this shim, which is also what the corpus asks for in that
file's own comment: "we want a constant number of spec examples across Ruby
implementations, even if they don't define these constants".

Checked: MRI now scores 13 on library/socket/tcpserver/accept_spec.rb where it
previously stopped at load.

Refs #916

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ved two

TestReadmeStubbedHelpersMatchTheShim failed on EVERY lane — all three test
lanes and all three arch lanes — for a change that touches one Ruby file:

    README.md cites `ruby_exe` at spec_helper.rb:882; it is at :904
    README.md cites `ruby_cmd` at spec_helper.rb:883; it is at :905

The MSpec block is 22 lines and both citations sit below it. 904 - 882 = 22.

That test is a drift gate on PROSE, and it earned its keep here: nothing about
adding a module to a Ruby file suggests that two line numbers in a README three
hundred lines away are now wrong, and no reviewer reads for that. It is also
why the failure showed up on the arch lanes, which build and vet rather than
run specs — the test lives in the repository's root package, so every lane that
runs `go test ./...` sees it.

⛔ The citation style is the brittle part, not the test: a line number in prose
is a fact about the file's current layout, and any insertion above it is a
silent edit to the documentation. The test converts that silence into a
failure, which is the right trade until the README cites names instead of
lines. Left as it is here — changing the convention is a separate change from
making it true.

Verified locally before pushing this time: the test passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit bd5579c into main Nov 14, 2026
26 checks passed
@tannevaled
tannevaled deleted the conformance-mspec branch November 14, 2026 23:08
tannevaled added a commit that referenced this pull request Nov 15, 2026
…938)

Every figure here was already measured and merged; none of it needed a new run.
What it needed was someone to ask which published sentences the merges had made
false, which is a different question from "what did we fix".

## The four

1. `library/socket` **8 killed -> 7**, and the headline **75 -> 9** is **75 -> 8**.
   #935 measured it in the sentence "nine files killed by the clock across
   `library/socket` + `library/timeout` before, eight after".

2. A withdrawn figure. The README said the remaining hangs were "275 passing
   against MRI's 371". That control scored **7** on those 190 files, because
   `library/socket/spec_helper.rb` calls `MSpec.enable_feature` and the shim
   defined no `MSpec` (#936). A number counted against a broken control is not a
   smaller number, it is not a number; it is removed rather than adjusted.

3. The corpus map's "the rest is concentrated: matrix 233, bigdecimal 180, ..."
   list was compiled while `library/socket` contributed a flat **zero on both
   sides**. Repaired, socket is MRI 1790 / rbgo 224 -- about **-1560**, larger
   than every module on that list put together (#937). The two totals are
   deliberately **not summed**: different runs, different MRI patch releases
   (4.0.7 and 4.0.6), and a sum across two measurements is how two errors cancel.
   The published gap is marked a **floor** instead.

4. `docs/SHARED-ENGINES.md` still concluded, at the top of its `net-ftp` section,
   that **"#916 is not closed, because the measurement did not move ... still 39
   killed"**, and told the reader `net-ftp` was not available for extraction.
   That was true on 2026-11-13 and is the opposite of the outcome: #927 released
   the GVL on the **data** connection and #928 added `Net::FTP.default_passive`,
   and net-ftp is **0 killed, -128**. The section now records why the inference
   was wrong -- "the fix did not move the measurement" was read as "the fix is
   incomplete" when the mechanism was wired at only some of its doors -- and
   lifts the extraction blocker while naming the -128 the new home inherits.

## Why a doc gate did not catch any of them

Seven README drift gates run in this repository's root package and all seven are
green both before and after this change: they check the conformance total
against `BASELINE`, the Ruby version against the runtime, the Go floor, the
numeric surface, the wasm lane, and the shim line numbers the prose cites. Not
one of them looks at a figure in a dated prose table, because there is nothing
to compare such a figure against -- the run that produced it is gone.

That is the honest limit, and it is stated in the README rather than papered
over: the dated map is explicitly "not a gate", and the only number CI enforces
is per-file. The gates caught the thing they can catch -- when a 22-line
insertion moved two cited line numbers, every lane went red.

Refs #916, #933, #935, #936, #937

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant