From 3f0785c126d1c42dd535c61921bf7483d2df9b7c Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 15 Nov 2026 10:30:26 +0100 Subject: [PATCH] docs: four published claims the last twelve pull requests falsified Every figure here was already measured and merged; none of it needed a new run. What it needed was someone to ask which published sentences the merges had made false, which is a different question from "what did we fix". ## The four 1. `library/socket` **8 killed -> 7**, and the headline **75 -> 9** is **75 -> 8**. #935 measured it in the sentence "nine files killed by the clock across `library/socket` + `library/timeout` before, eight after". 2. A withdrawn figure. The README said the remaining hangs were "275 passing against MRI's 371". That control scored **7** on those 190 files, because `library/socket/spec_helper.rb` calls `MSpec.enable_feature` and the shim defined no `MSpec` (#936). A number counted against a broken control is not a smaller number, it is not a number; it is removed rather than adjusted. 3. The corpus map's "the rest is concentrated: matrix 233, bigdecimal 180, ..." list was compiled while `library/socket` contributed a flat **zero on both sides**. Repaired, socket is MRI 1790 / rbgo 224 -- about **-1560**, larger than every module on that list put together (#937). The two totals are deliberately **not summed**: different runs, different MRI patch releases (4.0.7 and 4.0.6), and a sum across two measurements is how two errors cancel. The published gap is marked a **floor** instead. 4. `docs/SHARED-ENGINES.md` still concluded, at the top of its `net-ftp` section, that **"#916 is not closed, because the measurement did not move ... still 39 killed"**, and told the reader `net-ftp` was not available for extraction. That was true on 2026-11-13 and is the opposite of the outcome: #927 released the GVL on the **data** connection and #928 added `Net::FTP.default_passive`, and net-ftp is **0 killed, -128**. The section now records why the inference was wrong -- "the fix did not move the measurement" was read as "the fix is incomplete" when the mechanism was wired at only some of its doors -- and lifts the extraction blocker while naming the -128 the new home inherits. ## Why a doc gate did not catch any of them Seven README drift gates run in this repository's root package and all seven are green both before and after this change: they check the conformance total against `BASELINE`, the Ruby version against the runtime, the Go floor, the numeric surface, the wasm lane, and the shim line numbers the prose cites. Not one of them looks at a figure in a dated prose table, because there is nothing to compare such a figure against -- the run that produced it is gone. That is the honest limit, and it is stated in the README rather than papered over: the dated map is explicitly "not a gate", and the only number CI enforces is per-file. The gates caught the thing they can catch -- when a 22-line insertion moved two cited line numbers, every lane went red. Refs #916, #933, #935, #936, #937 Co-Authored-By: Claude Opus 5 --- README.md | 35 ++++++++++++++++---- docs/SHARED-ENGINES.md | 46 ++++++++++++++++++++------ scripts/conformance/rubyspec/README.md | 6 ++++ 3 files changed, 70 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 0dc9cf86..664a44d6 100644 --- a/README.md +++ b/README.md @@ -206,7 +206,7 @@ through the same shim, so the gap is a number rather than a guess | spec files in the corpus | 3,814 — of which 2,206 are swept and 1,608 are not | | files where the two engines differ | 970 | | examples MRI passes in them / rbgo passes | 3,676 / 1,335 | -| the gap | 2,341 | +| the gap | 2,341 — a **floor**, not a total: see `library/socket` below | | files rbgo produces no result for | 152 — **75 of them hang**, the rest are genuine absences | Three readings matter more than the total: @@ -228,17 +228,40 @@ Three readings matter more than the total: | --- | --- | --- | --- | | `library/net-ftp` | 39 killed | **0** | a blocking socket call held the emulated GVL | | `library/net-http` | 27 killed | **0** | `TCPSocket.open { … }` never closed the socket, so the peer waited for an EOF that never came | - | `library/socket` | 8 killed | 8 | `Thread#kill` cannot interrupt a blocking native call | - | `library/timeout` | 1 killed | 1 | ditto | + | `library/socket` | 8 killed | **7** | `Thread#kill` could not interrupt a blocking native call; [#935](https://github.com/go-embedded-ruby/ruby/pull/935) landed the unblock-function primitive on the first of thirteen sites | + | `library/timeout` | 1 killed | 1 | ditto — one of the twelve sites still listed in [#933](https://github.com/go-embedded-ruby/ruby/issues/933) | - **75 → 9.** The nine that remain are the last two rows, measured the same way - on 2026-11-14: 321 files across those three modules, 275 passing against - MRI's 371, nine killed under rbgo and none under MRI. + **75 → 8.** The eight that remain are the last two rows, and they name their + own causes rather than needing a new measurement: `recv`, `send`, `recvfrom`, + `connect_nonblock`, `tcp_server_loop`, `udp_server_loop` and + `unixserver/accept`. + + ⛔ A passing-example figure for those two modules stood here and has been + **withdrawn**: it was counted against a control that scored **7** on their 190 + files, because `library/socket/spec_helper.rb` calls `MSpec.enable_feature` + and the shim defined no `MSpec` + ([#936](https://github.com/go-embedded-ruby/ruby/pull/936), merged). What that + repair revealed is the next item. ⛔ None of this was measurable until the reference interpreter was repaired: it had been scoring a clean **0** on every file because its bundled gems were off the load path, which a sweep reports as a *result* rather than as a broken instrument. See `cmd/mriref`. +- ⛔ **The largest gap is `library/socket`, and the list below was compiled + without it.** Measured 2026-11-14 against MRI 4.0.6, same corpus, 30 s per + file, with the oracle repaired: `library/socket` + `library/timeout` is + **MRI 1790 / rbgo 224** over 190 files, so socket alone is ≈ **−1560** — + more than every module in the next bullet put together + ([#937](https://github.com/go-embedded-ruby/ruby/issues/937)). In the run + tabled above it read as a flat **zero on both sides**, for the reason in the + previous item, and a module whose control scores zero is indistinguishable + from a module with no gap. It is **unimplemented surface, not hangs**: + `Addrinfo#initialize` alone is −151, `BasicSocket#recvmsg` + `#recvmsg_nonblock` + −100, the address and option accessors −188 together. + + The two figures are deliberately **not added up here**: they come from + different runs against different MRI patch releases (4.0.7 and 4.0.6), and a + sum across two measurements is how two errors cancel. - **The rest is concentrated:** `matrix` 233, `bigdecimal` 180, `stringscanner` 154, `date`+`datetime` 179, `zlib` 130, `stringio` 94, `openssl` 93. The outright absences are small by comparison — `Coverage` 53, `StringScanner`'s diff --git a/docs/SHARED-ENGINES.md b/docs/SHARED-ENGINES.md index d84ab96a..7d3e1532 100644 --- a/docs/SHARED-ENGINES.md +++ b/docs/SHARED-ENGINES.md @@ -87,17 +87,41 @@ from the code without changing the conclusion above: deadline made the accept complete at 2.308 s, a 5 s one at 5.307 s, and `ioBlock` at 0.305 s. -⛔ **And yet #916 is not closed, because the measurement did not move.** The -`library/net-ftp` specs are byte-identical after both changes: still 39 killed, -still 459/17, still −442. Both fixes are necessary and proved on a minimal -reproduction; at least one further cause remains, and it is in the -spec-harness path. Three candidates have been eliminated by reading rather than -by timing — `setsockopt`/`SO_OOBINLINE` exist, every read in `socket.go` -already releases the GVL, and the fixture's `ruby_version_is ""..."4.1"` guard -does open at rbgo's `RUBY_VERSION`. - -⛔ `net-ftp` is therefore still **mid-change in this repository** and is not -available for extraction until #916 settles. +#### ⛔ What stood here was written mid-change, and the measurement has since moved + +On 2026-11-13 this section reported that the `library/net-ftp` specs were +byte-identical after the two fixes above — still 39 killed, still 459/17, still +−442 — and concluded that a further cause remained. The conclusion was right and +the inference drawn from it was not: *"the fix did not move the measurement"* was +read as *"the fix is incomplete or wrong"*, when the mechanism was simply **wired +at only some of its doors**. + +[#921](https://github.com/go-embedded-ruby/ruby/pull/921) bounded the *control* +connection and [#924](https://github.com/go-embedded-ruby/ruby/pull/924) released +the GVL around it. `dialData`, `acceptData`, `sendData`, `retrlines` and +`retrbinary` — the **data** connection — were still holding it +([#927](https://github.com/go-embedded-ruby/ruby/pull/927)). In **active** mode, +which ruby/spec's fixture selects, the client listens and the server dials back, +so a program that serves itself has both ends in one process: holding the GVL +inside that `Accept` parks every other Ruby thread, *including the one that would +make the call it is waiting for*. `Net::FTP.default_passive`, absent until +[#928](https://github.com/go-embedded-ruby/ruby/pull/928), was what had kept the +fixture from ever reaching that path. + +Re-measured 2026-11-13 against MRI 4.0.6, same corpus, 30 s per file: +**`Net::FTP` is 128 behind (459 → 331), with no file killed at all** — from 39. + +✅ `net-ftp`'s blocking behaviour is therefore **settled here**, and the +extraction argument above no longer waits on #916: the engine should take a +deadline in its own API because an unbounded read in a shared engine is a shared +denial of service, not because rbgo is mid-fix. As with `digest` and +`securerandom`, the new home starts with the number rather than inheriting an +unstated gap — **`net-ftp` carries −128**. + +⛔ #916 stays open for a different module: eight `library/socket` + +`library/timeout` files still hang, on `Thread#kill` being unable to interrupt a +blocking native call, with twelve conversion sites left +([#933](https://github.com/go-embedded-ruby/ruby/issues/933)). ## First inventory of the 178 `go-ruby-*` modules rbgo binds diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index a118de22..9bde0ea5 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -618,6 +618,12 @@ library/stringscanner 33 215 61 154 3 (27) TOTAL 970 3676 1335 2341 152 (620) ``` +⛔ That sample is a **2026-11-10 run shown to illustrate the tool's output, not +a current figure**, and two of its rows have since been overtaken: `library/net-ftp` +is 128 behind with 0 files killed (not 442/39), and the `TOTAL` was counted while +`library/socket`'s 190 files scored zero **on both sides** — so it is a floor. +The repository README carries the dated figures; this table carries the shape. + It **checks its arithmetic against each report's own footer** and refuses when they disagree — a rollup is exactly where a dropped row is invisible, since the table still looks tidy and the only symptom is a total nobody re-derives. The