Skip to content

socket: open with a block must close, which is 27 of the hangs in #916 - #932

Merged
tannevaled merged 3 commits into
mainfrom
socket-open-block
Nov 14, 2026
Merged

tannevaled merged 3 commits into
mainfrom
socket-open-block

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

socket: open with a block must close, which is 27 of the hangs in #916

TCPSocket.open, TCPServer.open and Socket.open were bare aliases for
.new, so the block form closed nothing: the socket outlived the block and the
peer waited for an EOF that never came.

⛔ That is the whole net-http column of issue 916

library/net-http/http/fixtures/http_server.rb serves one client at a time
and shuts down by opening a new connection to write an unterminated "CLOSE":

def close
  TCPSocket.open(ip, port) { |socket| socket.write "CLOSE" }
  @thread.join
end

With that socket still open the server's gets("\r\n\r\n") never returns, the
listen loop never comes back round to accept, and @thread.join never
completes. Every one of the 27 library/net-http files ends on that join.

Measured, against the restored MRI oracle

The ten-line reproduction, before:

MRI 4.0.6 rbgo
closed? after the block true false
the peer's gets "CLOSE" blocks for ever
join :joined hangs

After this change rbgo prints the same seven lines as MRI, in the same order,
and exits 0.

And over library/net-http + library/socket + library/timeout, 321 files,
30 s each:

files killed under rbgo passing
before 36 243
after 9 274

27 hangs gone, +31 examples. The nine that remain are the other two columns
of that issue, untouched here: library/socket ×8 and library/timeout ×1.
(net-http/http/start_spec.rb is now killed on the control — MRI hangs on
it and rbgo passes 12 — so the net-http column is 27 → 0.)

⛔ It is not the GVL, which is what I expected to find

A goroutine dump of a hung get_spec.rb puts the server thread correctly
inside ioBlock with the GVL released (readUntil → bufio.ReadByte →
net.(*conn).Read) and the main thread in threadJoin. The emulated GVL
behaves; the socket was simply still open.

Nor does Net::HTTP leak its own connection — my first guess, and wrong: the
reproduction shows peer closed (gets returned nil) after the request under
both engines.

How it closes, and why there is no fourth spelling of close

It sends the class's own close method, so TCPSocket, TCPServer and
Socket each keep the single close they already had. Each is guarded by
s.closed and therefore idempotent, so a block that closed the socket itself
is not closed twice. The close runs in a defer, so an exception or a break
leaves it closed too: this is an ensure, not a tidy-up on the happy path.

The test fails rather than hangs

It asserts the contract on the socket the block was handed, not through a
peer: the block's value comes back (not the socket), closed? is true after a
normal return and after an exception, and open without a block still gives
an open socket exactly as .new does. A regression therefore fails this test
instead of hanging it.

Refs #931, #916

(Both stay open: #931 also asks for an audit of the same block form on
UNIXSocket and UDPSocket, and for gets(sep) returning partial data at EOF,
neither of which this change settles; #916 still has nine hangs left.)

🤖 Generated with Claude Code



TCPSocket.open, TCPServer.open and Socket.open were bare aliases for .new, so
the block form closed nothing: the socket outlived the block and the peer
waited for an EOF that never came.

⛔ THAT IS THE WHOLE net-http COLUMN OF #916. ruby/spec's
library/net-http/http/fixtures/http_server.rb serves ONE client at a time and
shuts down by opening a NEW connection to write an unterminated "CLOSE":

    def close
      TCPSocket.open(ip, port) { |socket| socket.write "CLOSE" }
      @thread.join
    end

With that socket still open the server's `gets("\r\n\r\n")` never returns, the
listen loop never comes back to accept, and @thread.join never completes. All
27 library/net-http files end on that join.

Measured against the restored MRI oracle, the same ten-line reproduction:

    closed? after the block     MRI true    rbgo false
    the peer's gets             MRI "CLOSE" rbgo blocks for ever
    join                        MRI :joined rbgo hangs

and after this change rbgo prints the same seven lines as MRI, in the same
order, and exits 0.

⛔ IT IS NOT THE GVL, which is what I expected to find. A goroutine dump of a
hung get_spec.rb puts the server thread correctly INSIDE ioBlock with the GVL
released (readUntil -> bufio.ReadByte -> net.(*conn).Read) and the main thread
in threadJoin. The emulated GVL behaves; the socket was simply still open. Nor
does Net::HTTP leak its own connection: the reproduction shows "peer closed"
after the request under BOTH engines.

HOW IT CLOSES, and why there is no fourth spelling of close: it sends the
CLASS'S OWN `close` method, so TCPSocket, TCPServer and Socket each keep the
one close they already had -- each guarded by `s.closed`, hence idempotent, so
a block that closed the socket itself is not closed twice. The close runs in a
defer, so an exception or a break leaves the socket closed as well: this is an
ensure, not a tidy-up on the happy path.

The test asserts the contract on the socket the block was handed rather than
through a peer, so a regression FAILS it instead of HANGING it: the block's
value comes back (not the socket), closed? is true after a normal return and
after an exception, and `open` with no block still answers with an open
socket exactly as .new does.

Refs #931, #916

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ix exposed

The ruby/spec ratchet refused the previous commit with +27 examples overall
and ONE file down:

    passing examples: 23699 (baseline 23672, +27)
    -- REGRESSED (1) --
       core/io/nonblock_spec.rb: 3 passing, baseline 4 (-1)

⛔ The gate was right to refuse, and nothing had regressed. The example is

    TCPServer.open(0) { |socket| socket.nonblock?.should == true }

and while `open` ignored its block that body NEVER RAN: the example passed with
no assertion evaluated, and the baseline recorded that pass. Honouring the
block made it run, and it raised NoMethodError — TCPServer#nonblock? does not
exist. A test that had never run started running.

Measured both ways rather than assumed, because the +27 includes other work
merged into main today and the -1 could have been someone else's:

    core/io/nonblock_spec.rb, main ba89ba6:  pass=4 fail=0 error=0
    core/io/nonblock_spec.rb, this branch:    pass=3 fail=0 error=1
      NoMethodError  undefined method 'nonblock?' for an instance of TCPServer

So it was mine.

IO#nonblock? is installed on IO alone and reads an *IOObj field, which a socket
is not. This installs `nonblock?` on the socket classes FROM THE SAME
`io/nonblock` feature hook, so one place decides what that require adds and the
two cannot drift; the socket classes exist from startup, so they are there when
it fires.

The answer is `true` because it is true here: Go puts every socket descriptor
in non-blocking mode and polls it through the runtime, which is also why MRI
reports true for a socket by default.

⛔ `nonblock=` is deliberately absent. The blocking mode of these descriptors
belongs to the Go runtime, and a writer that accepted `false` and changed
nothing would be a more expensive lie than a missing method. MRI does allow it;
the divergence is stated here rather than papered over.

The file is back to 4 passing — but the fourth example now RUNS and asserts,
where on main it passes with nothing evaluated. Same number, different meaning.

Refs #931

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The coverage ratchet named installSocketNonblock at 83.3%, newly below 100%.
The uncovered statement was the `continue`:

    cls, ok := vm.consts[name].(*RClass)
    if !ok {
        continue          // ← all six classes exist, so nothing reaches this
    }

All six socket classes are registered at startup, so that arm is unreachable on
this build and no test can honestly cover it. Writing the guard as

    if cls, ok := vm.consts[name].(*RClass); ok { … }

keeps exactly the same protection — a name that is not a class is skipped
rather than type-asserted into a panic — and has no statement on the false
path to explain away.

⛔ The alternative was a test that deletes a class from vm.consts to drive the
`continue`. That buys the percentage by asserting a state the VM never has, and
leaves the next reader wondering which of the two is the lie. Removing the dead
statement is cheaper and truer.

Measured before pushing, with covratchet -require (#925), which is what that
flag exists for after two commits in one day failed to move this same number:

    required 2 function(s), matched 2 record(s) in a profile of 7351
    all required functions at 100%

The ruby/spec ratchet is already green on this head — the nonblock commit
settled core/io/nonblock_spec.rb — so this is the last red.

Refs #931

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 6b51deb into main Nov 14, 2026
26 checks passed
@tannevaled
tannevaled deleted the socket-open-block branch November 14, 2026 06:43
tannevaled added a commit that referenced this pull request Nov 14, 2026
#932 merged while this was in review, so the table's net-http row said 27
as the CURRENT figure for a module that no longer hangs at all. Publishing
that would have been the same defect this pull request exists to correct.

Re-measured on main at 6b51deb, 321 files across library/net-http +
library/socket + library/timeout, MRI 4.0.6 as the control, 30 s per file:
275 passing against MRI's 371, NINE killed under rbgo and none under MRI.

So the tally is 75 -> 9, and the nine are the last two rows: Thread#kill
cannot interrupt a thread blocked in a native call (#933).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Nov 14, 2026
#934)

* README: the Net::FTP hang figures describe code that has since changed

The map of where the work is said:

    The largest single debt is a hang, not a missing feature. Net::FTP alone
    is 442 examples behind, and 39 of its 56 spec files have to be killed by
    the clock on rbgo and none on MRI (#916).

Re-measured today against MRI 4.0.6, same corpus, 30 s per file, on main at
ba89ba6: Net::FTP is 128 behind (459 -> 331) and NO file is killed at all.
Only the figures this measurement actually covers are touched; the whole-corpus
numbers above that bullet are left exactly as they were, because I have not
re-swept the corpus.

The replacement states the thing that took a day to establish and that a
single number hides: ONE SYMPTOM, TWO UNRELATED CAUSES. Half the files hung
because a blocking socket call held the emulated GVL (#924); the other half
because TCPSocket.open with a block never closed the socket, so the peer waited
for an EOF that never came (#932, open). Neither fix moves the other's files,
which is why the first one looked like it had done nothing when it was measured
against the second one's specs.

The table also carries what is NOT fixed, with its cause named rather than left
as a count: library/socket 8 and library/timeout 1, both because Thread#kill
cannot interrupt a thread blocked in a native call (#933). A reader should be
able to tell the fixed from the merely counted.

⛔ And the note that outlives all of these numbers: none of it was measurable
until the reference interpreter was repaired. It had been scoring a clean 0 on
every file because its bundled gems were off the load path, and a sweep reports
that as a RESULT -- 56 files of "control 0 passing" reads as the implementation
under test beating the reference. cmd/mriref (#930) is named so the next person
measuring starts from a working oracle.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* README: net-http is now 0 too, measured on the merged main

#932 merged while this was in review, so the table's net-http row said 27
as the CURRENT figure for a module that no longer hangs at all. Publishing
that would have been the same defect this pull request exists to correct.

Re-measured on main at 6b51deb, 321 files across library/net-http +
library/socket + library/timeout, MRI 4.0.6 as the control, 30 s per file:
275 passing against MRI's 371, NINE killed under rbgo and none under MRI.

So the tally is 75 -> 9, and the nine are the last two rows: Thread#kill
cannot interrupt a thread blocked in a native call (#933).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Nov 15, 2026
#932 gave `open` MRI's block semantics on TCPSocket, TCPServer and Socket. #931's
"shape of the fix" named the rest of the family in the same breath — UNIXSocket,
UDPSocket, UNIXServer — and nothing carried it out. Those three went on aliasing
`open` straight to `new`, so a block handed to them was DISCARDED WITHOUT A WORD:
no error, no warning, a body that simply never ran.

Three of six doors wired is this week's recurring shape, and the reason it keeps
recurring is that the wrapper is shared while the WIRING is per class.

| class | was | now |
| --- | --- | --- |
| `UDPSocket` (socket_udp.go) | `native: newFn` | `socketOpenWithBlock(udp, newFn)` |
| `UNIXSocket` (socket_unix.go) | `native: newSock` | `socketOpenWithBlock(sock, newSock)` |
| `UNIXServer` (socket_unix.go) | `native: newSrv` | `socketOpenWithBlock(srv, newSrv)` |

## Checked before writing: the wrapper could have been a no-op here

`socketOpenWithBlock` closes through `cls.methods["close"]` — the class's OWN
method table, not its ancestors. A class whose `close` is inherited would have
taken the wrapper and closed nothing, and the fix would have read as done. All
three define `close` directly: UNIXSocket through `installStreamIO`
(socket.go:480), UNIXServer at socket_unix.go:144, UDPSocket at
socket_udp.go:170. That is why the test asserts `closed?` rather than trusting
the wrapper.

## ⛔ Each door proved by MUTATION, and each by ASSERTION

A test that passes before and after proves nothing, so each wiring was reverted
in turn:

| reverted | the test said |
| --- | --- |
| `UDPSocket` | `[:udp, #<UDPSocket>, :block_never_ran]` + `[:udp_raise, false, :block_never_ran]` |
| `UNIXSocket` | `[:unixsocket, #<UNIXSocket>, :block_never_ran]` |
| `UNIXServer` | `[:unixserver, #<UNIXServer>, :block_never_ran, true]` |

Three separate reds, each naming its own door and leaving the other two lines
intact — so this is one case per door, not one case that happens to cover one of
them.

The first draft failed by `NoMethodError: undefined method 'closed?' for nil`
instead. That is a panic standing in for an assertion: it detects the regression
but does not say what happened. Each case now prints `:block_never_ran` for a
variable the block never set, which makes a discarded block a readable diff.

⛔ And the UNIXSocket case deliberately has its server `close` AT ONCE without
reading. A server waiting for EOF would, with the door unwired, leave the client
open and HANG this test rather than fail it — the exact failure this whole issue
is about, reproduced inside its own regression test.

## Two traps recorded in the code rather than worked around

- **`t.TempDir()` puts the TEST'S NAME in the path, and `sun_path` is 104 bytes
  on darwin.** This test's name alone pushed the socket over, and `bind(2)`
  answers `invalid argument` — which reads as a bug in the code under test. The
  neighbouring tests fit only because their names are shorter, which is not a
  property anyone would think to preserve when renaming one. Short `os.MkdirTemp`
  dir, with the reason in a comment.
- A **backtick inside the Ruby source** closes the Go raw string that holds it.
  The test's Ruby comments say `.new`, not a quoted `new`, for that reason.

## A fourth site, outside the socket family, filed rather than fixed

Deriving the population instead of trusting #931's list found
`internal/vm/resolv.go:199`:

    dns.smethods["open"] = dns.smethods["new"]

`Resolv::DNS.open` with a block is the same defect. MRI 4.0.7
(`lib/ruby/4.0.0/resolv.rb:301-309`, read on this machine) yields, returns the
block's value and closes in an `ensure`. It is NOT fixed here, because rbgo's
`Resolv::DNS#close` currently raises `NotImplementedError` (resolv.go:207, in the
"needs real DNS sockets" list, where closing a resolver that opened nothing does
not belong) — so wiring the block form naively would trade a silent discard for a
raise. Two steps, its own change, filed separately.

Full `internal/vm` suite green (386 s under a 1-minute load average of 61);
`gofmt -l .` silent.

Refs #931, #932, #916

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tannevaled added a commit that referenced this pull request Nov 17, 2026
…lter (#958)

#950's second half asked whether Images::Image.open discarding its block
is a gap, the way the socket doors' discarded blocks were (#931/#932: 27
hung spec files) and Augeas.open's was (#952). The row in
testdata/open-doors.txt said "UNSPECIFIED, not a known defect" because
nobody had read the reference.

Read now. go-ruby-images' README calls itself "a MiniMagick / ruby-vips
replacement", and MiniMagick's lib/mini_magick/image.rb declares:

    def self.open(path_or_url, ext = nil, **options)

with NO block parameter — no `yield`, no `block_given?`, no `ensure`.
The only blocks in its body are internal, handed to `uri.open` and
`pathname.open`. ruby-vips has no `Image.open` at all; it spells this
`Vips::Image.new_from_file`. A Ruby method that takes no block and is
handed one ignores it, so rbgo is already right. Nothing to fix.

⛔⛔ AND THE "FIX" WOULD BE WORSE THAN POINTLESS. socketOpenWithBlock
closes through the CLASS'S OWN `close`, and on Images::Image `close` is
the MORPHOLOGICAL CLOSING operator — dilate then erode, taking a radius —
exactly as `#open` is morphological opening. Wrapping the door would run
an image filter on the way out of every block, believing it had released
something.

That is measured, not reasoned: wiring it makes the program die with

    ArgumentError: wrong number of arguments (given 0, expected 1)

raised by that operator, called with no radius from the helper's defer.

So three things land, none of them a behaviour change:

  - TestImagesImageOpenTakesNoBlock pins the contract. Without it the
    correct behaviour is INDISTINGUISHABLE from the defect it resembles:
    three other `open` doors that discarded a block were real bugs, so
    the next reader has every reason to assume a fourth. It asserts both
    halves a yield-only change would get wrong — the block does not run,
    and `open` answers the IMAGE, not the block's value.
  - the open-doors row is upgraded from UNSPECIFIED to reference-read,
    and says why the door STAYS listed: the row is what protects a
    correct door from being repaired.
  - socketOpenWithBlock's comment now states the assumption it was
    carrying silently — that `close` means "release" on the class being
    wrapped — with this counter-example. Nothing in the helper can tell
    the two meanings apart, since a native checks its own arguments and
    even arity is not introspectable; the door table is the check that
    can, so the instruction is to read what a new class's `close` does
    before wrapping it.

⛔ What I did NOT file, having looked: rbgo's `Image.open` takes one
argument where MiniMagick takes `(path_or_url, ext = nil, **options)`,
and `imagesArity` is a MINIMUM, so a second argument is silently ignored
rather than refused. That is cosmetic for decoding — go-ruby-images'
Open calls image.Decode, which SNIFFS the format from content via the
registered PNG/JPEG/GIF/BMP/TIFF/WebP decoders, so the `ext` hint changes
nothing either way. The URL half is a real gap (MiniMagick fetches
http/https/ftp) and is #957 rather than smuggled in here, because
implementing it gives the interpreter an outbound fetch driven by a
path-shaped argument — a judgement for the maintainer, not a parity
checkbox.

Verified by mutation, not by reading: wiring the door red-lines BOTH
gates independently — the behavioural test, and the ratchet, which
reports the row as one that no longer needs one. Reverted, both green.
gofmt clean, buildscan --vet under all 19.

Closes #950
Refs #957, #931, #932, #952

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant