From ddb3039cfda77fec39ac34a2ef9ce2da363b1cda Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 15 Nov 2026 22:22:29 +0100 Subject: [PATCH 1/4] vm: flush every open file at interpreter teardown, as MRI does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Ruby program that writes a file and exits without closing it lost its data. Measured against MRI 4.0.7, writing 12 bytes and never closing: MRI 4.0.7 12 bytes on disk rbgo 0 bytes The program exited 0 and the file existed, empty, so nothing reported it. This is ordinary practice in a short script. MRI closes every open IO at shutdown — ruby_cleanup runs rb_ec_teardown (the at_exit handlers), then the object-space finalizer reaches each stream's fptr_finalize, which fflushes it. rbgo's runAtExit ran only the first half. It is now that teardown, in MRI's order: handlers, then the flush, so a file an at_exit handler itself writes reaches disk too. ⛔ The ordering trap. "Close everything at the end" is right for exit, abort and falling off the end, and WRONG for exit!: rb_f_exit_bang calls _exit(2), which bypasses the libc exit handlers and the finalizer alike, so MRI loses the buffer there. Measured: exit leaves 12 bytes, exit! leaves 0. runAtExit's exitImmediate return now drops the tracked streams instead of flushing them, and the test takes all the exits rather than the common one. ⛔ A second divergence, found by the differential rather than by reading: an UNCAUGHT exception ran no teardown at all. MRI reports the exception only after tearing down (ruby_cleanup: rb_ec_teardown, then error_handle), so at_exit { puts "handler ran" }; puts "before"; raise "boom" MRI exit 1, stdout "before\nhandler ran" rbgo exit 1, stdout "before" lost its handler and its bytes both. Run now tears down on that path too and still returns the error. ⛔ And the door this mechanism was missing when first written with four: IO.for_fd — and IO.new(fd), and File.new(fd) — builds a SECOND IOObj over the descriptor, sharing the first one's path and buffer, and the program writes through THAT one. Tracking IO.sysopen's own stream left the file empty where MRI held "payload". The fifth site is ioAdoptDescriptor, the chokepoint all three reach. ⛔ And a teardown flush must NOT RAISE. This was written the other way first — re-raise the first Errno, because a silent teardown failure is worse than a loud one — and TestIOForFd and TestIOForFdModeCompat refuted it: ioFlush re-opens the path, having no descriptor, so Dir.mktmpdir { |d| IO.for_fd(File.open(File.join(d,"f"),"w").fileno,"w").write("foo") } gets ENOENT once the block has removed the directory, and a program that had ended cleanly came back as Errno::ENOENT @ rb_sysopen. MRI ignores a failing fflush at shutdown, and by then no rescue can run. The cost is stated rather than hidden: such a stream loses its bytes, where MRI — holding a descriptor to an unlinked file — would still write them. That is the buffer-backed model's limit, not a decision taken here. The tracked set is the streams that can lose data: writable, path-backed, not yet closed. A read-only sweep over thousands of files retains nothing. Closed entries are compacted out at the slice's growth point rather than untracked at each of the five places that set IOObj.closed — one site to get right instead of five, and the slice cannot outgrow the set of streams open at once. Measured with the differential in rbgo-643-oracle: 16 cases, 0 diverge. Closes #643 Refs #676, #941 Co-Authored-By: Claude Opus 5 --- internal/vm/io.go | 88 ++++++++ internal/vm/io_class.go | 11 + internal/vm/io_descriptors.go | 14 ++ internal/vm/kernel_introspect.go | 20 +- internal/vm/shutdown_flush_test.go | 330 +++++++++++++++++++++++++++++ internal/vm/vm.go | 31 +++ 6 files changed, 491 insertions(+), 3 deletions(-) create mode 100644 internal/vm/shutdown_flush_test.go diff --git a/internal/vm/io.go b/internal/vm/io.go index 8448297f..a87ac4fa 100644 --- a/internal/vm/io.go +++ b/internal/vm/io.go @@ -899,6 +899,7 @@ func (vm *VM) openFileArgs(cls *RClass, args []object.Value) *IOObj { } o := openFileSpec(cls, path, &ms, perm, permGiven) o.extEnc, o.intEnc, o.binmode, o.newline = ms.extEnc, ms.intEnc, ms.binmode, ms.newline + vm.trackOpenFile(o) // door 1 of 5: File.new / File.open / IO.new on a path return o } @@ -1291,6 +1292,93 @@ func ioFlush(o *IOObj) { } } +// trackOpenFile records a writable, path-backed stream so interpreter teardown +// can flush it (flushOpenFiles). A stream with nothing to lose — no path, or not +// writable — is not recorded, which keeps IO.foreach and ARGF (both read-only, +// and both deliberately absent from the call sites) out of the slice and keeps a +// program that reads thousands of files from retaining their buffers. +// +// Call it wherever a writable path-backed stream becomes visible to Ruby. There +// are FIVE such sites: File.new/File.open (openFileArgs), IO.sysopen, IO#reopen, +// IO#pwrite, and ioAdoptDescriptor — the last reached by IO.new(fd), IO.for_fd +// and File.new(fd), which build a SECOND IOObj over a descriptor and are the +// door this mechanism was missing when it was first written with four. +// TestEveryWayToGetAWritableFileFlushesAtShutdown covers them one by one, and the +// differential in the pull request found the fifth, because a mechanism wired at +// only some of its doors is this repository's commonest defect shape. +func (vm *VM) trackOpenFile(o *IOObj) { + if o == nil || o.path == "" || !o.writable { + return + } + for _, p := range vm.openFiles { + if p == o { // idempotent: #pwrite and #reopen can land on a tracked stream + return + } + } + // Compact at the growth point: the filter is O(n) exactly when append would + // have copied O(n) anyway, so tracking stays amortised O(1) while the slice + // cannot outgrow the set of streams open at once by more than one step. + if len(vm.openFiles) == cap(vm.openFiles) { + live := vm.openFiles[:0] + for _, p := range vm.openFiles { + if !p.closed { + live = append(live, p) + } + } + vm.openFiles = live + } + vm.openFiles = append(vm.openFiles, o) +} + +// flushOpenFiles flushes every tracked stream still open, which is what MRI's +// shutdown does by closing every IO (see openFiles). It runs at the end of +// runAtExit — AFTER the handlers, as MRI's rb_ec_teardown runs +// rb_ec_exec_end_proc before rb_objspace_call_finalizer, so a file an at_exit +// handler itself wrote reaches disk too — and NOT at all under Kernel#exit! / +// Process.exit!, whose _exit(2) skips the handlers and the flush alike. +// +// ⛔ A TEARDOWN FLUSH MUST NOT RAISE, and getting that wrong is how this +// function was first written. ioFlush re-opens the path (see its comment: a +// buffer-backed stream has no descriptor), so it can fail where MRI's fflush on +// a live fd cannot — most ordinarily when the directory has been removed under +// the stream: +// +// Dir.mktmpdir { |d| IO.for_fd(File.open(File.join(d, "f"), "w").fileno, "w").write("foo") } +// +// The block's exit deletes the directory, so the flush that follows gets ENOENT. +// Raising it turned that clean program into `Errno::ENOENT @ rb_sysopen`, which +// is what TestIOForFd and TestIOForFdModeCompat caught. MRI's shutdown ignores a +// failing fflush (rb_io_fptr_finalize keeps no error), and at teardown there is +// no longer anyone who could rescue one, so a Ruby error here is swallowed and +// the remaining streams are still attempted. +// +// What that COSTS is stated rather than hidden: a stream whose path can no +// longer be written loses its bytes, where MRI — holding a descriptor to an +// unlinked file — would still write them. That is the buffer-backed model's own +// limit (IOObj), not a decision taken here. +func (vm *VM) flushOpenFiles() { + files := vm.openFiles + vm.openFiles = nil + for _, o := range files { + if o.closed || !o.writable || !o.wbufDirty { + continue + } + vm.flushOneAtTeardown(o) + } +} + +func (vm *VM) flushOneAtTeardown(o *IOObj) { + defer func() { + if r := recover(); r != nil { + if _, ok := r.(RubyError); ok { + return // see flushOpenFiles: teardown reports nothing, as MRI does + } + panic(r) // a broken VM invariant stays loud + } + }() + ioFlush(o) +} + // curStdout / curStderr / curStdin return the IO currently bound to the global, // falling back to the raw VM writer when a host rebinds it to a non-IO value. // diff --git a/internal/vm/io_class.go b/internal/vm/io_class.go index aee08ab6..f4cfae68 100644 --- a/internal/vm/io_class.go +++ b/internal/vm/io_class.go @@ -64,6 +64,10 @@ func (vm *VM) registerIOClassMethods(cIO, cFile *RClass) { mode = vm.vmodeString(args[1]) // Integer flag set or fopen-style String } o := openFileIO(cFile, path, mode) + // Door 2 of 5: IO.sysopen's stream, which lives in fdTable. A program that + // writes through this very object (rather than through an IO.for_fd wrapper, + // which is door 5) needs it tracked here. + vm.trackOpenFile(o) vm.nextFd++ o.fd = vm.nextFd vm.fdTable[o.fd] = o @@ -288,6 +292,13 @@ func (vm *VM) ioAdoptDescriptor(o, src *IOObj, pos []object.Value, opts *object. } } } + // Door 5 of 5 (see trackOpenFile): the wrapper o is a DIFFERENT IOObj from the + // descriptor's src, sharing only its path and its buffer at this instant, and + // it is o that the program writes through. Tracking src at IO.sysopen is not + // enough — measured: IO.sysopen + IO.for_fd + write, unclosed, left the file + // empty under rbgo and held "payload" under MRI 4.0.7, with the sysopen site + // already tracked. + vm.trackOpenFile(o) } // splitIOOpts separates a trailing options Hash (IO.read/write keyword arguments) diff --git a/internal/vm/io_descriptors.go b/internal/vm/io_descriptors.go index 8a99afd6..47a629ed 100644 --- a/internal/vm/io_descriptors.go +++ b/internal/vm/io_descriptors.go @@ -348,6 +348,14 @@ func defIOSeekable(cls *RClass) { } copy(o.buf[off:], data) o.writable = true // a written File flushes its buffer back on flush/close + // Door 4 of 5. MRI's rb_io_pwrite calls pwrite(2) on the descriptor, so + // there the bytes are on disk the moment #pwrite returns; here they are in + // the buffer, which makes them losable exactly like a buffered #write. + // Marking them dirty and tracking the stream is what keeps them (#643); + // that they are not on disk UNTIL a flush is a separate divergence of the + // buffer-backed model, not one this records as fixed. + o.wbufDirty = true + vm.trackOpenFile(o) return object.IntValue(int64(len(data))) }) cls.define("sysseek", func(vm *VM, self object.Value, args []object.Value, _ *Proc) object.Value { @@ -733,6 +741,12 @@ func ioReopenPath(vm *VM, o *IOObj, pos []object.Value, opts *object.Hash) objec o.isStr, o.closed, o.lineno = true, false, 0 o.w, o.pipe, o.isWriteEnd, o.pipeSynced, o.reopened = nil, nil, false, 0, nil o.closeOnExecOff = false + // Door 3 of 5: the stream now writes a DIFFERENT path, and the one that must + // be flushed at teardown is o — never fresh, which is a throwaway holding the + // same path and the file's pre-reopen bytes, and flushing it would write those + // stale bytes back over o's. That is why openFileIO does not track and its + // callers do. + vm.trackOpenFile(o) if explicit { ms := vm.ioResolveModeEnc(pos, opts) o.extEnc, o.intEnc, o.binmode, o.newline = ms.extEnc, ms.intEnc, ms.binmode, ms.newline diff --git a/internal/vm/kernel_introspect.go b/internal/vm/kernel_introspect.go index 5931fe3f..ad018f68 100644 --- a/internal/vm/kernel_introspect.go +++ b/internal/vm/kernel_introspect.go @@ -784,22 +784,36 @@ func (vm *VM) frameFileLabel(i int) string { } } -// runAtExit runs the registered at_exit blocks in LIFO order. A raise inside one -// is swallowed so the remaining handlers still run (MRI reports it but keeps -// going); other control-flow signals propagate. +// runAtExit is this VM's interpreter teardown — MRI's rb_ec_teardown (eval.c), +// which has the same two jobs in this order: run the registered at_exit blocks +// in LIFO order (rb_ec_exec_end_proc), then let every open IO reach disk +// (rb_objspace_call_finalizer, whose fptr_finalize fflushes each one). A raise +// inside a handler is swallowed so the remaining handlers still run (MRI reports +// it but keeps going); other control-flow signals propagate. +// +// The order is MRI's and it matters: a file an at_exit handler itself writes is +// flushed, because the flush comes second. func (vm *VM) runAtExit() { // Kernel#exit! / Process.exit! runs none of them: process.c rb_f_exit_bang // calls _exit(2), which is precisely "terminate without running the exit // handlers". Measured against MRI 4.0.5: `at_exit { puts "x" }; exit!(4)` // prints nothing and exits 4. + // + // _exit(2) skips the IO flush too — it is the libc exit handlers and the + // object-space finalizer that it bypasses, not just the Ruby ones — so an + // unflushed buffer is LOST under exit!, and this return is what reproduces + // that. Measured against MRI 4.0.7, writing 12 bytes to a file and never + // closing it: `exit` leaves 12 bytes on disk, `exit!` leaves 0. if vm.exitImmediate { vm.atExit = nil + vm.openFiles = nil return } for i := len(vm.atExit) - 1; i >= 0; i-- { vm.runAtExitOne(vm.atExit[i]) } vm.atExit = nil + vm.flushOpenFiles() } func (vm *VM) runAtExitOne(blk *Proc) { diff --git a/internal/vm/shutdown_flush_test.go b/internal/vm/shutdown_flush_test.go new file mode 100644 index 00000000..04bb4660 --- /dev/null +++ b/internal/vm/shutdown_flush_test.go @@ -0,0 +1,330 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +//go:build !rbgo_core + +package vm + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +// ⛔ SILENT DATA LOSS: a file written and not closed was EMPTY on disk. +// +// MRI closes every open IO at interpreter shutdown — rb_ec_teardown reaches +// rb_objspace_call_finalizer, whose fptr_finalize fflushes each stream — so the +// ordinary short script +// +// f = File.open(path, "w"); f.write("never closed") +// +// leaves its bytes on disk. rbgo buffers a file write in IOObj.buf and nothing +// flushed it at shutdown, so the program exited 0, the file existed, and it held +// 0 bytes. Measured against MRI 4.0.7 through cmd/oracle: 12 bytes against 0. +// +// ⛔ And the DISCRIMINATOR that said which defect it was: adding `f.flush` made +// rbgo write the 12 bytes, so the write was real and the buffer was the whole +// story — a fix aimed at the write path would have been aimed at nothing. +// +// Each test below asserts the FILE'S CONTENT read back from disk, never stdout: +// the program's own output was already correct while the data was being lost. +func TestAnUnclosedFileReachesDiskAtShutdown(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "unclosed.txt") + runSrc(t, fmt.Sprintf(`f = File.open(%q, "w"); f.write("never closed")`, p)) + if got := readBack(t, p); got != "never closed" { + t.Errorf("an unclosed write must survive shutdown:\ngot %q\nwant %q", got, "never closed") + } +} + +// ⛔ THE ORDERING TRAP. "Close everything at the end" is right for exit, abort +// and falling off the end, and WRONG for exit!: process.c rb_f_exit_bang calls +// _exit(2), which bypasses the libc exit handlers and the object-space finalizer +// alike, so MRI loses the buffer there. A fix that flushed unconditionally would +// have handed exit! a flush MRI does not perform. +// +// Measured against MRI 4.0.7, 12 bytes written and never closed: +// +// exit → 12 bytes +// abort → 12 bytes +// falling off the end → 12 bytes +// exit! → 0 bytes +// +// The last row is the NEGATIVE control, and it is the reason this test takes all +// four exits rather than the common one. +func TestEachExitPathFlushesExceptTheImmediateOne(t *testing.T) { + for _, tc := range []struct { + name string + tail string + want string + }{ + {"exit", "exit 0", "payload"}, + {"exit_with_status", "exit 3", "payload"}, + {"abort", `begin; abort "stop"; rescue SystemExit; end; exit 1`, "payload"}, + {"falls_off_the_end", "", "payload"}, + {"at_exit_handler_runs_first", `at_exit { f.write("+handler") }`, "payload+handler"}, + + // ⛔ exit! must keep LOSING it, as MRI does. + {"exit_bang", "exit! 4", ""}, + {"process_exit_bang", "Process.exit! 4", ""}, + } { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "x.txt") + runSrc(t, fmt.Sprintf("f = File.open(%q, \"w\")\nf.write(\"payload\")\n%s\n", p, tc.tail)) + if got := readBack(t, p); got != tc.want { + t.Errorf("%s:\ngot %q\nwant %q", tc.name, got, tc.want) + } + }) + } +} + +// ⛔ AN UNCAUGHT EXCEPTION IS A PROGRAM END TOO, and it was the second +// divergence the differential found. MRI's ruby_cleanup tears the interpreter +// down — rb_ec_teardown runs the at_exit handlers, then the object-space +// finalizer flushes every IO — and only THEN does error_handle report the +// exception. Run returned the error having done neither, so this program lost +// its handler and its bytes both. +// +// Measured against MRI 4.0.7: +// +// at_exit { puts "handler ran" }; puts "before"; raise "boom" +// MRI exit 1, stdout "before\nhandler ran" +// rbgo exit 1, stdout "before" +// +// The error is still RETURNED: an uncaught exception is what happened, and the +// host decides what to print. Only the teardown was missing. +func TestAnUncaughtExceptionStillTearsTheInterpreterDown(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "x.txt") + _, out, err := runSrcErr(t, fmt.Sprintf(` +f = File.open(%q, "w") +f.write("payload") +at_exit { puts "handler ran" } +raise "boom" +`, p)) + if err == nil { + t.Fatal("an uncaught raise must still be reported to the host") + } + if got := readBack(t, p); got != "payload" { + t.Errorf("an uncaught exception lost the file's bytes:\ngot %q\nwant %q", got, "payload") + } + if !strings.Contains(out, "handler ran") { + t.Errorf("at_exit must run on an uncaught exception, as MRI does; stdout was %q", out) + } +} + +// ⛔ A MECHANISM WIRED AT ONLY SOME OF ITS DOORS is this repository's commonest +// defect shape — three of six socket open classes, one of two exit! spellings, +// the interpreter's missing safepoint. A writable path-backed stream can reach +// Ruby four ways, and this test takes them ONE BY ONE in Ruby's own vocabulary +// rather than trusting the four trackOpenFile call sites to be all of them. +// +// A failure here names the door: whichever subtest is red is the constructor +// that hands out an untracked stream. +func TestEveryWayToGetAWritableFileFlushesAtShutdown(t *testing.T) { + t.Run("File.open_w", func(t *testing.T) { + assertSurvives(t, `f = File.open(%q, "w"); f.write("payload")`, "payload") + }) + t.Run("File.new_w", func(t *testing.T) { + assertSurvives(t, `f = File.new(%q, "w"); f.write("payload")`, "payload") + }) + t.Run("File.open_append", func(t *testing.T) { + // The file does not exist yet, so "a" creates it: the whole content is the + // appended bytes. + assertSurvives(t, `f = File.open(%q, "a"); f.write("payload")`, "payload") + }) + t.Run("File.open_r_plus", func(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "x.txt") + if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil { + t.Fatal(err) + } + runSrc(t, fmt.Sprintf(`f = File.open(%q, "r+"); f.write("AB")`, p)) + if got := readBack(t, p); got != "AB23456789" { + t.Errorf("r+ write lost:\ngot %q\nwant %q", got, "AB23456789") + } + }) + // ⛔ THE DOOR THIS MECHANISM WAS MISSING when it was first written. IO.for_fd + // (and IO.new(fd), and File.new(fd)) build a SECOND IOObj over the descriptor, + // sharing the first one's path and buffer; the program writes through that + // second one, so tracking IO.sysopen's own stream left the file empty. Found + // by the differential against MRI 4.0.7, not by reading the call sites. + t.Run("IO.sysopen_then_for_fd", func(t *testing.T) { + assertSurvives(t, `fd = IO.sysopen(%q, "w"); f = IO.for_fd(fd); f.write("payload")`, "payload") + }) + t.Run("IO#pwrite", func(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "x.txt") + if err := os.WriteFile(p, []byte("0123456789"), 0o644); err != nil { + t.Fatal(err) + } + // "r+", not "r": #pwrite on a read-only stream raises IOError "not opened + // for writing" — rb_io_pwrite calls rb_io_check_writable — and rbgo and MRI + // 4.0.7 agree on that message, so it is not a door, it is a refusal. This + // test asserted "r" at first and was wrong where the code was right. + runSrc(t, fmt.Sprintf(`f = File.open(%q, "r+"); f.pwrite("AB", 0)`, p)) + if got := readBack(t, p); got != "AB23456789" { + t.Errorf("pwrite lost:\ngot %q\nwant %q", got, "AB23456789") + } + }) + t.Run("IO#reopen", func(t *testing.T) { + dir := t.TempDir() + first := filepath.Join(dir, "first.txt") + second := filepath.Join(dir, "second.txt") + if err := os.WriteFile(second, []byte("0123456789"), 0o644); err != nil { + t.Fatal(err) + } + runSrc(t, fmt.Sprintf( + "f = File.open(%q, \"w\")\nf.write(\"ignored\")\nf.reopen(%q, \"r+\")\nf.write(\"AB\")\n", + first, second)) + // ⛔ The stream that must be flushed is the REOPENED one, never the + // throwaway the reopen built to read the new path: that throwaway holds the + // same path and the file's PRE-REOPEN bytes, so flushing it would write + // "0123456789" back over the "AB". + if got := readBack(t, second); got != "AB23456789" { + t.Errorf("write after reopen lost (or the throwaway was flushed over it):\ngot %q\nwant %q", + got, "AB23456789") + } + }) +} + +// ⛔ THE CONTROL the four doors above cannot provide: that teardown does not +// write where the program did not ask it to. A stream explicitly closed, then +// the file changed from outside, must keep the OUTSIDE content — a teardown that +// re-flushed every tracked stream regardless of IOObj.closed would put the stale +// buffer back and this is the only test that would notice. +func TestShutdownDoesNotReFlushAClosedStream(t *testing.T) { + dir := t.TempDir() + p := filepath.Join(dir, "x.txt") + runSrc(t, fmt.Sprintf(` +f = File.open(%q, "w") +f.write("from ruby") +f.close +File.write(%q, "written after the close") +`, p, p)) + if got := readBack(t, p); got != "written after the close" { + t.Errorf("teardown re-flushed a closed stream:\ngot %q\nwant %q", got, "written after the close") + } +} + +// trackOpenFile compacts the closed entries out at the growth point rather than +// untracking at each of the five places that set IOObj.closed. This drives that +// branch directly: a stream that is closed before the next track must not keep +// its slot, or a long-running program that opens and closes files forever grows +// the slice forever. +func TestTrackOpenFileDropsClosedStreamsInsteadOfGrowing(t *testing.T) { + vm := New(os.Stdout) + for i := range 64 { + o := &IOObj{isStr: true, writable: true, path: fmt.Sprintf("/tmp/none-%d", i)} + vm.trackOpenFile(o) + o.closed = true // as IO#close does, before the next open + } + if n := len(vm.openFiles); n > 2 { + t.Errorf("64 opened-and-closed streams left %d tracked; the slice must not grow with them", n) + } + + // Not writable, and no path: nothing to lose, so nothing to retain. A read-only + // sweep over thousands of files must not pin their buffers. + before := len(vm.openFiles) + vm.trackOpenFile(&IOObj{isStr: true, path: "/tmp/read-only"}) + vm.trackOpenFile(&IOObj{isStr: true, writable: true}) + vm.trackOpenFile(nil) + if after := len(vm.openFiles); after != before { + t.Errorf("a stream with nothing to flush was tracked: %d → %d", before, after) + } + + // Idempotent: #pwrite and #reopen both land on a stream already tracked. + o := &IOObj{isStr: true, writable: true, path: "/tmp/twice"} + vm.trackOpenFile(o) + n := len(vm.openFiles) + vm.trackOpenFile(o) + if len(vm.openFiles) != n { + t.Errorf("tracking the same stream twice added an entry: %d → %d", n, len(vm.openFiles)) + } +} + +// ⛔ A TEARDOWN FLUSH MUST NOT RAISE. This function first re-raised the first +// Errno a flush produced, on the reasoning that a silent teardown failure is +// worse than a loud one. Two existing tests refuted it within the hour: +// +// Dir.mktmpdir { |d| IO.for_fd(File.open(File.join(d,"f"),"w").fileno,"w").write("foo") } +// +// The block's exit removes the directory, so the teardown flush — which re-opens +// the path, having no descriptor — gets ENOENT, and a program that had ended +// cleanly came back as `Errno::ENOENT @ rb_sysopen`. MRI ignores a failing +// fflush at shutdown, and by then no rescue can run anyway. +// +// So: an unwritable stream must cost nothing but its own bytes, and the streams +// after it must still be flushed — the half of the original reasoning that was +// right, and the only part a test can see. +func TestATeardownFlushNeitherRaisesNorSkipsTheRest(t *testing.T) { + dir := t.TempDir() + good := filepath.Join(dir, "good.txt") + vm := New(os.Stdout) + vm.openFiles = []*IOObj{ + {isStr: true, writable: true, wbufDirty: true, path: filepath.Join(dir, "no-such-dir", "bad.txt"), buf: []byte("lost")}, + {isStr: true, writable: true, wbufDirty: true, path: good, buf: []byte("kept")}, + } + if r := func() (r any) { + defer func() { r = recover() }() + vm.flushOpenFiles() + return nil + }(); r != nil { + t.Errorf("teardown must not raise, got %#v", r) + } + if got := readBack(t, good); got != "kept" { + t.Errorf("the stream after the failing one was skipped:\ngot %q\nwant %q", got, "kept") + } +} + +// And the same through Ruby, which is the shape the two existing tests have: the +// directory is gone by the time teardown runs, and the program must still end +// cleanly. +func TestAFileWhoseDirectoryIsGoneDoesNotBreakTheExit(t *testing.T) { + dir := t.TempDir() + sub := filepath.Join(dir, "sub") + if err := os.Mkdir(sub, 0o755); err != nil { + t.Fatal(err) + } + // runSrc t.Fatals on a run error, so it is itself the assertion here. + runSrc(t, fmt.Sprintf(` +f = File.open(%q, "w") +f.write("payload") +Dir.rmdir(%q) rescue nil +File.unlink(%q) rescue nil +Dir.rmdir(%q) +puts "ended cleanly" +`, filepath.Join(sub, "f.txt"), sub, filepath.Join(sub, "f.txt"), sub)) +} + +// assertSurvives runs src — a format holding exactly one %q for the path — with a +// fresh temp file and checks the file's content afterwards. +func assertSurvives(t *testing.T, src, want string) { + t.Helper() + dir := t.TempDir() + p := filepath.Join(dir, "x.txt") + runSrc(t, fmt.Sprintf(src, p)) + if got := readBack(t, p); got != want { + t.Errorf("content lost at shutdown:\ngot %q\nwant %q", got, want) + } +} + +// readBack reads the file the Ruby program wrote. A missing file reads as "" so a +// caller expecting the empty content (exit!) is not told a different story from +// one expecting a truncated file. +func readBack(t *testing.T, p string) string { + t.Helper() + b, err := os.ReadFile(p) + if os.IsNotExist(err) { + return "" + } + if err != nil { + t.Fatalf("read back %s: %v", p, err) + } + return string(b) +} diff --git a/internal/vm/vm.go b/internal/vm/vm.go index d9f8e764..ed51ee9a 100644 --- a/internal/vm/vm.go +++ b/internal/vm/vm.go @@ -833,6 +833,20 @@ type VM struct { // normally. GVL-guarded. atExit []*Proc + // openFiles holds the writable, path-backed streams this interpreter has + // handed to Ruby, so interpreter teardown can flush them — MRI closes every + // open IO at shutdown (rb_objspace_call_finalizer reaches each one's + // fptr_finalize, which fflushes), which is why a script that writes a file + // and exits without closing it still finds its bytes on disk. See + // trackOpenFile and flushOpenFiles (io.go), and runAtExit. + // + // Entries are never removed on #close; flushOpenFiles skips a closed stream, + // and trackOpenFile compacts the closed ones out whenever the slice would + // grow. That bounds the slice by the number of streams open AT ONCE with one + // site to get right, instead of an untrack call at each of the five places + // that set IOObj.closed. GVL-guarded, like atExit. + openFiles []*IOObj + // frameNames is the running method-name stack (innermost last), maintained by // exec for Kernel#__method__ and #caller. GVL-guarded (VM code is serialized // by the GVL). @@ -1522,6 +1536,23 @@ func (vm *VM) Run(iseq *bytecode.ISeq) (result object.Value, err error) { vm.frameCrefs = vm.frameCrefs[:0] vm.frameMethods = vm.frameMethods[:0] vm.fileStack = vm.fileStack[:0] + // An UNCAUGHT exception is still a program end, and MRI tears the + // interpreter down before it reports it: ruby_cleanup runs + // rb_ec_teardown (at_exit handlers, then the IO flush) and only then + // error_handle prints the message. rbgo returned the error from here + // having run NEITHER, so `at_exit { … }; raise "boom"` lost its handler + // and an unclosed file lost its bytes. Measured against MRI 4.0.7: + // + // at_exit { puts "handler ran" }; puts "before"; raise "boom" + // MRI exit 1, stdout "before\nhandler ran" + // rbgo exit 1, stdout "before" (before this line) + // + // Teardown runs with the frame stack already reset, as it does on the + // SystemExit and Thread#kill paths above. A handler that itself calls + // exit sets vm.terminalExit (runAtExitOne), which is how MRI lets it + // override the status; the error is still returned, because an uncaught + // exception is what happened and the host decides what to print. + vm.runAtExit() result, err = nil, rerr } }() From 35e63ebcf6562b560aaf4bf4006306a89af40d18 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 15 Nov 2026 22:27:20 +0100 Subject: [PATCH 2/4] README: record the two IO divergences the teardown flush leaves behind MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The flush closes #643, so the table needs no row for "an unclosed file loses its data". It needs the two things that are STILL true and were measured against MRI 4.0.7 while fixing it, both the same root as the `File#stat` row above them — rbgo's streams are buffered by path and hold no descriptor: - `IO#pwrite` puts its bytes in the buffer, not on disk. MRI's rb_io_pwrite calls pwrite(2) and is on disk the moment it returns. The bytes are no longer lost, only late. - a file whose directory is removed before the program ends loses its bytes, because teardown re-opens the path to flush it and gets ENOENT. MRI writes through its descriptor to the unlinked file. Teardown reports nothing, as MRI's shutdown ignores a failing fflush. Co-Authored-By: Claude Opus 5 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 8dfd396e..3907b4af 100644 --- a/README.md +++ b/README.md @@ -875,6 +875,8 @@ Otherwise: | `Numeric#to_int` | not defined — `Complex(2.9, 0).to_int` raises `NoMethodError`; MRI returns `2` | | `File::Stat#==` | `a == b` is `false` for two stats of the same file (MRI: `true`); `a.==(b)` answers `true` | | `File#stat` on a file unlinked while open | raises `Errno::ENOENT`; MRI answers from the open descriptor. rbgo's streams are buffered by path and hold no descriptor | +| `IO#pwrite` | the bytes are in the buffer, not on disk, until a flush — MRI's `rb_io_pwrite` calls `pwrite(2)` on the descriptor and is on disk the moment it returns. They are no longer *lost*: interpreter teardown flushes them like any buffered write ([#643](https://github.com/go-embedded-ruby/ruby/issues/643)), so a program that `pwrite`s and exits finds them there. Only the *immediacy* differs, and it is the same root as the row above: no descriptor | +| an unflushed file whose directory is removed before the program ends | loses its bytes; MRI, holding a descriptor to an unlinked file, still writes them. Teardown re-opens the path to flush it, so it gets `ENOENT` — and reports nothing, as MRI's shutdown ignores a failing `fflush`. Same root again: streams are buffered by path | | `Errno` | all **158** of MRI's constant names are present, but **32** report errno `0` where MRI has a real number (`EAUTH`, `EBADRPC`, `EDEVERR`, …). Measured by set difference on darwin: 81 names are zero-valued under rbgo, 49 under MRI, and every MRI zero is also zero under rbgo. Go's portable `syscall` package does not expose the platform-only names | | `pp` | neither `Kernel#pp` nor `require "pp"` exists (`require "prettyprint"` does work) | | `Process.fork` | does not exist — Go's runtime cannot be forked safely | From 03fb979fdfff46c8342b055cbd5aa35250c4b5c7 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 15 Nov 2026 22:32:41 +0100 Subject: [PATCH 3/4] README: cite #949 for the pwrite immediacy row and #635 for the teardown ENOENT one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both rows were added without a tracker reference, which is the one thing every other row in that table carries. #949 is the pwrite/pread immediacy, filed with the measurement; the ENOENT-at-teardown row has no issue of its own because it is the same root as #635 — streams buffered by path, holding no descriptor — and now says so. Co-Authored-By: Claude Opus 5 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3907b4af..d763dd7c 100644 --- a/README.md +++ b/README.md @@ -875,8 +875,8 @@ Otherwise: | `Numeric#to_int` | not defined — `Complex(2.9, 0).to_int` raises `NoMethodError`; MRI returns `2` | | `File::Stat#==` | `a == b` is `false` for two stats of the same file (MRI: `true`); `a.==(b)` answers `true` | | `File#stat` on a file unlinked while open | raises `Errno::ENOENT`; MRI answers from the open descriptor. rbgo's streams are buffered by path and hold no descriptor | -| `IO#pwrite` | the bytes are in the buffer, not on disk, until a flush — MRI's `rb_io_pwrite` calls `pwrite(2)` on the descriptor and is on disk the moment it returns. They are no longer *lost*: interpreter teardown flushes them like any buffered write ([#643](https://github.com/go-embedded-ruby/ruby/issues/643)), so a program that `pwrite`s and exits finds them there. Only the *immediacy* differs, and it is the same root as the row above: no descriptor | -| an unflushed file whose directory is removed before the program ends | loses its bytes; MRI, holding a descriptor to an unlinked file, still writes them. Teardown re-opens the path to flush it, so it gets `ENOENT` — and reports nothing, as MRI's shutdown ignores a failing `fflush`. Same root again: streams are buffered by path | +| `IO#pwrite` / `IO#pread` | positional reads and writes go through the buffer, not a descriptor, so a `#pwrite`'s bytes are not on disk until a flush — MRI's `rb_io_pwrite` calls `pwrite(2)` and they are there the moment it returns. They are no longer *lost*: interpreter teardown flushes them like any buffered write ([#643](https://github.com/go-embedded-ruby/ruby/issues/643)), so a program that `pwrite`s and exits finds them. Only the *immediacy* differs, and it is the same root as the row above: no descriptor. [#949](https://github.com/go-embedded-ruby/ruby/issues/949) | +| an unflushed file whose directory is removed before the program ends | loses its bytes; MRI, holding a descriptor to an unlinked file, still writes them. Teardown re-opens the path to flush it, so it gets `ENOENT` — and reports nothing, as MRI's shutdown ignores a failing `fflush`. Same root again: streams are buffered by path, which is also [#635](https://github.com/go-embedded-ruby/ruby/issues/635) | | `Errno` | all **158** of MRI's constant names are present, but **32** report errno `0` where MRI has a real number (`EAUTH`, `EBADRPC`, `EDEVERR`, …). Measured by set difference on darwin: 81 names are zero-valued under rbgo, 49 under MRI, and every MRI zero is also zero under rbgo. Go's portable `syscall` package does not expose the platform-only names | | `pp` | neither `Kernel#pp` nor `require "pp"` exists (`require "prettyprint"` does work) | | `Process.fork` | does not exist — Go's runtime cannot be forked safely | From 754181cbae60dbdf780fad2f92a26b2b2eaf40d0 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 15 Nov 2026 23:42:17 +0100 Subject: [PATCH 4/4] test: vet for every target, and cover the fault branch the ratchet caught MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI was red on NINE lanes for two causes, both mine. ⛔ 1. `undefined: runSrcErr` — seven lanes, including every "compile on every target" one. The helper lives behind `//go:build !windows && !wasm`, so the test compiled on darwin and broke wasip1, js, windows and the windows test lane's vet. THE ANSWER WAS ALREADY WRITTEN, WITH ITS WHY, IN A SIBLING FILE: runKeepingError (alloc_bounds_internal_test.go:20) is the same function without the constraint, and its comment says exactly this — runSrcErr does the same but lives behind `//go:build !windows && !wasm`, and these tests must compile on every lane — CI vets the test binary for GOOS=wasip1 and GOOS=js. I reached for the first helper I had seen rather than the one whose comment answers the question. The cheap local gate is three commands: GOOS=windows / GOOS=wasip1 / GOOS=js `go vet ./internal/vm/`, all three clean now. ⛔ 2. The coverage ratchet, on macOS AND ubuntu: flushOneAtTeardown 83.3%, "was fully covered". The untested branch is the one that matters most — a Ruby error is swallowed at teardown, a GO FAULT IS NOT. The repository draws that line deliberately in callNative: a fault caused by Ruby input becomes a rescuable exception so a host cannot be crashed by a program it ran, while a broken VM invariant stays loud instead of being laundered into something a host would log and move past. TestATeardownFlushSwallowsARubyErrorButNotAGoFault asserts it, with the fault synthesised because no real flush produces one: ioFlush raises Errno, which is the other branch. flushOneAtTeardown, flushOpenFiles and trackOpenFile all measure 100.0% now. Co-Authored-By: Claude Opus 5 --- internal/vm/shutdown_flush_test.go | 37 +++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/internal/vm/shutdown_flush_test.go b/internal/vm/shutdown_flush_test.go index 04bb4660..17edcad2 100644 --- a/internal/vm/shutdown_flush_test.go +++ b/internal/vm/shutdown_flush_test.go @@ -98,10 +98,18 @@ func TestEachExitPathFlushesExceptTheImmediateOne(t *testing.T) { // // The error is still RETURNED: an uncaught exception is what happened, and the // host decides what to print. Only the teardown was missing. +// +// ⛔ It uses runKeepingError, NOT runSrcErr. The two do the same thing, and +// runSrcErr lives behind `//go:build !windows && !wasm` — so reaching for it +// here compiled on darwin and broke SEVEN CI lanes with `undefined: runSrcErr`, +// including every `compile on every target` one, because CI vets the test binary +// for GOOS=wasip1 and GOOS=js. runKeepingError exists for exactly that reason and +// says so in its own comment (alloc_bounds_internal_test.go:20); the answer was +// already written, with its why, in a sibling file of this package. func TestAnUncaughtExceptionStillTearsTheInterpreterDown(t *testing.T) { dir := t.TempDir() p := filepath.Join(dir, "x.txt") - _, out, err := runSrcErr(t, fmt.Sprintf(` + _, out, err := runKeepingError(t, fmt.Sprintf(` f = File.open(%q, "w") f.write("payload") at_exit { puts "handler ran" } @@ -282,6 +290,33 @@ func TestATeardownFlushNeitherRaisesNorSkipsTheRest(t *testing.T) { } } +// ⛔ THE OTHER HALF OF "does not raise": a Ruby error is swallowed, and a GO +// FAULT IS NOT. The repository draws that line deliberately and in one place +// already (callNative): a fault caused by Ruby INPUT becomes a rescuable Ruby +// exception so a host can never be crashed by a program it ran, while a broken VM +// INVARIANT stays loud rather than being laundered into something a host would +// log and move past. flushOneAtTeardown's recover must respect it, and the +// coverage ratchet caught this branch untested — 83.3%, "was fully covered" — on +// macOS and ubuntu both. +// +// The fault is synthesised (a nil stream faults inside ioFlush at its first +// field read) because no real flush produces one: ioFlush raises Errno through +// raiseOpenErrno, which is the OTHER branch. That is what a test may build to +// prove a propagation it cannot provoke. +func TestATeardownFlushSwallowsARubyErrorButNotAGoFault(t *testing.T) { + vm := New(os.Stdout) + defer func() { + r := recover() + if r == nil { + t.Fatal("a Go fault at teardown must propagate, not be swallowed with the Errno") + } + if _, ok := r.(RubyError); ok { + t.Errorf("a Go fault was laundered into a Ruby error: %#v", r) + } + }() + vm.flushOneAtTeardown(nil) +} + // And the same through Ruby, which is the shape the two existing tests have: the // directory is gone by the time teardown runs, and the program must still end // cleanly.