From bc0029bad0a26f24cd9aefc0c5536560a1daf773 Mon Sep 17 00:00:00 2001 From: Reuben Brooks Date: Tue, 11 Aug 2026 12:37:31 -0500 Subject: [PATCH 1/2] docs+api: make the golden-runner quiet mode discoverable (#46 item 3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--hush-load` / `SHEN_HUSH_LOAD=1` — the mode issue #46 asked for (silence `load`'s own echo, keep the program's `(output ...)`) — shipped in #47 but was documented only in source comments and `bin/shen --help`, and had no regression coverage. A runner author reading the README still found only `-q`, which on the 41.2 kernel gates `pr` itself and so returns an empty run: exactly the reported blocker. - README: document `--hush-load` in the launcher block and add a "Batch and golden-suite runners" subsection explaining why `-q` cannot work here, with a worked before/after and the cache-independence guarantee. - `shen.boot{hush_load = true}`: the embedder spelling of the same switch, alongside `jit = false`. Applied before the idempotency guard, since it is a pure output-mode flag with no boot work behind it — an embedder handed an already-booted kernel can still ask for it. - `test/cli_spec.lua`: lock in all three modes (default / `-q` / `--hush-load`), the `SHEN_HUSH_LOAD=1` spelling being byte-identical to the flag, `--hush-load` output being identical cold and on a warm fasl hit, and the `shen.boot{hush_load=true}` path. - `doc/PERF-URDR-RESULTS.md`: an independent re-measurement of issue #46's own repro at 12fab4b (startup, the three urdr suites vs shen-cl, and the per-mode stdout line counts), plus where the residual boot cost actually lives. No behaviour change to existing modes. Co-Authored-By: Claude Fable 5 --- README.md | 31 +++++++++++ doc/PERF-URDR-RESULTS.md | 53 ++++++++++++++++++ shen.lua | 14 ++++- test/cli_spec.lua | 114 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 211 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index db0430b..d98b28e 100644 --- a/README.md +++ b/README.md @@ -227,9 +227,40 @@ inference budget (an over-budget check returns `false`, fail-closed). bin/shen # interactive REPL bin/shen prog.shen ... # (load) each file, then exit bin/shen -e "(+ 1 2)" # evaluate and print (mixes with files, in order) +bin/shen --hush-load prog.shen # silence load's echo only; (output ...) still prints bin/shen -q prog.shen # -q sets *hush*: silences load echo AND (output ...) ``` +#### Batch and golden-suite runners: `--hush-load`, not `-q` + +On the 41.2 kernel the `*hush*` global gates **`pr` itself**, so `-q` silences +*all* standard output — including the program's own `(output ...)`. That makes +`-q` useless for a runner that diffs a suite's printed results against a golden +file (issue #46): the file comes back empty. + +Use **`--hush-load`** (or **`SHEN_HUSH_LOAD=1`** where the argv is fixed) +instead. It silences only what `load` itself writes — the per-form +`(fn name)` / value / type echo and the `run time:` / `typechecked in N +inferences` banners — and leaves everything the loaded program prints alive: + +```sh +$ bin/shen suite.shen # default: user output buried in load echo +(fn double) +... +run time: 0.0013 secs +loaded +RESULT: 42 +"RESULT: 42 +" +$ bin/shen --hush-load suite.shen # just the program's own output +RESULT: 42 +``` + +The mode composes with the fasl cache in both directions: a cache written +under `--hush-load` replays correctly in the default (echoing) mode and vice +versa, so warm and cold runs produce identical bytes. Embedders get the same +switch as `shen.boot{hush_load = true}`. + The REPL reads multiline forms (it tracks paren balance through strings and comments), keeps history (`~/.shen_history` with linenoise/readline installed, or run under `rlwrap`), and translates Lua-level failures into useful errors: diff --git a/doc/PERF-URDR-RESULTS.md b/doc/PERF-URDR-RESULTS.md index c3c8e13..bc78490 100644 --- a/doc/PERF-URDR-RESULTS.md +++ b/doc/PERF-URDR-RESULTS.md @@ -91,6 +91,59 @@ cons churn (bits as cons lists). - **Issue #46**: cold-start and hush-load items addressed; suite compute gap largely structural. +## Re-measured at `12fab4b` (2026-08-11) + +An independent re-run of issue #46's own repro on main, after the whole +series landed. Same host; shen-cl `8df94be` at +`/Users/reuben/projects/shen-cl/bin/sbcl/shen`; urdr `ba85b09`. Interleaved +CL/Lua pairs, **min of N** (see the thermal-noise note above), shen-lua warm +(kernel bytecode cache + fasl), `--hush-load`. + +### Startup — `(output "hi~%")` + +| Configuration | wall | +|---|---:| +| shen-lua, warm (both caches) | **0.16 s** (median 0.18) | +| shen-lua, first ever run (empty caches) | 0.86 s | +| shen-lua, `SHEN_KERNEL_CACHE=off SHEN_FASL=off` | 0.71 s | +| shen-cl `script` | 0.01 s | + +Warm boot CPU breaks down (min of 8) as `require` 0.003 s + `load_kernel` +**0.051 s** + `load_stdlib` **0.100 s** = 0.154 s. The issue's reported +`initialise` **1.77 s → 0.10 s**; the 1.5 s trivial-script cold start no +longer reproduces. + +`load_stdlib` is now the whole remaining boot cost, and it is 21 warm fasl +**hits** (verified with `SHEN_FASL_DEBUG=1`: 21/21 hit), not compilation. +`jit.p` over it is flat — kernel `EQ` 8%, `append` 6%, `kdata_de` 5%, +`is_cons`/`shen.assoc->` 5% each, `fasl_read` 3% — i.e. the cost is the +replay *rebuilding the environment*, spread across the kernel's own list and +assoc primitives, with no hot spot to cut. Closing the remaining ~16× to +shen-cl means not rebuilding it at all (an image/snapshot of the booted +state, shen-cl's `save-lisp-and-die` equivalent), not micro-optimisation. + +### urdr suites (all **ALL PASS** on both ports) + +| Suite | shen-cl (min) | shen-lua (min) | ratio | at issue open | +|---|---:|---:|---:|---| +| `shen/tests/prng` | 0.25 s | 0.41 s | **1.6×** | ~12× | +| `shen/tests/search` | 0.80 s | 2.52 s | **3.1×** | ~12× | +| `shen/tests/world` | 0.45 s | 1.56 s | **3.5×** | — | + +### Output modes (issue #46 item 3), urdr prng + +| Invocation | lines on stdout | +|---|---:| +| `shen-cl script run-tests.shen` | 232 | +| `bin/shen run-tests.shen` (default) | 239 | +| `bin/shen --hush-load run-tests.shen` | **71** | +| `bin/shen -q run-tests.shen` | 0 | + +`-q` is empty — the reported blocker — and is kernel-faithful (*hush* gates +`pr` on 41.2), so it stays. `--hush-load` yields exactly the 71 lines the +suite itself prints; shen-cl's 232 are the same 71 plus its own 161-line +load echo. Regression-locked in `test/cli_spec.lua`. + ## Gates - `make test`: **500 pass / 0 fail** diff --git a/shen.lua b/shen.lua index d3f9cf7..fe57d54 100644 --- a/shen.lua +++ b/shen.lua @@ -29,6 +29,14 @@ shen.runtime = R -- silent session do shen.eval("(hush +)") — in 41.2 the -- *hush* global gates `pr` itself, i.e. ALL output. -- verbose = true -> log each kernel file to stderr as it loads +-- hush_load = true -> silence ONLY what (load ...) itself prints (the +-- per-form value/type echo and the run time/typechecked +-- banners); output the loaded program writes itself still +-- prints. The mode a batch or golden-suite runner wants — +-- unlike quiet/*hush*, which gates `pr` and so silences +-- the program too (issue #46). Same switch as bin/shen +-- --hush-load and SHEN_HUSH_LOAD=1; stays on for the +-- session (clear it with shen.prims.HUSH_LOAD_ECHO=nil). -- jit = false -> disable the LuaJIT compiler before loading the kernel -- (jit.off()). Mitigates the aarch64 boot-time trace -- compiler SIGSEGV (issue #43); equivalent to setting @@ -36,8 +44,12 @@ shen.runtime = R -- the JIT is already off. Leave unset to keep the JIT on. local booted = false function shen.boot(opts) - if booted then return shen end opts = opts or {} + -- A pure output-mode flag with no boot work behind it: honour it even on a + -- repeat call, so an embedder that inherited an already-booted kernel can + -- still ask for the quiet-load mode. + if opts.hush_load then P.HUSH_LOAD_ECHO = true end + if booted then return shen end if opts.jit == false then P.disable_jit() end local hush0 if opts.quiet then diff --git a/test/cli_spec.lua b/test/cli_spec.lua index 50af9dd..da4484e 100644 --- a/test/cli_spec.lua +++ b/test/cli_spec.lua @@ -15,6 +15,12 @@ -- suppresses writes to STANDARD OUTPUT. A `pr` to a FILE stream writes the -- payload regardless of *hush*, matching shen-cl/shen-go/ShenScript. (This -- used to diverge: -q produced a zero-byte file; fixed in #22.) +-- * the three output modes a batch/golden runner picks between (issue #46): +-- default (echo + program output), -q (nothing at all — *hush* gates pr +-- itself on 41.2), --hush-load / SHEN_HUSH_LOAD=1 (load's echo dropped, +-- the program's own (output ...) kept), including that --hush-load output +-- is byte-identical cold and on a warm fasl hit; +-- * shen.boot{hush_load = true}, the embedder spelling of that mode. -- -- Every subprocess is wrapped in `timeout` when available, so an EOF-loop -- regression FAILS (nonzero/empty output) rather than HANGS the whole suite. @@ -286,5 +292,113 @@ do os.remove(fpath); os.execute("rm -rf " .. sh_quote(fdir)) end +-- --------------------------------------------------------------------------- +-- --hush-load / SHEN_HUSH_LOAD=1 (pyrex41/shen-lua#46 item 3). +-- +-- A batch or golden-suite runner wants the loaded program's own output and +-- nothing else. `-q` cannot give it that: on the 41.2 kernel *hush* gates `pr` +-- itself, so -q silences the program too (the run comes back EMPTY — that is +-- the "-q unusable for golden runners" report). --hush-load silences only what +-- `load` itself writes: the per-form value/type echo, `loaded`, and the +-- run-time banner. Lock in all three modes at once, plus the env-var spelling +-- and the fasl-cache interaction (warm hit must not re-emit the echo the mode +-- suppressed, so a runner's golden file is cache-state independent). +-- --------------------------------------------------------------------------- +do + local fpath = os.tmpname() .. ".shen" + local h = io.open(fpath, "w") + -- A define (so `load` echoes "(fn hush-load-fn)") plus a program write that + -- must SURVIVE the mode. The output marker is distinct from anything the + -- echo prints, so the two are never confused. + h:write('(define hush-load-fn -> ok)\n(output "USER_46_MARKER~%")\n') + h:close() + local file = sh_quote(fpath) + + -- Each mode gets a private, fresh fasl dir so its first run is a guaranteed + -- MISS (the real compile path) regardless of the developer's ~/.cache state. + local function fresh_dir() + local d = os.tmpname(); os.remove(d) + return d + end + local dq, dh, dd, de = fresh_dir(), fresh_dir(), fresh_dir(), fresh_dir() + local function envp(d) return "env SHEN_FASL_DIR=" .. sh_quote(d) .. " " end + + -- 1. default: BOTH the load echo and the user output appear (the gate exists) + local outd, coded = run(envp(dd) .. SHEN .. " " .. file) + check(coded == 0, "#46: default load exits 0") + check(outd:find("USER_46_MARKER", 1, true) ~= nil, + "#46: default load prints the program's own (output ...)") + check(outd:find("(fn hush-load-fn)", 1, true) ~= nil, + "#46: default load prints load's per-form echo") + + -- 2. -q: silences EVERYTHING, program output included. This is the reported + -- defect and it is kernel-faithful behaviour, so it is asserted, not fixed. + local outq, codeq = run(envp(dq) .. SHEN .. " -q " .. file) + check(codeq == 0, "#46: -q load exits 0") + check(outq:find("(fn hush-load-fn)", 1, true) == nil, + "#46: -q silences load's echo") + check(outq:find("USER_46_MARKER", 1, true) == nil, + "#46: -q also silences the program's own (output ...) — why golden runners need --hush-load") + + -- 3. --hush-load: echo gone, program output kept. The mode the issue asked for. + local outh, codeh = run(envp(dh) .. SHEN .. " --hush-load " .. file) + check(codeh == 0, "#46: --hush-load exits 0") + check(outh:find("USER_46_MARKER", 1, true) ~= nil, + "#46: --hush-load KEEPS the program's own (output ...)") + check(outh:find("(fn hush-load-fn)", 1, true) == nil, + "#46: --hush-load drops load's per-form value echo") + check(outh:find("loaded", 1, true) == nil, + "#46: --hush-load drops load's `loaded` return echo") + check(outh:find("run time:", 1, true) == nil, + "#46: --hush-load drops the run-time banner") + + -- 4. SHEN_HUSH_LOAD=1 is the same mode, for runners whose argv is fixed. + local oute, codee = run(envp(de) .. "SHEN_HUSH_LOAD=1 " .. SHEN .. " " .. file) + check(codee == 0, "#46: SHEN_HUSH_LOAD=1 exits 0") + check(oute == outh, + "#46: SHEN_HUSH_LOAD=1 produces byte-identical output to --hush-load") + + -- 5. Cache-state independence: the second (warm fasl HIT) run under + -- --hush-load must produce the SAME bytes as the cold run, so a golden + -- file captured on one does not break on the other. + local warm, wcode = run(envp(dh) .. SHEN .. " --hush-load " .. file) + check(wcode == 0, "#46: warm --hush-load run exits 0") + check(warm == outh, + "#46: --hush-load output is identical cold and on a warm fasl hit") + + os.remove(fpath) + for _, d in ipairs{dq, dh, dd, de} do os.execute("rm -rf " .. sh_quote(d)) end +end + +-- --------------------------------------------------------------------------- +-- shen.boot{hush_load = true} — the embedder spelling of the same switch. +-- Drives a fresh LuaJIT process that boots in-library and (load)s a file, so +-- the assertion is on the public API, not on bin/shen's argv handling. +-- --------------------------------------------------------------------------- +do + local fpath = os.tmpname() .. ".shen" + local h = io.open(fpath, "w") + h:write('(define boot-hush-fn -> ok)\n(output "BOOT_46_MARKER~%")\n') + h:close() + local lpath = os.tmpname() .. ".lua" + local lh = io.open(lpath, "w") + lh:write(([[ +package.path = %q .. "?.lua;" .. package.path +local shen = require("shen") +shen.boot{quiet = true, hush_load = true} +shen.call("load", %q) +]]):format(here, fpath)) + lh:close() + + local out, code = run("luajit " .. sh_quote(lpath)) + check(code == 0, "#46: shen.boot{hush_load=true} run exits 0") + check(out:find("BOOT_46_MARKER", 1, true) ~= nil, + "#46: shen.boot{hush_load=true} keeps the program's own (output ...)") + check(out:find("(fn boot-hush-fn)", 1, true) == nil, + "#46: shen.boot{hush_load=true} drops load's per-form echo") + + os.remove(fpath); os.remove(lpath) +end + io.write(string.format("cli_spec: %d pass, %d fail\n", npass, nfail)) os.exit(nfail == 0 and 0 or 1) From e8e0d34c463f6d1089d0da629e91f0f2bced3931 Mon Sep 17 00:00:00 2001 From: Reuben Brooks Date: Tue, 11 Aug 2026 12:45:11 -0500 Subject: [PATCH 2/2] doc: state the exact cl/lua --hush-load stdout relationship (deletion-only diff) Co-Authored-By: Claude Fable 5 --- doc/PERF-URDR-RESULTS.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/PERF-URDR-RESULTS.md b/doc/PERF-URDR-RESULTS.md index bc78490..d6053e7 100644 --- a/doc/PERF-URDR-RESULTS.md +++ b/doc/PERF-URDR-RESULTS.md @@ -141,8 +141,12 @@ state, shen-cl's `save-lisp-and-die` equivalent), not micro-optimisation. `-q` is empty — the reported blocker — and is kernel-faithful (*hush* gates `pr` on 41.2), so it stays. `--hush-load` yields exactly the 71 lines the -suite itself prints; shen-cl's 232 are the same 71 plus its own 161-line -load echo. Regression-locked in `test/cli_spec.lua`. +suite itself prints, and the relationship to shen-cl is exact: `diff` turns +shen-cl's 232 lines into shen-lua's 71 by **deletion only** (161 deletions, +0 insertions, 0 changes), i.e. shen-lua `--hush-load` stdout is byte-for-byte +shen-cl's stdout minus shen-cl's own load echo — ordering included. That is +the property a Bifrost-style exact-golden runner needs. Regression-locked in +`test/cli_spec.lua`. ## Gates