perf: cache the recomputable half of a warm boot; standard-library boot image (#46)

[pyrex41]

Warm start-up was ~0.16 s and, as issue #46's last re-measurement established, none of it was compilation — the kernel bytecode cache and the user fasl cache were both hitting. The time went on state that the caches had no way to express and therefore rebuilt from scratch on every single start. This PR caches that state.

What was actually being recomputed

Measured on arm64 LuaJIT 2.1.1774638290, warm, os.clock inside the process (a quiet-machine reference boot: require 0.002 + load_kernel 0.046 + initialise 0.095 = 0.143 s):

item	cost	what it is
klambda/types.kl's 161 (declare …)	43 ms	declare runs the type theory for real: shen.variancy under the Prolog machine + (eval-kl (shen.prolog-abstraction Type)), a full KL→Lua compile and loadstring per signature
(shen.build-lambda-table (external shen))	12 ms	shen.lambda-entry runs the whole compiler once per external symbol (~280)
shen.process-datatype ×2 (stdlib)	34 ms	the fasl "dt" record re-ran the shen.<datatype> yacc parser + type theory, because shen.*datatypes* entries hold closures
shen.*lambdatable* rebuild ×285	19 ms	one shen.lambda-entry compile per entry, each shen.assoc->'d into a 568-entry list by non-tail KL recursion
install.shen driver	23 ms	reading it through the kernel reader (12 ms) + the (external stlib)/systemf/preclude-all-but block — outside any file, so no per-file cache could ever hold it

What this PR does

Five changes, each replacing a recomputation with a cache of the state it produces.

1. Kernel type signatures in the bytecode cache (SHENKC2 → SHENKC3). types.kl's trailing declare block is hoisted out of the concatenated chunk so boot.lua runs it, and one dumped prolog abstraction per signature goes in the cache. The variance check is a static check of the kernel's own signatures against sources the cache key already covers; the abstraction is deterministic. Only ever a contiguous trailing run is hoisted, so hoisting cannot reorder a file's effects.

2. "dv" fasl record replaces "dt". shen.*datatypes* / shen.*alldatatypes* entries are all (TypeName . (fn TypeName)) (sequent.kl shen.remember-datatype), so the record stores the names in order and rebuilds the list; the datatype's own work then replays from its recorded chunks. Whole-list rather than a delta, so (preclude-all-but []) shrinking the table replays correctly.

3. Native shen.lambda-entry and shen.assoc->. shen.lambda-entry compiled a Lua chunk to produce a curried chain over a late-bound F lookup and nothing else — built directly instead. It is installed early (its own installer, no dependency on the compiled original) so it is in place before declarations.kl builds the lambda table; reaching that point is what generalises hoist_declares into hoist_tail, which puts a file's trailing non-defun run into a separate ":init" chunk. shen.assoc-> was non-tail KL recursion over lists hundreds of entries long — native iterative prefix copy.

4. Standard-library boot image. The whole stdlib phase — install.shen's own forms and the ~20 nested loads alike — is recorded as one fasl record stream and replayed in one go: the nearest a Lua host gets to shen-cl's save-lisp-and-die, which is the actual reason shen-cl starts in 0.01 s. Nested loads splice their record streams into the enclosing recording; a fasl replay is marked in-chunk for the enclosing span, since a replay drives the very functions the recorder wraps.

Invalidation and semantics

The image is keyed on the kernel key + record format + Prolog engine + the on-disk install.shen, and it stores the path and content hash of every file the recorded span loaded, all re-verified on each boot. The per-file fasl caches are still written on a miss, so an image that cannot be serialized costs nothing beyond not existing. FASL_ROLL is folded from the same contents in the same order, so a user program's fasl key is identical either way. SHEN_STDLIB_IMAGE=off disables it; a single-file bundle opts out (it materialises the stdlib to a fresh temp dir each boot).

A cached boot is indistinguishable from an uncached one at the Shen level: same shen.*sigf* contents and order, same lambda table, same datatypes, same shen.*gensym* and (inferences), same typechecking. shen.*gensym* is lower than main's by the ~1300 names the skipped abstractions used to consume — the counter only grows within a session, so no name it can still produce was ever handed out, and cached/uncached agree exactly, which is the invariant that matters.

Results

Interleaved A/B against main @ 67e2f43, min-of-N child CPU. This box has heavy, bursty background load (load average 13-14 during these runs), which inflates every absolute number by roughly 2×; the ratios are what hold. Reference quiet-machine numbers for main reproduce issue #46's 0.154 s.

bin/shen -e '(output "hi~%")', warm, min of 40 interleaved:

	CPU	vs shen-cl
shen-cl script	0.029 s	1×
shen-lua main	0.352 s	12.3×
shen-lua this PR	0.143 s	5.0×

In-process phase split, min of 25 interleaved:

phase	main	this PR
load_kernel	0.0985	0.0394
initialise (stdlib)	0.2143	0.0746
total	0.3168	0.1185

Cold start (rm both caches first, min of 5) is unchanged: 1.89 s → 1.93 s CPU — the cold path does the same work plus writing the image.

Gates

• luajit run-kernel-tests.lua — 134 passed / 0 failed, ok
• make test — 896 pass / 0 fail across 19 specs (19 new)
• luajit examples/{openresty,openresty-authz,envoy}/selftest.lua — all OK
• bifrost python3 bifrost.py --impls shen-lua — 31 pass, 0 diverge, 0 FAIL
• build/make-bundle.lua rebuilt and exercised (kernel cache format changed)
• all gates run from a fully cold cache state

New test/boot_cache_spec.lua (19 assertions) pins cached-boot == uncached-boot across every configuration, asserts the image is genuinely hit (not vacuously skipped), asserts editing a standard-library file invalidates it (against a copy via SHEN_STDLIB_DIR, so the checkout is never mutated), and differential-tests both natives against the compiled-KL definitions compiled fresh out of klambda/ under an alias.