OSS release prep + production hardening - #1
Closed
ivarvong wants to merge 49 commits into
Closed
Conversation
…ardcoded paths
- New tooling.exs: single source of truth for the differential harnesses.
- node!(): $NODE override, else PATH node (24+), else portable auto-discovery
of a 24+ install under nvm/asdf (prefers the validated 24.x line over newer).
Fails fast with an actionable message instead of a confusing link error.
- wasmas!(): $WASM_AS / PATH / Homebrew fallback.
- wasm_as_args/3: canonical flag list (-all --disable-custom-descriptors) so
feature flags can't drift between harnesses again.
- Converted all 13 harness files (was 3 in the TODO; the hardcoded path and the
missing --disable-custom-descriptors flag were repo-wide). conformance + fuzz
now gain --disable-custom-descriptors (Node-24 compat) with no result change.
Verified: conformance 147/147, fuzz 33/33, gaps 18/20 (p01 lie + p18 stub-trap
unchanged from baseline), realistic_order 8/8. All 13 files syntax-valid.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… runaway Wasm
- tooling.exs gains Tooling.cmd/3: a System.cmd-style runner over a Port with an
absolute wall-clock deadline. On timeout it kills the OS child and returns
{out, :timeout}. stderr stays optional (driver stdout is parsed).
- All 5 node call sites in conformance/fuzz/gaps map :timeout to a TIMEOUT sentinel
that can never equal a real canonical result. Default 120s.
Verified: Tooling.cmd unit-tested (kill of a real node infinite loop, 0 survivors);
conformance 147/147, fuzz 33/33, gaps 18/20 unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New runtime/imports.mjs is the single source of truth for the host imports
(big, math, str, crypto, proc-stubs). Before this, every runner hand-rolled
its own `str` and they had already diverged: scheduler/driver had only
{upcase,downcase} while gaps/demo had re_split/re_run/titlecase/upchar — so a
module compiled against the richer surface LinkErrors under the leaner runner
(e.g. a Req/Regex module under the main scheduler).
Converted the 5 multi-module runners to the factories:
runtime/scheduler.mjs, conformance/driver.mjs, gaps/runner.mjs,
demo/runner.mjs, demo/bench.mjs
`str`/`crypto` take a () => exports getter (instance is built after the import
object). reRun/reSplit framing verified byte-identical to the prior inline
versions (gaps is the str-heavy regression check).
Verified: conformance 147/147, fuzz 33/33, gaps 18/20 (p01 lie + p18 trap as
baseline; str-heavy programs e.g. p17 still provably correct), demo runner +
bench run clean against the cached fixture (real Req + :crypto sha256, exit 0).
Scope note: 7 other .mjs still hand-roll imports (conformance/profile_wasm,
jason-demo/run, measurements/bignum, perf/{alloc,measure,scaling},
compiler/examples/runstrix). They are single-purpose — each hosts exactly one
fixed wasm with known imports — so they can't hit the multi-module drift bug;
converting them is DRY-only and left as an optional follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… raises 5.1: operand/1 STUB fallback emitted "(ref.null none)" — a usable nil that flows on as a silent wrong value (the one STUB-mode path that can turn a miscompile into a lie), and it wasn't counted so it hid as "0 stubs". Now emits "(unreachable)" and increments the STUBS meter, exactly like opcode-level stubs: trap-if-reached + visible. (Used a block comment; a ;; line comment inline would swallow the enclosing expression's closing parens.) 5.2: the two unsupported-construct raises (bs_match cmd; unhandled opcode) now include mod.name/arity and a "set STUB=1 to trap and continue" hint. Verified: conformance 147/147, fuzz 33/33, gaps 18/20 unchanged — no program relied on the silent nil, and none grew a new operand stub. Note: p01's lie is UNCHANGED, so it is a genuine miscompile of a supported op, not the operand fallback (tracked separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion order
A checkpoint bisector localized p01_text_analytics's wrong checksum (0 stubs) to
top_keys/3, which ranked bigram frequencies by count ALONE and broke ties by map
iteration order. Decomposition proved the map CONTENTS were bit-identical (an
order-independent fold matched); only iteration order differed.
Not a miscompile — a representation boundary:
- WasmGC runtime: maps are a weight-balanced BST -> key-sorted iteration.
- BEAM: <=32-key maps are term-sorted flatmaps (agree), but >32-key maps are a
16-ary HAMT keyed by the OTP-internal hash (verified vs OTP-28 erl_map.c /
erl_term_hashing.c) -> NOT key-sorted. Elixir documents map order as
unspecified and it isn't stable across OTP versions, so the runtime should
not (and does not) replicate it.
That's why the unigram map (<=27 keys -> flatmap -> sorted) matched but the
bigram map (>32 -> HAMT) diverged.
Fix: make top_keys total-ordered ({-c, k}), like its own sibling `ranked` sort
which always agreed. p01 is now provably correct (8/8); gaps is 19/20 (only p18's
non-byte-aligned bitstring stub remains — an honest trap, 0 lies).
Also: corrected the stale FINDINGS note about `exact` heap types / p16 — resolved
by --disable-custom-descriptors, now canonical in tooling.exs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…imers
4.1 Kill-by-unwind + cleanup (the documented spike-B leak):
- Park points (recv_wait, sched.yield, recv_wait_timeout) now stash reject as
well as resolve. finish() unwinds a dying process's PARKED stack by rejecting
it with ProcKill — surfaces as a non-$exc exception, so compiled try/rescue
(catch $exc only) can't trap it, like BEAM's exit(pid,:kill). Link-cascade to
parked linkers now unwinds instead of abandoning the engine-rooted stack.
- finish() also deletes the dead process record and prunes spent/owned monitors,
so a long-lived scheduler doesn't accumulate {mailbox,links,dict} objects.
- Global unhandledRejection net.
4.2 erlang:exit/2 (Process.exit/2): new compiler handler + proc.exit2 import.
Non-trapping target dies (unwound if parked); trapping target gets
{:EXIT,from,reason}; :normal to another process is a no-op.
4.3 Fairness: replaced strict toStart>toResume>toReady priority with ONE FIFO run
queue, so a spawn-heavy process can't starve resumes/readies (the roadmap's
"biggest unbuilt piece").
4.4 receive ... after: finite literal timeouts now honored — recv_wait_timeout +
a scheduler timer + idle-loop change (a timer-blocked process keeps the loop
alive and yields to a macrotask so the timer fires). after :infinity (->:wait)
and after 0 (-> fall-through) already worked. Variable timeouts still block
(documented limitation).
Tests: conformance +8 cases across new `kill` and `recv-after` categories incl.
kill_after_park (the real unwind path: a parked child killed by a separate
process) -> 155/155 bit-exact. New runtime/kill_memory_test.exs drives the real
scheduler under --expose-gc: +9900 spawned-then-killed parkers add 0.03 MB and
leave 1 live proc (vs ~50MB if stacks/records leaked). fuzz 33/33, gaps 19/20.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
6.2 Floats were supported (f64 register file + :math via libm; export wrapper already unboxes a :float return to f64) but UNTESTED in conformance — the canonicalizer had no float case. Added canon(:float) on both sides comparing the exact IEEE-754 bit pattern (Erlang <<v::float-64>> hex == JS DataView.setFloat64), so a 1-ULP error can't hide behind decimal rounding. New `floats` category: :math.sqrt/sin/cos/tan/ pow/log/atan2, int->float coercion, and the haversine LAX->JFK case. 6/6 bit-exact. 6.3 Fuzz bisection reported only the op COUNT at the first divergence. Added Ledger.op_name/2 (replays the PRNG exactly as loop/6) and the harness now names the op(s) around the divergence (e.g. "#4=open"), making a future finding actionable. op_name/advance are unreachable from run/2 so DCE drops them from the fuzz wasm. Verified: conformance 161/161, fuzz 33/33; op_name returns correct deterministic ops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5.4: int_literal/1 now guards on @i31_lo/@i31_hi instead of repeating the literal bounds (provably equivalent: `< 1_073_741_824` == `<= @i31_hi`). Verified: arith 11/11, bignum 11/11. 5.3 (module split) + 5.5 (call-variant dedup): deliberately deferred, with a precise plan in compiler/REFACTOR_PLAN.md. Rationale: it's pure maintainability on a correct, exhaustively-tested compiler (161 conformance + 33 fuzz + 19/20 gaps), and the "self-contained" builtins function actually reaches 4 sibling helpers + 8 pdict keys, so a clean split needs a shared-helper module or circular deps — a real restructure best done with the byte-identical-WAT gate under dedicated focus, not rushed as a tail task. The plan stages it (Common -> Builtins -> Emit -> thin entry) and lists the threading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo's stated purpose is a package a skeptical reviewer can trust, but the docs trailed the code badly. Corrected the false/stale claims and added the missing evidence: - ARCHITECTURE.md: maps are a weight-balanced BST (O(log n)), not a flat O(n) array, with the key-sorted vs BEAM-HAMT iteration-order delta documented; exact bignums are the DEFAULT (not opt-in) and bignum equality is done; compiler is ~3,800 lines (not "~320") with the split staged; §8 scheduler and §9 termination rewritten to reflect the real run queue + kill-by-unwind + finite receive-after (implemented, memory-verified), not "modeled". - README.md: status table gains the differential rows (conformance 161/161, fuzz 33/33 + gaps 19/20, the real runtime, real hex libs); "Read in this order" now includes conformance/runtime/fuzz/gaps/perf/interp/jason-demo/demo/durable-*. - compiler/README.md: maps O(log n); arithmetic exact-by-default. - ROADMAP.md: process plumbing + kill-by-unwind marked done; BIF/term-order status corrected. - BUILD.md: canonical wasm-as flags; portable toolchain discovery (tooling.exs, NODE/WASM_AS, auto-discovery); verified OTP 27 / Elixir 1.17. - TODO.txt: added an executed-status header. Final state verified green: conformance 161/161, fuzz 33/33, gaps 19/20, memory PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…) + thin shim
The compiler was one 3,740-line .exs script with a top-level run call. It's now a
small Elixir library, addressing "it should be a module, not a script":
compiler/beam2wasm.exs 11 thin CLI shim (require lib/, call run/1) — invocation unchanged
compiler/lib/codegen_common.ex 55 Codegen.Common: shared leaf helpers (term_eq, sanitize, bin_literal)
compiler/lib/codegen_runtime.ex 1481 Codegen.Runtime: the hand-written WAT runtime library
compiler/lib/beam2wasm.ex 2219 Beam2Wasm: run/1 orchestration + emit path (imports the above)
Method: AST call-graph analysis (heredocs are opaque to it, unlike grep) showed the
runtime library's ONLY external deps are {term_eq, sanitize, bin_literal} -> those
are Codegen.Common; everything else stayed. Used `import` so every call site is
byte-for-byte unchanged.
Safety gate — PROVABLY behavior-preserving: all 109 generated .wat files across
conformance/gaps/fuzz are byte-identical to the pre-split baseline (the compiler
emits literally the same output). Plus conformance 161/161, fuzz 33/33, gaps 19/20,
all unchanged.
Remaining (deferred, see REFACTOR_PLAN.md): split Codegen.Emit out of beam2wasm.ex;
the 5.5 call-variant dedup; full Mix packaging (mix wasm.compile). The harness
invocation (`elixir beam2wasm.exs <beams>`) is untouched, so no harness churn.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…odules Extracted the per-function emit path (compile_fun/2 + the opcode case + value/operand renderers + arith/comparison/bitstring/map/const helpers + instruction preprocessing) into Codegen.Emit, leaving Beam2Wasm as pure run/1 orchestration. fq + type_test_i32 joined Codegen.Common (shared with run-orchestration). beam2wasm.exs 12 thin CLI shim lib/codegen_common.ex 74 shared leaf helpers lib/codegen_runtime.ex 1481 WAT runtime library lib/codegen_emit.ex 1123 emit path lib/beam2wasm.ex 1085 orchestration (was 2219) From one 3,740-line script -> four modules whose largest is the runtime library (mostly static WAT data). Used `import`, so call sites are unchanged. PROVABLY behavior-preserving: all 109 generated .wat byte-identical to the pre-split baseline; conformance 161/161, fuzz 33/33, gaps 19/20. Process note: AST call-graph analysis drove the partition, but its name regex initially missed `?`/`!` functions (fits_i31?, int_typed?) — caught immediately by the standalone compile + byte-diff gates, then fixed. Clause grouping in the splitter keeps multi-clause fns (operand, materialize) contiguous. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, bit-exact
A real-world content pipeline: real unmodified Jason 1.4.5 decodes a JSON article,
a markdown->HTML renderer + template produce an HTML page. The harness renders on
WasmGC AND the VM and asserts byte-identical HTML, then benches.
Result: 3/3 pages byte-identical, ~36,700 renders/sec, 218 KB wasm, 0.09ms
instantiate. Headline: **Jason DECODE runs on WasmGC** (real lib, real JSON ->
Elixir maps, bit-exact) — previously only encode was shown.
Findings this demo surfaced:
- Earmark (real markdown lib) COMPILES (152k WAT, 28 stubs) but traps at runtime on
Kernel.struct/1 (dynamic struct-from-module-atom via __struct__ dispatch) in
Earmark.Options.make_options. Documented as a next target; render is hand-written.
- Jason number decode needs :erlang.binary_to_integer (+ float parse) — unsupported;
the sample articles are string/array/object only. The renderer likewise avoids
Integer.parse via binary pattern matching.
Compiler fix (verified suite-safe — conformance 161/161, fuzz 33/33, gaps 19/20):
- erlang:exit/1 outside proc mode referenced the proc-only $exit_raw import and
failed to assemble; now it traps (no process to unwind on a non-proc error path).
Proc-mode exit/1 is unchanged.
app/ is a mix project pinning {:jason, "~> 1.4"} (mix.lock committed; deps/_build
gitignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…'t work around) The existing binary_to_integer/1 accumulated in an i32 (overflowed, truncated to i31) and /2 (base-N) was missing entirely (traps). Replaced with a proper implementation: accumulates acc = acc*base + digit through the tiered int helpers ($int_mul/$int_add/$int_sub on term values), so results promote i31->i64->$big instead of truncating; digits 0-9/A-Z/a-z; base from the /2 arg (/1 = base 10); empty/invalid-digit traps (badarg). This unblocks Integer.parse/1 and JSON *number* decode in Jason — both route through binary_to_integer. Verified bit-exact (Integer.parse + 20-digit bignum + base-16 + negative all match the VM), and suite-safe: conformance 161/161, fuzz 33/33, gaps 19/20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…the Regex API
Per the mission (all pure Elixir must run): extending the source-extraction + host
JS RegExp shim (the proven approach for Regex.run/2 + split/3) to more of the Regex
surface that real libraries need.
- Regex.run/3 with `return: :index`: new re_run_index host shim returns byte-accurate
match positions; the WAT wrapper builds the [{offset, length}, ...] list ({-1,0} for
a non-participating group, like :re), and delegates non-index opts to run/2.
- Regex.replace/3 (global string replace): new re_replace host shim, translating Elixir
replacement syntax (\\N backrefs, \\0 whole-match) to JS ($N, $&); WAT wrapper extracts
the source and calls it.
- :return/:index added to the forced atom table (the run/3 shim references them).
Verified bit-exact vs the VM in isolation (run/3: single/multi-capture/nomatch;
replace/3: literal patterns) and suite-safe: conformance 161/161, fuzz 33/33, gaps
19/20. These unblock real Earmark several layers deeper into its parser; the live
frontier is Regex.replace with a FUNCTION replacement + an upstream type divergence
in LineScanner (next).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… bug inventory
Per the mission ("any non-native code should work; it's a bug if it doesn't"):
a rigorous, evidence-grounded taxonomy.
§1 TRUE LIMITS (the only legitimate "can't"s): NIF/BIF host-boundary shim
existence + semantic fidelity (PCRE edges, Ryu float formatting, unicode
tables — each with its endgame path); platform effects (no fs/OS, fetch-only,
128MB/10MB caps); runtime code creation (answered by the interp tier);
scale/scheduling capacity.
§2 DELIBERATE DELTAS: >32-key map order, empty stacktraces, closed-world apply,
no live introspection.
§3 THE BUG INVENTORY: ~190 functions enumerated from the Earmark probe + gap
corpus, classified by work kind ([beams]/[builtin]/[compiler]/[exact]/[proc]/
[interp]) — exceptions/error machinery, term primitives (phash2, cmp_term,
int/float conversions), dynamic atoms, Regex completion, pure-stdlib feeding
(:array/Calendar/Stream/leex output/...), proc-mode completion, host-effect
shims, + the live LineScanner divergence.
§4 HOW WE HOLD THE BAR: differential harnesses, the never-lie stub meter, and
the planned stdlib API scoreboard (every public function diffed vs the VM —
limitations as a shrinking measured number).
CLAUDE.md + README now point at it as the canonical line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a valid backing Doctrine correction (user directive): File/network/OS effects are NOT a platform "can't" — they lower to host imports at the same boundary as NIFs, and the HOST decides the backing: real fs/sockets on a Node host; a virtual filesystem (in-memory or KV/R2/DO-backed) + fetch on Workers. File.read/1 works wherever the host wires it; an unwired effect traps honestly, never lies. - §1.2 reframed from "no filesystem, File.read will never work" to the host-effects boundary model (the same model the Req demo already uses for http.get/:crypto). True limits shrink to: what the platform can physically provide (no raw sockets on Workers, no ports) + the hard caps (128MB isolate / 10MB module). - §3 inventory: "host-effect shims" expanded into the effects ABI work item (File.* -> fs_read/fs_write/fs_stat imports with per-host backings; IO.* -> console/stream imports). - CLAUDE.md + README aligned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…x surface
Per the mission (any pure Elixir runs; build, don't work around):
ERROR/RAISE (topped the gap-corpus frequency list — every program's error paths):
- erlang:error/1,2,3, throw/1, nif_error/1 builtins, 3-mode gated like raise/3
(exc -> throw $exc with the right class; proc-only -> crash the process; else trap).
- maps:update/3 builtin ({:badkey, K} error) — what Kernel.struct!/2 validates with;
found because feeding the real :maps beam gives nif_error BIF bodies, and the new
nif_error correctly raised where it used to trap-in-place.
- Real `raise ArgumentError, msg` / `raise "msg"` / rescue-in / catch class+term all
work with REAL exception structs (feed Kernel+Exception+ArgumentError+... beams).
TERM PRIMITIVES:
- erts_internal:cmp_term/2 -> $term_compare wire.
- integer_to_binary/2 + integer_to_list/1,2: base-N (uppercase, like Erlang),
bignum-safe via $int_div/$int_rem tiers; integer_to_binary/1 keeps its i31 fast
path and delegates non-i31 (was a silent-overflow class). list_to_integer/1,2
now delegates to the same single parser. binary_to_integer + the new parsers are
mode-gated (BIGNUM=0 gets wrapping-i32 fallbacks).
- make_ref/0 as a real builtin (covers apply/capture/tail-call forms).
REGEX — the full surface, host-shimmed (NIF boundary per LIMITATIONS):
- match?/2, scan/2 (group lists, "" for non-participating), escape/1 (Elixir's exact
set), split/2, replace/4 (global: false), compile/1 + compile!/1,2 (runtime regexes:
the "compiled" form IS source+opts under the shim model).
- replace/3,4 with FUNCTION replacements: host drives JS replace, calling back into
the exported $re_fun_call which arity-dispatches the closure ($clos1/$clos2 via
ref.test on the funcref) — fn(match) and fn(match, cap1) both bit-exact.
- PCRE->JS translation layer (pcre2js): Elixir 1.13+ opts-as-atom-list canonicalized
WAT-side ($regex_opts) to flag chars; x-mode whitespace/#-comment stripping outside
char classes; (?'name' -> (?<name>; \A \z \Z \h \R; \# and escaped-space; (?| branch
reset -> (?: (exact when the first alternative participates — documented edge);
:unicode deliberately NOT mapped to JS u (PCRE default is byte-mode).
- All shims take (source, opts); shared $regex_src/$regex_opts accessors.
Conformance grew 161 -> 185/185 bit-exact (new categories: raise 6, term-prims 7,
regex 11/11 incl. fn-replacements + runtime compile!). Fuzz 33/33. Gaps 19/20.
Earmark now clears JSON decode, options/struct building, and the ENTIRE LineScanner
(every line-typing regex); the live frontier moved to Parser._parse/4 (a map-op
divergence under investigation).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tual fs)
Per the user directive (IO is a host effect; a virtual filesystem is a valid
backing), File and console IO now lower to host imports at the same boundary as
NIFs:
- Host (runtime/imports.mjs): makeFs(getExports, backing) with TWO backings —
nodeFsBacking (real filesystem) and memFsBacking (in-memory virtual fs, the
Workers/test story); makeIo (console, with an optional capture sink). Frames:
read -> <<1, bytes>> | <<0, errcode>>; write -> errcode (1 enoent/2 eacces/3 eio).
- Compiler: :file.read_file/1 + :file.write_file/2,3 and IO.puts/1,2 + IO.warn/1
as BUILTINS (so they beat fed File/IO beam bodies — the builtin-overrides-beam
rule), bodies gated on fs/io detection so ungated builds keep honest traps; posix
errcodes map to real :enoent/:eacces/:eio atoms. Feed the REAL File beam on top
and File.read/write/read! are pure Elixir over the two BIF shims.
- All six runners provide fs (virtual, in-memory) + io (console) defaults — fed
stdlib beams (Kernel/Exception/OTP) reference IO.warn internally, so the imports
must exist everywhere (caught by the suite: feeding beams flipped io? on and
LinkErrored runners that lacked the module).
New demo/effects: the SAME compiled program does File.read -> transform ->
File.write -> IO.puts -> missing-file {:error, :enoent} handling, run on the VM
(REAL fs, captured stdout) and on Wasm (VIRTUAL in-memory fs, captured console):
return value, WRITTEN FILE BYTES, and printed lines all identical.
Suite: conformance 185/185, fuzz 33/33, gaps 19/20.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ons bit-exact (91%) The measurement instrument from LIMITATIONS §4: for EVERY public function of the core stdlib modules (via __info__(:functions)), generate a concrete call from a typed input pool (VM-validated; nondeterministic fns like shuffle/random classified out), compile a wrapper alongside the REAL stdlib beams, run on Wasm AND the VM, diff identical checksums (maps folded in sorted-key order so the documented >32-key iteration delta can't bite). Writes SCOREBOARD.md. Enum 72/77 List 28/32 Map 24/27 Keyword 32/34 Tuple 6/6 Integer 17/17 String 62/66 Range 8/8 MapSet 18/20 Float 5/12 TOTAL 272/299 bit-exact (91.0%) · 93 nogen (input-pool gaps) · 392 public fns The 27 failures cluster EXACTLY into the LIMITATIONS §3 inventory — the scoreboard independently re-derives the gap list: Float/Ryu formatting (7: to_string, to_charlist, ceil/2, floor/2, round/2, ratio), Stream.Reducers (chunk_*, splitter), dynamic atoms (to_existing_atom x2), Map.pop family (5), MapSet.filter/reject (v2 set internals), String.equivalent? (unicode normalize). Also fixed en route (caught by the scoreboard — it's already earning its keep): - atoms_json: control-character atoms (stdlib has :"\n") now \u-escaped — the @Atoms table was emitting invalid JSON and breaking the driver wholesale. - conformance/driver.mjs provides benign proc/sched stubs (fed Kernel flips proc mode on without running processes) — same default-everything rule as fs/io. Suite: conformance 185/185, fuzz 33/33, gaps 19/20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…99 (91%) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…codegen bug fixed
THE HEADLINE: demo/markdown now renders 3/3 pages BYTE-IDENTICAL vs the VM through
the complete REAL-dependency pipeline — unmodified Jason 1.4.5 (decode) -> unmodified
Earmark 1.4.49 (full markdown->HTML: LineScanner, Parser, AST renderer, the
leex-generated link lexers, OTP's erl_scan/io_lib) -> template. 166 beams, 2.9MB
wasm, 890 renders/sec.
THE REAL COMPILER BUG (found by differential staging into Earmark's _parse/4):
- get_map_elements is ONE instruction in BEAM — every key fetched from the ORIGINAL
map even when a destination register aliases the source (Earmark's _parse does
dst1 == src). Our sequential lowering re-read the clobbered register for key 2+.
Fixed by stashing the map in $tmp first. Could corrupt ANY multi-key map pattern.
- element/2 in GUARD context (real fail label) now guard-fails on non-tuple/range
instead of cast-trapping (:lists.keydelete/keytake walk lists of non-tuples).
BURN-DOWN (all bit-exact in isolation + suite-safe):
- maps:take/2 (Map.pop family), erts_internal:mc_iterator/mc_refill (the OTP-27 map
cursor; sets/MapSet.filter+reject), unicode NF* normalize host shims
(String.equivalent?), characters_to_list charlist passthrough (List.to_charlist).
- binary_to_existing_atom/1,2 + list_to_existing_atom/1 via the closed-world
atom-names table (badarg if not an existing atom).
- Erlang float formatting: flt_fmt host shim — :short (empirically-derived exact
rule: plain iff -3 <= dp <= 15 and dp-len <= 2; Ryu digits are unique so JS
provides them), default 20-digit scientific, {:decimals,D} + :compact.
float_to_binary/1,2 + float_to_list/1,2 shims (Float.to_string works).
- Sub-byte bitstrings: $bits_read64/$bits_write/$term_i64; bs_match :integer >30
bits narrows through the exact tier; :float (f64) match command; bit-mode
bs_create_bin (<<s::1, e::11, m::52>> construct+match round-trips bit-exact;
Float.ratio works).
- apply_N dispatch miss now raises {:undef, Mod, Fun} (was a bare unreachable) and
the $exc tag is exported — escaped exceptions are host-decodable via the existing
term-introspection exports. This diagnosed the List.Chars.BitString miss in
minutes.
- epp:default_encoding, io:printable_range, iolist_size builtins; pcre2js gains
atomic-group (?> translation (exact for single-token groups).
SCOREBOARD: 296/299 bit-exact (99.0%) — Enum/List/Map/Keyword/Tuple/Integer/String/
Range/MapSet ALL PERFECT. Generation made deterministic (10s probe timeout) and
map-order-unspecified results (Map.keys/values/to_list, Keyword.new, Enum.unzip)
sort-normalized per LIMITATIONS §2. Remaining 3: Float.ceil/floor/round w/precision
(bit-length-carrying binary values — the one remaining representation class, shared
with gaps p18).
Suite: conformance 185/185, fuzz 33/33, gaps 19/20, effects 3/3 identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…NS inventory burned down to one class Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The final representation class, built: sub-byte bitstrings as first-class values. - New $bitstr type (bytes MSB-padded + bit length), DISTINCT from $binary — so is_binary(<<1::4>>) is correctly false and every byte-aligned code path is untouched. $mctx gains an end-bit field (all remaining-bits checks use it; a context can now be backed by a bitstring). - Match side: sub-byte :binary extraction via $bits_extract (byte fast-path kept when statically byte-sized AND runtime-aligned); get_tail returns $bitstr when unaligned; f64 float-segment match; bit_size works on binary/$bitstr/CONTEXT (BEAM optimizes rest::bitstring + bit_size into context-remaining!). - Construction: bit-mode create_bin generalized — dynamic-size integer segments (0::size(8-r)), binary/append/bitstring-append segments from $binary OR $bitstr sources, string literals, runtime two-pass bit totals; result is $bitstr when unaligned else $binary. Bitstring LITERALS (<<5::3>>) materialize properly. - Endianness: id::little-16 style segments honored BOTH directions (a new $bits_read64_le + flag-aware byte blits); byte-mode integer blits now go through $term_i64 (an i31 cast trapped on $i64-tier values like 32-bit payloads). Relics deleted (the pre-mission workaround pattern): - float_builtins: a precision-0-ONLY Float.round/ceil/floor shim + a DCE bypass that deliberately refused to follow Float's real IEEE machinery. The real Elixir Float code (decompose over 52-bit bitstrings) now runs — round/ceil/ floor with precision are bit-exact. - A silent-nil $erlang.integer_to_list_1 builtin (returned null!) and the materialize/1 STUB fallback that emitted a usable nil for unsupported literals (now traps + counts, like the operand-fallback fix). FINAL STATE — everything green: gaps 20/20 PROVABLY CORRECT (p18 first-ever pass; 0 stubs anywhere) scoreboard 299/299 bit-exact (100.0%) — all ten modules perfect conformance 185/185 · fuzz 33/33 · Earmark 3/3 byte-identical (886 renders/s) effects 3/3 identical · kill-memory PASS Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New demo/markdown/bench_vs_js.exs: a realistic 3.7KB document through real Earmark on WasmGC, Earmark on the native BEAM, marked, and markdown-it — byte-identical gate vs the VM before any timing. Result: 3.1x of native BEAM (consistent with the perf suite); the gap to marked is ~28x library design (AST parser vs line regexes), not Wasm overhead. Cold start 17.1ms total (compile 1.9 + instantiate 1.5 + first render 13.5). Built on first run of the new doc (gaps, not workarounds): - binary:split with a LIST of patterns (leftmost, longest at equal pos) + :trim/:trim_all — exactly what String.split/1 whitespace uses. New $bin_find_any builtin; bsplit rewritten; conformance "bin-split" category added (13 cases) -> 198/198. - host regex shim compiled per CALL (new RegExp + PCRE->JS translation every time); now cached with lastIndex reset. -22%/render; the 3-page demo went 899 -> 1,224 renders/sec. Also: perf/measure.mjs + perf/alloc.mjs now default-provide proc/sched/ fs/io imports (decimal-portfolio bench silently read 0.0 without io). Verified: conformance 198/198, fuzz 33/33, gaps 20/20 provably correct, scoreboard 299/299 (100.0%), markdown 3/3 byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rify.exs New instruments (the suite the suite was missing): - genfuzz/: GENERATIVE program fuzzer — seeded random Elixir over the full term algebra (bignum-boundary constants, floats, binaries, maps, closures, try/rescue, bounded recursion), diffed vs the VM. 12 default + 40-sweep verified; every program saved for instant repro. - regexdiff/: the :re shim corpus — 76 cases (patterns x subjects x the full Regex API), every divergence CLASSIFIED: 68 exact, 4 documented deltas, 4 honest refusals, 0 lies. - verify.exs: the one-command manifest with pinned floors; exit 1 on any drop. 8/8 in 142s. Found and FIXED at the root (each by a new instrument on its first runs): 1. binary:split list-patterns + :trim_all (String.split/1) — built earlier today, locked by conformance bin-split. 2. trunc/float->int beyond 2^63 hard-trapped — new $f64_to_int covers all tiers via big.from_float (trunc(Float.max_finite()) = exact bignum). 3. UNconsolidated protocol beams fell to runtime Module.concat — the compiler now CONSOLIDATES PROTOCOLS ITSELF against the fed impls (closed-world default; mix-consolidated beams pass through untouched). 4. Constant-hoisting emitted >i64 bignums into global initializers (host calls aren't constant exprs) — const_exprable? gates hoisting; also fixed Enum-on-struct crash in the checker (Protocol.UndefinedError made every struct literal a stub). 5. div/rem by zero was an uncatchable Wasm trap — now raises catchable :badarith (rescue ArithmeticError works) via a canonical-zero guard. 6. map_get in GUARD context (rescue class tests!) cast-trapped on non-maps — guard form now jumps to the fail label. 7. Regex shim lies: PCRE $/\Z match before a final newline (JS doesn't) — translated; JS .split() injects captures + drops edge empties — replaced with an exec loop; \K/\G silently matched literal K/G — now refused. Plus: Regex.named_captures/2 built (re_named host call + WAT shim); split parts:/include_captures: forwarded to the host. 8. Scoreboard: 91-entry @special call table (specialized fun shapes, 4-ary combos, option atoms) + nogen names printed + huge-float checksum fix -> 389/389 (100%) of 392 public fns; only @nondet (3) unmeasured. Stack cliff (LIMITATIONS 1.4): measured precisely (~10^4 frames default; JSPI doesn't help). runtime/deepstack.mjs ships the Node mitigation — 5,000,000 frames verified in a 256MB-stack worker. workerd stays platform- fixed; compiler trampolining is the roadmap fix. Workers prod-prep: demo/markdown/worker/ — worker.mjs + config.capnp + wrangler.toml + smoke.exs. Local workerd gate: 4/4 responses BYTE-IDENTICAL to the VM over HTTP (incl. POST /render of the 3.7KB doc), p50=1.1ms. `npx wrangler deploy` ships the same three files. VERIFIED: verify.exs 8/8 ALL GREEN — conformance 198/198, fuzz 33/33, gaps 20/20, genfuzz 12/12 (+40/40 sweep), regexdiff 0 lies, scoreboard 389/389, markdown 3/3, effects byte-identical. workerd smoke 4/4. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-exact https://elixir-markdown.ivar.workers.dev (deploy 71c3b65f, 505 KiB gzipped) The full differential gate now closes over the public internet: all 4 production responses (3 article pages + POST /render of the 3.7KB doc) are BYTE-IDENTICAL to the real Elixir VM. Local workerd compute floor p50=1.0ms; production TTFB p50~88ms from a residential client (TCP+TLS to the EWR PoP dominates; p99 ~414ms shows cold isolates for the 2.9MB module). worker.mjs imports switched to relative specifiers ("./blog.wasm") so the SAME file bundles under wrangler and serves under raw workerd — re-gated locally (4/4 byte-identical, p50=1.04ms) before deploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
autocannon against https://elixir-markdown.ivar.workers.dev: - GET render: 15,000 requests (25 conns x 30s, ~500 req/s, 0 errors) p50=46ms p90=75ms p99=134ms p99.9=505ms (cold-isolate tail). The prior 414ms "p99" was sequential fresh-TLS curls, not service latency. - POST /render (3.7KB doc, ~12ms-CPU render): 2,000 requests, 0 errors, p50=152ms p99=567ms. - Byte-identical to the VM re-verified after the 17k requests. - Client network floor ~25ms RTT to EWR; local workerd compute floor p50=1.0ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ects
Track 1 — packaging (compiler is now a Mix package):
- compiler/mix.exs: add {:beam2wasm, path: ..., runtime: false} to any Mix
project and `mix wasm.build --module M --export "f:int->bin" --worker`
produces a deployable module + complete Cloudflare scaffold (worker.mjs
JSON-args dispatcher, imports.mjs via the priv->runtime symlink,
wrangler.toml, config.capnp for local workerd).
- Honest two-class report: unsupported CONSTRUCTS (compiler's own counted
stubs; 0 = provably supported, --strict enforces) vs called-but-not-fed
EXTERNALS (named traps if reached; real libraries carry some on cold
paths the apply-analysis keeps).
- Excludes its own beams from the sweep (it compiled itself on the first
run). Default stdlib surface = the exact proven markdown set.
- GATE: the markdown app built through the task is byte-identical to the
VM on all 4 render checks, and the generated scaffold serves correctly
on local workerd (health + both exports).
- Found+removed demo/markdown/app/lib/probe.ex (stale Earmark-bisection
debug module the curated builds never fed — the task's sweep honestly
surfaced its 158 cold externals).
Track 2 — the product thesis in production:
- durable-genserver/ deployed: https://elixir-durable-bank.ivar.workers.dev
(wrangler.toml w/ SQLite-backed DO class; works on every plan).
- Verified live: init 100 -> deposit 50 -> withdraw 30 -> withdraw 999
rejected by the COMPILED `when amt <= balance` guard (:insufficient),
per-actor isolation, and state SURVIVED 7 minutes idle (balance=120,
events=6). Real OTP callback logic in a real Durable Object, durable.
Suite-safe: verify.exs 8/8 ALL GREEN (142s).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…wn DB The database as a host effect, exactly the NIF model: Sqlite.query!/2 -> :sql_host.exec/2 -> a gated `sql` host import (fs-pattern: detection, import, builtin shim, extra_defined). Params and rows ride REAL Jason both directions, so a row is a plain Elixir map. Backings (runtime/imports.mjs): makeSql + nodeSqliteBacking (node:sqlite DatabaseSync; .all() drives DDL/DML/SELECT uniformly) + doSqliteBacking (the DO's synchronous ctx.storage.sql). demo/durable-sql/: - app/lib/sql_ledger.ex: Sqlite client + a ledger (CREATE TABLE, INSERT ... RETURNING id with bound params, GROUP BY/ORDER BY aggregates, and an Elixir-side Enum fold that must agree with SQL SUM). - run.exs differential gate: same seed-driven session on the BEAM (via a node:sqlite line-server Port — the oracle hits the IDENTICAL engine) and on WasmGC. 6/6 sessions BYTE-IDENTICAL. - worker/: LedgerDO instantiates the module per actor, binding the `sql` import to THAT actor's SQLite. Verified on local workerd: inserts, balance, report (sql/elixir-fold agree), per-actor isolation. wrangler.toml ready (new_sqlite_classes); deploy pending go-ahead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
https://elixir-sqlite-ledger.ivar.workers.dev - SqlLedger gains the full CRUD surface (list/get/update/delete, all via SQL with RETURNING so row state comes from the database) — and the differential session now exercises it: 12 inserts + 3 updates + 2 deletes + a read-back, 6/6 sessions BYTE-IDENTICAL vs the BEAM on the identical engine. - worker.mjs: GET / serves the webapp (vanilla HTML/JS, inline edit, per-actor switch); POST /api?actor=X forwards the JSON op to the DO; curl-friendly GET form kept. - Verified in production: create/update/list/delete with not-found handling; per-actor isolation; and an actor's rows written before the redeploy were intact after it (durable across deploys). - Gotcha: stale local workerd from a prior smoke held the port and served the old worker — pkill before re-serving. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ntical ivarvong/pyex (a Python 3 interpreter in pure Elixir, 531 beams / 14,083 functions — the largest program this compiler has eaten) builds via `mix wasm.build` and runs real Python on WasmGC. demo/pyex battery: 16/16 programs with BYTE-IDENTICAL transcripts (print output + repr + error text) vs pyex-on-the-BEAM — recursion, comprehensions, classes, custom exceptions, closures, f-strings, slicing, floats (banker's round() via Decimal), 2**100 bignums. Warm evals 3-20ms; 15.1MB raw, 2.23MB gzipped (Workers-deployable). Compiler hardening it forced (all general, all suite-safe): - V8 array.new_fixed 10k cap, twice: atom-names tables >10k entries build in a (start) function; >10k-byte binary literals emit DATA SEGMENTS + array.new_data (size-gated out of constant expressions, where array.new_data is illegal). - :persistent_term as a mutable-global assoc table (real storage). - erlang.--/2 (list difference), binary_to_list/1, list_to_binary/1. - System.monotonic_time/0,1 + convert_time_unit/3 (+ erlang form) on the deterministic $monotime counter; time-unit atoms forced. - binary_to_float/1 + list_to_float/1 via host bin_to_float (both engines are correctly-rounded decimal->double; bad syntax throws). - BUG: base-N binary_to_integer/2 rejected leading '+' (Erlang accepts it) — surfaced by Python round() -> Decimal exponent parse '+2'. Known delta: pyex requires Elixir >=1.19 (machine runs 1.17), so one interpreter path (tuple-assign in __init__) fails identically on BOTH engines — a version issue upstream of the compiler. Suite-safe: verify.exs 8/8 ALL GREEN (146s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
https://elixir-python.ivar.workers.dev — POST Python source, get the transcript; GET / is a playground page. The eval path is two layers of compiled Elixir: request body -> pyex (Python 3 interpreter in pure Elixir) -> BEAM bytecode -> WasmGC -> the Worker. 2.3MB gzipped upload. - PyexWasm.eval error path now reads Pyex.Error struct fields directly (kind + message) instead of Exception.message/1 (whose Inspect machinery isn't in the closed world) — invalid Python returns a classified transcript ("error(python)\nNameError: ... (line 1)") on BOTH engines; battery re-verified 16/16 transcript-identical. - worker.mjs: 64KB source cap, x-eval-ms header, honest capability stubs (crypto/http trap cleanly), platform CPU cap as the runaway backstop. - Production-verified: fib battery, 2**100 f-strings, dict CRUD, except-handling, banker's round() — all correct over the public internet, sub-ms server eval after warmup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…th fixes demo/pyex/bench.exs: seeded random CONUS airport pairs (42-airport table), the same Python haversine run on pyex-on-BEAM locally and on the deployed elixir-python worker, with lat/lon/codes passed as QUERY PARAMS the worker injects as Python variables (numeric-safe prelude; identifier-filtered, 32-param/256-char caps). RESULT (N=40, SEED=1): 37/40 transcripts byte-identical, 3/40 within 1 ulp (Apple libm vs V8 transcendentals — the documented :math fidelity boundary, LIMITATIONS §1.1), 0 divergent. Local in-process eval p50=0.17ms; production server-side eval sub-ms (x-eval-ms p50=0); HTTP p50=40ms from this client (network-dominated). Compiler fixes the benchmark forced (dynamic code does float arithmetic through generic BIF paths): - UNARY minus emitted $int_sub even in float mode -> $num_sub (an interpreter's USub on -118.4085 reaches it with a $float). - abs/1 was integer-only (negative floats trapped in int_sub; positives passed by term-order luck) -> new tier-aware $num_abs builtin. worker.mjs: query-param -> Python variable injection deployed. Suite-safe: verify.exs 8/8 ALL GREEN; pyex battery 16/16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /json on elixir-python now returns structured JSON — and the JSON
is rendered at the LAST INCH: PyexWasm.eval_value/1 returns
{:ok, normalized_value, stdout} | {:error, kind, msg} as a TERM, and
the host walks the live graph. No serialization inside the module.
- Compiler: the term-introspection export surface completed (flag-gated):
head_term, tier-aware is_int + int_val (externref BigInt — exact
bignums cross intact), is_float/float_val, is_map/map_kv (interleaved
kv tuple-array), atom_name (reads the atom-names table, so hosts
decode atoms WITHOUT shipping the 13k-atom JSON).
- runtime/imports.mjs: termToJs(exports, term) — the generic walker:
[]/null, true/false/nil atoms, ":atom" strings, ints (Number when
safe, exact decimal string past 2^53), floats, binaries->strings,
cons->arrays, maps->objects, tuples->arrays, "#opaque" for funs/pids.
- App: eval_value normalizes pyex shapes (tuples/sets) in compiled,
differentially-testable Elixir; wasm-safe crash path (no
Exception.message in the closed world).
- Gate: 7/7 JSON results identical to a VM-side mirror (incl. bignum-as-
string and 0.1+0.2); prod verified: dicts/sets/tuples, 2**100,
haversine-with-params, classified syntax errors. Battery 16/16;
verify.exs fast 6/6.
This is the durable-genserver tuple decode generalized into THE
boundary primitive (and what richer DO state will ride).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same Node, same machine, identical programs. pyex-wasm: 82ms to first answer (22 compile + 43 instantiate + 17 first eval) vs pyodide 994ms load. Warm compute: pyodide wins 1.9-230x (it's CPython's bytecode VM vs a tree-walker; worst on deep recursion), EXCEPT bignums where pyex's 3-tier ints + host BigInt are 1.2x faster than CPython-on-wasm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The realistic workload — 100-order JSON analytics (group/aggregate/top-N/
flag), the agent-tool-call shape — FOUND THE CLIFF IN THE WILD: 6.8KB of
data embedded in source overflows the lexer's per-char recursion on
workerd's fixed stack. The fix is better service design: POST /json now
accepts {"code":..., "data":...} — data is decoded by compiled Jason
(tail-recursive, no lexer) and injected as params.data via pyex's
custom-modules capability (PyexWasm.eval_data/2, zero pyex patches).
Caps split: code 64KB (lexer guard), data 1MB.
Measured (same machine, /tmp payloads, demo committed in vs_pyodide.mjs
style inline):
pyex-wasm 100 orders: cold=124ms warm=5.3ms
pyodide 100 orders: cold=994ms warm=1.46ms (3.6x warm, 8x cold)
pyex-wasm 1000 orders: warm=30.3ms vs pyodide 3.8ms (8x)
cpython native ref: 0.05ms/0.49ms
prod (real Workers): 100 orders p50=93ms TTFB, 1000 orders (70KB
upload) p50=196ms — network-dominated; NOTE workerd freezes Date.now
during CPU work, so x-eval-ms reads 0 in prod; server-side timing
comes from local workerd.
Break-even vs pyodide ≈ 225 warm calls per cold start: bursty/
scale-to-zero/multi-tenant favors pyex-wasm decisively; sustained heavy
compute favors pyodide.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t stack
Tail recursion modulo cons, implemented on the existing dispatch-loop
architecture: detect `call self; (test_heap)?; put_list H, x0, x0;
(deallocate)?; return` (on trim-resolved ops) and replace the window
with {:trmc_cons, H} — allocate the cons with a NULL tail hole, link it
to the chain, re-enter $dispatch at the entry block. Every return path
runs the hole-patch epilogue. Self TAIL calls become loop re-entries
(chain locals survive; also a stack win); cross-function tail calls in
TRMC functions demote to call+epilogue via a line-level pass so the
patch can never be skipped. $cons tail is now `mut` — each hole written
exactly once before the list escapes; the mutation is unobservable.
Reduction accounting preserved on re-entry paths (preemption fairness).
Gates, all green:
- NEW conformance category deep-lists (5 cases): mk(1_000_000)+sum,
map/filter/order chains at 10^5-10^6 — "a differential test at depth
10^6" was not WRITABLE before today. 203/203; verify floor raised.
- scaling: Enum.map + Enum.uniq ❌ stack-overflow -> ✓ linear to 100k.
- pyex: the 7.6KB embedded-data payload that crashed PRODUCTION now
returns correct results (redeployed + verified live); 171KB of
Python source lexes (8k statements, 1.3s).
- verify.exs 8/8; perf zero-delta (jason 3.2x, realistic 2.4x,
markdown 1,250 renders/sec).
LIMITATIONS 1.4 rewritten: remaining cliff = non-cons body recursion
only (accumulators, tree folds); modulo-op extensions are mechanical
follow-ons; full CPS not planned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Block-local i64 chain fusion: runs of integer gc_bifs compile to raw
wrapping-i64 arithmetic in shadow locals, boxing only live-outs.
Soundness is a 3-domain lattice — {:s64,bounds} proven by beam_disasm
type bounds; :u64raw congruence-class mod 2^64 (only +,*,band,bor,bxor;
must be consumed in-run); :u64 canonical after `rem 2^64` / low-mask
(then shr_u/rem_u/bxor read the bits directly). New $term_u64bits /
$narrow_u64 helpers + big.to_u64/from_u64 host bridges (all 7 big
objects). bsl gated on shift bounds <= 63 (wasm shl masks the count).
`x rem 2^64` short-circuits before divisor resolution (2^64 is the
canonicalizer, not a value). NOFUSE=1 kill switch.
ledger/500: 1935us -> 122us. 0.3x of the BEAM — the compiled code is
now 3.3x FASTER than native, honestly: BEAM heap-allocates bignums for
the same 2^64-range PRNG values; we exploit the rem-2^64 congruence to
stay in machine words. Host calls/op 205,600 -> 5,640 (36x);
big.mul/big.rem eliminated entirely.
genfuzz caught 3 real miscompiles in the first lattice draft (fresh
GENSEED=11 universe): fused-prefix planning deleted the unfused tail
ops; prefix liveness judged against the wrong successor op; unbounded
variable bsl entered the congruence domain. Fixed; gates: genfuzz
40/40 on TWO universes, fuzz 33/33 (the ledger rolling-hash diff),
verify.exs 8/8 (203/203 conformance), ledger result unchanged after
the gating.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- attic/ now holds the preserved-but-dead lineage with a README table mapping each entry to what superseded it: spikes/, cloudflare-workers/ (early DO workers), durable-object/ (-> durable-genserver), jason-demo/ (-> demo/markdown), measurements/ (-> perf/), TODO.txt (executed; see WRITEUP.md). - ~330MB of untracked node_modules deleted from atticked dirs; node_modules/ confirmed gitignored. - Doc path references repointed; README gains a directory map. - verify.exs fast: all green after the move (no live code referenced any of it — checked before moving). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…AT gated)
- Namespacing: Codegen.{Common,Runtime,Emit} -> Beam2Wasm.Codegen.*;
files under lib/beam2wasm/codegen/. One top-level name claimed.
- Real options API: Beam2Wasm.run(beam_paths, opts) with documented
:exports/:stub/:bignum/:reds/:dce/:fuse. Environment variables are now
exclusively the CLI shim's interface (beam2wasm.exs translates env ->
opts); the Mix task passes opts directly (its System.put_env removed).
- priv/ is real files (hex archives can't follow symlinks): imports.mjs,
scheduler.mjs, deepstack.mjs copied from runtime/ with an ExUnit drift
test pinning them to the source of truth.
- Package hygiene: hex metadata (description/licenses/links/files),
LICENSE (MIT), CHANGELOG.md, .formatter.exs + mix format applied,
ex_doc config (extras include LIMITATIONS.md and WRITEUP.md),
@moduledoc on the public API, @moduledoc false on internals.
- ExUnit: 15 tests, 0 failures — including pure unit tests of the TRMC
matcher and the i64 fusion planner, a REGRESSION test for the
genfuzz-caught fused-prefix-deletes-tail miscompile, the bsl shift-
bounds gate, data-segment literals, and trim resolution. The
differential suites remain the outer ring.
Gates: WAT byte-identical to pre-refactor baselines across rename +
opts conversion + format + docs (two builds, md5-matched); mix test
15/15; mix wasm.build on the markdown app unchanged (2.93MB, 34 stubs);
verify.exs 8/8 ALL GREEN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…doctor
Walked the library as a consumer would and built what was missing:
- priv/host.mjs: instantiate(src, opts) — one call from .wasm file to callable
exports, with the full default import surface and honest stubs for unwired
effects (crypto/http/sql trap with a clear message instead of misbehaving).
Returns {exports, toBin, fromBin, toJs, callBin}.
- Beam2Wasm.compile/2 -> {:ok, %Beam2Wasm.Result{wat, stubs, externals}} |
{:error, e} — the programmatic API; run/2 stays the raw string path.
- Beam2Wasm.Toolchain — wasm-as/node discovery shared by tasks and tests.
- mix wasm.verify — the same differential discipline the compiler is built on,
for the consumer's own app: seeded type-directed cases per export, VM vs wasm,
compared in a typed grammar (ints exact, floats by IEEE-754 bit pattern,
binaries base64, maps key-sorted; both-raised = agreement).
- mix wasm.doctor — toolchain checks with install hints.
- README: consumer quickstart (doctor -> build -> verify -> instantiate).
Dogfooding wasm.verify on the markdown app immediately caught a real ABI bug:
export int params were i32, so render(2564971219) silently wrapped negative in
JS and rendered the wrong page. Export int params are now f64 — exact to 2^53,
JS Numbers pass natively, no caller changes. That one divergence is exactly the
class of bug the task exists to catch.
verify.exs: 8/8 suites at or above floors (conformance 203/203, fuzz 33/33,
gaps 20/20, genfuzz 12/12, regexdiff 0 lies, scoreboard 389/389, markdown 3/3,
effects byte-identical). mix test: 15/15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion demo/rebalancer is built exactly as a library consumer would build it: ordinary Mix app (pure Elixir + real unmodified Jason), one rebalance(json)::json entry, then doctor -> wasm.build -> wasm.verify (28/28 identical vs the VM: 13 realistic portfolios incl. tier-crossing dollar values + every validation error, plus seeded garbage bins pinning the malformed-JSON path) -> workerd smoke (13/13 byte-identical over HTTP, p50 0.42ms) -> wrangler deploy -> prod curl byte-compared vs mix run (identical). Live: elixir-rebalancer.ivar.workers.dev. Dogfood finding, fixed at the root: priv/host.mjs imported node:fs at module top level, which no edge runtime resolves — a consumer's first Worker deploy would have died at module resolution. node:fs is now a lazy dynamic import used only when instantiate() is given a file path; the worker passes the WebAssembly.Module and runs host.mjs unmodified on workerd/Cloudflare. This worker is the first to consume the library surface (instantiate()) instead of hand-wiring imports. compiler mix test: 15/15. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…found
mix wasm.verify grows --gen FILE: a consumer-supplied structured case
generator (%{"export" => fn index -> args end}) for exports whose inputs
have a shape typed random args can't reach. :rand reseeds from
{seed, index} per case, so any divergence regenerates standalone; runs
are batched (25k) and streamed, so --runs 1000000 works in bounded
memory with live progress and capped divergence samples.
demo/rebalancer/app/verify/gen.exs: ~90% well-formed portfolios across
every magnitude tier, ~10% deliberate hits on each validation branch
plus malformed JSON. Final run: 1,000,000/1,000,000 identical (seed 7).
The first 2,000 structured cases caught a real bug five suites and the
typed fuzzer had all missed: the host float->string shim's notation rule
("plain iff dp >= -3", derived from fuzz that never generated tiny
floats) rendered 2.07e-4 as 0.000207. The true Erlang :short rule,
measured via a 438-point (digits, exponent) sweep + boundary probes:
pick the SHORTER of plain/scientific, plain wins ties, never plain at
or above 2^53 (9007199254740991.0 plain, 9007199254740992.0 sci).
Reimplemented; exact on 1,000,000 random-bit-pattern doubles.
Pinning the fix in conformance exposed a second root bug: float_mode?
only saw float arithmetic/:math/literals, so text<->float conversions
with no arithmetic (String.to_float |> Float.to_string) never enabled
float mode and the fltfmt?/fltparse? gates stubbed them. The conversion
BIFs now imply float mode.
conformance grows a float-format category (16 boundary cases) -> 219/219,
floor raised in verify.exs. Full manifest 8/8 ALL GREEN; compiler mix
test 15/15; rebalancer workerd smoke 13/13 byte-identical (p50 0.42ms).
NOTE: the deployed elixir-rebalancer still runs the old formatter —
redeploy with `cd demo/rebalancer/worker && npx wrangler deploy`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deployed (version c721d568) with [observability] enabled = true: retained, queryable invocation logs with per-request CPU time in the dashboard. Telemetry surfaces verified live: wrangler tail shows per-invocation cpuTime/wallTime (warm ~2-4ms, cold ~42ms = module-scope instantiate of the 2.5MB wasm); GraphQL workersInvocationsAdaptive (wrangler OAuth token works as bearer) shows cpuTimeP50 1.57ms / P99 64ms over the prod test traffic. Post-redeploy: 8/8 responses byte-identical to the VM. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… local state Make the repo safe and reproducible for a public release: - Add MIT LICENSE, SECURITY.md, CONTRIBUTING.md, CODE_OF_CONDUCT.md, .nvmrc, .tool-versions. - Untrack local runtime state (.wrangler/, **/state/, *.sqlite*) and stale *.err / compiler work scratch; .gitignore now excludes them. - De-hardcode local toolchain paths: smoke tests resolve workerd via Tooling.workerd!() ($WORKERD / PATH / node_modules), interp/build.sh and runsort.mjs no longer assume /Users/ivar; pyex demo deps point at GitHub instead of /tmp/pyex. - Document the host-effect capability model: explicit SECURITY warnings on nodeFsBacking (full-FS authority) and nodeSqliteBacking (arbitrary SQL). - Doc polish across README/ARCHITECTURE/BUILD/LIMITATIONS/WRITEUP. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the beam2wasm Hex package production-quality against Elixir community conventions, keeping all 8 differential suites bit-exact (verify.exs: 8/8). - API surface: beam2wasm.ex exposed 139 public functions that were all compiler internals. Reduce to the two genuine entry points — compile/2 and run/2 (both now @SPEC'd) — and make the other 137 defp. No external or test caller referenced any of them; verified bit-exact after the change. - Linting: add Credo (mix credo clean). .credo.exs disables the cyclomatic- complexity / ABC-size / nesting checks (inherently wrong for a hand-written BEAM-opcode code generator) with a documented rationale; all correctness and readability checks stay on. - Cleanups Credo surfaced: remove two undocumented FUSEDBG-gated IO.inspect debug traces from the optimizer; Enum.map |> join -> Enum.map_join; one-armed cond -> if. - CI: .github/workflows/ci.yml — fast gate (mix format --check + credo + test) plus the differential verify.exs gate (OTP 27 / Elixir 1.17.1 / Node 24.16 / Binaryen 130). - Hygiene: gitignore generated doc/ and cover/; CHANGELOG notes the API/quality work. Clean under mix compile --warnings-as-errors and mix docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Superseded: repo history squashed to a single clean initial commit on main for the public release. Closing this branch-based PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gets the repo ready for a public release and brings the
beam2wasmpackage up toproduction-library standards. All 8 differential suites stay bit-exact vs the Elixir
VM (
elixir verify.exs: 8/8) across every change here.1 — Release prep (
f389d39)LICENSE,SECURITY.md,CONTRIBUTING.md,CODE_OF_CONDUCT.md,.nvmrc,.tool-versions..wrangler/,**/state/,*.sqlite*) and scratch;.gitignoreexcludes them.Tooling.workerd!(); pyex demo → GitHub dep).nodeFsBacking(full-host-FS) andnodeSqliteBacking(arbitrary SQL).2 — Production hardening (
4d19cbe)Beam2Wasmwent from 139 public functions (all compiler internals) to the two real entry points —compile/2,run/2, both@spec'd. The other 137 are nowdefp(no external/test caller referenced any; verified bit-exact).mix credoclean);.credo.exsdisables the complexity/ABC/nesting checks that don't fit a hand-written code generator, with rationale; all correctness checks stay on.FUSEDBGIO.inspectdebug traces;map |> join→Enum.map_join; one-armedcond→if..github/workflows/ci.yml— fast gate (format --check+credo+mix test) and the differentialverify.exsgate (OTP 27 / Elixir 1.17.1 / Node 24.16 / Binaryen 130).mix compile --warnings-as-errorsandmix docs; generateddoc//cover/gitignored.Verification
cd compiler && mix test→ 15/15.Notes for reviewer
verifyjob on first run — its Binaryen/Node toolchain setup couldn't be validated locally. Thelint-testjob is pure-Elixir, high-confidence.github.com/ivarvong/pyexpublic for the pyex demo to build (doesn't affectverify.exs).🤖 Generated with Claude Code