diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 93457cc87..8a7d91c66 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -30,7 +30,7 @@ Plan: `.claude/plans/2026-10-08-resident-projection-fold-mask-v1.md`. Mask = adm | D-id | scope | status | gate / falsifier | |---|---|---|---| -| **D-RPF-0** | Pearl3 / Epi5 / mantissa predicate over `MaterializedEdges` via `MatchFacet16Strided`, threshold as ≤ n+1 patterns | In PR (test-pinned, `mask-risc/tests/resident_ce64_predicate.rs`) | equals CE64 accessors; wrong-bit care disagrees; unnamed fields silent; both windows stay inside the 32 B tenant | +| **D-RPF-0** | Pearl3 / Epi5 / mantissa predicate over `MaterializedEdges` via `MatchFacet16Strided`, threshold as ≤ n+1 patterns | Shipped (#1413; test-pinned, `mask-risc/tests/resident_ce64_predicate.rs`) | equals CE64 accessors; wrong-bit care disagrees; unnamed fields silent; both windows stay inside the 32 B tenant | | **D-RPF-1** | Law tables → per-recipe ternary pattern set (≤ 24 × 8), run under an admission plane (class, rail, generation, provenance) | Queued | equals `measure_declared`; dropping one code loses exactly its rows; removing the admission plane admits a v1 row | | **D-RPF-2** | Census of eligibility and read-set classes on a real population | Queued | classes/rows reported; near 1 drops D-RPF-3; no population-sized seen-set | | **D-RPF-3** | Exact read-set dedup per CE64 instruction, key incl. handle context | Queued (after D-RPF-2) | removing cohort from key changes a result; dedup-then-fold equals per-row for Count/Sum/Avg | @@ -38,6 +38,9 @@ Plan: `.claude/plans/2026-10-08-resident-projection-fold-mask-v1.md`. Mask = adm | **D-RPF-5** | Moore tenants 18..20 under a declared `SlabReading`; per-slot masks | Blocked (`SlabReading` decision) | sign-triple consumer refuses Moore lanes through the declared reading | | **D-RPF-6** | Tile skip on zero gate measured on resident-row predicates (capstone pt 3) | Queued | identical answers; dead-tile beside dead-word fraction | | **D-RPF-7** | Rank spectrum of the 256×256 palette distance LUTs | Queued (measurement only) | reported; never a fold | +| **D-RPF-CONF-0** | Confidence surface: Rust vs exact integer form, (255,255), zero-weight identity, c=255 candidates, census of live revision laws | Shipped (#1424; measured) | T1 three one-code f32 deviations; (255,255) → (0,0); 24 defs → 7 laws → 4 behaviours; G1 open (`entries/2026-10-09-d-rpf-conf-0-confidence-surface.md`) | +| **D-RPF-TABLE-1** | Cost of `NarsTables` resolutions B1..B16 vs exact revision A: surface, replay drift, decision flips, throughput | Shipped (#1426; measured); G2 follow-up: law-A load-bearing fixture + frequency-bar rationale without B1's constant (this PR) | B1 2.2–4.4× hot replay, 37.9% verdict flips; no Bn decision-stable; A wins cold; law-A fixture 210 → 64 flips, B1–B8 miss it, B16 agrees (`entries/2026-10-09-d-rpf-table-1-fast-path-cost.md`) | +| **D-RPF-P5** | NARS ↔ Beta correspondence on the CE64 surface: Jeffreys identity, unit-evidence stall, duplicate inflation, variance coarseness, 255 unreachable | In PR (tests only) | E1 exact for k = 1, k = 2 distinguishable; E2 stall at c = 244 after 20 unit obs; E2b 12-row stall table; E3 4 copies → sd 2.04× too small; E5 max revision code 254; ceil-rounding disable turns E2/E2b/E5 red (`entries/2026-10-10-d-rpf-p5-nars-beta-correspondence.md`) | | **D-RPF-8** | Boundary: mask-risc never executes CE64 instructions; no VSA on this path | Queued (doc) | review-enforced | | **D-RPF-9** | Fold-Join deforestation probe: resident ∧ resident, predicate chains gated/ungated, CE64 in place, strided pattern merge | Shipped (probe; R2/R3/G1–G3 proposed) | every arm equals the oracle; care-shift fires, non-care rewrite silent; merge precondition refuses conflicting cares | @@ -84,7 +87,11 @@ Plan: `.claude/plans/2026-10-06-global-sudoku-replayable-orchestration-v1.md`. O | **D-ALPHA-G-0** | WorldG as Alpha-route provenance, not a CE64 field | Shipped (#1372) | `cognitive-shader-driver/examples/alpha_world_provenance_probe.rs`: event = (Alpha-attended address, CE64); WorldG = `graph_of(addr)`, order = `SpogTenants` route (`merge_in_claim_order`); same CE64 bits in two worlds = two events; replay of the same claims recovers the world sequence with all-zero edges; changing CE64 fields never moves the world while the events change; fed from the tunnel's per-world shadows each `MailboxSoA` holds only its own deliveries through the unchanged `apply_edges`; an undeclared world is refused at the tunnel (`NoTenant`). No production caller of `apply_edges` exists, so "no mixing" is absence of a path; 5 tests, 5 disable runs red | | **D-WA-0** | Witness × Angle as a declared sparse local source coordinate | Shipped (#1372) | `cognitive-shader-driver/examples/witness_angle_probe.rs`: `(class, generation, w_slot, direction bits 43..45) → SourceCoord` under a probe-local sparse law (6 entries); same bits mean different sources in another class of the same world, in another world and in another generation; undeclared class, a class reading the bits as pathology triad or cohort `WitnessTable`, an undeclared generation, Witness 0 and v1/unknown provenance refuse; other CE64 fields never move the coordinate; world from the attended key, never the edge; 0 allocations; 7 tests, 8 disable runs red | | **D-CE64-TIME-0** | `CausalEdge64` as the register that survives a cycle | In PR | `cognitive-shader-driver/examples/ce64_cycle_survival_probe.rs`: one declared (`set_populated`) `MailboxSoA` `edges` row (cycle-gated `write_row` + `tick`, read back through `MailboxSoaView`) is the only cross-cycle state; folds = shipped `CausalEdge64::learn` (NARS revision of F/C) over a per-cycle `SupportLedger`; next-cycle reading = the #1370 law, moved unchanged to `examples/shared/affordance_law.rs`. Trajectory over cycles 0..3: eligible `OBS, OBS, OBS\|STRATIFY, OBS` (accumulation, certification read next cycle, contradiction read next cycle). The EpistemicState5 step (3 ↔ 7) is probe-declared: no production writer of bits 59..63 from evidence exists (`ISS-NO-EVIDENCE-WRITER-FOR-EPISTEMIC-STATE`). F1–F7 tests; 1000 folds from one source never certify; sources split across cycles never combine (missing state, pinned); an undeclared row is not a register. 10 tests, 9 disable runs red | -| **D-MOORE-NARS-0** | Recipe learning from reasoning consequences on Gomoku (`cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs`): census of the 34 recipes, 9 realized on Moore lines, structural `situation → recipe` NARS learner | In PR | persistent structural learner 0.500 → 0.535 vs best fixed recipe 0.498 and reset-every-game ~0.47; literal-board memory and immediate credit learn nothing; W-delayed one-reply credit is the spine; transfer to unseen openings 0.526 vs 0.493; "less work" not achieved (~280 folds/step); unlearning after switch not demonstrated (`entries/2026-10-08-moore-nars-0-recipe-learning-gomoku.md`) | +| **D-MOORE-NARS-0** | Recipe learning from reasoning consequences on Gomoku (`cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs`): census of the 34 recipes, 9 realized on Moore lines, structural `situation → recipe` NARS learner | Shipped (#1400) | persistent structural learner 0.500 → 0.535 vs best fixed recipe 0.498 and reset-every-game ~0.47; literal-board memory and immediate credit learn nothing; W-delayed one-reply credit is the spine; transfer to unseen openings 0.526 vs 0.493; "less work" not achieved (~280 folds/step); unlearning after switch not demonstrated (`entries/2026-10-08-moore-nars-0-recipe-learning-gomoku.md`) | +| **D-MOORE-PERT-AB-0** | Moore perturbation field A/B probe with an independent arm and statistical comparison (`examples/MoorePertubationsfeld.rs`) | Shipped (#1427; probe) | CI line; percentile fix; timings dropped from asserts; falsifiable tests | +| **D-MOORE-OBSERVABLE-FIRST-0** | Certify N-1 overload decisions before field convergence (linear flow model) | Shipped (#1428; measured) | solver-agnostic ‖r‖₁ bound + warm start certifies most decisions early; PCG/LODF beat Jacobi; bound proven for the linear flow model only (`entries/2026-10-09-d-moore-observable-first-0.md`) | +| **D-LAB-P1** | Certified argmax on Gomoku minimax: deterministic `L_best ≥ U_j` cutoff (`minimax_certified`) | In PR (probe) | same pick as `minimax` on 400 positions at k 4/4, 5/5, 8/8; folds −36.2 / −44.8 / −61.0 %; cut at `L + 1` changes a pick (`entries/2026-10-10-d-lab-p1-certified-argmax-gomoku.md`) | +| **D-SELF-CALIBRATING-LAB-0** | Hypothesis lab v0: shared paired/Holm/verdict/reliability statistics (`examples/shared/lab.rs`) + Gomoku arm (`moore_nars_gomoku_probe --lab`) | In PR (probe; Gomoku + crossword) | crossword (16 fresh puzzles): C1 Support −36× nodes SUPPORTED, C2–C4 ops FALSIFIED (Support 13×, Single 2.9×, Pair 2.2× more ops); Gomoku fresh-seed confirmation, 16 blocks: H2 learned > fixed HPM +0.024 SUPPORTED; H4 certified −29 folds/step, identical games at cost 0 SUPPORTED; H7 cost term −143 folds/step SUPPORTED (−0.026 score); H3 INCONCLUSIVE; H5 FALSIFIED; e ECE 0.05 over non-neutral outcomes (`entries/2026-10-10-d-self-calibrating-lab-0.md`) | | **D-CE64-NEXTSTATE-0** | EpistemicState5 as a sparse next-state mutation of the CE64 register | In PR | `cognitive-shader-driver/examples/ce64_nextstate_probe.rs`: N1 only bits 59..63 move, all 32 codes on an edge with every field set; N2 written in cycle k through `MailboxSoA::write_row`, read in k+1 through `MailboxSoaView`; N3 the mailbox does not double-buffer (a same-cycle read sees the write), and next-cycle authority is a read discipline: start-of-cycle copy plus `last_write_cycle == cycle` treated as pending; N4 canonical LE round trip (bits 59..63 in byte 7, bits 3..7); N5 a fresh mailbox from the persisted word continues identically; N6 the shipped `with_spare` + `with_truth` compile to `bzhi; shl; or`, identical to the masked insert (LLVM folded the two), so no 5-bit writer is added. Production change: `CausalEdge64::to_le_bytes` / `from_le_bytes` (mirroring `EpisodicEdges64`) and the register contract on the type. 6 tests, 7 disable runs red | | **D-ANCESTRY-K1-0** | StreamDto / PerturbationDto / BusDto / p64 / shader-driver ancestry vs code | Shipped (this PR) | `entries/2026-10-06-streamdto-circuit-ancestry.md`: v3 docs first, then code (git history is shallow at #1327). The k → k+1 loop was intended (`EmitMode::Persist`, `persist_cycle`, `mailbox-thoughtspace` read shim, W4a cast pairing) and left open: the driver owned its mailbox read-only. Path B | | **D-STREAMDTO-0** | A real `StreamDto` through the shipped path to next-cycle #1370 eligibility | In PR | `cognitive-shader-driver/examples/streamdto_circuit_probe.rs`: `StreamDto` → the new `ingest_codebook_indices_soa` (straight into the SoA, cycle-gated `write_row`, encoding shared with the superseded BindSpace arm) → `dispatch` k → `persisted_row` gets `emitted_edges[0]` via the new `ShaderDriver::mailbox_mut` → `tick` → `dispatch` k+1 reads it and changes → `measure`. The edge steers k+1 only through `s_idx / 4`; the shader writes code 0, so eligibility is unchanged by the circuit (`0x20`, same as an unwritten word; code 7 gives `0x24`). Timestamp ≠ cycle; restart from LE bytes reproduces k+1. CI step added (feature-gated targets incl. `w2_differential` were never run). 6 tests + 1 lib test, 9 disable runs red | diff --git a/.claude/board/entries/2026-10-10-d-lab-p1-certified-argmax-gomoku.md b/.claude/board/entries/2026-10-10-d-lab-p1-certified-argmax-gomoku.md new file mode 100644 index 000000000..74924f8bb --- /dev/null +++ b/.claude/board/entries/2026-10-10-d-lab-p1-certified-argmax-gomoku.md @@ -0,0 +1,52 @@ +# D-LAB-P1 — certified argmax on Gomoku: a deterministic certificate (2026-10-10) + +**Status:** MEASURED. Probe only +(`crates/cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs`, +`minimax_certified` + `certified_minimax_picks_what_minimax_picks_with_fewer_folds`). +Stockfish O1 untouched. `Op::Asc` and `Opp::Lookahead` still call `minimax`. + +## Certificate (kind: deterministic) + +A candidate's value is the minimum over its replies, so the running minimum +is an upper bound U that only falls. The best finished candidate has an exact +value L. When U ≤ L, the candidate cannot be chosen (the choice needs a strict +`>`), and its remaining replies are skipped. No value exceeds 7, so a finished +7 ends the search. This is the alpha-beta cutoff at the root, stated as a +certificate: `L_best ≥ U_j` ⇒ j is excluded. It holds by construction, not by +measurement; the corpus test checks the implementation. + +#1428's residual bound was not reused: it is proven for the linear flow model +only. + +## Measured (400 reasoning positions; propagation-decided positions excluded) + +| k1/k2 | folds full | folds certified | saved | positions with a cut | pick changed | +|---|---|---|---|---|---| +| 4/4 | 442,720 | 282,480 | 36.2 % | 400/400 | 0 | +| 5/5 | 665,076 | 367,344 | 44.8 % | 400/400 | 0 | +| 8/8 | 1,601,258 | 624,386 | 61.0 % | 400/400 | 0 | + +Savings grow with branching, as expected for cutoffs. Overhead: one +comparison per evaluated reply. Abstentions: none (an exact search never +abstains; it only skips). + +CI: the probe is `test = true` and runs under rust-test.yml's shader-driver +step. Disable run: cutting at `U ≤ L + 1` (unsound by one) → the test fails on +k = 4/4 with a changed pick. Pinned fold counts are exact (deterministic +corpus). + +## For the lab + +This is strategy S7 (certified early stopping) on one task, with a +decision-identical guarantee. It answers H4 on Gomoku minimax: computation +drops 36–61 % with zero changed decisions. It does not touch D-MOORE-NARS-0's +"less work not achieved" by itself: that probe's recipes would need to call +`minimax_certified`, which would change its reported folds but not its games. +That swap is the first lab arm to measure, not an edit made here. + +``` +PR (this) | STATUS: measured | OUTCOME: deterministic certified argmax on +Gomoku minimax: identical picks on 400 positions, 36-61% fewer folds; unsound +cut changes a pick | OPEN: recipes still call minimax by default; no +certificate yet for statistical (sampled) values +``` diff --git a/.claude/board/entries/2026-10-10-d-rpf-g2-law-a-boundary.md b/.claude/board/entries/2026-10-10-d-rpf-g2-law-a-boundary.md new file mode 100644 index 000000000..5684d36bb --- /dev/null +++ b/.claude/board/entries/2026-10-10-d-rpf-g2-law-a-boundary.md @@ -0,0 +1,60 @@ +# D-RPF-G2 — the law for decision-bearing wiring; P8 audit closed (2026-10-10) + +Follows `entries/2026-10-09-d-rpf-table-1-fast-path-cost.md` (#1426), which +already holds the P8 semantic audit. This entry re-checks it against code, +records the G2 boundary, and closes two of its four open items. + +## P8 — re-checked in code + +| question | answer | where | +|---|---|---| +| Which law does replay run? | Whatever `NarsTables` the caller passes; every caller passes `build(1)` (B1) except `tests/chain_confidence.rs` (16) | `chain_replay.rs::replay_step` | +| Who calls it outside tests? | Nobody. `replay_chain`, `counterfactual_replay` and `pearl::reason`'s `CutContext` are built only under `#[cfg(test)]` (`chain_admission.rs:165`, `pearl.rs:427`, `chain_counterfactual.rs:272`), in `tests/` and in examples | read 2026-10-10 | +| `NarsEngine::tables` consumed? | No production reader. `NarsEngine::new` (production instance `strategy/chat_bundle.rs:42`) allocates `build(1)`; `revise_fast`/`deduce_fast` have test callers only | `cache/nars_engine.rs` | +| Does replay overwrite forward's truth? | **Yes, confirmed.** `CausalEdge64::execute` computes `op.truth(..)` and packs it with `op.inference_type()`; `replay_step` then writes revision's F/C over it. The emitted edge names the weight's opcode (e.g. Deduction) and carries a revision truth | `causal-edge/src/edge.rs` `execute`; `chain_replay.rs` `replay_step` step 3 | + +No production behaviour depends on any of this today. + +## G2 — recommendation (DECISION, scoped) + +- DECISION: future decision-bearing consumers of replay/revision use exact + revision (law A). A table resolution is admissible only for a hot, + dependent replay whose measured decision-flip rate on that workload's own + corpus is below a bar stated before the run. +- SCOPE: new wiring only. `NarsEngine::new`'s `build(1)` default and every + existing public default stay as they are (operator-held). +- BASIS: #1426 — no Bn is decision-stable (B16: 4.7 % verdict flips); A beats + every table cold and beats B8/B16 on random access. +- REVISIT WHEN: a real recorded chain corpus exists (both #1426 corpora are + synthetic) or a consumer needs replay throughput A cannot meet. + +G1 (the (255,255) cell and finite vs dogmatic evidence) is untouched and stays +open; it is a separate decision. + +## Closed here + +1. **Load-bearing fixture re-derived under law A.** + `tests/d_rpf_table_1.rs::a_load_bearing_fixture_under_law_a`: seed 200/200, + support 250/250, refutation 10/230, cut 0. Law A: `LoadBearing {210 → 64}`. + Tables `[B1, B2, B4, B8, B16]` → load-bearing `[false, false, false, false, + true]`. The mirror image of the old fixture (load-bearing under B1 only). + Disable run: law A replaced by B1 → red, B1 reads `TruthOnly {118 → 105}`; + with B1 even the supported factual arm falls below the bar. +2. **`DEFAULT_FREQUENCY_BAR` rationale** no longer cites B1's constant 170 as + evidence; the semantic reason (consistency is strength, not amount of + evidence) stands alone. `the_bar_is_not_inert` keeps pinning 170, labelled + as B1's constant. + +## Still open + +- forward's discarded truth (compute waste) and the opcode/truth label + mismatch on replayed edges — recorded, unchanged. +- G1. + +``` +PR (this) | STATUS: done | OUTCOME: P8 re-checked in code (no production +consumer; replay overwrites forward's truth); G2 recorded as law A for new +decision-bearing wiring, defaults untouched; law-A load-bearing fixture +pinned; frequency-bar rationale corrected | OPEN: G1; forward truth discard +and opcode label mismatch +``` diff --git a/.claude/board/entries/2026-10-10-d-rpf-p5-nars-beta-correspondence.md b/.claude/board/entries/2026-10-10-d-rpf-p5-nars-beta-correspondence.md new file mode 100644 index 000000000..3f871c76e --- /dev/null +++ b/.claude/board/entries/2026-10-10-d-rpf-p5-nars-beta-correspondence.md @@ -0,0 +1,48 @@ +# D-RPF-P5 — NARS ↔ Beta correspondence on the CE64 surface (2026-10-10) + +**Status:** MEASURED. Tests only (`crates/causal-edge/tests/d_rpf_p5_beta.rs`). +No production semantics changed. G1 untouched. + +The live revision law has horizon k = 1 (`w = c/(255−c)`, revision adds `w`; +`d_rpf_conf_0.rs` T1). Under Bernoulli evidence with prior Beta(k/2, k/2), +`e = c(f−½)+½ = (w⁺+k/2)/(w+k)`, so k = 1 is the Jeffreys prior Beta(½, ½). + +| id | finding | pinned | +|---|---|---| +| E1 | The identity holds exactly for k = 1 over every code (max 3.3e-16); k = 2 differs by ≥ 1.15e-5; the f32 `expectation()` stays within 5.8e-8 | yes | +| E2 | A unit observation (w = 1 → c = 127.5) is not a code. Sequential revision of unit Bernoulli observations stops at **c = 244** (w ≈ 22.2) after 20 observations, in every one of 32 runs. From then on old evidence is forgotten as in a fixed window, while the conjugate posterior keeps narrowing: mean \|Δe\| < 0.002 at n = 8, > 0.05 at n = 256 (max 0.16) | yes | +| E2b | The stall code depends on observation strength: c_obs = 1 stalls at 75 (w 0.42), 64 at 236, 128 at 244, 240 at 252, 250 at 254 | 12-row table | +| E3 | Revision cannot tell duplicated from independent evidence. Four copies of one observation claim a posterior sd 2.04× too small; eight copies 3.48× | yes | +| E4 | A variance rebuilt from `(f, c)` moves 0.3 % per code at c = 128 and 29 % at c = 253 | yes | +| E5 | Revision of finite codes never reaches 255: the maximum, (254, 254), is 254.499 → 254. The singular cell is entered only by writing 255 directly | yes | + +Disable run: production confidence rounding changed from `round` to `ceil` → +E2, E2b and E5 red. + +## What this means for the lab and for G1 + +- **The u8 confidence code is an evidence horizon, not a counter.** Past the + stall, NARS-on-CE64 behaves like an exponentially weighted estimate with a + window set by observation strength, not like a conjugate Beta update. That is + a usable forgetting behaviour (it adapts after a change), but it is an + implicit one: no parameter names it. +- Any lab strategy that reads `(f, c)` as a Beta posterior is valid only below + the stall; past it, intervals must come from an explicit evidence count kept + elsewhere (e.g. a PowerSums fold), not from the code. +- E3 is why contradiction and pooling need provenance: the operands carry none. +- G1: E5 shows that (255, 255) is unreachable by accumulation, so the G1 policy + only governs values written directly. + +``` +PR (this) | STATUS: measured | OUTCOME: e is the Jeffreys posterior mean +exactly; u8 confidence stalls (unit evidence at 244 after 20 obs) so CE64 +revision is a fixed-window estimate past the stall; duplicates inflate +precision; 255 unreachable by revision | OPEN: G1; whether the lab keeps a +separate evidence count; decay is not an operation on CE64 truth (not tested) +``` + +MIRROR +- BIAS CHECK: the brief framed P5 as "is the correspondence exact"; the + identity was never in doubt. The stall (E2) is the result that matters, and + it showed up only because the probe ran long sequences rather than single + revisions. diff --git a/.claude/board/entries/2026-10-10-d-self-calibrating-lab-0.md b/.claude/board/entries/2026-10-10-d-self-calibrating-lab-0.md new file mode 100644 index 000000000..94d890a23 --- /dev/null +++ b/.claude/board/entries/2026-10-10-d-self-calibrating-lab-0.md @@ -0,0 +1,148 @@ +# D-SELF-CALIBRATING-LAB-0 — the hypothesis lab, v0: shared statistics + Gomoku and crossword arms (2026-10-10) + +**Status:** MEASURED (one task family). Probe only; no production code. +**Code:** `crates/cognitive-shader-driver/examples/shared/lab.rs` (task-agnostic +statistics) and `moore_nars_gomoku_probe.rs --lab` (the Gomoku arm). Tests run +in CI (`test = true`, rust-test.yml shader-driver step). +**Predecessors:** D-MOORE-NARS-0 (the learner), D-LAB-P1 (certified minimax), +D-RPF-P5 (what the expectation means). + +## What v0 is + +A task probe runs each strategy arm on matched blocks (seeds) and hands +`lab.rs` one outcome per (arm, block). `lab.rs` provides: + +| piece | certificate kind | note | +|---|---|---| +| paired comparison: per-block difference, exact sign-flip p (exhaustive ≤ 20 blocks), 95 % t-interval | statistical | t-interval assumes roughly normal block differences; the p-value assumes nothing beyond exchangeability of signs | +| Holm over a pre-registered family | statistical | α = 0.05 | +| verdict per pre-registration | statistical | SUPPORTED needs Holm rejection, the declared direction and ≥ the declared minimum effect; FALSIFIED needs the interval to rule the minimum out; otherwise INCONCLUSIVE; a wrong block count is never SUPPORTED | +| reliability bins, ECE, Brier | empirical | | +| deterministic checks (e.g. identical games) | deterministic | reported separately, never folded into a p-value | + +Experiment record = (task, context, strategy arm, hypothesis, block seed, +outcome, cost). Provenance is the seed set, the arm's `Cfg`, and the commit. +Not yet a SoA record; v0 keeps it in the probe (see OPEN). + +Falsifiers: Holm's step-down stop and the min-effect gate are each +disable-verified red (`holm_matches_the_textbook_step_down`, +`a_planted_effect_is_found_and_a_null_is_not`). The first version of the +min-effect test was vacuous (the gate's removal stayed green); a significant +tiny effect was added and the disable then failed. + +## Pilot → amendments → confirmation + +The pilot (seeds 1001–1012, 100 games, whole-run score) was exploratory. +It exposed three design errors, each recorded in code before the +confirmatory run: + +1. **H4's identical-games check failed** — not the certificate: the chooser + has a fold-cost term, so a cheaper ASC changes which recipe it picks. + H4 is now measured at cost weight 0, and the interaction is pinned + (`the_certificate_changes_no_game_for_a_cost_blind_learner`: identical at + cost 0, not identical at 0.02). +2. **100 games measured the learning phase.** Endpoint is now the last + 100-game window of 300. +3. **Calibration needs the right population** (see below). + +Confirmation: 16 fresh blocks (seeds 2001–2016), 9×9 five vs Threat, oracle +off, 28.6 s wall. + +## Hypothesis ledger (Gomoku, confirmatory) + +| id | hypothesis | effect (treatment − baseline, + = better) | 95 % CI | p | Holm | verdict | +|---|---|---|---|---|---|---| +| H2 | learned chooser beats best fixed recipe (HPM), score | +0.0238 | [+0.0060, +0.0415] | 0.0156 | reject | **SUPPORTED** | +| H4 | certified minimax cuts folds/step (cost 0) | +28.9 | [+20.3, +37.6] | <0.0001 | reject | **SUPPORTED** (+ deterministic: identical games on all 16 blocks) | +| H7 | fold-cost term cuts folds/step | +142.7 | [+118.3, +167.1] | <0.0001 | reject | **SUPPORTED** | +| H3 | delayed (one-reply) credit beats immediate, score | +0.0225 | [+0.0036, +0.0414] | 0.0267 | keep | INCONCLUSIVE | +| H5 | decay 0.95 adapts faster after Threat → Aggressor, post-switch score | +0.0072 | [−0.0051, +0.0195] | 0.2322 | keep | **FALSIFIED** (CI rules out the 0.02 minimum) | + +H2 replicates D-MOORE-NARS-0's central claim on fresh seeds. H3 points the +same way as D-MOORE-NARS-0 but does not survive the family correction at +16 blocks. + +## Pareto (mean over blocks; empirical) + +| arm | score | folds/step | +|---|---|---| +| cost 0 + certified | 0.5331 | 368.6 | +| cost 0 | 0.5331 | 397.6 | +| decay 0.95 | 0.5159 | 231.9 | +| base (cost 0.02) | 0.5069 | 254.8 | +| immediate credit | 0.4844 | 0.0 | +| fixed HPM | 0.4831 | 0.0 | + +`cost 0` is dominated by `cost 0 + certified` (same score, fewer folds). +The cost term is a trade-off, not a free win: −0.026 score for −143 +folds/step. Immediate credit collapses onto HPM (zero-fold recipes). + +## Calibration — what the NARS expectation predicts + +The chooser's expectation e of the chosen recipe vs whether the step was +productive (retained activation > 0), base arm: + +- over **all** outcomes: ECE 0.317 (e overstates); +- over **non-neutral** outcomes (activation ≠ 0): ECE 0.050. + +A zero activation adds no evidence, so e estimates P(positive | non-neutral), +and it is well calibrated for that. Reading it as P(productive step) is the +error. Same lesson as D-RPF-P5: e is a posterior mean of the evidence the +substrate actually records, nothing more. + +## Crossword arm (`crossword_crossing_care_probe lab`) + +Same `lab.rs`, second task family, separate Holm family. Blocks = 16 fresh +puzzles (seed 0x1AB5EED; the original used the 0xC0FFEE sets), side 5, open +workload (half givens, cap 50). Endpoints in natural logs (min effect 0.1 ≈ +10 %); ops is the deterministic cost counter, not wall time. 3.4 s wall. + +Deterministic half: every arm sound on every puzzle; same fill count wherever +two arms both solved; 16/16 solved by each arm. + +| id | hypothesis | effect (ln, + = treatment better) | 95 % CI | Holm | verdict | +|---|---|---|---|---|---| +| C1 | Support cuts search nodes vs Baseline | +3.59 (≈36×) | [+3.00, +4.17] | reject | **SUPPORTED** | +| C2 | Support cuts total ops vs Baseline | −2.57 (≈13× more ops) | [−2.90, −2.24] | reject | **FALSIFIED** | +| C3 | Single counterfactuals cut ops vs Baseline | −1.07 (≈2.9× more) | [−1.71, −0.43] | reject | **FALSIFIED** | +| C4 | Pair nogoods cut ops beyond Single | −0.78 (≈2.2× more) | [−1.13, −0.43] | reject | **FALSIFIED** | + +Geometric means: Baseline 2,427 nodes / 4,718 ops; Support 67 / 61,700; +Single 471 / 13,717; Pair 434 / 29,895. Replicates D-SCF-CARE-PAIR-0 on +fresh puzzles. One difference worth noting: here Single needs 5× fewer nodes +than Baseline, while the original's open set showed similar node counts; +the original attributed that to enumeration order under the cap. + +## Across the two tasks (empirical, not yet a tested meta-hypothesis) + +On both tasks the strategy that wins on a work-avoidance proxy (nodes; +learned value) is not automatically the cheapest end to end, and the lab +separates the two only because cost is a declared endpoint. On Gomoku the +deterministic certificate is the one change that cuts cost with zero quality +change; on crossword no tested arm beats Baseline on ops. A cost-aware +strategy selector is therefore the next lab question, not a conclusion. + +## OPEN + +- Held-out evaluation (Moore #1428) and the cross-task meta-experiment need + both tasks in one process: the record should move to a SoA-shaped carrier + then, not before. +- Strategy arms not yet present: Thompson sampling, eligibility traces over W, + context-conditioned selection, cost-aware experiment selection. +- H3 needs more blocks to decide; that is a pre-registered follow-up, not a + re-reading of this run. + +``` +PR (this) | STATUS: measured | OUTCOME: lab v0 (paired sign-flip + Holm + +verdicts + reliability) with a Gomoku arm; pilot exposed 3 design errors, +fixed before a fresh-seed confirmation: H2 H4 H7 supported, H3 inconclusive, +H5 falsified; e calibrated (ECE 0.05) only over non-neutral outcomes | +OPEN: held-out domain, SoA record, Thompson/eligibility arms, cost-aware +selector; crossword arm: C1 supported, C2-C4 falsified (fresh puzzles) +``` + +MIRROR +- BIAS CHECK: the H4 pilot "failure" looked like a certificate bug; the + disable-verified P1 test already proved the certificate, so the divergence + had to come from elsewhere, and the cost term was it. +- BLIND SPOT: one opponent family (Threat, Aggressor), one board size. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index b83f7c164..691baa278 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,14 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -283 entries, 2026-08-06 .. 2026-10-09. +287 entries, 2026-08-06 .. 2026-10-10. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-10 | `D-SELF-CALIBRATING-LAB-0` | | [2026-10-10-d-self-calibrating-lab-0.md](2026-10-10-d-self-calibrating-lab-0.md) | +| 2026-10-10 | `D-RPF-P5` | | [2026-10-10-d-rpf-p5-nars-beta-correspondence.md](2026-10-10-d-rpf-p5-nars-beta-correspondence.md) | +| 2026-10-10 | `D-RPF-G2` | | [2026-10-10-d-rpf-g2-law-a-boundary.md](2026-10-10-d-rpf-g2-law-a-boundary.md) | +| 2026-10-10 | `D-LAB-P1` | | [2026-10-10-d-lab-p1-certified-argmax-gomoku.md](2026-10-10-d-lab-p1-certified-argmax-gomoku.md) | | 2026-10-09 | `D-SCF-CARE-PAIR-0` | | [2026-10-09-d-scf-care-pair-0-crossing-counterfactuals.md](2026-10-09-d-scf-care-pair-0-crossing-counterfactuals.md) | | 2026-10-09 | `D-RPF-TABLE-1` | | [2026-10-09-d-rpf-table-1-fast-path-cost.md](2026-10-09-d-rpf-table-1-fast-path-cost.md) | | 2026-10-09 | `D-RPF-CONF-0` | | [2026-10-09-d-rpf-conf-0-confidence-surface.md](2026-10-09-d-rpf-conf-0-confidence-surface.md) | diff --git a/crates/causal-edge/tests/d_rpf_p5_beta.rs b/crates/causal-edge/tests/d_rpf_p5_beta.rs new file mode 100644 index 000000000..06da97198 --- /dev/null +++ b/crates/causal-edge/tests/d_rpf_p5_beta.rs @@ -0,0 +1,294 @@ +//! D-RPF-P5 — the NARS ↔ Beta correspondence, measured on the CE64 surface. +//! +//! The live revision law has evidential horizon k = 1: a confidence code +//! `c` stands for evidence weight `w = c / (255 − c)` and revision adds +//! weights (`d_rpf_conf_0.rs` T1). Under Bernoulli evidence with a +//! Beta(k/2, k/2) prior, the NARS expectation +//! `e = c·(f − ½) + ½` equals the posterior mean `(w⁺ + k/2) / (w + k)`. +//! With k = 1 the prior is Jeffreys, Beta(½, ½). +//! +//! What this file pins, and what it does not claim: +//! +//! - **E1** the identity holds exactly in real arithmetic for k = 1 and fails +//! for k = 2, and the f32 `expectation()` stays within its rounding. +//! - **E2** a unit observation (w = 1, c = ½) is not representable as a code; +//! sequential revision of Bernoulli observations therefore drifts from the +//! conjugate posterior, and the code range caps the evidence it can hold. +//! - **E3** revision cannot tell duplicated evidence from independent +//! evidence; duplication shrinks the posterior as if it were new data. +//! - **E2b** the stall code depends on observation strength (weak evidence +//! stalls low: observations at c = 1 stop at c = 75). +//! - **E4** a posterior variance rebuilt from `(f, c)` codes is coarse at +//! high confidence, because successive codes are far apart in `w`. +//! - **E5** revision of finite codes never produces c = 255. +//! +//! Confidence is not a calibrated credible interval. A variance is derived +//! from the evidence model (E4), never read off the confidence code. +//! There is no decay operation on the CE64 truth fields, so decay is not +//! tested here. +#![cfg(feature = "causal-edge-v2-layout")] + +use causal_edge::CausalEdge64; + +fn edge(f: u8, c: u8) -> CausalEdge64 { + let mut e = CausalEdge64(0); + e.set_frequency_u8(f); + e.set_confidence_u8(c); + e +} + +/// Evidence weight of a confidence code under horizon k = 1. +fn weight(c: u8) -> f64 { + f64::from(c) / (255.0 - f64::from(c)) +} + +/// Beta posterior mean with prior Beta(k/2, k/2). +fn posterior_mean(w_pos: f64, w: f64, k: f64) -> f64 { + (w_pos + k / 2.0) / (w + k) +} + +/// Beta posterior variance with prior Beta(k/2, k/2). +fn posterior_var(w_pos: f64, w: f64, k: f64) -> f64 { + let a = w_pos + k / 2.0; + let b = w - w_pos + k / 2.0; + a * b / ((a + b) * (a + b) * (a + b + 1.0)) +} + +/// NARS expectation in exact arithmetic from codes. +fn expectation_exact(f: u8, c: u8) -> f64 { + let (f, c) = (f64::from(f) / 255.0, f64::from(c) / 255.0); + c * (f - 0.5) + 0.5 +} + +// ─── E1 ───────────────────────────────────────────────────────────────── + +#[test] +fn e1_expectation_is_the_jeffreys_posterior_mean() { + let mut max_k1 = 0.0_f64; + let mut min_k2 = f64::INFINITY; + let mut max_f32 = 0.0_f64; + for c in 0..255u8 { + let w = weight(c); + for f in 0..=255u8 { + let w_pos = w * f64::from(f) / 255.0; + let e = expectation_exact(f, c); + max_k1 = max_k1.max((e - posterior_mean(w_pos, w, 1.0)).abs()); + if c > 0 && f != 128 && f != 127 { + min_k2 = min_k2.min((e - posterior_mean(w_pos, w, 2.0)).abs()); + } + max_f32 = max_f32.max((f64::from(edge(f, c).expectation()) - e).abs()); + } + } + eprintln!("E1 k=1 max {max_k1:e} k=2 min {min_k2:e} f32 max {max_f32:e}"); + assert!(max_k1 < 1e-12, "k = 1: e is the posterior mean"); + assert!(min_k2 > 1e-6, "k = 2 is a different prior; the test sees k"); + assert!(max_f32 < 1e-6, "f32 expectation stays within rounding"); +} + +// ─── E2 ───────────────────────────────────────────────────────────────── + +struct Lcg(u64); +impl Lcg { + fn next(&mut self) -> u64 { + self.0 = self + .0 + .wrapping_mul(6_364_136_223_846_793_005) + .wrapping_add(1_442_695_040_888_963_407); + self.0 >> 33 + } +} + +/// Revise `n` Bernoulli(p) unit observations into one edge. Returns, at each +/// step, `(e_code_path, e_conjugate, c_code)`. +fn bernoulli_run(p_num: u64, n: usize, seed: u64) -> Vec<(f64, f64, u8)> { + let unit_c = 128u8; // round(255 · ½): the nearest code to w = 1 + let mut rng = Lcg(seed); + let mut acc: Option = None; + let mut successes = 0.0_f64; + let mut out = Vec::with_capacity(n); + for i in 0..n { + let hit = rng.next() % 100 < p_num; + if hit { + successes += 1.0; + } + let obs = edge(if hit { 255 } else { 0 }, unit_c); + let next = match acc { + None => obs, + Some(a) => a.revision(obs), + }; + acc = Some(next); + let e_conj = posterior_mean(successes, (i + 1) as f64, 1.0); + out.push(( + expectation_exact(next.frequency_u8(), next.confidence_u8()), + e_conj, + next.confidence_u8(), + )); + } + out +} + +#[test] +fn e2_unit_observation_is_not_a_code_and_the_horizon_is_bounded() { + // w = 1 needs c = 127.5; the nearest codes carry 127/128 and 128/127. + assert!((weight(127) - 1.0).abs() > 0.007 && (weight(128) - 1.0).abs() > 0.007); + + let runs: Vec> = (0..32u64).map(|s| bernoulli_run(70, 400, s)).collect(); + for n in [1usize, 2, 4, 8, 16, 32, 64, 128, 256, 400] { + let mut max_d = 0.0_f64; + let mut sum_d = 0.0_f64; + let mut cmin = u8::MAX; + let mut cmax = 0u8; + for r in &runs { + let (e, ec, c) = r[n - 1]; + let d = (e - ec).abs(); + max_d = max_d.max(d); + sum_d += d; + cmin = cmin.min(c); + cmax = cmax.max(c); + } + eprintln!( + "E2 n={n:4} mean|Δe| {:.4} max|Δe| {:.4} c in {cmin}..={cmax}", + sum_d / runs.len() as f64, + max_d + ); + } + // First step at which any run's confidence code stops rising. + let mut first_stall = usize::MAX; + for r in &runs { + for i in 1..r.len() { + if r[i].2 <= r[i - 1].2 { + first_stall = first_stall.min(i + 1); + break; + } + } + } + eprintln!("E2 first confidence stall at n = {first_stall}"); + // Measured: every run's confidence stops at code 244 (w ≈ 22.2); the + // first stall is at n = 21. From there the code path forgets old + // evidence like a fixed window while the conjugate posterior keeps + // narrowing, so e drifts. + assert_eq!(first_stall, 21); + assert!(runs.iter().all(|r| r[399].2 == 244)); + let mean_d_256: f64 = runs + .iter() + .map(|r| (r[255].0 - r[255].1).abs()) + .sum::() + / runs.len() as f64; + let mean_d_8: f64 = + runs.iter().map(|r| (r[7].0 - r[7].1).abs()).sum::() / runs.len() as f64; + assert!( + mean_d_8 < 0.002, + "before the stall the code path tracks the posterior" + ); + assert!(mean_d_256 > 0.05, "after the stall it does not"); +} + +// ─── E3 ───────────────────────────────────────────────────────────────── + +#[test] +fn e3_duplicated_evidence_is_counted_as_new() { + let one = edge(255, 128); + let mut dup = one; + for k in 2..=8u32 { + dup = dup.revision(one); + let w = weight(dup.confidence_u8()); + let var_claimed = posterior_var(w, w, 1.0); + let w1 = weight(128); + let var_true = posterior_var(w1, w1, 1.0); + let ratio = (var_true / var_claimed).sqrt(); + eprintln!( + "E3 copies {k}: c {} w {:.3} sd claimed {:.4} sd true {:.4} ratio {:.3}", + dup.confidence_u8(), + w, + var_claimed.sqrt(), + var_true.sqrt(), + ratio + ); + // Four copies of one observation claim twice the precision it has. + if k == 4 { + assert!((2.0..2.1).contains(&ratio), "k = 4 ratio {ratio}"); + } + } + // Independent and duplicated evidence are indistinguishable to revision: + // the operands carry no provenance, so this is by construction. + assert_eq!(one.revision(one), edge(255, 128).revision(edge(255, 128))); +} + +// ─── E4 ───────────────────────────────────────────────────────────────── + +#[test] +fn e4_variance_from_codes_is_coarse_at_high_confidence() { + for c in [64u8, 128, 192, 224, 240, 248, 252, 253] { + let w = weight(c); + let w_next = weight(c + 1); + let v = posterior_var(0.7 * w, w, 1.0); + let v_next = posterior_var(0.7 * w_next, w_next, 1.0); + let step = 1.0 - (v_next / v).sqrt(); + eprintln!( + "E4 c {c:3}: w {w:8.3} → {w_next:8.3} sd {:.4} → {:.4} step {:.1}%", + v.sqrt(), + v_next.sqrt(), + 100.0 * step + ); + if c == 128 { + assert!(step < 0.005, "mid-range: one code is a fine step"); + } + if c == 253 { + assert!(step > 0.25, "top of range: one code moves sd by > 25%"); + } + } +} + +// ─── E2b: the quantization fixed point ────────────────────────────────── + +/// Repeatedly revise an edge with observations of confidence code `obs_c`; +/// return the code where confidence stops rising and the step it got there. +fn stall_code(obs_c: u8) -> (u8, u32) { + let obs = edge(255, obs_c); + let mut acc = obs; + for step in 2..10_000u32 { + let next = acc.revision(obs); + if next.confidence_u8() <= acc.confidence_u8() { + return (acc.confidence_u8(), step - 1); + } + acc = next; + } + (acc.confidence_u8(), 10_000) +} + +/// The horizon depends on the strength of each observation: weak evidence +/// stalls at low confidence. Measured and pinned. +#[test] +fn e2b_the_stall_code_depends_on_observation_strength() { + const PINNED: [(u8, u8, u32); 12] = [ + (1, 75, 75), + (8, 193, 87), + (32, 225, 48), + (64, 236, 34), + (96, 241, 26), + (128, 244, 20), + (160, 247, 16), + (192, 249, 12), + (224, 251, 8), + (240, 252, 5), + (250, 254, 4), + (254, 254, 1), + ]; + for (obs, stall, after) in PINNED { + assert_eq!(stall_code(obs), (stall, after), "obs c {obs}"); + } +} + +/// Revision of two finite codes never reaches 255: the maximum, (254, 254), +/// is 254.499… and rounds to 254. So the singular cell (G1) is entered only +/// by writing 255 directly, never by accumulating evidence. +#[test] +fn e5_revision_never_reaches_the_singular_code() { + let mut max = 0u8; + for c1 in 0..255u8 { + for c2 in 0..255u8 { + max = max.max(edge(128, c1).revision(edge(128, c2)).confidence_u8()); + } + } + assert_eq!(max, 254); +} diff --git a/crates/cognitive-shader-driver/examples/crossword_crossing_care_probe.rs b/crates/cognitive-shader-driver/examples/crossword_crossing_care_probe.rs index aa892d14d..54dbfb475 100644 --- a/crates/cognitive-shader-driver/examples/crossword_crossing_care_probe.rs +++ b/crates/cognitive-shader-driver/examples/crossword_crossing_care_probe.rs @@ -52,6 +52,9 @@ use population_fold::Rng; mod crossword_core; use crossword_core::*; +#[path = "shared/lab.rs"] +mod lab; + const SYMS: usize = MooreSymbol8::COUNT; /// `|a ∧ b|` without writing anything. @@ -1449,8 +1452,139 @@ fn chain_report(hot: &Hot) { } } +// ── D-SELF-CALIBRATING-LAB-0: the crossword arm of the hypothesis lab ────── + +/// Pre-registered before the run that reports it. Directions come from +/// D-SCF-CARE-PAIR-0 (support: far fewer nodes, more ops; single and pair +/// counterfactuals do not pay). Endpoints are natural logs, so a minimum +/// effect of 0.1 means about 10 %. Fresh puzzles (seed 0x1AB5EED, not the +/// 0xC0FFEE sets the original measured), side 5, open workload. +const LAB_PUZZLES: usize = 16; +const LAB_PREREG: [lab::Prereg; 4] = [ + lab::Prereg { + id: "C1", + hypothesis: "letter support cuts search nodes", + baseline: "Baseline", + treatment: "Support", + endpoint: "ln nodes", + higher_is_better: false, + min_effect: 0.1, + blocks: LAB_PUZZLES, + }, + lab::Prereg { + id: "C2", + hypothesis: "letter support cuts total ops", + baseline: "Baseline", + treatment: "Support", + endpoint: "ln ops", + higher_is_better: false, + min_effect: 0.1, + blocks: LAB_PUZZLES, + }, + lab::Prereg { + id: "C3", + hypothesis: "single-crossing counterfactuals cut total ops", + baseline: "Baseline", + treatment: "Single", + endpoint: "ln ops", + higher_is_better: false, + min_effect: 0.1, + blocks: LAB_PUZZLES, + }, + lab::Prereg { + id: "C4", + hypothesis: "pair nogoods cut total ops beyond single", + baseline: "Single", + treatment: "Pair", + endpoint: "ln ops", + higher_is_better: false, + min_effect: 0.1, + blocks: LAB_PUZZLES, + }, +]; + +fn lab_stage(hot: &Hot) { + let t0 = Instant::now(); + println!("D-SELF-CALIBRATING-LAB-0 — crossword arm (side 5, open: half givens, cap 50)"); + let set = make(hot, 5, LAB_PUZZLES, 0x1AB5EED); + let arms = [Arm::Baseline, Arm::Support, Arm::Single, Arm::Pair]; + let runs: Vec> = arms + .iter() + .map(|&arm| { + set.iter() + .map(|x| { + run( + hot, + &x.c.puz, + &half(&x.c.givens), + arm, + Broken::default(), + 40_000_000, + 50, + ) + }) + .collect() + }) + .collect(); + // Deterministic half: every arm is sound and reaches the same answer. + let sound_all = arms.iter().enumerate().all(|(k, _)| { + set.iter() + .zip(&runs[k]) + .all(|(x, r)| sound(hot, &x.c.puz, r, &x.c.solution)) + }); + let same = (1..arms.len()).all(|k| { + runs[k] + .iter() + .zip(&runs[0]) + .all(|(a, b)| a.solved() != b.solved() || !a.solved() || a.fills == b.fills) + }); + let solved: Vec = runs + .iter() + .map(|v| v.iter().filter(|r| r.solved()).count()) + .collect(); + println!( + "deterministic: every arm sound {sound_all}; same fill count where both solved {same}; solved {solved:?} of {LAB_PUZZLES}" + ); + let ln = |v: &[Run], f: fn(&Run) -> u64| { + v.iter() + .map(|r| (f(r).max(1) as f64).ln()) + .collect::>() + }; + let nodes = |r: &Run| r.nodes; + let ops = |r: &Run| r.ops; + let results = [ + lab::paired(&ln(&runs[0], nodes), &ln(&runs[1], nodes), false), + lab::paired(&ln(&runs[0], ops), &ln(&runs[1], ops), false), + lab::paired(&ln(&runs[0], ops), &ln(&runs[2], ops), false), + lab::paired(&ln(&runs[2], ops), &ln(&runs[3], ops), false), + ]; + let rejected = lab::holm(&results.map(|r| r.p), 0.05); + println!("\nFamily (Holm, alpha 0.05; effects in ln units, + = treatment better):"); + for i in 0..LAB_PREREG.len() { + println!( + " {}", + lab::report(&LAB_PREREG[i], &results[i], rejected[i]) + ); + } + println!("\nPareto (geometric mean over puzzles; empirical):"); + for (k, arm) in arms.iter().enumerate() { + let g = |v: Vec| (v.iter().sum::() / v.len() as f64).exp(); + println!( + " {:<9} nodes {:>10.0} ops {:>12.0}", + format!("{arm:?}"), + g(ln(&runs[k], nodes)), + g(ln(&runs[k], ops)) + ); + } + println!("\nlab wall time {:.1}s", t0.elapsed().as_secs_f64()); +} + fn main() { let (hot, _cold) = build_lexicon(Lang::En, &english_ranked()); + if std::env::args().nth(1).as_deref() == Some("lab") { + lab_stage(&hot); + return; + } println!("D-SCF-CARE-PAIR-0 (English, {} ids)", hot.len.len()); if std::env::args().nth(1).as_deref() == Some("l2") { uncapped_report(&hot, &make(&hot, 5, 10, 0xC0FFEE + 5)); diff --git a/crates/cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs b/crates/cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs index 4874470ce..d11708beb 100644 --- a/crates/cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs +++ b/crates/cognitive-shader-driver/examples/moore_nars_gomoku_probe.rs @@ -34,6 +34,9 @@ use lance_graph_contract::epistemic_state5::{Certification3, Epi5Gen, EpistemicS #[cfg(test)] use lance_graph_contract::recipes::RECIPES; +#[path = "shared/lab.rs"] +mod lab; + // ── Census of the 34 catalogue recipes ───────────────────────────────────── /// How a catalogue recipe stands in this domain. @@ -550,6 +553,57 @@ fn minimax(b: &Board, me: u8, s: &Snap, k1: usize, k2: usize, folds: &mut u64) - best.1 } +/// D-LAB-P1: `minimax` with a deterministic certificate. A candidate's value +/// is the minimum over its replies, so the running minimum is an upper bound +/// that only falls. Once it is `<=` the best finished value, the candidate +/// cannot be chosen (the choice needs a strict `>`), and its remaining +/// replies are skipped. Values never exceed 7, so a finished 7 ends the +/// search. Picks the same move as `minimax` by construction; the probe +/// measures what the skipped replies were worth in folds. +fn minimax_certified(b: &Board, me: u8, s: &Snap, k1: usize, k2: usize, folds: &mut u64) -> usize { + let mut best = (i32::MIN, s.cells[0].i); + for m in s.top(k1, hpm) { + if best.0 >= 7 { + break; + } + let mut c = b.clone(); + c.play(m.i, me); + let worst = if m.c_me == FIVE { + 7 + } else { + let s1 = snapshot(&c, 3 - me, folds); + let mut worst = i32::MAX; + for r in s1.top(k2, hpm) { + let v = if r.c_me == FIVE { + -7 + } else { + let mut cc = c.clone(); + cc.play(r.i, 3 - me); + let s2 = snapshot(&cc, me, folds); + if s2.cells.is_empty() { + 0 + } else { + balance(&s2) + } + }; + worst = worst.min(v); + if worst <= best.0 { + break; + } + } + if worst == i32::MAX { + 0 + } else { + worst + } + }; + if worst > best.0 { + best = (worst, m.i); + } + } + best.1 +} + struct Rng(u64); impl Rng { fn next(&mut self) -> u64 { @@ -565,7 +619,15 @@ impl Rng { } /// Run one recipe over the population. Returns the move; adds its folds. -fn run_op(op: Op, b: &Board, me: u8, s: &Snap, rng: &mut Rng, folds: &mut u64) -> usize { +fn run_op( + op: Op, + b: &Board, + me: u8, + s: &Snap, + rng: &mut Rng, + folds: &mut u64, + certified: bool, +) -> usize { match op { Op::Hpm => s.argmax(hpm).i, Op::Rcr => { @@ -615,6 +677,7 @@ fn run_op(op: Op, b: &Board, me: u8, s: &Snap, rng: &mut Rng, folds: &mut u64) - } best.1 } + Op::Asc if certified => minimax_certified(b, me, s, 4, 4, folds), Op::Asc => minimax(b, me, s, 4, 4, folds), Op::Cr => { let a = s.argmax(|c| i64::from(c.s_me)).i; @@ -633,9 +696,9 @@ fn run_op(op: Op, b: &Board, me: u8, s: &Snap, rng: &mut Rng, folds: &mut u64) - } Op::Tcf => { let picks = [ - run_op(Op::Hpm, b, me, s, rng, folds), - run_op(Op::Rcr, b, me, s, rng, folds), - run_op(Op::Tcp, b, me, s, rng, folds), + run_op(Op::Hpm, b, me, s, rng, folds, certified), + run_op(Op::Rcr, b, me, s, rng, folds, certified), + run_op(Op::Tcp, b, me, s, rng, folds, certified), ]; if picks[1] == picks[2] { picks[1] @@ -717,6 +780,8 @@ struct Cfg { cost_weight: f64, /// Update from experience (false = frozen evaluation). learn: bool, + /// ASC runs `minimax_certified` (same pick, fewer folds; D-LAB-P1). + certified: bool, } const BASE: Cfg = Cfg { @@ -727,6 +792,7 @@ const BASE: Cfg = Cfg { decay: 1.0, cost_weight: 0.02, learn: true, + certified: false, }; /// NARS evidence for one (signature, recipe) pairing. @@ -775,6 +841,9 @@ struct Pending { dir: u8, /// The question the step asked (bits 40..42). question: CausalMask, + /// The chooser's NARS expectation of this recipe at choice time (0.5 + /// when not learned): the lab's calibration prediction. + pred: f64, } #[derive(Default)] @@ -890,6 +959,12 @@ struct Stats { regret_n: u64, /// Steps where the chosen recipe was among the best. optimal: u64, + /// Reliability bins over `Pending::pred`: (count, sum of pred, retained > 0). + cal: [(u64, f64, u64); 10], + /// The same bins over non-neutral outcomes only (retained != 0): the + /// population a NARS expectation actually estimates, since a zero + /// activation adds no evidence. + cal_nn: [(u64, f64, u64); 10], } impl Stats { @@ -910,6 +985,16 @@ impl Stats { self.regret += o.regret; self.regret_n += o.regret_n; self.optimal += o.optimal; + for (a, b) in self + .cal + .iter_mut() + .chain(self.cal_nn.iter_mut()) + .zip(o.cal.iter().chain(o.cal_nn.iter())) + { + a.0 += b.0; + a.1 += b.1; + a.2 += b.2; + } } fn score(&self) -> f64 { (self.wins as f64 + 0.5 * (self.games - self.wins - self.losses) as f64) @@ -959,7 +1044,7 @@ fn oracle_regret( for (k, &op) in OPS.iter().enumerate() { let mut f = 0; let mut rng = Rng(rng_seed); - let mv = run_op(op, b, me, s, &mut rng, &mut f); + let mv = run_op(op, b, me, s, &mut rng, &mut f, false); acts[k] = consequence(b, me, mv, opp, before); } let best = *acts.iter().max().unwrap(); @@ -1066,6 +1151,15 @@ fn game( if retained > 0 { st.productive += 1; } + let bin = ((p.pred * 10.0) as usize).min(9); + st.cal[bin].0 += 1; + st.cal[bin].1 += p.pred; + st.cal[bin].2 += u64::from(retained > 0); + if retained != 0 { + st.cal_nn[bin].0 += 1; + st.cal_nn[bin].1 += p.pred; + st.cal_nn[bin].2 += u64::from(retained > 0); + } let act = match cfg.signal { Signal::Retained(_) => Some(retained), Signal::Immediate => Some(p.immediate), @@ -1091,11 +1185,22 @@ fn game( }; let back = coarse(&s); let (op, hit) = l.choose(cfg, key, back, &mut rng); + let pred = match cfg.chooser { + Chooser::Learned => l + .stats + .get(&key) + .or_else(|| match cfg.memory { + Memory::Structural => l.stats.get(&back), + Memory::Literal => None, + }) + .map_or(0.5, |row| row[op.index()].expectation()), + _ => 0.5, + }; st.hits += u64::from(hit); let step_seed = rng.next(); let mut r2 = Rng(step_seed); let mut f = 0; - let mv = run_op(op, &b, me, &s, &mut r2, &mut f); + let mv = run_op(op, &b, me, &s, &mut r2, &mut f, cfg.certified); st.steps += 1; st.folds += f; st.ops[op.index()] += 1; @@ -1129,6 +1234,7 @@ fn game( immediate, dir: s.cells.iter().find(|c| c.i == mv).map_or(0, |c| c.dir), question: question(&s), + pred, }; ring[wslot as usize] = Some(p); queue.push_back((wslot, st.moves)); @@ -1265,7 +1371,234 @@ fn dist(s: &Stats) -> String { .join(" ") } +// ── D-SELF-CALIBRATING-LAB-0: the Gomoku arm of the hypothesis lab ───────── + +/// One arm on one block: the LAST window's stats (the learned policy, not +/// the learning phase). +fn lab_block(cfg: &Cfg, seed: u64, games: usize) -> Stats { + let w = World { + n: 9, + win: 5, + opp: Opp::Threat, + oracle: false, + }; + run(w, cfg, &mut Learner::default(), seed, games, LAB_WINDOW) + .pop() + .expect("at least one window") +} + +/// Train against Threat, then switch to Aggressor; return the post-switch +/// stats only. +fn lab_switch_block(cfg: &Cfg, seed: u64, games: usize) -> Stats { + let mut l = Learner::default(); + let threat = World { + n: 9, + win: 5, + opp: Opp::Threat, + oracle: false, + }; + let aggressor = World { + opp: Opp::Aggressor, + ..threat + }; + let _ = run(threat, cfg, &mut l, seed, games, games); + total(&run(aggressor, cfg, &mut l, seed ^ 0xA55A, games, games)) +} + +/// The confirmatory family, pre-registered before the run that reports it. +/// +/// Pilot (2026-10-10, seeds 1001..1012, 100 games, whole-run score) was +/// exploratory and amended three things before this confirmation: +/// H4 is compared at cost weight 0 (a cost-aware chooser reacts to the +/// cheaper ASC, so games diverge for a reason that is not the certificate); +/// the endpoint is the last 100-game window of 300 (the pilot measured the +/// learning phase); calibration is also read over non-neutral outcomes. +/// Confirmation uses fresh seeds 2001.. (D-MOORE-NARS-0 used 11, 22, 33). +const LAB_BLOCKS: usize = 16; +const LAB_GAMES: usize = 300; +const LAB_WINDOW: usize = 100; +const LAB_SEED0: u64 = 2001; +const LAB_PREREG: [lab::Prereg; 5] = [ + lab::Prereg { + id: "H3", + hypothesis: "delayed (one-reply) credit beats immediate credit", + baseline: "Immediate", + treatment: "Retained(1)", + endpoint: "score", + higher_is_better: true, + min_effect: 0.02, + blocks: LAB_BLOCKS, + }, + lab::Prereg { + id: "H4", + hypothesis: "certified minimax reduces folds per step (cost weight 0)", + baseline: "cost 0", + treatment: "cost 0 + certified", + endpoint: "folds/step", + higher_is_better: false, + min_effect: 1.0, + blocks: LAB_BLOCKS, + }, + lab::Prereg { + id: "H7", + hypothesis: "a fold-cost term reduces folds per step", + baseline: "cost 0", + treatment: "cost 0.02", + endpoint: "folds/step", + higher_is_better: false, + min_effect: 5.0, + blocks: LAB_BLOCKS, + }, + lab::Prereg { + id: "H2", + hypothesis: "the learned chooser beats the best fixed recipe", + baseline: "fixed HPM", + treatment: "learned", + endpoint: "score", + higher_is_better: true, + min_effect: 0.02, + blocks: LAB_BLOCKS, + }, + lab::Prereg { + id: "H5", + hypothesis: "evidence decay adapts faster after an opponent switch", + baseline: "decay 1.0", + treatment: "decay 0.95", + endpoint: "post-switch score", + higher_is_better: true, + min_effect: 0.02, + blocks: LAB_BLOCKS, + }, +]; + +fn lab_stage() { + let t0 = std::time::Instant::now(); + println!("D-SELF-CALIBRATING-LAB-0 — Gomoku arm (9x9 five vs Threat, oracle off)"); + println!( + "blocks {LAB_BLOCKS} (seeds {LAB_SEED0}..), {LAB_GAMES} games per arm per block, endpoint = last {LAB_WINDOW}\n" + ); + let seeds: Vec = (0..LAB_BLOCKS as u64).map(|i| LAB_SEED0 + i).collect(); + let arms: [(&str, Cfg); 6] = [ + ("base", BASE), + ( + "immediate", + Cfg { + signal: Signal::Immediate, + ..BASE + }, + ), + ( + "cost 0 + cert", + Cfg { + certified: true, + cost_weight: 0.0, + ..BASE + }, + ), + ( + "cost 0", + Cfg { + cost_weight: 0.0, + ..BASE + }, + ), + ( + "fixed HPM", + Cfg { + chooser: Chooser::Fixed(Op::Hpm), + ..BASE + }, + ), + ( + "decay 0.95", + Cfg { + decay: 0.95, + ..BASE + }, + ), + ]; + let mut per: Vec> = Vec::new(); + for (_, cfg) in &arms { + per.push( + seeds + .iter() + .map(|&s| lab_block(cfg, s, LAB_GAMES)) + .collect(), + ); + } + let switch_base: Vec = seeds + .iter() + .map(|&s| lab_switch_block(&BASE, s, LAB_GAMES)) + .collect(); + let switch_decay: Vec = seeds + .iter() + .map(|&s| lab_switch_block(&arms[5].1, s, LAB_GAMES)) + .collect(); + + let score = |v: &[Stats]| v.iter().map(Stats::score).collect::>(); + let fps = |v: &[Stats]| v.iter().map(Stats::folds_per_step).collect::>(); + + // H4's deterministic half: the certificate never changes a pick, so every + // block's games must be identical, not merely similar. + let identical = per[3] + .iter() + .zip(&per[2]) + .all(|(a, b)| a.wins == b.wins && a.losses == b.losses && a.moves == b.moves); + println!( + "H4 deterministic: certified and minimax games identical on every block: {identical} (deterministic)" + ); + + let results = [ + lab::paired(&score(&per[1]), &score(&per[0]), true), + lab::paired(&fps(&per[3]), &fps(&per[2]), false), + lab::paired(&fps(&per[3]), &fps(&per[0]), false), + lab::paired(&score(&per[4]), &score(&per[0]), true), + lab::paired(&score(&switch_base), &score(&switch_decay), true), + ]; + let rejected = lab::holm(&results.map(|r| r.p), 0.05); + println!("\nFamily (Holm, alpha 0.05):"); + for i in 0..LAB_PREREG.len() { + println!( + " {}", + lab::report(&LAB_PREREG[i], &results[i], rejected[i]) + ); + } + + println!("\nPareto (mean over blocks; empirical):"); + for (k, (name, _)) in arms.iter().enumerate() { + let sc = score(&per[k]).iter().sum::() / LAB_BLOCKS as f64; + let f = fps(&per[k]).iter().sum::() / LAB_BLOCKS as f64; + println!(" {name:<12} score {sc:.4} folds/step {f:7.1}"); + } + + let mut cal = Stats::default(); + for s in &per[0] { + cal.add(s); + } + for (label, bins) in [ + ("all outcomes", &cal.cal), + ("non-neutral outcomes", &cal.cal_nn), + ] { + let (bins, ece) = lab::reliability(bins); + println!( + "\nReliability of the learned chooser's expectation, {label} (base arm; empirical):" + ); + for b in bins.iter().filter(|b| b.n > 0) { + println!( + " [{:.1},{:.1}) n {:6} mean e {:.3} observed productive {:.3}", + b.lo, b.hi, b.n, b.mean_pred, b.observed + ); + } + println!(" ECE {ece:.4}"); + } + println!("\nlab wall time {:.1}s", t0.elapsed().as_secs_f64()); +} + fn main() { + if std::env::args().any(|a| a == "--lab") { + lab_stage(); + return; + } let t0 = std::time::Instant::now(); println!("D-MOORE-NARS-0: recipe learning on Gomoku\n"); println!("Census of the 34 catalogue recipes:"); @@ -1553,6 +1886,113 @@ mod tests { DIRS.iter().any(|&d| c.line(i, p, d).0 >= c.win) } + /// Positions where reasoning (not propagation) decides: played from the + /// empty 9x9 board by a noisy threat player, stopped at a random depth. + fn reasoning_corpus(count: usize) -> Vec<(Board, u8)> { + let mut out = Vec::new(); + let mut rng = Rng(0x5EED_0001); + while out.len() < count { + let mut b = Board::new(9, 5); + let depth = 4 + rng.below(30); + let mut me = 1u8; + let mut ok = true; + for _ in 0..depth { + let mut f = 0; + let s = snapshot(&b, me, &mut f); + if s.cells.is_empty() { + ok = false; + break; + } + let i = if rng.below(10) < 7 { + opp_move(&b, me, Opp::Threat) + } else { + s.cells[rng.below(s.cells.len())].i + }; + b.play(i, me); + if DIRS.iter().any(|&d| b.line(i, me, d).0 >= b.win) { + ok = false; + break; + } + me = 3 - me; + } + if !ok { + continue; + } + let mut f = 0; + let s = snapshot(&b, me, &mut f); + if s.cells.len() >= 2 && propagate(&s).is_none() { + out.push((b, me)); + } + } + out + } + + /// D-SELF-CALIBRATING-LAB-0, H4's deterministic half: inside the learner, + /// the certificate changes no game when the chooser is cost-blind. With + /// a cost term the cheaper ASC changes what the chooser picks, so games + /// may diverge; that interaction is pinned too, so a future reading of + /// "identical" is not made without the cost-weight condition. + #[test] + fn the_certificate_changes_no_game_for_a_cost_blind_learner() { + let pairs = |cost: f64| -> Vec { + (0..4u64) + .map(|i| { + let off = Cfg { + cost_weight: cost, + ..BASE + }; + let on = Cfg { + certified: true, + ..off + }; + let a = lab_block(&off, 3001 + i, 200); + let b = lab_block(&on, 3001 + i, 200); + a.wins == b.wins && a.losses == b.losses && a.moves == b.moves + }) + .collect() + }; + assert!(pairs(0.0).iter().all(|&x| x), "cost 0: identical games"); + assert!( + !pairs(0.02).iter().all(|&x| x), + "cost 0.02: the chooser reacts to the cheaper ASC" + ); + } + + /// D-LAB-P1: the certificate never changes the pick, and it saves folds. + #[test] + fn certified_minimax_picks_what_minimax_picks_with_fewer_folds() { + let corpus = reasoning_corpus(400); + // Measured, deterministic: (k1, k2, full folds, certified folds). + const PINNED: [(usize, usize, u64, u64); 3] = [ + (4, 4, 442_720, 282_480), + (5, 5, 665_076, 367_344), + (8, 8, 1_601_258, 624_386), + ]; + for (k1, k2, pin_full, pin_cert) in PINNED { + let (mut full, mut cert, mut cut_positions) = (0u64, 0u64, 0usize); + for (b, me) in &corpus { + let mut f0 = 0; + let s = snapshot(b, *me, &mut f0); + let (mut ff, mut fc) = (0u64, 0u64); + let a = minimax(b, *me, &s, k1, k2, &mut ff); + let c = minimax_certified(b, *me, &s, k1, k2, &mut fc); + assert_eq!(a, c, "the certificate changed the pick (k {k1}/{k2})"); + assert!(fc <= ff); + if fc < ff { + cut_positions += 1; + } + full += ff; + cert += fc; + } + eprintln!( + "P1 k {k1}/{k2}: folds {full} -> {cert} ({:.1}% saved), cut on {cut_positions}/{} positions", + 100.0 * (1.0 - cert as f64 / full as f64), + corpus.len() + ); + assert_eq!((full, cert), (pin_full, pin_cert), "k {k1}/{k2}"); + } + } + #[test] fn census_covers_the_catalogue_exactly() { let c = census(); @@ -1609,6 +2049,7 @@ mod tests { immediate: 0, dir: 2, question: CausalMask::SO, + pred: 0.5, }; let epi = Ev::default().state(Topology2::Direct); for a in [-7, 0, 3] { diff --git a/crates/cognitive-shader-driver/examples/shared/lab.rs b/crates/cognitive-shader-driver/examples/shared/lab.rs new file mode 100644 index 000000000..56bdf94a4 --- /dev/null +++ b/crates/cognitive-shader-driver/examples/shared/lab.rs @@ -0,0 +1,311 @@ +//! D-SELF-CALIBRATING-LAB-0 — the statistics every lab probe reports through. +//! +//! A task probe runs a strategy on matched blocks (seeds, puzzles) and hands +//! this module one outcome per (arm, block). Nothing here knows the task. +//! +//! - **Paired comparison** on matched blocks: the per-block difference, an +//! exact sign-flip p-value (no distributional assumption; exhaustive up to +//! 20 blocks), and a t-interval (which does assume roughly normal block +//! differences, and says so). +//! - **Holm** correction across a pre-registered family. +//! - **Verdict** per pre-registration: SUPPORTED needs the Holm rejection, +//! the declared direction and a mean effect of at least the declared +//! minimum. FALSIFIED needs the interval to rule the minimum effect out. +//! Everything else is INCONCLUSIVE. +//! - **Reliability** of predicted probabilities: bins, Brier score, ECE. +//! +//! Every result carries the kind of certificate it is. A statistical verdict +//! is never a deterministic guarantee, and an empirical number is neither. +#![allow(dead_code)] + +/// What kind of claim a result is. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum CertKind { + /// Proved bound; holds for every input within its assumptions. + Deterministic, + /// Probability statement under stated assumptions (test, interval). + Statistical, + /// A measured number, no guarantee. + Empirical, +} + +/// A hypothesis, fixed before the confirmatory run. +#[derive(Debug, Clone, Copy)] +pub struct Prereg { + pub id: &'static str, + pub hypothesis: &'static str, + pub baseline: &'static str, + pub treatment: &'static str, + pub endpoint: &'static str, + /// Larger endpoint values are better. + pub higher_is_better: bool, + /// Smallest effect worth acting on, in endpoint units. + pub min_effect: f64, + /// Number of matched blocks the run must have. + pub blocks: usize, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Verdict { + Supported, + Falsified, + Inconclusive, +} + +/// Paired comparison of treatment against baseline on matched blocks. +#[derive(Debug, Clone, Copy)] +pub struct Paired { + pub n: usize, + /// Mean of (treatment − baseline), signed so that positive = better. + pub mean: f64, + pub sd: f64, + /// Exact two-sided sign-flip p-value. + pub p: f64, + /// 95 % t-interval for `mean`. + pub lo: f64, + pub hi: f64, +} + +/// Two-sided 97.5 % t quantiles for df 1..=30; df > 30 uses 1.96. +const T975: [f64; 30] = [ + 12.706, 4.303, 3.182, 2.776, 2.571, 2.447, 2.365, 2.306, 2.262, 2.228, 2.201, 2.179, 2.160, + 2.145, 2.131, 2.120, 2.110, 2.101, 2.093, 2.086, 2.080, 2.074, 2.069, 2.064, 2.060, 2.056, + 2.052, 2.048, 2.045, 2.042, +]; + +/// Exact two-sided sign-flip p-value of the mean of `d` (n ≤ 20). +pub fn sign_flip_p(d: &[f64]) -> f64 { + assert!( + d.len() <= 20, + "exhaustive sign flip is limited to 20 blocks" + ); + let obs = d.iter().sum::().abs(); + let n = d.len(); + let mut extreme = 0u64; + for mask in 0u64..(1 << n) { + let s: f64 = d + .iter() + .enumerate() + .map(|(i, &x)| if mask >> i & 1 == 1 { -x } else { x }) + .sum(); + if s.abs() >= obs - 1e-9 { + extreme += 1; + } + } + extreme as f64 / (1u64 << n) as f64 +} + +/// Compare `treatment` with `baseline`, block for block. +pub fn paired(baseline: &[f64], treatment: &[f64], higher_is_better: bool) -> Paired { + assert_eq!(baseline.len(), treatment.len(), "blocks must be matched"); + let sign = if higher_is_better { 1.0 } else { -1.0 }; + let d: Vec = baseline + .iter() + .zip(treatment) + .map(|(b, t)| sign * (t - b)) + .collect(); + let n = d.len(); + let mean = d.iter().sum::() / n as f64; + let var = if n > 1 { + d.iter().map(|x| (x - mean).powi(2)).sum::() / (n - 1) as f64 + } else { + 0.0 + }; + let sd = var.sqrt(); + let t = if n >= 2 && n - 1 <= 30 { + T975[n - 2] + } else { + 1.96 + }; + let half = t * sd / (n as f64).sqrt(); + Paired { + n, + mean, + sd, + p: sign_flip_p(&d), + lo: mean - half, + hi: mean + half, + } +} + +/// Holm step-down over a family. Returns which hypotheses are rejected. +pub fn holm(p: &[f64], alpha: f64) -> Vec { + let m = p.len(); + let mut order: Vec = (0..m).collect(); + order.sort_by(|&a, &b| p[a].total_cmp(&p[b])); + let mut out = vec![false; m]; + for (rank, &i) in order.iter().enumerate() { + if p[i] <= alpha / (m - rank) as f64 { + out[i] = true; + } else { + break; + } + } + out +} + +/// The verdict a pre-registration earns from its paired result and its +/// Holm decision. +pub fn verdict(pr: &Prereg, r: &Paired, rejected: bool) -> Verdict { + if r.n != pr.blocks { + return Verdict::Inconclusive; + } + if rejected && r.mean >= pr.min_effect { + Verdict::Supported + } else if r.hi < pr.min_effect { + Verdict::Falsified + } else { + Verdict::Inconclusive + } +} + +/// One reliability bin: predictions in `[lo, hi)`. +#[derive(Debug, Clone, Copy)] +pub struct Bin { + pub lo: f64, + pub hi: f64, + pub n: u64, + pub mean_pred: f64, + pub observed: f64, +} + +/// Reliability from pre-binned counts `(n, Σ pred, hits)` over equal bins +/// on [0, 1]. Returns the bins, the expected calibration error, and the +/// Brier score lower bound computable from bins (exact Brier needs the +/// per-prediction values; see `brier`). +pub fn reliability(binned: &[(u64, f64, u64)]) -> (Vec, f64) { + let k = binned.len(); + let total: u64 = binned.iter().map(|b| b.0).sum(); + let mut out = Vec::with_capacity(k); + let mut ece = 0.0; + for (i, &(n, sp, hits)) in binned.iter().enumerate() { + let (mean_pred, observed) = if n == 0 { + (0.0, 0.0) + } else { + (sp / n as f64, hits as f64 / n as f64) + }; + if n > 0 && total > 0 { + ece += n as f64 / total as f64 * (mean_pred - observed).abs(); + } + out.push(Bin { + lo: i as f64 / k as f64, + hi: (i + 1) as f64 / k as f64, + n, + mean_pred, + observed, + }); + } + (out, ece) +} + +/// Brier score of `(prediction, outcome)` pairs. +pub fn brier(pairs: &[(f64, bool)]) -> f64 { + pairs + .iter() + .map(|&(p, y)| (p - if y { 1.0 } else { 0.0 }).powi(2)) + .sum::() + / pairs.len().max(1) as f64 +} + +/// One printed line per pre-registration. +pub fn report(pr: &Prereg, r: &Paired, rejected: bool) -> String { + format!( + "{:<6} {:<34} {} vs {}: mean {:+.4} [{:+.4}, {:+.4}] p {:.4} holm {} -> {:?} (statistical)", + pr.id, + pr.endpoint, + pr.treatment, + pr.baseline, + r.mean, + r.lo, + r.hi, + r.p, + if rejected { "reject" } else { "keep" }, + verdict(pr, r, rejected) + ) +} + +#[cfg(test)] +mod lab_tests { + use super::*; + + #[test] + fn sign_flip_p_is_exact_on_a_small_case() { + // All four diffs positive: only the all-plus and all-minus flips are + // as extreme, 2 of 16. + assert!((sign_flip_p(&[1.0, 2.0, 3.0, 4.0]) - 2.0 / 16.0).abs() < 1e-12); + // A symmetric set has p = 1. + assert!((sign_flip_p(&[1.0, -1.0]) - 1.0).abs() < 1e-12); + } + + #[test] + fn holm_matches_the_textbook_step_down() { + // m = 4, alpha 0.05: thresholds 0.0125, 0.0167, 0.025, 0.05. + assert_eq!( + holm(&[0.010, 0.040, 0.030, 0.005], 0.05), + vec![true, false, false, true] + ); + // Both pass their step thresholds (0.025, then 0.05). + assert_eq!(holm(&[0.03, 0.001], 0.05), vec![true, true]); + // Stops at the first failure: 0.03 > 0.025 halts, so 0.04 is kept + // even though it is below its own threshold 0.05. + assert_eq!(holm(&[0.001, 0.03, 0.04], 0.05), vec![true, false, false]); + assert_eq!(holm(&[0.06, 0.001], 0.05), vec![false, true]); + } + + #[test] + fn a_planted_effect_is_found_and_a_null_is_not() { + let base: Vec = (0..12).map(|i| 0.5 + 0.01 * (i % 3) as f64).collect(); + let better: Vec = base.iter().map(|x| x + 0.05).collect(); + let r = paired(&base, &better, true); + assert!(r.p < 0.001 && r.mean > 0.049); + let pr = Prereg { + id: "T", + hypothesis: "", + baseline: "b", + treatment: "t", + endpoint: "e", + higher_is_better: true, + min_effect: 0.02, + blocks: 12, + }; + assert_eq!(verdict(&pr, &r, true), Verdict::Supported); + // Null: alternating ± noise, no effect. + let noisy: Vec = base + .iter() + .enumerate() + .map(|(i, x)| x + if i % 2 == 0 { 0.01 } else { -0.01 }) + .collect(); + let r0 = paired(&base, &noisy, true); + assert!(r0.p > 0.5); + assert_eq!( + verdict(&pr, &r0, holm(&[r0.p], 0.05)[0]), + Verdict::Falsified + ); + // Direction matters: "lower is better" flips the sign. + assert!(paired(&base, &better, false).mean < 0.0); + // Significant but below the minimum effect: never SUPPORTED. A tiny + // shift on every block is rejected by Holm, and its interval sits + // under 0.02, so the verdict is FALSIFIED. + let tiny: Vec = base.iter().map(|x| x + 0.005).collect(); + let rt = paired(&base, &tiny, true); + assert!(rt.p < 0.001); + assert_eq!(verdict(&pr, &rt, true), Verdict::Falsified); + // A run with the wrong block count is never SUPPORTED. + let short = paired(&base[..6], &better[..6], true); + assert_eq!(verdict(&pr, &short, true), Verdict::Inconclusive); + } + + #[test] + fn reliability_sees_calibration_and_miscalibration() { + // Perfectly calibrated: bin with mean 0.25 hits 25 %. + let (_, ece) = reliability(&[(0, 0.0, 0), (0, 0.0, 0), (4, 1.0, 1), (0, 0.0, 0)]); + assert!(ece < 1e-12); + // Overconfident: predicts 0.9, hits 50 %. + let mut bins = [(0u64, 0.0f64, 0u64); 10]; + bins[9] = (10, 9.0, 5); + let (_, ece) = reliability(&bins); + assert!((ece - 0.4).abs() < 1e-12); + assert!((brier(&[(1.0, true), (0.0, false)])).abs() < 1e-12); + assert!((brier(&[(1.0, false)]) - 1.0).abs() < 1e-12); + } +} diff --git a/crates/lance-graph-planner/src/chain_counterfactual.rs b/crates/lance-graph-planner/src/chain_counterfactual.rs index 2142a8e8f..443c6c986 100644 --- a/crates/lance-graph-planner/src/chain_counterfactual.rs +++ b/crates/lance-graph-planner/src/chain_counterfactual.rs @@ -94,18 +94,17 @@ pub enum Verdict { /// derived number) — but the choice of **frequency over confidence is /// measured, and it matters**. /// -/// `NarsTables::build(1)` drives revision's confidence to a fixed point: -/// measured across a weak 3-chain, a strong 4-chain, and a mixed one, the -/// terminal confidence was **170 in every case**, while frequency separated -/// them cleanly (78 / 247 / 93). A confidence-based verdict would therefore -/// have been a **vacuous threshold** — every chain on the same side of every -/// bar, a gate that cannot fire and cannot stay silent, discriminating -/// nothing while looking rigorous. +/// The reason is semantic: "is this chain consistent?" asks how strongly the +/// composed relation holds (frequency), not how much evidence stands behind +/// it (confidence). A verdict read off confidence would call a well-attested +/// refutation "consistent". /// -/// It is also the semantically right axis: "is this chain consistent?" asks -/// how strongly the composed relation holds, not how much evidence has piled -/// up behind it. Confidence saturating is the substrate working as designed; -/// reading a verdict off it was the error. +/// Corrected 2026-10-10 (D-RPF-TABLE-1): this doc used to cite a measured +/// "terminal confidence of 170 in every chain" as the evidence. 170 is not +/// saturation: it is `NarsTables::build(1)` discarding both confidences and +/// returning one constant. Under exact revision (law A) confidence varies +/// along every chain, so that measurement says nothing about confidence as a +/// verdict axis. The choice of frequency stands on the semantic reason alone. /// /// [`the_bar_is_not_inert`] pins that the knob does something in BOTH /// directions — the inertness test a threshold parameter owes. @@ -642,9 +641,9 @@ mod tests { } /// The bar is a real knob, not decoration: raising it must silence a /// verdict that fires, and lowering it must admit one that does not. The - /// inertness test a threshold parameter owes — and the test that would - /// have caught the confidence axis, where every chain sat at 170 and no - /// bar could separate anything. + /// inertness test a threshold parameter owes. (This fixture runs under + /// `NarsTables::build(1)`, which returns one constant confidence, 170; see + /// `DEFAULT_FREQUENCY_BAR` for why that is not evidence about confidence.) #[test] fn the_bar_is_not_inert() { let (tables, t) = fixture(); @@ -671,14 +670,11 @@ mod tests { "lowering admits" ); - // ...and the axis matters: the SAME trace's confidence is saturated, - // so a confidence bar cannot discriminate. Pinned so nobody switches - // the verdict back onto it. + // B1 (`build(1)`) returns one constant confidence. Pinned as a + // property of B1, not of revision: under law A the confidence varies + // (D-RPF-TABLE-1). If the fixture's table changes, this moves. let c = trace.last().unwrap().edge.confidence_u8(); - assert_eq!( - c, 170, - "confidence saturates — measured across every fixture here" - ); + assert_eq!(c, 170, "B1's constant confidence"); // An empty replay asserts nothing and fails closed at any bar. assert_eq!(verdict_at(&[], 0), Verdict::Inconsistent); diff --git a/crates/lance-graph-planner/tests/d_rpf_table_1.rs b/crates/lance-graph-planner/tests/d_rpf_table_1.rs index 30d5402da..0ed03d192 100644 --- a/crates/lance-graph-planner/tests/d_rpf_table_1.rs +++ b/crates/lance-graph-planner/tests/d_rpf_table_1.rs @@ -847,6 +847,40 @@ fn the_load_bearing_fixture_is_load_bearing_only_under_b1() { assert_eq!(lb, vec![true, false, false, false, false]); } +/// G2 re-derivation: a load-bearing fixture under law A, the law +/// recommended for decision-bearing wiring. Seed 200/200, a strong support +/// step 250/250, then a confident refutation 10/230. Cutting the support +/// leaves the refutation to dominate, so the verdict flips under exact +/// revision. B1's verdict on the same chain is pinned beside it as the +/// historical record, not as the contract. +#[test] +fn a_load_bearing_fixture_under_law_a() { + let c = compose_tables(); + let compose = compose_of(&c); + let ch = Chain { + seed: edge(200, 200, InferenceType::Deduction), + steps: vec![ + (0x91, edge(250, 250, InferenceType::Deduction)), + (0x92, edge(10, 230, InferenceType::Deduction)), + ], + }; + let a = decide(&law_a(), &ch, 0, compose); + assert_eq!( + a.reaction, + Reaction::LoadBearing { + factual: 210, + counterfactual: 64 + } + ); + // The mirror image of the old fixture: B1, B2, B4 and B8 call this cut + // NOT load-bearing; only B16 agrees with A. + let lb: Vec = tables() + .iter() + .map(|t| decide(&law_table(t), &ch, 0, compose).load_bearing) + .collect(); + assert_eq!(lb, vec![false, false, false, false, true]); +} + /// Anti-vacuity for the flip counter: a counter that compared only /// frequencies would miss TruthOnly decisions that come from confidence /// alone. This fixture has A moving confidence with frequency held, so a