diff --git a/docs/W7_3_FUZZ_BASELINE.md b/docs/W7_3_FUZZ_BASELINE.md index 0fc09a48..c8f0cc61 100644 --- a/docs/W7_3_FUZZ_BASELINE.md +++ b/docs/W7_3_FUZZ_BASELINE.md @@ -1,8 +1,10 @@ # W7.3 E1+E2 baseline — parse-invariance run -Status: **BASELINE LANDED** (2026-07-05). -Branch: `w7/testing/fuzz-baseline` (PR #46). -Provenance: `tests/fuzz/grammar_v2/{Cargo.toml,src/gen.rs,roundtrip.py}` @ this commit. +Status: **FROZEN SUBSET BASELINE LANDED** (2026-07-05, via PR #46 @ `3272583`) + **EXPANDED BASELINE (collection-params)** landed 2026-07-05 (this commit). +Frozen-subset provenance: `tests/fuzz/grammar_v2/{Cargo.toml,src/gen.rs,roundtrip.py}` @ `3272583`. +Expanded provenance: same paths @ this commit (base `main` @ `3272583` + `\b`-fix + collection-params). + +Two baselines are reported side-by-side. The frozen subset remains the reference citation-point; the expanded run adds coverage of collection-typed parameter positions (isolate-variables: only collection-params, not yet Call/If/Index). ## Setup @@ -70,18 +72,71 @@ The E1 grammar subset does not exercise function parameters. The W6.2 audit foun Between now and that deadline, E1 grammar expansion is the primary open workstream on this branch. +## Expanded baseline — collection-params increment (N=1000, seed range `0xC0FFEE..0xC0FFEE+999`) + +### Corpus-mismatch resolution and syntax choice + +An earlier revision of this section reported a `[]const T` / `[]T` / `[N]T` (Zig-style) syntax. That syntax was chosen from a workspace-wide `grep`, which included a parallel non-tri-net spec corpus at `../t27/specs/` (830 `[]const T` occurrences). Peer-review flagged the mismatch: the audit corpus targeted by W6.2 lives at `tri-net/specs` on the PR #39 branch (`feat/strategic-audit-2026-07-04`), and contains 0 `[]const T` occurrences. Its collection syntax is Rust-style `[T; NAMED_CONST]` with module-scope const-decls — 159 total occurrences across 68 spec files, 100% `u32` element type. Top declared consts by count: `MAX_NODES` (29), `MAX_PARAMS` (18), `MAX_METRICS` (12), `MAX_FLOWS` (11), and so on. + +This revision replaces the Zig-style forms with the Rust-style form actually present in the tri-net audit corpus. Zig-style forms were dropped entirely. Element type is fixed at `u32` (matches 100% of audit-corpus occurrences). Anchor recorded: any claim verified against ground truth requires scoping the verification tool to the same corpus as the claim. + +### Scope of change vs frozen subset + +Function parameter lists (0–4 per fn) with mixed scalar and collection-typed params. One collection form: `[u32; NAMED_CONST]` where `NAMED_CONST` is drawn from a fixed 10-name pool matching the audit-corpus top-10 (`MAX_NODES`, `MAX_PARAMS`, `MAX_METRICS`, `MAX_FLOWS`, `MAX_ENTRIES`, `MAX_MODULES`, `MAX_TASKS`, `MAX_FUNCTIONS`, `MAX_SAMPLES`, `MAX_RESULTS`). Each module emits 1–4 module-scope `const NAME: u32 = ;` declarations before its fns, with literals in `[2, 32]`. Collection-typed params reference only consts declared in the same module (tracked via `Ctx.declared_consts`). Isolation constraint retained — collection-typed idents are recorded in `Ctx.coll_params` (signature only) and NOT pushed into `Ctx.idents` (which feeds `gen_expr`'s scalar pool). + +Call / Index / If are deferred to separate commits per isolate-variables discipline. + +### Results + +| Metric | Frozen subset (PR #46 @ `3272583`) | Expanded (`[u32; NAMED_CONST]`) | +|---|---|---| +| Parse-success | 1000 / 1000 (100.0%) | **1000 / 1000 (100.0%)** | +| Determinism | 1000 / 1000 (100.0%) | **1000 / 1000 (100.0%)** | +| Whitespace-invariance | 1000 / 1000 (100.0%) | **1000 / 1000 (100.0%)** | +| Parse errors | 0 | **0** | +| Panics | 0 | **0** | +| Non-determinism | 0 | **0** | +| Corpus size | ~3.6 MB | 4.0 MB | +| Fns emitted | ~1900 (0-params only) | 1951 (mix 0-4 params) | +| Fns with ≥1 collection-param | 0 | 1177 (60.3%) | +| Fns with 0 params | ~1900 | 392 (20.1%) | +| Const-decls emitted (module-scope) | 0 | 2510 (avg 2.5 per module) | + +### Coverage delta + +- 1924 `[u32; NAMED_CONST]` occurrences across the corpus (avg ~1 per fn), distributed across all 10 named-const identifiers with roughly uniform weight (166–224 per name). +- 2510 module-scope `const NAME: u32 = ;` declarations, values drawn from `[2, 32]`, 1–4 per module, no repeats within a module. +- Every collection-typed param references a const declared in the enclosing module — no dangling references by construction. +- Body of every fn still exercises only scalar operations (isolation constraint holds by construction — `gen_expr` never sees a collection-typed ident). + +### Interpretation + +**What this expanded claim IS**: t27c's parser accepts `[u32; NAMED_CONST]` collection params (the syntactic form actually present in the tri-net audit corpus) together with module-scope const-decls, with the same 100% parse / determinism / whitespace-invariance behavior as the scalar-only subset. Adding param-position variety and const-decls did not introduce any new invariance failures. The parser's param-list handling and const-decl handling are whitespace-robust for the tested forms. + +**What this expanded claim IS NOT**: +- Not a claim that collection *values* are handled correctly — bodies still only touch scalar idents. Index / Call / If are the next increments. +- Not a differential test — still parser self-consistency only. E3 still timer-blocked (backstop 2026-07-19 12:24 UTC per PR #44). +- Not coverage of the W6.2 Class 2 defect surface *in operation* — that requires Index expressions to actually reference collection-typed params. The current increment establishes param-position parser-exercise plus const-decl parser-exercise; Index will drive body-exercise. +- Not coverage of Zig-style collection forms (`[]const T`, `[]T`, `[N]T`) — those are absent from the tri-net audit corpus and were dropped. If a future audit target introduces them, they will be re-added as a separate increment. + +### Frozen citation-point preserved + +The frozen subset baseline (PR #46 @ `3272583`, N=1000, 100/100/0 on the pre-params grammar) remains the reference point. Any future expansion whose invariance signal drops from 100% can be cited against both this expanded number and the frozen subset — the pair localizes whether the drop came from the newly-added grammar region or from the pre-existing subset. + ## Reproducibility ```bash # From tri-net workspace root. cd tests/fuzz/grammar_v2 cargo build --release -W73_OUT=/tmp/w73_baseline_1000 ./target/release/gen 1000 0xC0FFEE +W73_OUT=/tmp/w73_expanded_v2_1000 ./target/release/gen 1000 0xC0FFEE cd ../../.. -python3 tests/fuzz/grammar_v2/roundtrip.py /tmp/w73_baseline_1000 --out /tmp/w73_baseline_1000_report.json +python3 tests/fuzz/grammar_v2/roundtrip.py /tmp/w73_expanded_v2_1000 --out /tmp/w73_expanded_v2_1000_report.json ``` -Expected: `ok=1000 parse_err=0 mut_fail=0 non_det=0`, elapsed <15 sec on a modern x86_64 sandbox. +Expected on the expanded generator (this commit): `ok=1000 parse_err=0 mut_fail=0 non_det=0`, elapsed ~11 sec on a modern x86_64 sandbox. + +To reproduce the frozen subset baseline (PR #46 @ `3272583`), check out that commit and run the same command against `/tmp/w73_baseline_1000`: `ok=1000 parse_err=0 mut_fail=0 non_det=0`, ~10 sec. ## Anchor diff --git a/docs/W7_3_FUZZ_BASELINE_PLAN.md b/docs/W7_3_FUZZ_BASELINE_PLAN.md index 6c026b05..d5ad3b92 100644 --- a/docs/W7_3_FUZZ_BASELINE_PLAN.md +++ b/docs/W7_3_FUZZ_BASELINE_PLAN.md @@ -76,10 +76,10 @@ E1+E2 baseline считается **зелёным**, если: GLM-5.2 peer-review PR #46 @ 6c0c93d выявил coverage-gap: текущий E1 эмитит zero-param functions. Но W6.2 audit нашёл Vec<>-defect (E0107 Class 2) в **param-position** — E3 backend-differential будет слепым к этому дефекту, если grammar не расширить. Backstop таймер t27#1401 = 2026-07-19 12:24 UTC (14 дней). За это окно: -- [ ] Extend `gen_fn` to emit **function parameters** (от 0 до 4, mixed primitives + хотя бы один collection type). -- [ ] Add `Call` expression to `gen_expr` с recursion на другие генерируемые функции. -- [ ] Add `Index` expression если grammar поддерживает (верифицировать через `parse` на minimal specs). -- [ ] Add `If` statement branching (grammar уже в plan’e, но в code нет). +- [x] Extend `gen_fn` to emit **function parameters** (0–4, mixed scalar + collection types). **LANDED** this commit — collection-params in signature-position with isolation constraint (scalar-eligible pool untouched to prevent ill-typed body expressions). Expanded baseline N=1000 = 100/100/0. See `W7_3_FUZZ_BASELINE.md` §Expanded baseline. +- [ ] Add `Index` expression (`arr[i]`) to make collection-params live in body — the actual W6.2 Class 2 defect-surface exercise. Isolate-variables: land alone in a separate commit, re-baseline, then next. +- [ ] Add `Call` expression to `gen_expr` с recursion на другие генерируемые функции. Requires fn-signature registry in `Ctx` (arg-type matching to param-types). +- [ ] Add `If` statement branching (grammar уже в plan’e, но в code нет). Terminal-Return invariant must extend to both branches. - [ ] Reduce dead-let частоту (вес ident-branch в `gen_expr` → 40%+). Статус обновлять в этом файле по мере выполнения. diff --git a/tests/fuzz/grammar_v2/roundtrip.py b/tests/fuzz/grammar_v2/roundtrip.py index 0c292531..b4298eb0 100755 --- a/tests/fuzz/grammar_v2/roundtrip.py +++ b/tests/fuzz/grammar_v2/roundtrip.py @@ -49,7 +49,7 @@ def normalize_ast(ast_text: str) -> str: If a whitespace mutation changes the AST after this normalization, it is a real structural change, not a metadata artifact. """ - stripped = re.sub(r"line:\s*\d+,?", "", ast_text) + stripped = re.sub(r"\bline:\s*\d+,?", "", ast_text) return re.sub(r"\s+", " ", stripped).strip() diff --git a/tests/fuzz/grammar_v2/src/gen.rs b/tests/fuzz/grammar_v2/src/gen.rs index 2d0c09dc..027cfc24 100644 --- a/tests/fuzz/grammar_v2/src/gen.rs +++ b/tests/fuzz/grammar_v2/src/gen.rs @@ -7,14 +7,43 @@ // round-trip (E2) and backend-differential (E3, blocked on upstream Stmt::Let) // analysis. This binary only generates; parsing / round-tripping lives in E2. // -// Grammar coverage (initial subset): -// Module ::= UseDecl* ConstDecl* FnDecl+ -// FnDecl ::= "fn" ident "(" params? ")" ("->" Type)? "{" Stmt+ Return "}" -// Stmt ::= LetStmt | IfStmt | ExprStmt -// LetStmt ::= "let" ident ":" Type "=" Expr ";" -// IfStmt ::= "if" "(" Expr ")" "{" Stmt* Return "}" ("else" "{" Stmt* Return "}")? -// Expr ::= Literal | Ident | BinOp | Cast -// Type ::= u8 | u16 | u32 | u64 | usize | bool +// Grammar coverage (initial subset + collection-params expansion): +// Module ::= "module" ident "{" ConstDecl* FnDecl+ "}" +// ConstDecl ::= "const" IDENT ":" "u32" "=" IntLiteral ";" +// FnDecl ::= "fn" ident "(" Params? ")" ("->" Type)? "{" Stmt+ Return "}" +// Params ::= Param ("," Param)* +// Param ::= ident ":" (Type | CollType) +// CollType ::= "[" "u32" ";" IDENT "]" // [u32; NAMED_CONST] +// Stmt ::= LetStmt | IfStmt | ExprStmt +// LetStmt ::= "let" ident ":" Type "=" Expr ";" +// Expr ::= Literal | Ident | BinOp | Cast +// Type ::= u8 | u16 | u32 | u64 | usize | bool +// IntType ::= u8 | u16 | u32 | u64 | usize +// +// Collection-params grammar (W7.3 target #1) — CORRECTED per PR #47 GLM +// re-review @ 247427d + ground-truth verification against tri-net/specs +// on PR #39 branch (feat/strategic-audit-2026-07-04): +// +// - Corpus of interest: tri-net/specs/*.t27 (68 modules, W6.2 audit corpus) +// - Only collection form observed there: [u32; NAMED_CONST] (159 occurrences) +// - Element type always u32 +// - Named-const size always resolves via a module-scope `const NAME: u32 = ...` +// - This is the exact syntactic path that t27c lowers to `Vec<>` in gen/rust +// (see specs/anomaly_detector.t27:68 -> gen/rust/anomaly_detector.rs +// calculate_baseline(history: Vec<>, count: u32)). +// +// Zig-style forms ([]const T / []T / [N]T literal) previously emitted by +// this generator were dropped: they exist in the parallel ../t27/ Zig +// backend tree but NOT in tri-net/specs, and they exercise different +// parser/lowering paths than the Vec<>-defect surface the audit targets. +// +// Isolation constraint (unchanged from initial expansion): +// Collection-typed params are declared in the signature (parser-exercise +// of param-position for the Vec<>-defect class W6.2 Class 2 surface), +// but are NOT pushed into the scalar ident pool used by gen_expr. This +// prevents ill-typed expressions (e.g. vec + 1u32) that t27c would +// reject and would confound the whitespace-invariance signal. Body-use +// of collections (Index expressions) is deferred to a separate commit. // // Depth-bounded (max_depth = 6). Seed-reproducible via ChaCha20 RNG. // See docs/W7_3_FUZZ_BASELINE_PLAN.md. @@ -32,11 +61,42 @@ const INT_TYPES: &[&str] = &["u8", "u16", "u32", "u64", "usize"]; const BIN_OPS: &[&str] = &["+", "-", "*", "&", "|", "^", "<<", ">>"]; const CMP_OPS: &[&str] = &["==", "!=", "<", "<=", ">", ">="]; +// Named-const pool for [u32; NAMED_CONST] collection params. Names match +// tri-net/specs vocabulary (top-10 by occurrence: MAX_NODES 29, MAX_PARAMS 18, +// MAX_METRICS 12, MAX_FLOWS 11, MAX_ENTRIES 11, MAX_MODULES 10, MAX_TASKS 7, +// MAX_FUNCTIONS 7, MAX_SAMPLES 6, MAX_RESULTS 6). Values match the range +// observed in real const-decls (2..32). +const NAMED_CONST_POOL: &[&str] = &[ + "MAX_NODES", + "MAX_PARAMS", + "MAX_METRICS", + "MAX_FLOWS", + "MAX_ENTRIES", + "MAX_MODULES", + "MAX_TASKS", + "MAX_FUNCTIONS", + "MAX_SAMPLES", + "MAX_RESULTS", +]; + struct Ctx { rng: ChaCha20Rng, + // Scalar idents visible to gen_expr (BinOp/Cast eligible). Populated by + // let-stmts and by scalar-typed params. Collection-typed params live in + // `coll_params` instead — see isolation constraint at top of file. idents: Vec<(String, String)>, // (name, type) + // Collection-typed params: name + rendered type string (only "[u32; NAME]" + // form is emitted). Recorded for signature emission only. Not exposed to + // gen_expr — collection idents are dead in the body pending Index support. + coll_params: Vec<(String, String)>, + // Module-scope const-decls emitted for this module. Any [u32; NAME] used + // in a param signature must reference a NAME that is declared here. + // Populated by gen_module before any fn is generated. + declared_consts: Vec<(String, u32)>, // (NAME, value) max_depth: u32, max_stmts_per_fn: u32, + max_params_per_fn: u32, + coll_param_prob: u32, // percent chance a param is a collection (else scalar) } impl Ctx { @@ -44,8 +104,12 @@ impl Ctx { Self { rng: ChaCha20Rng::seed_from_u64(seed), idents: Vec::new(), + coll_params: Vec::new(), + declared_consts: Vec::new(), max_depth: 6, max_stmts_per_fn: 20, + max_params_per_fn: 4, + coll_param_prob: 50, } } @@ -61,6 +125,13 @@ impl Ctx { fn pick_int_type(&mut self) -> String { INT_TYPES[self.rng.gen_range(0..INT_TYPES.len())].to_string() } + + /// Pick a random module-scope named const to reference. Assumes + /// `declared_consts` is non-empty (guarded by caller). + fn pick_const_name(&mut self) -> String { + let idx = self.rng.gen_range(0..self.declared_consts.len()); + self.declared_consts[idx].0.clone() + } } fn gen_literal(ctx: &mut Ctx, ty: &str) -> String { @@ -130,8 +201,47 @@ fn gen_return(ctx: &mut Ctx, ret_ty: &str) -> String { format!(" return {};", expr) } +/// Emit the parameter list. Mixes scalar and collection params. +/// +/// Scalar params are pushed into `ctx.idents` so `gen_expr` can reference +/// them in BinOp / Cast / return. Collection params ([u32; NAMED_CONST]) +/// are pushed into `ctx.coll_params` only — they appear in the signature +/// but are dead in the body. This is the isolation constraint that lets +/// whitespace-invariance signal reflect param-position parser behavior +/// without ill-typed body noise. +/// +/// A collection param can only be emitted if `ctx.declared_consts` is +/// non-empty (which it will be after `gen_module` initializes it). If +/// somehow empty, we fall back to a scalar param for that slot. +fn gen_params(ctx: &mut Ctx) -> String { + let n_params = ctx.rng.gen_range(0u32..=ctx.max_params_per_fn); + if n_params == 0 { + return String::new(); + } + + let mut parts: Vec = Vec::with_capacity(n_params as usize); + for _ in 0..n_params { + let name = ctx.fresh_ident("p"); + let want_coll = ctx.rng.gen_range(0u32..100) < ctx.coll_param_prob; + let can_coll = !ctx.declared_consts.is_empty(); + if want_coll && can_coll { + let const_name = ctx.pick_const_name(); + let ty = format!("[u32; {}]", const_name); + parts.push(format!("{}: {}", name, ty)); + ctx.coll_params.push((name, ty)); + } else { + let ty = ctx.pick_type(); + parts.push(format!("{}: {}", name, ty)); + ctx.idents.push((name, ty)); + } + } + parts.join(", ") +} + fn gen_fn(ctx: &mut Ctx, name: &str) -> String { ctx.idents.clear(); + ctx.coll_params.clear(); + let params = gen_params(ctx); let ret_ty = ctx.pick_type(); let n_stmts = ctx.rng.gen_range(1u32..=ctx.max_stmts_per_fn); @@ -143,13 +253,45 @@ fn gen_fn(ctx: &mut Ctx, name: &str) -> String { body.push_str(&gen_return(ctx, &ret_ty)); format!( - " fn {}() -> {} {{\n{}\n }}", - name, ret_ty, body + " fn {}({}) -> {} {{\n{}\n }}", + name, params, ret_ty, body ) } +/// Emit module-scope const-decls. Samples 1-4 names from `NAMED_CONST_POOL` +/// (no repeats within a module) and assigns each a literal u32 value in +/// [2, 32] — the range observed in tri-net/specs const-decls. Populates +/// `ctx.declared_consts` so subsequent `pick_const_name` calls only +/// reference names that were actually declared. +fn gen_const_decls(ctx: &mut Ctx) -> String { + ctx.declared_consts.clear(); + let n_consts = ctx.rng.gen_range(1u32..=4) as usize; + let n_consts = n_consts.min(NAMED_CONST_POOL.len()); + + // Sample without replacement via index shuffle. + let mut indices: Vec = (0..NAMED_CONST_POOL.len()).collect(); + for i in (1..indices.len()).rev() { + let j = ctx.rng.gen_range(0..=i); + indices.swap(i, j); + } + let picked: Vec<&str> = indices.iter().take(n_consts).map(|&i| NAMED_CONST_POOL[i]).collect(); + + let mut lines = Vec::with_capacity(n_consts); + for name in picked { + let value = ctx.rng.gen_range(2u32..=32); + ctx.declared_consts.push((name.to_string(), value)); + lines.push(format!(" const {}: u32 = {};", name, value)); + } + lines.join("\n") +} + fn gen_module(ctx: &mut Ctx, mod_idx: u32) -> String { let mod_name = format!("W73Fuzz{}", mod_idx); + // Emit const-decls FIRST so gen_params can reference them via + // ctx.declared_consts. This ordering matters — collection params depend + // on the const pool being populated. + let const_decls = gen_const_decls(ctx); + let n_fns = ctx.rng.gen_range(1u32..=3); let mut fns = Vec::new(); for i in 0..n_fns { @@ -162,9 +304,12 @@ fn gen_module(ctx: &mut Ctx, mod_idx: u32) -> String { \n\ module {} {{\n\ {}\n\ + \n\ + {}\n\ }}\n", mod_idx, mod_name, + const_decls, fns.join("\n\n") ) }