Five open issues, three of them misdiagnosed by their reports, and staging as a service (2026.9.11.2)

.agents/docs/2026-09-11-distribution-plugins-and-platform-decomposition.md

@@ -1001,3 +1001,149 @@ on the one above it being released rather than merely merged.
 
 The engine tasks are the only ones on the critical path. P1 and X1 do not wait
 on anything.
+
+## 11. The plan read from nine angles
+
+Section 9 reviewed the proposal on its own terms. This section reads the
+*implemented* result from the angles a reviewer would apply independently of it,
+because each angle catches a different class of mistake and several of them
+caught one.
+
+### 11.1 Architecture
+
+The load-bearing claim is that **the engine holds the dispatch and no format**.
+It survives one test the proposal did not anticipate: a format that consumes
+nothing. `dist-wix` packages one named program and never reads the staged tree,
+which is what §6's own guidance recommends — and the first implementation of the
+dispatch refused exactly that member, because it identified the distributable by
+"which action named `${mcpp.stage_dir}`". The criterion was a property of the
+*mechanism* rather than of the *request*. It is now "which artifact actions the
+request introduced", which needs nothing of the member.
+
+The same mistake occurred one layer down and was found by a real macOS runner:
+staging ran before the dispatch and its failure was fatal, so every dispatched
+format was unreachable on a target whose built-in bundling is refused. Staging
+is a service to the provider, not a precondition.
+
+Both are the same error in different clothes: **the engine deciding something on
+the provider's behalf.** That is the failure this architecture is most exposed
+to, because the whole point of it is that the provider decides.
+
+### 11.2 Stability
+
+Three axes now ride `build.ninja`'s header line — shape, schedule, device
+variant — and `dist=` is the fourth. Each was added after the same defect: a
+graph written for one purpose replayed for another, in a directory the two
+share. The format deliberately does not enter the fingerprint, because it would
+cost a full recompile to package an already-built tree; the header line is the
+cheaper half of that pair and is the half the fast paths ask.
+
+The measurement that matters is the one that says the criterion is worth
+having: on 2026-09-11 a plain build after a pack pass regenerates the graph
+*even with the field ignored*, so an end-to-end assertion would pass whether or
+not the field works. The unit test is where the invariant is held.
+
+### 11.3 Elegance
+
+Two additions were withdrawn as duplicates of something that ships.
+`--stage-only` is `--format dir`. And `rule_module` on a `dist-*` feature was
+refused by the engine, correctly: that key means "the module that reaches a
+rule" and implies `device_extensions`, which a member compiling nothing cannot
+have. The `dist-*` members take the `tools-*` shape instead, and the consumer
+writes one line more than a rule needs — which says something true.
+
+### 11.4 User experience
+
+`--format` gained values rather than a second flag, because `tar`, `dir`, `msi`
+and `appimage` answer one question. An unknown value names what *is* available
+rather than a fixed list, and the refusal arrives before anything is compiled.
+
+The failure mode this category is most exposed to is a step that succeeds while
+carrying nothing — §2's measured 52 KB installer. Every member therefore
+asserts a floor on the success path through `mcpp::warning`, because stderr on
+a successful build is discarded. The first such floor was a size bound and was
+wrong on its first real fixture: a stripped hello-world stages at 14999 bytes,
+under a 16 KB bound, so a correct AppImage was reported as empty. A size is a
+proxy for a question that can be asked directly.
+
+### 11.5 Compatibility
+
+The engine's rule is unchanged: no per-package floor exists, and the index-level
+`min_mcpp` does not move for a package, because raising it makes the whole index
+unreadable to clients stopped below it. What an older client gets is legible at
+the point of use, and for these members it is the best case of that rule —
+`mcpp::provides_pack_format` does not exist in an older engine's bundled module,
+so a consumer fails at the `build.mcpp` **compile**, naming the missing
+function, rather than at a link or in an artifact.
+
+### 11.6 Cross-platform
+
+Three members, three platforms, and the honest asymmetry is that only one of
+them could be measured where it was written. `plan_for()` returning
+`applies == false` on the wrong OS says the gate works and says nothing about
+whether the tool accepts what the member renders. That gap was closed by adding
+CI steps that actually run `wix build` and assemble a real `.app` — and the
+first thing they did was fail, twice, for unrelated reasons: WiX 7 refuses to
+run without an out-of-band licence acceptance (`WIX7015`), and the Mach-O
+staging refusal above. Both are findings the plan-level assertions could not
+have produced.
+
+### 11.7 Consistency
+
+`rules-*`, `tools-*` and `dist-*` is one taxonomy with one rule: the prefix
+says which of three questions a member answers. The engine's own three-value
+`ObjectFormat` is the same discipline applied to a fact rather than to a
+package — the binary format was re-derived at roughly 35 sites, which is
+affordable at two values and becomes an addition at every site at three, where
+a missed site silently answers ELF.
+
+Two sites answered the object format by searching for `"apple"` in a string
+that never contains it, so an explicit `--target aarch64-macos` — a *verified*
+row — linked and recorded as ELF while a native build on the same machine said
+`macho`. One function, two paths, only the exercised one right.
+
+### 11.8 Seamless upgrade
+
+Every new field is absent-tolerant in the direction that matters. An older
+graph's missing `dist=` reads as a miss and never as `none`. A build program on
+an older engine gets empty strings from the new accessors, which a member reads
+as "fall back to what you did before". The `pack-format` directive carries a
+non-empty cache tag, so a declaration survives a cache hit — the pass that
+reads it is `mcpp pack`, which is never a project's first build, and an
+unpersisted declaration would be absent exactly when a user names a format.
+
+`kCacheEpoch` is deliberately not bumped: an entry written before the row
+carries no such line and the program that wrote it could not emit one, so
+replaying it yields what that program said.
+
+### 11.9 Test coverage
+
+The count is not the measure; what each test excludes is. Two are worth naming.
+
+`638` holds nine properties, each paired with the wrong answer it excludes, and
+two of them were verified load-bearing by removing the guard and watching the
+test fail — including the one that distinguishes "the request introduced this
+action" from "any artifact action".
+
+And one CI assertion was itself the defect: it grepped for `struct
+embedded_file`, the *default* `row_type`, while the fixture sets
+`row_type = "shader_entry"` precisely because that option exists. The code was
+correct and the assertion was wrong, which is what a check tied to a spelling
+the fixture chooses will eventually always be. It now reads the struct's name
+out of the file and asserts the table's element type *is* that struct, for
+every generated header rather than whichever `find` listed first.
+
+### 11.10 The dependency order, and why it is not the demand order
+
+    mcpp engine  ──────────────►  released, because plugins CI pins a release
+        │
+        ├─► xim payloads  ─────►  independent of the engine; merged first
+        │
+        └─► mcpp-plugins  ─────►  needs the release, so it cannot precede it
+                │
+                └─► mcpp-index  ►  needs the plugins tag's sha256
+
+Four repositories, one pull request each. The engine is the only thing on the
+critical path, and every measurement that changed the plan came from the layer
+*above* it — which is the argument for doing the payloads early even though the
+rows land last.

.agents/docs/2026-09-11-six-open-issues-analysis.md

@@ -321,7 +321,7 @@ default. Someone who sets a machine-wide number almost certainly means it there
 too -- a test runner at 10 concurrent processes has the same memory shape as a
 compile at 10 -- so it should apply, and be documented as applying. What must
 not happen is for it to apply silently: this is a second behaviour under one
-key, and `docs/03-configuration.md` has to say so.
+key, and `docs/04-mcpp-toml.md` has to say so.
 
 **`default_backend` should be removed.** `BackendKind` has `Ninja` and `Native`
 (`src/build/backend.cppm:10`) and `src/build/` contains one backend
@@ -410,3 +410,124 @@ leaving it in place:
 5. **#597** -- graduating a row that already exists, in the batch that added it.
 
 The first four are one release. #597 belongs to the platform batch.
+
+## 8. Self-review, and the one plan a measurement changed
+
+Written after §1-§7 and before any implementation. Four of the six plans
+survive unchanged. One is wrong, one has an unstated cost, and the review
+found the defect the plan for #606 would have half-fixed.
+
+### 8.1 #606: the defect is bidirectional, and the other direction is worse
+
+§1 proposed "a `bool in_block` carried across iterations, with the stripping
+done before `strip_line_comment`", and gave as a criterion that
+`/* */ import x;` on one line "still records the import". Both are wrong.
+
+Measured on 2026.9.10.2, by whether mcpp emits its own
+`imported but not provided` warning (which only the scanner can produce, so it
+separates "the scanner saw it" from "the compiler saw it"):
+
+| source | scanner | correct |
+|---|---|---|
+| `import x;` | sees it | sees it |
+| `const char* s = "a /* b";` then `import x;` | sees it | sees it |
+| `/* */ import x;` | **misses it** | sees it |
+| `// R"(` then `import x;` | **misses it** | sees it |
+| `/*`, `R"(`, `*/` then `import x;` | **misses it** | sees it |
+| `/*`, `export module y;`, `*/` | records a phantom producer | ignores it |
+| `/*`, `module (exe)`, `*/` | refuses the build | ignores it |
+
+So `/* */ import x;` is not a behaviour to preserve -- it is a fourth wrong
+answer. And two of the wrong answers run in the **opposite** direction to the
+reported one: a `//`-commented or block-commented raw-string opener puts
+`strip_raw_strings` into raw mode, which blanks every following line until a
+`)"` that never comes, and real declarations after it are invisible to the
+scanner while remaining visible to the compiler.
+
+A missed `import` is worse in kind than a refused build. It is a **missing
+dependency edge**: the compile is not ordered after the BMI it needs, so the
+failure is a build-order race that appears under parallelism as
+`failed to read compiled module` and disappears on a retry. #606's reported
+form is at least deterministic.
+
+The two directions have one cause. The scanner has three lexical states --
+code, block comment, raw string -- which are mutually exclusive and decided by
+whichever opener comes first. It implements one and a half: raw strings fully,
+line comments as an unconditional `find("//")`, block comments not at all, and
+the three passes run in a fixed order that cannot express "whichever came
+first". Fixing block comments alone, in either order relative to the existing
+passes, produces one of the two wrong directions:
+
+* strip comments first, and `R"( /* )"` opens a comment inside a string;
+* strip raw strings first, and `// R"(` opens a string inside a comment --
+  which is the defect measured above.
+
+### 8.2 The revised plan for #606
+
+One pass over the line with the three states, replacing `strip_raw_strings` and
+`strip_line_comment` at the call site. It blanks non-code and preserves
+offsets, so the reported column stays correct. State carried across lines is
+what it already is (`in_raw`, `raw_close`) plus `in_block`.
+
+Not a lexer: character and string literals need no tokenising, because the only
+question asked of the result is whether the trimmed line *starts with* a
+keyword, and an ordinary `"..."` cannot begin a line with one. The one thing
+the pass must respect about them is `"a /* b"` -- a `/*` inside an ordinary
+string must not open a comment -- which is one state, not a literal parser.
+
+Criteria, one per row of the table above, with the last two being the pair that
+separates a fix from a mute:
+
+* the phantom-producer case generates a graph with no `gcm.cache/y.gcm` output;
+* the four-line file from the report builds;
+* `/* */ import x;` records the import -- a *new* property, and the one that a
+  cheap "skip any line starting with `/*`" would fail;
+* `"a /* b"` then `import x;` still records the import -- currently correct by
+  luck, and the property that stops the fix from treating every `/*` as an
+  opener.
+
+### 8.3 #603: one function, and the two answers must be measured to agree
+
+§3 left open whether the MSVC path keeps the cl banner. It must not: two
+readers of one question is what this codebase treats as the defect, and the
+`std.ixx` path is the better input on both paths, because it describes the STL
+that will actually be compiled rather than the one a fresh search finds first.
+The unit test therefore asserts that for a well-formed VC layout the path
+answer equals what the banner answer would have been -- otherwise the change
+is a silent behaviour change on the one path that was verified.
+
+### 8.4 #599: the cost of running the check is not stated
+
+§4 proposes `submodules: true` on the checkout of whichever shard runs `233`.
+The bench workloads are pinned full source trees of mcpp and xlings, so this is
+not free, and the plan does not say what it costs. Measure before choosing;
+if it is large, the cheaper shape is a job that checks out **only**
+`bench/projects` and runs `233` alone, since the check needs trees and no
+toolchain at all.
+
+### 8.5 #604 and #564 stand, with one narrowing each
+
+#604: append verbatim, and de-duplicate nothing. The alternatives considered --
+de-duplicate by logical module name, or by contiguous subsequence -- are both
+correct and both add a rule to keep an argv tidy. The rule being removed was
+wrong; replacing it with a better rule for the same cosmetic purpose is the
+kind of trade this codebase records as a mistake. The comment says the repeat
+is harmless; the fix should rely on that sentence rather than work around it.
+
+#564: the e2e must assert the *precedence*, not the plumbing. A fixture that
+only sets `default_jobs` and reads `-j3` would pass if the global value were
+wired in above `MCPP_JOBS` instead of below it. Two invocations of one
+fixture, with and without `MCPP_JOBS`, is the smallest thing that distinguishes
+them.
+
+### 8.6 #597 stands, and route A now makes a second row measurable
+
+Unchanged. Noted here because the platform record's Android rows were resolved
+by the same kind of measurement in the same session: `qemu-aarch64 -L <root
+extracted from the system image>` executes the **default, dynamic**
+configuration for `aarch64-linux-android`, with `libc++_shared.so` supplied
+from the NDK's own directory outside the `-L` prefix. The emulator route is
+refuted for every build the vendor manifest currently serves, measured across
+all four Linux host entries rather than the pinned one. So both remaining
+platform rows are executable on an x86_64 Linux runner with no device and no
+virtualization, which is what a CI lane needs.

.github/actions/setup-macos-llvm/action.yml

@@ -107,7 +107,6 @@ runs:
 
         [build]
         default_jobs    = 0
-        default_backend = "ninja"
         EOF
 
         cat "$HOME/.mcpp/config.toml"

.github/workflows/ci-linux-e2e.yml

@@ -42,7 +42,22 @@ jobs:
       # set only in the fresh-install workflows (cold bootstrap, no such asserts).
       # A specific test that needs verbose passes `--verbose` itself.
     steps:
+      # `submodules: recursive` so tests/e2e/233_bench_matrix.sh can check that
+      # each `hub`/`body` in bench/matrix.json exists in the tree it names.
+      # Without the trees that check reads "submodule not initialised" and
+      # reports nothing, which is how a hub path written for the CURRENT
+      # layout stayed in matrix.json while the workload it named is a
+      # HISTORICAL mcpp -- three cells reporting `skipped` on every bench run
+      # and the job still green (#599).
+      #
+      # It is not free and it is not expensive: the three pinned workloads are
+      # 725 + 701 + 806 tracked files, under 10 MB of source in total, and
+      # nothing here builds them. Both shards carry it because run_all.sh
+      # slices the file list round-robin, so which shard holds 233 moves when
+      # a test is added.
       - uses: actions/checkout@v4
+        with:
+          submodules: recursive
 
       # Same cache lineage as ci-linux.yml so this job lands on a warm
       # toolchain/sandbox instead of re-installing it.

.github/workflows/ci-macos-e2e.yml

@@ -28,7 +28,14 @@ jobs:
     # NOTE: no MCPP_VERBOSE — the e2e suite asserts mcpp's default quiet
     # output (tests 48/53).
     steps:
+      # `submodules: recursive` so tests/e2e/233_bench_matrix.sh can check that
+      # each `hub`/`body` in bench/matrix.json exists in the tree it names --
+      # the check reads "submodule not initialised" without them and reports
+      # nothing, which is how a stale hub path survived (#599). Under 10 MB of
+      # source across the three pins, and nothing here builds them.
       - uses: actions/checkout@v4
+        with:
+          submodules: recursive
       - uses: ./.github/actions/setup-macos-llvm
 
       - name: Build mcpp from source (self-host)

.github/workflows/ci-windows-e2e.yml

@@ -36,7 +36,14 @@ jobs:
       # assert mcpp's DEFAULT (quiet) output — e.g. 48_build_error_output and
       # 53_namespaced_cache_label — which forced verbose would break.
     steps:
+      # `submodules: recursive` so tests/e2e/233_bench_matrix.sh can check that
+      # each `hub`/`body` in bench/matrix.json exists in the tree it names --
+      # the check reads "submodule not initialised" without them and reports
+      # nothing, which is how a stale hub path survived (#599). Under 10 MB of
+      # source across the three pins, and nothing here builds them.
       - uses: actions/checkout@v4
+        with:
+          submodules: recursive
       - uses: ./.github/actions/bootstrap-mcpp
 
       - name: Build mcpp from source (self-host)