Skip to content

refactor(cli): make the refactor's guarantees provable — determinism, mutation, golden, dead code #750

Description

@blafourcade

Problem

The context refactor closed with a green suite, and three of its guarantees were weaker than they read.

Concurrent test runs failed on a difference that was not there. pnpm test built dist/cli.js with clean: true, every e2e file read that same path, and a second run's rebuild landed between the golden suites' two captures. It was chased as a phantom twice before the cause was measured, and the answer written down was a rule for humans — "run one vitest at a time" — which nobody can enforce.

The mutation scores were not reproducible. stryker.conf.json listed seventeen kernel files by hand, so a file added to the kernel escaped mutation in silence: the score did not drop, because the mutants that would have died were never generated. The four other per-context figures on record came from command lines typed once and not kept, and measured each context's domain/ layer alone — visible in phase-20.md's target column, invisible to anyone reading the numbers.

The golden byte-compared one cell of nine. FROZEN_CELLS = new Set(["claude"]), and claude's content rewrite is the identity — so the only guarded cell was structurally incapable of catching a change in any other profile's. Three of the eight unfrozen cells turned out to have been stale since the migration commit of 2026-07-22, the file's only write.

Behind all three: src/ carried an entire reverse API implemented in every tool profile and called by nothing, and a placeholder-rewriting path superseded by the decision to drive each tool's own plugin CLI.

Scope

  • e2e determinism — each run builds its own binary under a gitignored .e2e-build/, published through vitest's provide/inject; pnpm test stops building at all. AIDD_BUILD_OUT_DIR refuses any target but dist or a run directory, because clean: true empties what it is given.
  • Reproducible mutationmutation-scopes.json declares a glob per context plus what is excluded and why; scripts/run-mutation.mjs <scope> files each report under reports/mutation/<scope>/; break threshold removed, since a threshold that fails a command is a gate whatever it is called.
  • Golden freeze — all nine target/mode cells byte-compared, every re-baseline carrying its cause in the file header.
  • Coverage where a user is exposed — the source spellings aidd plugin add accepts (kernel/source.ts, 71 uncovered mutants to 0), Copilot's reference rewriting, and the settings file this CLI shares with whoever edits it by hand.
  • Dead code removed — the reverse API (reverseRewriteContent, detectUserFileSectionKey, reverseConvertFrontmatter, detectSectionKeyFromPrefixes), the {{TOOLS}}/{{DOCS}} placeholder substitution with the docsDir threading that existed only to feed it, and the marketplace registration path every plugin-capable tool now covers through its own CLI.
  • Guards that prove they still fire — every new rule is checked by injecting the violation it exists to catch, including the biome import overrides, which is how a translate rule naming six deleted directories was found forbidding nothing.

Acceptance criteria

  • Two concurrent vitest run --project e2e both pass; a process creating, corrupting and deleting dist/cli.js 200 times during the run leaves 104/104 green
  • pnpm test:mutation:<scope> reproduces every score quoted in aidd_docs/, seven scopes, 23 minutes for all
  • tests/architecture/mutation-covers-source.arch.test.ts fails when a .ts under src/ matches no scope and no declared exclusion
  • All nine golden cells byte-compare; a corrupted hash in any of them fails the test naming that cell
  • tests/architecture/import-rules-bite.arch.test.ts fails when a restricted-import pattern names a path that no longer exists
  • Nine build outputs byte-identical between a binary built at this branch's base and HEAD; setup + plugin install + sync identical for all five tools
  • 2032 tests over 1001 suites, tsc 0, biome 0, knip 0, smoke 98/0 across 22 of 22 leaf commands

Prior art in this repo

  • cli/aidd_docs/tasks/2026_08/2026_08_20_refactor-contextes-cli/ — the twenty-phase context refactor this builds on. phase-20.md recorded the five per-context mutation scores whose scope this work corrected.
  • cli/tests/architecture/tool-addition-cost.arch.test.ts — the ratchet pattern every new guard follows: a baseline that may only shrink, plus a probe of the rule over a synthetic violation.
  • cli/aidd_docs/memory/testing.md — held "run one vitest at a time" as a rule; now records the mechanism that made it unnecessary.
  • Five task folders under cli/aidd_docs/tasks/2026_09/ carry the plan, the phase records and what each review found.

Out of scope

  • Three adapters still weakly coveredplugin-fetcher-adapter, self-update/git-adapter, auth/auth-provider-adapter, roughly 105 uncovered mutants. Phase 5 of 2026_09_03_mutants-sans-couverture/, not written.
  • contexts/framework is 88 files and 8248 lines — 38% of src/ in one context, against 16 files for translate. Whether it is one context or three (install, sync, restore) is a design question that wants a brainstorm, not a cleanup.
  • presentation scores 14.1% — 1250 of its mutants sit in commander wiring, whose real net is the e2e suite and the smoke script, which the mutation run cannot see. Recorded as an artifact of the measurement's blind spot rather than a debt. global-options.ts and doctor.ts's pure helpers are the exception and belong to a later phase.
  • The four kanban dependenciesink, react, cli-table3, gray-matter, 25 MB at install. They are runtime requirements of the shipped aidd kanban, so removing them is a product decision, not a cleanup. React is also pinned at 19.2.8 in cli/ and 19.2.7 in kanban/.
  • Whether the command surface is intuitivetranslate, sync, and doctor absorbing status were reviewed as scope changes, but nobody has watched someone use them.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

None yet

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions