Skip to content

docs(testing): the evidence bar — a test executes the path, a fix records its mutation (#2829) - #2833

Merged
vybe merged 2 commits into
devfrom
fix/2829-execution-coverage-docs
Sep 16, 2026
Merged

vybe merged 2 commits into
devfrom
fix/2829-execution-coverage-docs

Conversation

@dolho

@dolho dolho commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

The public half of #2829. Three of five ejections on the 2026-09-15 merge train — the third train running — were tests that prove the code was written rather than that it runs: source-text regexes over the module under test (#2811), a bound check at the one value where both bounds coincide (#2817), a docstring claim about CI never negative-controlled (#2805). All green CI. The question that caught all three lived only in /merge-train Phase 2.

The issue's four directions, split by where they live:

Direction Where
1. Ask the coverage question first in /validate-pr and /review trinity-dev#29/validate-pr §5.4 and /review Step 2.5, answered before the checklist with a report block; /implement gains two done-criteria
2. Mutation as the evidence standard for a fix this PR's PR-template Mutation: line + STRATEGY.md; enforced by /review (a fix with no red mutation is CRITICAL)
3. A declared-bound-never-assigned ref is conclusive STRATEGY.md + both skills
4. Bound tests parameterised away from the coincidence STRATEGY.md + both skills

Changes

Test Plan

  • test_2350_journey_impact.py (reads the template) — green
  • test_2338_journey_catalog.py — green
  • After merge: the next fix: PR carries a Mutation: line; /validate-pr reports it under Execution coverage

Mutation: n/a — docs and process

Journey Impact: none: documentation and PR-template only, touches no journey promise

Fixes #2829

🤖 Generated with Claude Code

https://claude.ai/code/session_01VpvcfgWkmQPD7DrDLmATTf

…ords its mutation (#2829)

Three of five ejections on the 2026-09-15 merge train — the third train
running — were tests that prove the code was written rather than that it
runs: source-text regexes over the module under test (#2811), a bound
check at the one value where both bounds coincide (#2817), a docstring
claim about CI never negative-controlled (#2805). All green.

- docs/testing/STRATEGY.md: a new "Evidence bar for a test" section beside
  the harness bar — the three spellings, the two greps (the live-consumer
  grep is the one that decides), guard-vs-source-only with the train's own
  pair (#2819 kept, #2811 ejected, same shape), mutation as the fix
  standard, bound tests away from the coincidence — each with what
  enforces it.
- .github/pull_request_template.md: a Testing checkbox for "every new test
  executes the changed path" and a `Mutation:` line naming the test(s)
  that go red with the fix reverted ("n/a — not a fix" otherwise). The
  trailing space after the colon matches the existing `Journey Impact:`
  line — a fill-in prompt.
- docs/memory/learnings.md: the class, with the prior occurrences.

The skill half — /review Step 2.5 and /validate-pr §5.4 answered first
and in writing, /implement's two done-criteria — is trinity-dev#29.

Fixes #2829

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpvcfgWkmQPD7DrDLmATTf
@dolho
dolho requested a review from vybe September 16, 2026 04:18
@dolho

dolho commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

/validate-pr — head c309d0573 (merge-train pre-validation, lane A)

Docs/process only: docs/testing/STRATEGY.md, .github/pull_request_template.md, learnings.md. No executable path changes, so no coverage question; the one test touching the diff (test_2350_journey_impact.py::test_the_templates_carry_the_field) is the sanctioned template guard and still passes (79 passed at head). Secret/email/IP greps clean. Fixes #2829 bare form ✓. 22 checks green.

⚠️ non-blocking: the STRATEGY.md text forward-references /review Step 2.5, /validate-pr §5.4 and the Mutation: handling, which exist only on trinity-dev#29 (open) — the pinned .claude (35133b6) has none of them. Harmless to merge; the process lands once trinity-dev#29 merges and the pointer is bumped.

Verdict: READY.

@vybe

vybe commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

merge-train: merged dev into this branch to resolve the docs/memory/learnings.md append collision left by the previous train member (keep both entries, theirs first). Mechanical; no other file touched.

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge-train: batch validated on train/20260916-0919 (train PR #2839)

@vybe
vybe merged commit aa3ebeb into dev Sep 16, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants