fix(scoring): test the arithmetic this benchmark's numbers are made of

[webdevsamran]

Phase 6 calls for mutation-testing the scoring logic, on the grounds that a measurement tool's measurement is its product. Doing it put tooltrace/scoring/builtin.py at a 57.4% mutation score — 23 of 54 mutants survived, meaning 23 behaviours no test constrained.

Three findings, in order of severity.

1. A scorer that could be inverted silently

In file_not_contains, mutating in → not in survived. That scorer would have returned 1.0 for a file containing the forbidden text and 0.0 for a clean one — doing exactly the opposite of its name, with the whole suite green.

2. A constraint that has never worked

Writing a boundary test for git_diff's max_changed_files crashed it:

changed = {line.split()[2] for line in ... if line.startswith("+++ b/")}

A git diff header is +++ b/path — two whitespace-separated fields — so [2] raises IndexError on every real diff. The constraint could not return a score, only crash. No shipped task uses it, which is why nothing noticed. I verified the format against real git diff HEAD output rather than reasoning about it.

3. The pytest scorer's core calculation was entirely unconstrained

Every operator survived:

total = passed + failed + errors                                  # Add → Sub survived
ratio = passed / total if total else 0.0                          # Div → Mult survived
score = 1.0 if ratio >= min_ratio and errors == 0 else round(...)  # GtE, Eq, and → all survived

The total could have been computed by subtraction, the ratio by multiplication, the threshold inverted. It was untested because it was unreachable — welded inside _tests_pass behind a subprocess.run of a real pytest in a temp workspace. Extracting score_pytest_output is what made the assertions possible.

Result

Mutation score 57.4% → 77.8% (42/54). Every comparison and arithmetic mutant is killed.

The 12 survivors are boolean short-circuits and bool constants, mostly isinstance(...) and ... guards whose second operand is unreachable without the first — equivalent mutants rather than gaps. Saying so rather than chasing the number: a mutation score is a diagnostic, not a target.

One of my own mistakes

My first attempt to kill the JSON-path Lt → LtE mutant used index 9 on a three-element array, where 9 < 3 and 9 <= 3 are both false — so the mutant survived a test written specifically to kill it. Only idx == len distinguishes them.

Verification

Gate	Result
ruff / ruff format --check	clean, 123 files
mypy	clean, 73 source files
pytest	340 passed
coverage	85.48% against the 80% floor

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.