Record what produced a run, not just what it scored

[johnl-amd]

Summary

• Three fields added to the meta block of every routing and behavioral report: engine, agent_cli_version, and model_resolved. New helpers agent.cli_version() and agent.resolved_model(), joined by cli._provenance() and splatted into both meta dicts.
• Why: a report says which prompts passed and nothing about what graded them. model is the alias the caller asked for, and the target behind opus moves. The CLI that does the discovering and activating updates on its own schedule and is not recorded at all. Two runs a week apart can disagree and the artifacts hold nothing that separates a skill getting worse from a dependency that changed.
• Risk: low. Three keys added to an artifact, no key changed or removed, no new flag, no extra agent call.

What it looks like

Before, two reports a week apart:

routing-20260907-143545.json   6 of 8
routing-20260914-101200.json   4 of 8

After:

"model": "sonnet",
"engine": "legacy",
"agent_cli_version": "2.1.270",
"model_resolved": "claude-sonnet-5",

Same model, new CLI build, so the CLI is where to look.

Where each field comes from

• engine is a constant. There is one engine, and naming it means a reader does not have to date a report to work out what graded it.
• agent_cli_version is claude --version, cached for the process. One subprocess, no tokens.
• model_resolved is read out of the preflight call every graded run already pays for, so it costs nothing. It is null when --skip-preflight was passed, which is honest: nothing spoke to the API, so nothing knows.

The preflight result reports usage per model rather than naming the one that ran, and the CLI bills auxiliary work to a second model in the same block (a run asking for sonnet can come back with a session-title model listed beside it). The alias that was asked for is what picks the right entry; with no alias, the entry that did the most work.

Test plan

• 11 new cases in TestARunRecordsWhatProducedIt. Full suite 237, green on python -m unittest discover -s tests -t ..
• The auxiliary-model case is not hypothetical. The first version of the extraction took the first entry and reported gpt-5.5 for a run that asked for sonnet, against a live API. Reverting the fix with the tests in place turns that test red and nothing else, so it is pinned to the behaviour rather than to the shape of the code.
• End to end against a real catalog: skillscope routing --skills-dir 'skills/*' --routing-room all --only finetune-on-laptop --model sonnet, meta block as shown above.
• Extraction also checked against a missing modelUsage, an empty one, a list where an object belongs, and unparseable output. All degrade to null rather than raising.