[TEST] MUT-027, MUT-048, MUT-052: CLI, reliability, and adapter extract tests miss product paths #75

Problem

The Python suite is strong on policy matrices, Hosted security removal, ingest, and auth. It is weak where the product actually ships:

• MUT-027: Execution-mode tests replace _run_local. Minimize, regression save, Featherless wiring, and the campaign loop in run_cmd are mostly uncovered. mutiny test SKIPPED / corrupt JSON is untested. AGENT_REF path that mutiny init tells users to set has no tests. Coverage misses run_cmd.py ~229–315, 338–410.
• MUT-048: The “reliability” job is a ~0.08s boundary-seed smoke (stop_on_first_violation, gate ≥2/3 hits). That is not stressing mutation search. Stronger check test_mutation_path_finds_violation_from_sub_boundary_seeds is unit, not reliability. Highly likely from timing; not a failing test today.
• MUT-052: OpenAI adapter extract unit tests only cover JSON-string args. Invalid JSON, dict raw_item, missing new_items type variants, agent_ref resolution untested. Integration does exercise the real offline sample (good). Highly likely coverage hole.

CLI smoke and verify_release_artifacts.sh do run real paths; the unit suite (what the PR template asks for) does not.

Why it matters

Tests can pass while _maybe_minimize_and_save, SKIPPED reporting, or AGENT_REF loading is broken. Search/mutator regressions can pass the “reliability” job. SDK shape changes can pass unit tests.

Evidence

• Audit IDs: MUT-027 (confirmed testing gap), MUT-048 / MUT-052 (highly likely — coverage/timing, not failing tests).
• tests/unit/test_cli_execution_mode.py mocks _run_local.
• tests/reliability/test_smoke_gate.py; config/demo_pin.json.
• tests/unit/test_openai_agents_adapter.py vs extract.py missed lines.

Related #27 is friendlier AGENT_REF error messages, not loader tests.

Reproduction

Coverage/timing as measured in the audit. Not independently re-measured in this filing pass. Do not treat these as production defects; they are test-gap work.

Expected behavior

Unit/reliability jobs fail if CLI minimize/save, SKIPPED handling, AGENT_REF load, extract dict/invalid JSON, or mutation-from-sub-boundary search regress.

Suggested implementation direction

1. One unmocked mutiny run in tmp_path; fixture with invalid JSON for mutiny test; temp modules for valid/invalid AGENT_REF.
2. Reliability: smoke without boundary seeds, or require a generation ≥ 1 violator (move or duplicate the unit mutation-path test into the reliability job).
3. Parametrize extract on dict/raw/invalid JSON; one test for agent_ref.

Keep tests offline (MUTINY_SAMPLE_OFFLINE=1). No live OpenAI.

Acceptance criteria

• At least one unmocked CLI run test in tmp_path (sample or stub adapter) covering minimize/save or documenting why smoke already covers it and adding the SKIPPED/AGENT_REF unit cases.
• Invalid regression JSON path is unit-tested (pairs with MUT-023).
• Reliability job is not only 0.08s boundary-seed smoke or is renamed/documented as smoke and a real search assertion runs in CI.
• Extract tests cover non-JSON-string arguments.
• Existing 3.11/3.12 suite still passes; no new network calls.

Scope

tests/unit, tests/reliability, maybe tiny CLI hooks. Out of scope: Hosted UI tests (MUT-026), MUT-024 dirty git.

Related audit findings

MUT-027, MUT-048, MUT-052. Related: MUT-023, #27.

Contributor notes

Generated from the Mutiny deep audit.

help wanted. Not a good-first-issue as a combined testing epic; splitting PRs (CLI vs extract vs reliability job) is fine if each links this issue.