Part of #20 (slice F). Design doc + D-026 + bench/a320_bench/scoring.py.
Metric (full math in docs/fase5-metrica.md): commands = set_control calls with is_error==false; canonical deterministic matching against the ground truth blocks (injective, multiplicity counts); every command classified as matched_required / optional / forbidden / extraneous. Scored vector: coverage, order (applicable constraints only; null when <1 applicable — no vacuous 1.0), end_state (fraction of success_eval checks), safety (dangerous→hard floor 0, anti_procedure→×0.5), extraneous (mild tidiness factor, floor 0.75). Informative: efficiency, reads, observed_before_acting, judge_bundle (future LLM-judge input, no judge implemented). Derived scalar (rankings only): safety × tidiness × (0.4·end_state + 0.4·coverage + 0.2·order) with order-weight redistribution. Acceptance case (real smokes): full 1.00 > omitted-isolation 0.90 > ext-pwr-only 0.55 > nothing 0.00.
invalid_scenario doesn't score; provider_error scores but is excluded from means by default. Contamination section required (open declaration; closed-loop execution vs recall, with the smoke evidence: both models knew the procedure and still omitted the isolation step).
Implementation: pure module (no Sim/network/binding — automated import test without a320_sim); reuse read_trajectory and scenario.py dataclasses via a minimal refactor extracting parse_ground_truth/parse_success from load_scenario; lazy exports. Synthetic tests: every edge case (repeat multiplicity, OFF→ON pair, wrong-block, wrong value, toggle-back, errored forbidden, extraneous cap, truncated trajectory) = one named test.
Success criteria: CI green; import-without-binding test; every edge case covered.
Part of #20 (slice F). Design doc + D-026 +
bench/a320_bench/scoring.py.Metric (full math in
docs/fase5-metrica.md): commands =set_controlcalls withis_error==false; canonical deterministic matching against the ground truth blocks (injective, multiplicity counts); every command classified as matched_required / optional / forbidden / extraneous. Scored vector:coverage,order(applicable constraints only; null when <1 applicable — no vacuous 1.0),end_state(fraction of success_eval checks),safety(dangerous→hard floor 0, anti_procedure→×0.5),extraneous(mild tidiness factor, floor 0.75). Informative: efficiency, reads, observed_before_acting, judge_bundle (future LLM-judge input, no judge implemented). Derived scalar (rankings only):safety × tidiness × (0.4·end_state + 0.4·coverage + 0.2·order)with order-weight redistribution. Acceptance case (real smokes): full 1.00 > omitted-isolation 0.90 > ext-pwr-only 0.55 > nothing 0.00.invalid_scenariodoesn't score;provider_errorscores but is excluded from means by default. Contamination section required (open declaration; closed-loop execution vs recall, with the smoke evidence: both models knew the procedure and still omitted the isolation step).Implementation: pure module (no Sim/network/binding — automated import test without
a320_sim); reuseread_trajectoryandscenario.pydataclasses via a minimal refactor extractingparse_ground_truth/parse_successfromload_scenario; lazy exports. Synthetic tests: every edge case (repeat multiplicity, OFF→ON pair, wrong-block, wrong value, toggle-back, errored forbidden, extraneous cap, truncated trajectory) = one named test.Success criteria: CI green; import-without-binding test; every edge case covered.