diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f1bc45..4091f91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,7 +31,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (`blocking: false`) entries included, though only blocking ones count toward the budget; those rows are tagged "advisory". `report.json` carries the same rows under a new top-level `invariant_violations` key, each with - the entry's `blocking` flag. + the entry's `blocking` flag. The "Overall, by evaluator" table tags such an + entry's row too and says what a trace-rule row measures, and the "Top + regression causes" panel counts `INVARIANT_VIOLATION` as the budget does — + distinct examples with a blocking target violation — so the narrative never + quotes a figure the verdict did not use. - `evalshift init` scaffolds `max_invariant_violations: 0` in every `--profile` policy block, beside the other budgets. Older CLIs reject the field, so bump the CI action's `evalshift-version` pin to this release @@ -43,6 +47,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- The report's per-prompt Executive summary — and the header's mean score Δ, + which averages its rows — now picks each prompt's worst severity with ties + broken toward the most negative effect size (the order the hero panel and + the narrative already used) instead of whichever evaluator `analyze` listed + first, and skips advisory (`blocking: false`) evaluators while the prompt + has a gating one. An advisory rule the target satisfied by calling no tools + at all could otherwise read as the prompt's "+9% improved" above a failed + verdict. - Upgrading moves the resume `config_hash`: the new config fields are part of every config's canonical dump, so a run left in progress by an older CLI cannot be `--resume`d after upgrading (it fails with the usual config-hash diff --git a/DOCS.md b/DOCS.md index a4139f3..d82df50 100644 --- a/DOCS.md +++ b/DOCS.md @@ -608,7 +608,7 @@ Rules are checked per **response** (one model turn) against the tool calls alrea | `call_count` | `max_calls`: a model call takes the running count (context first) past it, one violation per excess call. `min_calls`: context plus model calls at the end of the last response fall short (one violation). Both equal = exact | context calls count toward both bounds | | `arguments` | a model call to `tool` fails `json_schema` (Draft 2020-12, validated at config load); unparseable arguments fail any schema that requires a field | none | -`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants). +`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). The **Overall, by evaluator** table tags an advisory entry's row the same way and says what a trace-rule row measures (each side judged against the entry's rules, not against the other side); the **Top regression causes** panel (`failure_categories` in the decision) counts `INVARIANT_VIOLATION` as the budget does — distinct examples with a blocking target violation, never per row and never for advisory entries; and the per-prompt executive summary (with the header's mean score Δ, the mean of its rows) is drawn from blocking evaluators only — worst severity first, ties to the most negative effect size — falling back to advisory ones on a prompt that has nothing else. Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants). --- diff --git a/docs/configuration.md b/docs/configuration.md index 2fb13ac..91637fc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -728,14 +728,27 @@ as one broken once, and `delta = target − source`. The record's kind is `trace_invariants`; its metadata carries `rules_checked`, `source_violations` and `target_violations` (each a list of `{rule_id, rule_type, tool, round_index, detail}`), `owner` when set, and the failure category -`INVARIANT_VIOLATION` when the target broke anything. With -`samples_per_example` above 1 the example's row keeps every scored sample's -violations, each tagged with its `sample` ordinal. The HTML report's -**Trace rules broken** panel and `report.json`'s `invariant_violations` list -every rule the target broke, whatever the delta — including those of advisory -(`blocking: false`) entries, which never count toward the budget — with the -owner and whether the source broke it too. Each row carries the entry's -`blocking` flag, and the panel tags `blocking: false` rows **advisory**. +`INVARIANT_VIOLATION` when the target broke anything. The report's **Top +regression causes** panel (and `failure_categories` in +`migration_decision.json`) counts that category the way the budget does — +distinct examples on which the target broke a *blocking* rule — so an advisory +violation, or a second entry broken on the same example, never inflates it +past the figure the verdict used. With `samples_per_example` above 1 the +example's row keeps every scored sample's violations, each tagged with its +`sample` ordinal. The HTML report's **Trace rules broken** panel and +`report.json`'s `invariant_violations` list every rule the target broke, +whatever the delta — including those of advisory (`blocking: false`) entries, +which never count toward the budget — with the owner and whether the source +broke it too. Each row carries the entry's `blocking` flag, and the panel tags +`blocking: false` rows **advisory**. So does the **Overall, by evaluator** +table, where every `trace_invariants` row also says what it measures: each +side judged against the entry's rules, not against the other side. The +per-prompt **Executive summary** — and the header's mean score Δ, which +averages its rows — is drawn from blocking evaluators only (worst severity +first, ties to the most negative effect size), falling back to advisory ones +on a prompt that has nothing else, so an advisory rule the target satisfied by +calling no tools cannot read as the prompt's improvement above a failed +verdict. Four things to know: diff --git a/llms-full.txt b/llms-full.txt index 108ce90..00dfb6a 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -646,7 +646,9 @@ evaluators: # every evaluator config also takes blocking # Score per side = (rules - rules broken) / rules; kind slug trace_invariants; metadata # rules_checked, source_violations, target_violations (each [{rule_id, rule_type, tool, # round_index, detail}], with sample ordinal when samples_per_example > 1), owner; - # failure_categories [INVARIANT_VIOLATION] when the target broke anything. + # failure_categories [INVARIANT_VIOLATION] when the target broke anything (the decision's + # failure_categories count it as the budget does: distinct examples with a blocking + # target violation -- never per row, never for advisory entries). suites: # managed block; capture sync rewrites between : # ">>> evalshift suites" markers source: captured|jsonl = captured @@ -859,7 +861,12 @@ scored successfully (NOT the raw sample_index). Advisory (blocking: false) entri ARE listed; only blocking ones count toward max_invariant_violations. blocking = the record's flag (false = advisory entry). report.html renders the same rows as a "Trace rules broken" panel (rule, owner, detail, "Source too?"), tagging blocking=false rows with a muted -"advisory" badge. +"advisory" badge. The "Overall, by evaluator" table tags an advisory entry's row the same way +and says what a trace-rule row measures (each side judged against the entry's rules). The +per-prompt executive summary (and the header's mean score Δ, the mean of its rows) is drawn +from blocking evaluators only -- worst severity first, ties to the most negative effect size, +the order the hero panel and the narrative use -- falling back to advisory ones on a prompt +with nothing else. ## Migration policy verdict algorithm diff --git a/src/evalshift_cli/analysis/policy.py b/src/evalshift_cli/analysis/policy.py index e8a498b..7f1813b 100644 --- a/src/evalshift_cli/analysis/policy.py +++ b/src/evalshift_cli/analysis/policy.py @@ -22,6 +22,7 @@ from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.evaluators.failures import ( BROKEN_HARNESS_CAUSES, + INVARIANT_VIOLATION, SEMANTIC_REGRESSION, TOOL_GROUND_TRUTH_MISS, ) @@ -1485,13 +1486,26 @@ def _blocking_regressions(comparisons: list[ComparisonResult]) -> list[BlockingR def _failure_categories(records: list[EvalRecord]) -> list[FailureCategoryCount]: + """Count each failure category over the rows that carry it. + + ``INVARIANT_VIOLATION`` is the exception: every ``trace_invariants`` row + carries it when the target broke a rule, advisory or not, source too or + not -- so counting rows read "8" in the report's causes panel and the + narrative built on it, above a verdict that counted 5. The category is + counted the way ``max_invariant_violations`` is, by + :func:`_invariant_violation_counts` over the blocking rows, so the two can + never disagree. + """ counter: Counter[str] = Counter() for r in records: raw = r.metadata.get("failure_categories", []) if isinstance(raw, list): for item in raw: - if isinstance(item, str) and item: + if isinstance(item, str) and item and item != INVARIANT_VIOLATION: counter[item] += 1 + violated, _ = _invariant_violation_counts([r for r in records if r.blocking]) + if violated: + counter[INVARIANT_VIOLATION] = violated return [ FailureCategoryCount(category=category, count=count) for category, count in counter.most_common() diff --git a/src/evalshift_cli/reports/html.py b/src/evalshift_cli/reports/html.py index ef07d34..ac82a3c 100644 --- a/src/evalshift_cli/reports/html.py +++ b/src/evalshift_cli/reports/html.py @@ -19,6 +19,7 @@ from evalshift_cli.analysis.statistics import AXIS_NOTE_PREFIX, UNMEASURED_NOTE_PREFIX from evalshift_cli.evaluators.failures import category_label from evalshift_cli.evaluators.tool_selection import KIND_CONFORMANCE, KIND_DIVERGENCE +from evalshift_cli.evaluators.trace_invariants import KIND as KIND_INVARIANTS from evalshift_cli.insights.models import Insight from evalshift_cli.reports.json import ( SHARED_GROUND_TRUTH_NOTE_PREFIX, @@ -161,6 +162,14 @@ def _confidence_label(severity: str, p_corrected: float) -> str: KIND_DIVERGENCE: "the target graded against what the source did", } +# The same for a single-axis family whose name alone does not say what the +# score is measured against. A trace-rule row is judged against rules the +# team wrote, not against the other side: its delta is a difference in rules +# kept, and a rule both models broke is a zero that is not "no change". +_KIND_BLURBS: dict[str, str] = { + KIND_INVARIANTS: "each side judged against the entry's hand-written rules", +} + # Statistical tests, spelled out for the row tooltip. _TEST_LABELS: dict[str, str] = { "paired_t": "paired t-test", @@ -186,6 +195,11 @@ def _axis_blurb(kind: str) -> str: return _AXIS_BLURBS.get(kind, "") +def _kind_blurb(kind: str) -> str: + """One phrase saying what a family's score is measured against, or ``""``.""" + return _KIND_BLURBS.get(kind, "") + + def _evaluator_label(name: str, kind: str = "") -> str: """Return a human-readable name for an evaluator id like ``semantic.cosine``. @@ -479,6 +493,7 @@ def render_html(report: ReportData, *, insight: Insight | None = None) -> str: env.globals["evaluator_label"] = _evaluator_label env.globals["axis_of"] = _axis_of env.globals["axis_blurb"] = _axis_blurb + env.globals["kind_blurb"] = _kind_blurb env.globals["test_label"] = _test_label env.globals["regression_reason"] = _regression_reason env.globals["tool_list"] = _tool_list diff --git a/src/evalshift_cli/reports/json.py b/src/evalshift_cli/reports/json.py index c3022b6..f467b2d 100644 --- a/src/evalshift_cli/reports/json.py +++ b/src/evalshift_cli/reports/json.py @@ -21,7 +21,7 @@ from typing import Any from evalshift_cli.analysis.policy import is_shared_ground_truth_miss -from evalshift_cli.analysis.statistics import AXIS_NOTE_PREFIX +from evalshift_cli.analysis.statistics import ADVISORY_NOTE_PREFIX, AXIS_NOTE_PREFIX from evalshift_cli.cli.commands.analyze import ANALYSIS_FILENAME, MIGRATION_DECISION_FILENAME from evalshift_cli.cli.commands.evaluate import SCORES_FILENAME from evalshift_cli.evaluators.base import EvalRecord @@ -338,7 +338,8 @@ def build_report_payload( truncated = sum(1 for c in calls if c.truncated) cost = sum(c.cost_usd for c in calls) - summary = _build_executive_summary(analysis) + advisory = _advisory_evaluator_names(analysis, scores) + summary = _build_executive_summary(analysis, advisory) sections = _build_prompt_sections( analysis, scores, @@ -346,6 +347,7 @@ def build_report_payload( suite, tool_evaluator_names, agent_traces, + advisory=advisory, ) return ReportData( @@ -554,9 +556,39 @@ def _n_examples(calls: list[Call]) -> int: return len({c.example_id for c in calls}) -def _build_executive_summary(analysis: dict[str, Any]) -> list[dict[str, Any]]: - """One row per prompt summarising the worst non-all-slice severity.""" - by_prompt: dict[str, dict[str, Any]] = {} +def _advisory_evaluator_names(analysis: dict[str, Any], scores: list[EvalRecord]) -> set[str]: + """Names of the ``blocking: false`` evaluators, as the policy gate sees them. + + From the records when there are records -- every row carries the config + flag -- and from the ``advisory:`` note ``analyze`` stamps on the + synthesized comparison of an advisory evaluator that scored nothing. + Mirrors :func:`evalshift_cli.analysis.policy._advisory_evaluator_names`, + which works on parsed comparisons rather than the raw ``analysis.json`` + rows this module renders. + """ + names = {record.evaluator_name for record in scores if not record.blocking} + for comparison in analysis.get("comparisons", []): + notes = comparison.get("notes") or [] + if any(isinstance(note, str) and note.startswith(ADVISORY_NOTE_PREFIX) for note in notes): + names.add(comparison["evaluator_name"]) + return names + + +def _build_executive_summary( + analysis: dict[str, Any], + advisory: set[str], +) -> list[dict[str, Any]]: + """One row per prompt: the comparison that speaks for it. + + The worst severity wins; ties go to the most negative effect size, then + the smaller corrected p-value -- the order :func:`reports.html._headline_comparison` + and the insights use, so this table, the header's average score change + and the hero panel beside them quote the same row rather than whichever + evaluator ``analyze`` happened to list first. Advisory (``blocking: + false``) evaluators are passed over while the prompt has a gating one: a + rule the target satisfied by calling no tools at all once read as the + prompt's "+9% improved" directly above a failed verdict. + """ severity_rank = { "critical": 0, "high": 1, @@ -566,13 +598,25 @@ def _build_executive_summary(analysis: dict[str, Any]) -> list[dict[str, Any]]: "none": 5, "insufficient": 6, } + + def order(comparison: dict[str, Any]) -> tuple[int, float, float]: + return ( + severity_rank.get(comparison["severity"], 99), + float(comparison.get("effect_size", 0.0)), + float(comparison.get("p_value_corrected", 1.0)), + ) + + by_prompt: dict[str, list[dict[str, Any]]] = {} for c in analysis["comparisons"]: - prompt = c["prompt_id"] - rank = severity_rank.get(c["severity"], 99) - existing = by_prompt.get(prompt) - if existing is None or rank < severity_rank.get(existing["severity"], 99): - by_prompt[prompt] = c - return [by_prompt[k] for k in sorted(by_prompt)] + by_prompt.setdefault(c["prompt_id"], []).append(c) + rows: list[dict[str, Any]] = [] + for prompt in sorted(by_prompt): + gating = [c for c in by_prompt[prompt] if c["evaluator_name"] not in advisory] + # A prompt scored only by advisory evaluators keeps its row -- the + # table is one line per prompt -- and that row is then the best + # advisory one, tagged as such where it is listed in full. + rows.append(min(gating or by_prompt[prompt], key=order)) + return rows def _build_prompt_sections( @@ -582,6 +626,8 @@ def _build_prompt_sections( suite: Suite, tool_evaluator_names: frozenset[str], agent_traces: dict[TraceKey, AgentTrace], + *, + advisory: set[str], ) -> list[PromptSection]: # One displayed output per (prompt, example, role): sample 0. Keyed by # role below, so every sample must not overwrite the one before it. @@ -604,10 +650,15 @@ def _build_prompt_sections( # Aggregate rows: pick the implicit "all" slice rows for this prompt. # Only these are annotated: the slice table renders significant # severities only, and a shared miss is always ``none``. - aggregates_all = _annotate_shared_misses( - [c for c in prompt_comparisons if c["slice_name"] == "all"], - prompt_records, - ) + # ``advisory`` rides on the row so the evaluator table can tag a + # ``blocking: false`` entry the way the broken-rules panel does. + aggregates_all = [ + {**row, "advisory": row["evaluator_name"] in advisory} + for row in _annotate_shared_misses( + [c for c in prompt_comparisons if c["slice_name"] == "all"], + prompt_records, + ) + ] # Slice rows: every non-all comparison for this prompt. slice_rows = [c for c in prompt_comparisons if c["slice_name"] != "all"] # Top regressions: the 5 records with the most-negative delta. diff --git a/src/evalshift_cli/reports/templates/report.html.j2 b/src/evalshift_cli/reports/templates/report.html.j2 index a0bd0c0..e10cfdb 100644 --- a/src/evalshift_cli/reports/templates/report.html.j2 +++ b/src/evalshift_cli/reports/templates/report.html.j2 @@ -559,7 +559,9 @@ {{ evaluator_label(row.evaluator_name, axis) }} {{ row.evaluator_name }}{% if axis %} · {{ axis }}{% endif %} - {% if axis_blurb(axis) %}{{ axis_blurb(axis) }}{% endif %} + {% if row.get("advisory") %}advisory{% endif %} + {% set measures = axis_blurb(axis) or kind_blurb(row.get("kind", "")) %} + {% if measures %}{{ measures }}{% endif %} {{ row.n }} diff --git a/tests/unit/test_policy.py b/tests/unit/test_policy.py index aec1f39..c9bd18c 100644 --- a/tests/unit/test_policy.py +++ b/tests/unit/test_policy.py @@ -2975,3 +2975,34 @@ def test_a_breached_budget_overrides_the_failed_slice_demotion(self) -> None: assert decision.verdict == "fail" assert decision.reason is not None assert "max_invariant_violations" in decision.reason + + +class TestInvariantFailureCategory: + def test_the_category_counts_what_the_budget_counts(self) -> None: + """The causes panel must not read 3 where the budget reads 1. + + Two blocking entries broke a rule on ``e1`` (two rows, one example), an + advisory entry broke one on ``e2``, and the source shares every break: + the budget observes 1 and the category must agree -- counting rows + made the report's causes panel and the narrative built on it name a + figure the verdict never used. + """ + decision = _decide( + [ + _invariant_record("e1", 0.0, 0.0), + _invariant_record("e1", 0.0, 0.0, rule_id="one-charge").model_copy( + update={"evaluator_name": "refunds"} + ), + _invariant_record("e2", 0.0, 0.0, blocking=False), + _invariant_record("e3", 1.0, 1.0), + ], + ) + counts = {c.category: c.count for c in decision.failure_categories} + assert counts[INVARIANT_VIOLATION] == 1 + assert _budgets(decision)["max_invariant_violations"].observed == 1.0 + + def test_advisory_breaks_alone_write_no_category(self) -> None: + decision = _decide( + [_invariant_record("e1", 1.0, 0.0, blocking=False), _invariant_record("e2", 1.0, 1.0)], + ) + assert INVARIANT_VIOLATION not in {c.category for c in decision.failure_categories} diff --git a/tests/unit/test_reports.py b/tests/unit/test_reports.py index 18fbf76..c249c63 100644 --- a/tests/unit/test_reports.py +++ b/tests/unit/test_reports.py @@ -2615,3 +2615,128 @@ def test_family_label_reads_as_words(self) -> None: from evalshift_cli.reports.html import _kind_label assert _kind_label("trace_invariants") == "Trace rules" + + +def _append_comparisons(run_dir: Path, rows: list[dict[str, Any]]) -> None: + """Add aggregate comparisons to the scaffold's ``analysis.json``.""" + path = run_dir / ANALYSIS_FILENAME + analysis = json.loads(path.read_text(encoding="utf-8")) + analysis["comparisons"].extend(rows) + path.write_text(json.dumps(analysis), encoding="utf-8") + + +class TestAdvisoryEvaluatorsInSummary: + """A ``blocking: false`` evaluator never speaks for a prompt that has a gate.""" + + @staticmethod + def _comparison(evaluator_name: str, delta: float, **overrides: Any) -> dict[str, Any]: + row: dict[str, Any] = { + "prompt_id": "greet", + "evaluator_name": evaluator_name, + "kind": "trace_invariants", + "slice_name": "all", + "n": 2, + "test": "wilcoxon", + "statistic": 0.0, + "p_value": 1.0, + "p_value_corrected": 1.0, + "effect_size": 0.3 if delta > 0 else -0.3 if delta < 0 else 0.0, + "effect_size_ci_low": -0.6, + "effect_size_ci_high": 0.6, + "delta_avg_score": delta, + "severity": "none", + "notes": [], + } + row.update(overrides) + return row + + @staticmethod + def _record( + evaluator_name: str, delta: float, *, blocking: bool, prompt_id: str = "greet" + ) -> dict[str, Any]: + return { + "run_id": "r_20260601_aaaaaa", + "prompt_id": prompt_id, + "example_id": "ex1", + "evaluator_name": evaluator_name, + "kind": "trace_invariants", + "source_score": 1.0 if delta <= 0 else 0.0, + "target_score": 1.0 if delta >= 0 else 0.0, + "delta": delta, + "blocking": blocking, + } + + def _run( + self, tmp_path: Path, records: list[dict[str, Any]], comparisons: list[dict[str, Any]] + ) -> Path: + run_dir = _run_dir_with_scores(tmp_path, records) + _append_comparisons(run_dir, comparisons) + return run_dir + + def _mixed_run(self, tmp_path: Path) -> Path: + # The advisory row is listed first, so it won every severity tie. + return self._run( + tmp_path, + [ + self._record("notes_watch", 1.0, blocking=False), + self._record("payments", -1.0, blocking=True), + ], + [self._comparison("notes_watch", 0.5), self._comparison("payments", -0.5)], + ) + + def test_executive_summary_prefers_a_blocking_evaluator_over_an_advisory_one( + self, tmp_path: Path + ) -> None: + rows = build_report_payload(self._mixed_run(tmp_path)).executive_summary + assert [(r["prompt_id"], r["evaluator_name"]) for r in rows] == [("greet", "payments")] + + def test_executive_summary_breaks_a_severity_tie_toward_the_worst_effect( + self, tmp_path: Path + ) -> None: + run_dir = self._run( + tmp_path, + [ + self._record("a_improved", 1.0, blocking=True), + self._record("b_regressed", -1.0, blocking=True), + ], + [self._comparison("a_improved", 0.5), self._comparison("b_regressed", -0.5)], + ) + rows = build_report_payload(run_dir).executive_summary + assert [r["evaluator_name"] for r in rows] == ["b_regressed"] + + def test_executive_summary_keeps_a_prompt_whose_only_evaluator_is_advisory( + self, tmp_path: Path + ) -> None: + run_dir = self._run( + tmp_path, + [self._record("notes_watch", 1.0, blocking=False, prompt_id="side")], + [self._comparison("notes_watch", 0.5, prompt_id="side")], + ) + rows = build_report_payload(run_dir).executive_summary + assert [(r["prompt_id"], r["evaluator_name"]) for r in rows] == [ + ("greet", "structural.length"), + ("side", "notes_watch"), + ] + + def test_header_average_follows_the_blocking_row(self, tmp_path: Path) -> None: + html = render_html(build_report_payload(self._mixed_run(tmp_path))) + shown = re.search( + r'Avg score Δ\s*([^<]+)', html + ) + assert shown is not None + assert shown.group(1) == "-0.500" + + def test_evaluator_table_tags_an_advisory_evaluator(self, tmp_path: Path) -> None: + html = render_html(build_report_payload(self._mixed_run(tmp_path))) + tagged = r'{}\s* None: + run_dir = self._run( + tmp_path, + [self._record("payments", 0.0, blocking=True)], + [self._comparison("payments", 0.0)], + ) + html = render_html(build_report_payload(run_dir)) + assert "each side judged against the entry" in html