fix(report): advisory evaluators out of the summary; trace-rule causes counted as the budget does

CHANGELOG.md

@@ -31,7 +31,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   (`blocking: false`) entries included, though only blocking ones count
   toward the budget; those rows are tagged "advisory". `report.json` carries
   the same rows under a new top-level `invariant_violations` key, each with
-  the entry's `blocking` flag.
+  the entry's `blocking` flag. The "Overall, by evaluator" table tags such an
+  entry's row too and says what a trace-rule row measures, and the "Top
+  regression causes" panel counts `INVARIANT_VIOLATION` as the budget does —
+  distinct examples with a blocking target violation — so the narrative never
+  quotes a figure the verdict did not use.
 - `evalshift init` scaffolds `max_invariant_violations: 0` in every
   `--profile` policy block, beside the other budgets. Older CLIs reject the
   field, so bump the CI action's `evalshift-version` pin to this release
@@ -43,6 +47,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- The report's per-prompt Executive summary — and the header's mean score Δ,
+  which averages its rows — now picks each prompt's worst severity with ties
+  broken toward the most negative effect size (the order the hero panel and
+  the narrative already used) instead of whichever evaluator `analyze` listed
+  first, and skips advisory (`blocking: false`) evaluators while the prompt
+  has a gating one. An advisory rule the target satisfied by calling no tools
+  at all could otherwise read as the prompt's "+9% improved" above a failed
+  verdict.
 - Upgrading moves the resume `config_hash`: the new config fields are part of
   every config's canonical dump, so a run left in progress by an older CLI
   cannot be `--resume`d after upgrading (it fails with the usual config-hash

DOCS.md

@@ -608,7 +608,7 @@ Rules are checked per **response** (one model turn) against the tool calls alrea
 | `call_count` | `max_calls`: a model call takes the running count (context first) past it, one violation per excess call. `min_calls`: context plus model calls at the end of the last response fall short (one violation). Both equal = exact | context calls count toward both bounds |
 | `arguments` | a model call to `tool` fails `json_schema` (Draft 2020-12, validated at config load); unparseable arguments fail any schema that requires a field | none |
 
-`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants).
+`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). The **Overall, by evaluator** table tags an advisory entry's row the same way and says what a trace-rule row measures (each side judged against the entry's rules, not against the other side); the **Top regression causes** panel (`failure_categories` in the decision) counts `INVARIANT_VIOLATION` as the budget does — distinct examples with a blocking target violation, never per row and never for advisory entries; and the per-prompt executive summary (with the header's mean score Δ, the mean of its rows) is drawn from blocking evaluators only — worst severity first, ties to the most negative effect size — falling back to advisory ones on a prompt that has nothing else. Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants).
 
 ---
 

docs/configuration.md

@@ -728,14 +728,27 @@ as one broken once, and `delta = target − source`. The record's kind is
 `trace_invariants`; its metadata carries `rules_checked`, `source_violations`
 and `target_violations` (each a list of `{rule_id, rule_type, tool,
 round_index, detail}`), `owner` when set, and the failure category
-`INVARIANT_VIOLATION` when the target broke anything. With
-`samples_per_example` above 1 the example's row keeps every scored sample's
-violations, each tagged with its `sample` ordinal. The HTML report's
-**Trace rules broken** panel and `report.json`'s `invariant_violations` list
-every rule the target broke, whatever the delta — including those of advisory
-(`blocking: false`) entries, which never count toward the budget — with the
-owner and whether the source broke it too. Each row carries the entry's
-`blocking` flag, and the panel tags `blocking: false` rows **advisory**.
+`INVARIANT_VIOLATION` when the target broke anything. The report's **Top
+regression causes** panel (and `failure_categories` in
+`migration_decision.json`) counts that category the way the budget does —
+distinct examples on which the target broke a *blocking* rule — so an advisory
+violation, or a second entry broken on the same example, never inflates it
+past the figure the verdict used. With `samples_per_example` above 1 the
+example's row keeps every scored sample's violations, each tagged with its
+`sample` ordinal. The HTML report's **Trace rules broken** panel and
+`report.json`'s `invariant_violations` list every rule the target broke,
+whatever the delta — including those of advisory (`blocking: false`) entries,
+which never count toward the budget — with the owner and whether the source
+broke it too. Each row carries the entry's `blocking` flag, and the panel tags
+`blocking: false` rows **advisory**. So does the **Overall, by evaluator**
+table, where every `trace_invariants` row also says what it measures: each
+side judged against the entry's rules, not against the other side. The
+per-prompt **Executive summary** — and the header's mean score Δ, which
+averages its rows — is drawn from blocking evaluators only (worst severity
+first, ties to the most negative effect size), falling back to advisory ones
+on a prompt that has nothing else, so an advisory rule the target satisfied by
+calling no tools cannot read as the prompt's improvement above a failed
+verdict.
 
 Four things to know:
 

llms-full.txt

@@ -646,7 +646,9 @@ evaluators:                         # every evaluator config also takes blocking
       # Score per side = (rules - rules broken) / rules; kind slug trace_invariants; metadata
       #   rules_checked, source_violations, target_violations (each [{rule_id, rule_type, tool,
       #   round_index, detail}], with sample ordinal when samples_per_example > 1), owner;
-      #   failure_categories [INVARIANT_VIOLATION] when the target broke anything.
+      #   failure_categories [INVARIANT_VIOLATION] when the target broke anything (the decision's
+      #   failure_categories count it as the budget does: distinct examples with a blocking
+      #   target violation -- never per row, never for advisory entries).
 suites:                             # managed block; capture sync rewrites between
   <name>:                           # ">>> evalshift suites" markers
     source: captured|jsonl = captured
@@ -859,7 +861,12 @@ scored successfully (NOT the raw sample_index). Advisory (blocking: false) entri
 ARE listed; only blocking ones count toward max_invariant_violations. blocking = the record's
 flag (false = advisory entry). report.html renders the same rows as a "Trace rules broken"
 panel (rule, owner, detail, "Source too?"), tagging blocking=false rows with a muted
-"advisory" badge.
+"advisory" badge. The "Overall, by evaluator" table tags an advisory entry's row the same way
+and says what a trace-rule row measures (each side judged against the entry's rules). The
+per-prompt executive summary (and the header's mean score Δ, the mean of its rows) is drawn
+from blocking evaluators only -- worst severity first, ties to the most negative effect size,
+the order the hero panel and the narrative use -- falling back to advisory ones on a prompt
+with nothing else.
 
 ## Migration policy verdict algorithm
 

src/evalshift_cli/analysis/policy.py

@@ -22,6 +22,7 @@
 from evalshift_cli.evaluators.base import EvalRecord
 from evalshift_cli.evaluators.failures import (
     BROKEN_HARNESS_CAUSES,
+    INVARIANT_VIOLATION,
     SEMANTIC_REGRESSION,
     TOOL_GROUND_TRUTH_MISS,
 )
@@ -1485,13 +1486,26 @@ def _blocking_regressions(comparisons: list[ComparisonResult]) -> list[BlockingR
 
 
 def _failure_categories(records: list[EvalRecord]) -> list[FailureCategoryCount]:
+    """Count each failure category over the rows that carry it.
+
+    ``INVARIANT_VIOLATION`` is the exception: every ``trace_invariants`` row
+    carries it when the target broke a rule, advisory or not, source too or
+    not -- so counting rows read "8" in the report's causes panel and the
+    narrative built on it, above a verdict that counted 5. The category is
+    counted the way ``max_invariant_violations`` is, by
+    :func:`_invariant_violation_counts` over the blocking rows, so the two can
+    never disagree.
+    """
     counter: Counter[str] = Counter()
     for r in records:
         raw = r.metadata.get("failure_categories", [])
         if isinstance(raw, list):
             for item in raw:
-                if isinstance(item, str) and item:
+                if isinstance(item, str) and item and item != INVARIANT_VIOLATION:
                     counter[item] += 1
+    violated, _ = _invariant_violation_counts([r for r in records if r.blocking])
+    if violated:
+        counter[INVARIANT_VIOLATION] = violated
     return [
         FailureCategoryCount(category=category, count=count)
         for category, count in counter.most_common()

src/evalshift_cli/reports/html.py

@@ -19,6 +19,7 @@
 from evalshift_cli.analysis.statistics import AXIS_NOTE_PREFIX, UNMEASURED_NOTE_PREFIX
 from evalshift_cli.evaluators.failures import category_label
 from evalshift_cli.evaluators.tool_selection import KIND_CONFORMANCE, KIND_DIVERGENCE
+from evalshift_cli.evaluators.trace_invariants import KIND as KIND_INVARIANTS
 from evalshift_cli.insights.models import Insight
 from evalshift_cli.reports.json import (
     SHARED_GROUND_TRUTH_NOTE_PREFIX,
@@ -161,6 +162,14 @@ def _confidence_label(severity: str, p_corrected: float) -> str:
     KIND_DIVERGENCE: "the target graded against what the source did",
 }
 
+# The same for a single-axis family whose name alone does not say what the
+# score is measured against. A trace-rule row is judged against rules the
+# team wrote, not against the other side: its delta is a difference in rules
+# kept, and a rule both models broke is a zero that is not "no change".
+_KIND_BLURBS: dict[str, str] = {
+    KIND_INVARIANTS: "each side judged against the entry's hand-written rules",
+}
+
 # Statistical tests, spelled out for the row tooltip.
 _TEST_LABELS: dict[str, str] = {
     "paired_t": "paired t-test",
@@ -186,6 +195,11 @@ def _axis_blurb(kind: str) -> str:
     return _AXIS_BLURBS.get(kind, "")
 
 
+def _kind_blurb(kind: str) -> str:
+    """One phrase saying what a family's score is measured against, or ``""``."""
+    return _KIND_BLURBS.get(kind, "")
+
+
 def _evaluator_label(name: str, kind: str = "") -> str:
     """Return a human-readable name for an evaluator id like ``semantic.cosine``.
 
@@ -479,6 +493,7 @@ def render_html(report: ReportData, *, insight: Insight | None = None) -> str:
     env.globals["evaluator_label"] = _evaluator_label
     env.globals["axis_of"] = _axis_of
     env.globals["axis_blurb"] = _axis_blurb
+    env.globals["kind_blurb"] = _kind_blurb
     env.globals["test_label"] = _test_label
     env.globals["regression_reason"] = _regression_reason
     env.globals["tool_list"] = _tool_list