docs: cover trace invariants on the summary surfaces; fix advisory-tag scope and hosted-gate wording

CHANGELOG.md

@@ -10,7 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - `evaluators.trace_invariants`: hand-written rules over tool-call traces —
-  `forbidden`, `required`, `order`, `call_count` (min/max/exact per tool) and
+  `forbidden`, `required`, `order`, `call_count` (`min_calls` and/or
+  `max_calls` per tool; equal bounds = an exact count) and
   `arguments` (JSON Schema) — checked on both sides of every pair, on replayed
   traces or (`traces: imported`) on imported ones. A rule the target breaks
   counts whatever the source did. Its `applies_to` is enforced: a prompt
@@ -23,16 +24,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   rule, so an example that broke rules in two entries counts once. A breach
   fails the run even when the suite is too small for statistics, and a
   slice whose own budget breaches reads `fail` however few comparisons it has.
-  Pushed bundles' `decision.policy` now always carries the field, so pushing
-  needs an evalshift-server that accepts `max_invariant_violations` in policy
-  snapshots; this CLI release waits on that server deploy.
+  `analyze` adds a recommendation when the source broke the same rules the
+  target did: review the rule with its owner rather than raising the budget.
+  Pushed bundles' `decision.policy` now always carries the field; hosted
+  EvalShift accepts and stores it since 2026-10-08 (it never re-evaluates it —
+  a run that carries its own policy is answered with the CLI's decision).
 - The HTML report lists every broken trace rule in a "Trace rules broken"
   panel, with its owner and whether the source broke it too — advisory
   (`blocking: false`) entries included, though only blocking ones count
   toward the budget; those rows are tagged "advisory". `report.json` carries
   the same rows under a new top-level `invariant_violations` key, each with
-  the entry's `blocking` flag. The "Overall, by evaluator" table tags such an
-  entry's row too and says what a trace-rule row measures, and the "Top
+  the entry's `blocking` flag. The "Overall, by evaluator" table tags every
+  advisory (`blocking: false`) evaluator's row the same way and says what a
+  trace-rule row measures, and the "Top
   regression causes" panel counts `INVARIANT_VIOLATION` as the budget does —
   distinct examples with a blocking target violation — so the narrative never
   quotes a figure the verdict did not use.

README.md

@@ -263,7 +263,11 @@ shape, `fail-on` modes, and baseline behavior.
 
 Migrating an agent (a prompt that uses tools)? EvalShift detects
 regressions in *which* tools the new model calls, *what* arguments it
-passes, and in what order and parallelism within a response. By default
+passes, and in what order and parallelism within a response. Hand-written
+`trace_invariants` rules (auth before a charge, a tool that must never be
+called, at most one charge, argument bounds) hold *both* models to a
+contract whatever the source did, gated by
+`migration_policy.max_invariant_violations`. By default
 each example is one model call scored against the first recorded round;
 promote with `--rounds all` to replay every recorded round teacher-forced,
 with the recorded tool results fed back and each round scored on its own.
@@ -289,7 +293,10 @@ assets, works offline) has:
 
 * **Migration verdict** — the policy decision up top: which budgets failed, the
   top regression causes, and the recommendation.
-* **Executive summary** — one row per prompt with a severity badge.
+* **Executive summary** — one row per prompt with a severity badge, drawn from
+  blocking evaluators (advisory ones only when nothing else scored the prompt).
+* **Trace rules broken** — every hand-written trace rule the target broke,
+  with its owner and whether the source broke it too; advisory rows are tagged.
 * **What changed, in plain language** — the verdict, the economics and the
   behavioural drift explained by `defaults.insights_model`. Every figure in it
   is copied from the computed statistics, never generated. Needs a provider

docs/agents.md

@@ -22,8 +22,9 @@ The killer scenario it catches:
 
 * **`ToolTrace` data model**: provider-agnostic, populated from
   Anthropic / OpenAI / Gemini responses.
-* **Three new evaluators**: `tool_selection`, `tool_arguments`,
-  `tool_trace_structure`.
+* **Four new evaluators**: `tool_selection`, `tool_arguments`,
+  `tool_trace_structure`, and hand-written `trace_invariants` (both sides
+  judged against your rules, not against each other).
 * **Suite extension**: optional `expected_tools`, `expected_tool_count`,
   `expected_no_tools`, `expected_parallel` per example, plus a required
   `toolset_ref` or inline `tools` — see [Suite ground truth](#suite-ground-truth).
@@ -85,7 +86,9 @@ defaults:
 
 evaluators:
   # Top-level: what every suite is scored with. Tool evaluators do not belong
-  # here — `capture sync` writes them per suite (see below).
+  # here — `capture sync` writes them per suite (see below). Hand-written
+  # `trace_invariants` are the exception: top level is fine, and sync keeps
+  # them on the managed entries it rewrites.
   semantic:
     embedding_model: gemini/gemini-embedding-001
     blocking: false
@@ -600,6 +603,13 @@ identically is not a ✗ — it is the same pair before and after, and what
 is wrong is the ground truth. Look for **Ground truth missed by both**
 in the evaluator table when that happens.
 
+A **Trace rules broken** panel appears when the target broke a hand-written
+[`trace_invariants`](configuration.md#evaluatorstrace_invariants) rule: every
+broken rule with its owner and whether the source broke it too, whatever the
+delta, advisory (`blocking: false`) rows tagged **advisory**. `report.json`
+carries the same rows under `invariant_violations`, and the evaluator table
+tags an advisory entry's row the same way.
+
 ## Troubleshooting
 
 * **A suite reports tool evaluator scores you didn't expect on some rows** —

docs/configuration.md

@@ -504,7 +504,7 @@ A list. Each entry has:
 | `name`            | string | (required)    | Identifier surfaced in reports.  |
 | `conformance`     | enum   | `expected`    | Ground-truth axis: `expected` / `expected_set` / `off`. `expected_set` is `expected` made order-insensitive: multiset recall of `example.expected_tools` names. Use it when the expected calls are a parallel fan-out whose order carries no meaning. |
 | `divergence`      | enum   | `set`         | Target-vs-source axis: `set` (Jaccard on tool names) / `exact` (sequence equality) / `first` (first call only) / `off`. |
-| `applies_to`      | list   | `["*"]`       | Glob list of prompt ids. |
+| `applies_to`      | list   | `["*"]`       | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `severity_floor`  | enum   | `null`        | If set, surfaces in metadata so the analysis layer can floor severity. |
 
 The two axes are independent and each writes **its own record**, under
@@ -535,7 +535,7 @@ deprecated, and a config still carrying one fails validation.
 | Field                   | Type   | Default | Description |
 | ----------------------- | ------ | ------- | ----------- |
 | `name`                  | string | (required) | Identifier. |
-| `applies_to`            | list   | `["*"]` | Glob list. |
+| `applies_to`            | list   | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `against`               | enum   | `source` | What arguments are compared to: `source` (drift from the source model) or `expected` (correctness against `expected_tools[].arguments`, scored on both sides). |
 | `strategies`            | dict   | `{}`    | Per-field strategy overrides (`exact`/`subset`/`numeric`/`semantic`/`auto`). |
 | `default_strategy`      | enum   | `auto`  | Strategy for fields `strategies` does not name. `auto` is the ladder below; `exact` restores byte-equality scoring. |
@@ -611,7 +611,7 @@ unaffected: it counts calls whose *target* score fell below
 | Field                 | Type   | Default | Description |
 | --------------------- | ------ | ------- | ----------- |
 | `name`                | string | (required) | Identifier. |
-| `applies_to`          | list   | `["*"]` | Glob list. |
+| `applies_to`          | list   | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `check_call_count`    | bool   | `true`  | Score the number of tool calls. |
 | `check_parallelism`   | bool   | `true`  | Score parallel-vs-sequential alignment. |
 | `check_refusals`      | bool   | `true`  | Score refusal alignment; mismatches force `severity_floor: high`. |
@@ -626,7 +626,7 @@ the normal `raw.jsonl` model-call artifact.
 | Field                        | Type   | Default | Description |
 | ---------------------------- | ------ | ------- | ----------- |
 | `name`                       | string | required | Identifier surfaced in scores and reports. |
-| `applies_to`                 | list   | `["*"]` | Glob list of prompt ids. |
+| `applies_to`                 | list   | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). |
 | `check_tool_order`           | bool   | `true`  | Compare source and target tool-call order. |
 | `check_arguments`            | bool   | `true`  | Compare arguments for same-name matched tool calls. |
 | `check_missing_verification` | bool   | `true`  | Check dangerous tools have an earlier verification tool. |
@@ -740,9 +740,10 @@ example's row keeps every scored sample's violations, each tagged with its
 whatever the delta — including those of advisory (`blocking: false`) entries,
 which never count toward the budget — with the owner and whether the source
 broke it too. Each row carries the entry's `blocking` flag, and the panel tags
-`blocking: false` rows **advisory**. So does the **Overall, by evaluator**
-table, where every `trace_invariants` row also says what it measures: each
-side judged against the entry's rules, not against the other side. The
+`blocking: false` rows **advisory**. The **Overall, by evaluator** table tags
+every advisory (`blocking: false`) evaluator's row the same way, and every
+`trace_invariants` row there also says what it measures: each side judged
+against the entry's rules, not against the other side. The
 per-prompt **Executive summary** — and the header's mean score Δ, which
 averages its rows — is drawn from blocking evaluators only (worst severity
 first, ties to the most negative effect size), falling back to advisory ones

docs/evaluators.md

@@ -207,10 +207,12 @@ Every other tool evaluator treats "the source did it" as correct:
 the source, so a wrong call both models make scores as agreement.
 `trace_invariants` judges **both** sides against rules your team wrote —
 `forbidden` tools, `required` tools, an `order` between two tools, a
-`call_count` (min, max or exact per tool) and `arguments` that must satisfy a
-JSON Schema. A rule the target breaks counts against it whatever the source
-did, and `migration_policy.max_invariant_violations` (default `0`) fails the
-run on it, even on a suite too small for statistics.
+`call_count` (`min_calls` and/or `max_calls` per tool; equal bounds = an exact
+count) and `arguments` that must satisfy a JSON Schema. A rule the target
+breaks counts against it whatever the source did, and for a blocking entry
+(the default) `migration_policy.max_invariant_violations` (default `0`) fails
+the run on it, even on a suite too small for statistics; `blocking: false`
+entries are reported but never gate.
 
 Write rules only for invariants where a silent miss is expensive: auth before
 a write, a deprecated endpoint that must stay unused, at most one charge,
@@ -227,8 +229,9 @@ recording. Configuration, rule semantics and scoring:
 
 ## Mixing evaluators
 
-You can configure several at once. They all run on every (prompt,
-example) pair, and the analysis layer treats each as a separate
+You can configure several at once. Most run on every (prompt, example)
+pair — `trace_invariants` only on the prompts its `applies_to` matches,
+and only on pairs with a trace to check — and the analysis layer treats each as a separate
 comparison (so BH correction adjusts for the multiple-test count
 correctly).
 

docs/faq.md

@@ -98,7 +98,9 @@ since, and continues from where it left off. Already-completed calls
 (including ones that errored at the LLM layer) are skipped.
 
 A config change or a different suite path between attempts aborts the
-resume — start a fresh run instead. The suite's *contents* are not checked,
+resume — start a fresh run instead. Upgrading the CLI counts when the release
+added config fields (they join the hashed canonical dump), so an older CLI's
+in-progress run cannot be resumed after upgrading. The suite's *contents* are not checked,
 so after editing examples, start a fresh run yourself.
 
 ## How do I push a run to hosted EvalShift?
@@ -180,6 +182,11 @@ Three common causes, all by design:
    confirm it** at this suite size — grow the suite.
 3. **All comparisons were `insufficient`** (n < 5).
 
+A breached `max_invariant_violations` is the exception to all three: it is a
+count, conclusive however small the sample, so a blocking hand-written trace
+rule the target broke reads `fail`, never `inconclusive` — even when every
+comparison is insufficient.
+
 `analyze` and `compare` print the specific reason and the recommended fix
 under the verdict line, and record them in `migration_decision.json`
 (`reason` / `recommendations`).

docs/hosted.md

@@ -296,7 +296,7 @@ The bundle itself contains:
 | --- | --- |
 | `manifest` | Run id, `org/project` slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the **local suite file path** as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. |
 | `examples[]` — one row per prompt × example | The example's template variables (`inputs`) **verbatim**; its `expected` reference output **verbatim**; both models' **full output text**; tool-call traces from the replay: each side's tool calls (names, arguments, call ids) with round markers, any final text and refusal messages, capped at 256 KB per side. Imported agent traces (`traces import`) are not uploaded; per-evaluator scores and error strings; per-side cost and latency; tags and slice names. |
-| `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It therefore carries `max_invariant_violations` (default `0`) even when the config has no trace rules, which the server must accept. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. |
+| `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It therefore carries `max_invariant_violations` (default `0`) even when the config has no trace rules; hosted EvalShift accepts and stores it (since 2026-10-08) but never re-evaluates it — a run that carries its own policy is answered with the CLI's decision, which did check it, and only runs pushed without a policy fall back to the server's six-budget re-check. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. |
 | `methodology_notes` | The model ids and the statistical-contract sentences shown in every report. |
 | `insights` | The machine-written run narrative, when one was generated. It is prose *about* your run and can paraphrase or quote the regressions it summarizes. |
 | `evaluator_config` | Config version; the prompt list **metadata only** — prompt names, file paths, and variable names, with every prompt body replaced by a `content_hash`; the whole `defaults` block (model ids, concurrency, cache flag, cost ceiling, max_tokens, samples_per_example); an always-empty `slices` list (a legacy key, kept so `eval_config_hash` does not change — the top-level `slices:` config key was removed); and the full evaluators block — which includes each `llm_judge` entry's `criterion_prompt` text, so keep judge criteria free of secrets. An empty `trace_invariants` list is left out of it, so a config without trace rules keeps its `eval_config_hash`; adding rules changes the hash like any other evaluator edit. |

docs/methodology.md

@@ -336,10 +336,11 @@ counted every one of them and still measured nothing, so it reports a positive
 denominator beside `conclusive: false`. Reading either field as a stand-in for
 the other loses that.
 
-The field exists for the hosted gate. `GET /runs/{run_id}/policy-check`
-re-decides an uploaded run against the project's *current* policy and never saw
-the underlying records, so without a sample size it cannot tell a clean run from
-an empty one. Bundles written before the field omit it entirely, and a missing
+The field exists for the hosted gate. For a run pushed without a policy,
+`GET /runs/{run_id}/policy-check` re-decides it against the project's *current*
+policy (a run that carries its own `decision.policy` is answered with the CLI's
+stored decision instead), and the server never saw the underlying records, so
+without a sample size it cannot tell a clean run from an empty one. Bundles written before the field omit it entirely, and a missing
 `denominator` means **unknown, not zero** — the server falls back to
 `conclusive` for those. This CLI always knows its own denominators, so every
 budget it emits carries an integer.

docs/traces.md

@@ -137,7 +137,10 @@ evaluators:
 An imported trace is checked as one response with no prior context: every call
 in it, in `sequence_index` order, was the agent's own. A rule the target breaks
 counts toward `migration_policy.max_invariant_violations` whatever the source
-did.
+did, when the entry is blocking (the default); `blocking: false` entries are
+reported but never gate. The entry's `applies_to` scopes imported pairs by
+`prompt_id`, and `evaluate` fails on a run with no `traces.jsonl`, naming
+`trace_invariants (traces: imported)` in the error.
 
 Debug commands become trace-aware: