diff --git a/CHANGELOG.md b/CHANGELOG.md index 4acae47a..9f1bc457 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- `evaluators.trace_invariants`: hand-written rules over tool-call traces — + `forbidden`, `required`, `order`, `call_count` (min/max/exact per tool) and + `arguments` (JSON Schema) — checked on both sides of every pair, on replayed + traces or (`traces: imported`) on imported ones. A rule the target breaks + counts whatever the source did. Its `applies_to` is enforced: a prompt + outside the globs is never checked, and `evaluate` warns, naming the entry + and its `applies_to`, when the globs matched nothing in the run (so a typo + is not a silent clean run). A `traces: imported` entry on a run without + `traces.jsonl` fails `evaluate` with an error naming that family. +- `migration_policy.max_invariant_violations` (default `0`, per-slice + overridable): distinct examples on which the target broke a blocking trace + rule, so an example that broke rules in two entries counts once. A breach + fails the run even when the suite is too small for statistics, and a + slice whose own budget breaches reads `fail` however few comparisons it has. + Pushed bundles' `decision.policy` now always carries the field, so pushing + needs an evalshift-server that accepts `max_invariant_violations` in policy + snapshots; this CLI release waits on that server deploy. +- The HTML report lists every broken trace rule in a "Trace rules broken" + panel, with its owner and whether the source broke it too — advisory + (`blocking: false`) entries included, though only blocking ones count + toward the budget; those rows are tagged "advisory". `report.json` carries + the same rows under a new top-level `invariant_violations` key, each with + the entry's `blocking` flag. +- `evalshift init` scaffolds `max_invariant_violations: 0` in every + `--profile` policy block, beside the other budgets. Older CLIs reject the + field, so bump the CI action's `evalshift-version` pin to this release + before committing a freshly scaffolded config. +- `capture sync` keeps a managed entry's hand-written + `evaluators.trace_invariants` block when it regenerates the entry. The + marker comment still says hand edits are overwritten; this block is the one + exception. + +### Changed + +- Upgrading moves the resume `config_hash`: the new config fields are part of + every config's canonical dump, so a run left in progress by an older CLI + cannot be `--resume`d after upgrading (it fails with the usual config-hash + mismatch). Start a fresh run; cached responses are reused. + ## [1.2.2] - 2026-10-05 ### Fixed diff --git a/DOCS.md b/DOCS.md index c0da4e9e..a4139f3b 100644 --- a/DOCS.md +++ b/DOCS.md @@ -225,7 +225,7 @@ Run ids look like `r_20260722_golden_a1b2c3` (`r___`). Un | `scores.jsonl` | evaluate | One line per (pair × evaluator): source/target scores, delta, explanation | | `analysis.json` | analyze | Per-comparison statistics, severities, notes | | `migration_decision.json` | analyze | Policy verdict + per-budget detail (only when `migration_policy` set) | -| `report.json` | report | The report payload | +| `report.json` | report | The report payload, including `invariant_violations`: one row per [trace rule](#trace-invariants) the target broke | | `report.html` | report | Single-file HTML report | | `insights.json` | report | Cached [run insights](#run-insights) narrative (optional; skipped with `--no-insights` or no API key) | | `traces.jsonl` | traces import | Bring-your-own agent traces (optional) | @@ -294,6 +294,7 @@ migration_policy: min_equivalence_rate: 0.75 max_tool_argument_drift: 0.20 max_tool_divergence: 0.20 + max_invariant_violations: 0 max_cost_increase: 0.30 max_latency_increase: 0.30 @@ -389,7 +390,7 @@ suites: Resolution is **family-level replacement**, applied by one method (`EvalShiftConfig.evaluators_for`) that `evaluate`, the report and the hosted bundle all route through, so the scored set and the reported set cannot drift apart. A family the suite does not mention is inherited from the top level; a family it mentions replaces the top-level one wholesale (no deep merge, no per-name merge); a family written as `[]` or `null` **removes** what would have been inherited — absent and `null` are different instructions. A run launched with a raw `--suite `, or with a `--suite-name` that has no entry, resolves to the top-level block, so nothing that predates per-suite evaluators changes behaviour. -Derivation, on every `capture sync`: no row offered a toolset → **no block** (`structural` is never derived — nothing in a capture says what shape an answer must have); any row offered a toolset → `tool_selection` (`conformance: expected` + `divergence: set`); any row recorded call arguments → `tool_arguments` (`against: expected`, no `strategies:` — the default `auto` already grades free text by meaning). Generated names are stable by contract (`routing`, `routing_args`) because reports key on evaluator names across runs. Sync regenerates only the suites it just promoted and carries the rest of the region forward verbatim; `managed: false` on an entry freezes it, and sync prints the block it would have written instead. +Derivation, on every `capture sync`: no row offered a toolset → **no block** (`structural` is never derived — nothing in a capture says what shape an answer must have); any row offered a toolset → `tool_selection` (`conformance: expected` + `divergence: set`); any row recorded call arguments → `tool_arguments` (`against: expected`, no `strategies:` — the default `auto` already grades free text by meaning). Generated names are stable by contract (`routing`, `routing_args`) because reports key on evaluator names across runs. Sync regenerates only the suites it just promoted and carries the rest of the region forward verbatim; `managed: false` on an entry freezes it, and sync prints the block it would have written instead. One hand edit survives regeneration: a managed entry's `evaluators.trace_invariants` block is copied verbatim into the entry sync rewrites, because no capture can derive a team's rules. The marker comment still says hand edits inside the region are overwritten — its text is matched exactly to find the region, so it cannot change — and this is the documented exception to it. ### Model ids @@ -512,10 +513,11 @@ Pairwise A/B comparison per criterion: the judge sees the two outputs anonymised - **`tool_arguments`** — same tools, different arguments? Calls matched greedily by `(tool_name, nearest sequence_index)`; each argument field scored by a per-field strategy: `exact`, `subset`, `numeric` (relative error, linear decay to 0 at `numeric_tolerance`, default 0.05), `semantic` (embedding cosine, borrows the configured `semantic` evaluator's model and cache — without one it degrades to `exact`), or `auto`. Fields `strategies` does not name are scored by `default_strategy`, **`auto`** by default: a ladder that (1) scores two strings equal after normalizing case and whitespace at 1.0, (2) dispatches on the field's declared type in the toolset the example carries — identifiers (`*_id`/`*_ids`), enums, booleans and `date`/`date-time`/`uuid`/`email` formats → `exact`, numbers → `numeric`, objects/arrays → `subset`, and (3) *grades* whatever is left, which is free text: embedding similarity when a `semantic` block lent a model, `difflib` ratio when not, so partial credit survives with no embedding model. `default_strategy: exact` restores byte-equality scoring; a `strategies` entry always wins. A field present on one side only scores 0.5 (`optional_fields_scored: strict` restores 0.0): omitting an optional parameter is a real difference, not a wrong value. `against: expected` switches the comparison from drift-vs-source (where `source_score` is 1.0 by construction, so a hallucinating source defines the yardstick) to correctness — **both** sides scored against `expected_tools[].arguments`, with each expectation's `match_strategy` picking the compared keys (`exact` = union, `subset`/`contains_per_field` = recorded keys only). An expected call the model never made scores 0; an example with no expected arguments is skipped at a neutral 1.0/1.0. On a multi-round replay calls are paired **within** a round (a right call in the wrong round is a miss), the score is the mean over rounds that had something to score, and `metadata.rounds` carries the per-round detail. A ground-truth field **neither** side produced is dropped from that call's denominator on both sides and disclosed as `unmeasured_fields` in the per-call metadata — a stale expectation would otherwise cap the call below 1.0 forever, since no model change could lift it. - **`tool_trace_structure`** — shape of the trace: call-count drift (`call_count_tolerance`, default 1), parallelism match, refusal alignment (a refusal mismatch forces severity ≥ high and flags `REFUSAL_REGRESSION`), and `expected_tool_count` when set. Toggles: `check_call_count` / `check_parallelism` / `check_refusals` (all default on). - **`agent_trace`** — for imported external traces; see [External agent traces](#external-agent-traces). +- **`trace_invariants`** — hand-written rules checked on both sides, independent of what the source did; see [Trace invariants](#trace-invariants). ### Failure categories -Regressions carry machine-readable labels that the report and hosted diff group by: `FORMAT_FAILURE`, `SEMANTIC_REGRESSION`, `TOOL_SELECTION_DRIFT`, `TOOL_GROUND_TRUTH_MISS` (both models missed the same recorded tool ground truth — a broken-harness signal, not a migration finding; see [Migration policy and CI gating](#migration-policy-and-ci-gating)), `ARGUMENT_VALUE_DRIFT`, `TOOL_TRACE_STRUCTURE_DRIFT`, `TOOL_ORDER_DRIFT`, `DANGEROUS_ACTION_DRIFT`, `MISSING_VERIFICATION_STEP`, `UNNECESSARY_TOOL_CALL`, and `REFUSAL_REGRESSION`. That is the complete set — every label the evaluators emit is declared in `evaluators/failures.py`. The machine labels live in `scores.jsonl`, `report.json` and the bundle; every rendered surface (the HTML report, decision prose, the run narrative) shows the plain-language display name instead — `TOOL_SELECTION_DRIFT` renders as "Different tools chosen" — with the mapping declared beside the labels in `evaluators/failures.py`. +Regressions carry machine-readable labels that the report and hosted diff group by: `FORMAT_FAILURE`, `SEMANTIC_REGRESSION`, `TOOL_SELECTION_DRIFT`, `TOOL_GROUND_TRUTH_MISS` (both models missed the same recorded tool ground truth — a broken-harness signal, not a migration finding; see [Migration policy and CI gating](#migration-policy-and-ci-gating)), `ARGUMENT_VALUE_DRIFT`, `TOOL_TRACE_STRUCTURE_DRIFT`, `TOOL_ORDER_DRIFT`, `DANGEROUS_ACTION_DRIFT`, `MISSING_VERIFICATION_STEP`, `UNNECESSARY_TOOL_CALL`, `REFUSAL_REGRESSION`, and `INVARIANT_VIOLATION` (the target broke a hand-written [trace rule](#trace-invariants), stamped whatever the source did). That is the complete set — every label the evaluators emit is declared in `evaluators/failures.py`. The machine labels live in `scores.jsonl`, `report.json` and the bundle; every rendered surface (the HTML report, decision prose, the run narrative) shows the plain-language display name instead — `TOOL_SELECTION_DRIFT` renders as "Different tools chosen" — with the mapping declared beside the labels in `evaluators/failures.py`. `ARGUMENT_VALUE_DRIFT` counts **regressions**: it is stamped only when the target scored below the source. Under `against: expected` both models can miss the same recorded expectation by the same margin — a zero delta, and a fact about your ground truth rather than a migration defect, already reported as such. Policy budgets are unaffected by the label: `max_tool_argument_drift` counts calls whose *target* score fell below `tool_argument_drift_floor`. @@ -573,6 +575,41 @@ Per-example expectations (`expected_tools`, `expected_no_tools`, `expected_tool_ Derived expectations are marked `provenance: captured` (the default), meaning their arguments were transcribed verbatim from the source model's own recorded call — nobody has checked that they are *right*. On such a row the source scores 1.0 **by construction**, so an `against: expected` gate degenerates into what `against: source` already measured: target deviation from source. The run discloses that rather than letting `source_score: 1.0` read as evidence — when every scored row is `captured`, `migration_decision.json` gains a recommendation naming the count and the caveat. Set `provenance: reviewed` on a row once a human has confirmed its arguments; scoring is identical either way, and the disclosure goes silent as soon as one row is reviewed (a blanket disclaimer would understate the rows someone did check). +### Trace invariants + +Every other tool evaluator treats "the source did it" as correct, so a wrong call both models make reads as agreement. `evaluators.trace_invariants` holds **both** sides to rules your team wrote, and a rule the target breaks counts against it whatever the source did. Use it for the few invariants where a silent miss is expensive — auth before a write, a deprecated endpoint, at most one charge, argument bounds — and keep production traces as the broad corpus. An ordering that matters belongs in an `order` rule, not in the recorded `expected_tools`, whose order `conformance: expected` grades as recorded, incidental or not. + +```yaml +evaluators: + trace_invariants: + - name: payments_contract + owner: "@payments-team" # optional; shown on every violation + applies_to: ["checkout-*"] # prompt-id globs, enforced here; default ["*"] + blocking: true # default; false = reported, never gates + traces: replayed # default; `imported` checks `traces import` traces + rules: + - {id: auth-before-charge, type: order, before: authenticate, after: charge_card} + - {id: no-legacy-refund, type: forbidden, tools: [refund_v1]} + - {id: one-charge, type: call_count, tool: charge_card, max_calls: 1} + - {id: must-book, type: required, tools: [create_booking]} + - id: sane-amount + type: arguments + tool: charge_card + json_schema: {type: object, required: [amount], properties: {amount: {type: number, minimum: 0.01, maximum: 5000}}} +``` + +Rules are checked per **response** (one model turn) against the tool calls already in its context: on a replay, the `history` turns' calls plus the *recorded* earlier rounds a teacher-forced replay fed back (never the model's own earlier rounds); an imported trace is one response with an empty context. Context calls are never violations, but they can satisfy or count toward a rule: + +| `type` | Violated when | Context role | +|---|---|---| +| `forbidden` | the model calls any tool in `tools` (one violation per call) | none | +| `required` | some tool in `tools` is called in no response by the model | none — use `call_count` `min_calls` when the conversation may already contain the call | +| `order` | the model calls `after` with no `before` earlier, in the context or earlier in the same response | a context `before` satisfies it | +| `call_count` | `max_calls`: a model call takes the running count (context first) past it, one violation per excess call. `min_calls`: context plus model calls at the end of the last response fall short (one violation). Both equal = exact | context calls count toward both bounds | +| `arguments` | a model call to `tool` fails `json_schema` (Draft 2020-12, validated at config load); unparseable arguments fail any schema that requires a field | none | + +`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants). + --- ## Multi-turn conversations @@ -628,6 +665,8 @@ evaluators: Extra dangerous calls on the target flag `DANGEROUS_ACTION_DRIFT`; a dangerous call with no preceding verification tool flags `MISSING_VERIFICATION_STEP`; order changes flag `TOOL_ORDER_DRIFT`. +`dangerous_tools` compares the target against the source, so a dangerous call both sides made is not flagged. To hold both sides to rules instead, add a [`trace_invariants`](#trace-invariants) entry with `traces: imported`: each imported trace is checked as one response with an empty context, and a rule the target breaks counts toward `max_invariant_violations` whatever the source did. + Trace-aware debugging: `evalshift diff case ` (side-by-side trace diff when traces exist, text diff otherwise), `evalshift inspect case `, `evalshift replay case --model target --trace`. --- @@ -673,6 +712,7 @@ Budgets (fractions, not percents): | `min_equivalence_rate` | 0.75 | Floor on the non-regression rate (equivalent **or improved** both count) | | `max_tool_argument_drift` | 0.20 | Share of records with argument drift | | `max_tool_divergence` | 0.20 | Share of `tool_selection.divergence` records where the target called different tools than the source | +| `max_invariant_violations` | 0 | **Count** of examples on which the target broke a rule of a blocking [`trace_invariants`](#trace-invariants) evaluator, whatever the source did. A breach always fails | | `tool_argument_drift_floor` | 0.9 | Target argument score below which a call counts as drifted. Argument scores are continuous — without a floor every reworded query counts as drift. | | `max_cost_increase` | 0.30 | Relative avg-cost increase, target vs source | | `max_latency_increase` | 0.30 | Relative avg-latency increase | @@ -685,12 +725,13 @@ How the verdict is computed: - **Only blocking evaluators gate quality.** Advisory records are summarised separately (`advisory`, `advisory_regressions`) and never flip the verdict. If *every* configured evaluator is advisory (the fresh-`init` state), the verdict is `inconclusive` with guidance to promote evaluators to blocking as the suite grows — **unless** the call-derived `max_cost_increase`/`max_latency_increase` budgets are breached, which still **fails** the run: those measure the calls, not the evaluators. - **The four rate budgets are Wilson-CI-aware.** `max_overall_regression_rate`, `min_equivalence_rate`, `max_tool_argument_drift` and `max_tool_divergence` are each a count of records over a count of records, so each carries a 95% Wilson interval (`ci_low`/`ci_high`) on its rate — the first three are also the ones the hosted gate computes an interval for. A breach of one of them only **fails** when the interval confirms the breach (the CI's favourable bound still clears the budget → the suite is too small to be sure → `inconclusive`, not `fail`). A budget the observation *held* stays conclusive however wide its interval: a thin sample must never turn a clean run into a caveat. Cost and latency budgets are exact — a measured breach of one is never softened — but they are conclusive only when they measured something: their ratio falls back to `0.00` both when either arm has no error-free call and when **both** arms average zero (unpriced models, whose calls carry `cost_usd`/`latency_ms` of `0` on every row), and either way `conclusive: false`. The record-derived budgets (regression rate, equivalence rate, critical count, tool-argument drift, tool-selection divergence) report `conclusive: false` on a scope that scored zero records, since their `0/0` default measures nothing. The two per-axis budgets need one of *their own* rows: a scope that scored plenty but ran no `tool_arguments` evaluator, or set `divergence: off`, is unmeasured too. +- **A broken trace rule fails, however small the suite.** `max_invariant_violations` counts distinct examples — `(prompt_id, example_id)` pairs with any blocking `trace_invariants` row whose target score is below `1.0`, so a row averaged over repeated samples counts when any sample broke a rule, and an example that broke rules in two entries (each entry writes its own row) counts once. Its `denominator` is the distinct examples with a blocking `trace_invariants` row. It is conclusive by construction — no interval, no `1/n` warning — and a breach overrides `inconclusive` and `conditional_pass` to **`fail`**, even when every comparison is `insufficient`. The same holds per slice: a slice whose own `max_invariant_violations` breaches reads `fail` however few comparisons it has. The `reason` names the worst scope and its count. A violation the source shares still counts; it also adds a `recommendations` line naming the rule, since either both models break it or the rule no longer matches the toolset and its owner should review it. The budget's row appears only in scopes that scored at least one blocking `trace_invariants` row; advisory-only rules add none. - **A rate ceiling finer than `1/n` is flagged as zero-tolerance.** Rates are counted over whole rows, so on a ten-row suite the achievable tool-argument drift rates are `{0.0, 0.1, …}` and a `0.01` budget is really "any drift at all". `analyze` adds a line to the recommendations — shown in the terminal and the HTML report — naming the budget, its value, the granularity and the denominator: `The tool-argument drift budget of 1% (max_tool_argument_drift in evalshift.yaml) is below the 10% granularity of 10 tool-argument comparisons — effective tolerance is zero at this sample size.` It applies to `max_overall_regression_rate`, `max_tool_argument_drift` and `max_tool_divergence`, per scope (slices use their own row counts). Nothing is emitted for a deliberate `0.0` budget, for a denominator of `0` (already reported by `conclusive: false`), for the count/ratio budgets, or for the `min_equivalence_rate` floor. No default changes and no verdict moves — see [methodology](docs/methodology.md). - **A shared ground-truth miss is not equivalence.** A `tool_selection.conformance` row grades each side *absolutely* against the ground truth the suite recorded, so both sides can miss it at the same height — `0.0 / 0.0` on an example both models answered with a tool call the recording never made. The delta is zero, which used to read as equivalence and is the whole of the `equivalent_rate: 1.0` a real run once shipped over a suite where nine pairs in ten routed differently. Those rows now leave **every** policy rate: they measure the harness (wrong toolset attached, wrong prompt, a suite promoted from a different agent), not the migration. They are still reported — as a `TOOL_GROUND_TRUTH_MISS` count in `failure_categories` and as a recommendations line naming how many were excluded. Only the *shared-height* case is dropped: a conformance row the target lost ground on (`0.8 / 0.3`) is still a regression and one it improved on (`0.2 / 0.6`) is still an improvement. A run whose every blocking row was a shared miss is `inconclusive`, and its recommendation is **Fix the eval harness before collecting more examples** — more pairs from the same setup are more excluded rows, so the denominator stays empty however many are added. The exclusion is deliberately narrower than the diagnosis, which is why `evaluate` runs its own broken-harness check over the **source side alone**: a suite the source fails and the target happens to satisfy has a *positive* delta and no `TOOL_GROUND_TRUTH_MISS` tag, so the exclusion rule cannot see it and it is exactly as misconfigured. - **A cost/latency ratio measured only from zeroes says so.** When the calls exist but every `cost_usd` (or `latency_ms`) is `0` on both models, `analyze` adds a recommendations line beside the `conclusive: false`: `The cost increase budget could not be measured: all 4 error-free calls across both models recorded a cost of 0, so its observed 0.00 is a default, not a measurement.` A run with no calls gets no line — the empty `raw.jsonl` already explains itself. Emitted once per run, since every scope reads the same calls. -- **Every budget reports its own `denominator`** — the sample `observed` was computed over. Scored records for the regression rate, the equivalence rate and the critical count — counted over *measurements*, so an evaluator scoring two axes contributes two rows per example; `tool_arguments` rows for tool drift; `tool_selection.divergence` rows for tool divergence; the error-free calls behind both averages for the cost and latency ratios. Slices report their own counts. `0` means "counted, and the sample was empty", so `observed` is a default; a *missing* `denominator` means "no sample size reported" and is **not** zero — only bundles written before the field say that, and the hosted gate falls back to `conclusive` for them. It is the same number the `1/n` granularity warning is judged on, and it is orthogonal to `conclusive`: an all-zero cost ratio counted every call it averaged and still measured nothing, so it reports a positive denominator beside `conclusive: false`. The hosted gate derives its own Wilson interval from these denominators, over the same three rate budgets and with the same confidence constant the CLI uses, so a local verdict and a hosted one now agree on whether a breach was confirmed — see [methodology](docs/methodology.md). +- **Every budget reports its own `denominator`** — the sample `observed` was computed over. Scored records for the regression rate, the equivalence rate and the critical count — counted over *measurements*, so an evaluator scoring two axes contributes two rows per example; `tool_arguments` rows for tool drift; `tool_selection.divergence` rows for tool divergence; the distinct examples with a blocking `trace_invariants` row for the trace-rule count; the error-free calls behind both averages for the cost and latency ratios. Slices report their own counts. `0` means "counted, and the sample was empty", so `observed` is a default; a *missing* `denominator` means "no sample size reported" and is **not** zero — only bundles written before the field say that, and the hosted gate falls back to `conclusive` for them. It is the same number the `1/n` granularity warning is judged on, and it is orthogonal to `conclusive`: an all-zero cost ratio counted every call it averaged and still measured nothing, so it reports a positive denominator beside `conclusive: false`. The hosted gate derives its own Wilson interval from these denominators, over the same three rate budgets and with the same confidence constant the CLI uses, so a local verdict and a hosted one now agree on whether a breach was confirmed — see [methodology](docs/methodology.md). - Any conclusive budget failure, or any blocking critical/high-severity comparison → **`fail`**. Any lower-severity blocking regression → **`conditional_pass`**. Otherwise → **`pass`**. -- **`migration_policy` is the single source of truth for these budgets.** `migration_decision.json`'s `policy` field is the resolved policy `analyze` computed the verdict under — every top-level budget with its default applied, plus `slices`. `bundle`/`push` do not read `migration_decision.json`; `bundle` re-resolves the verdict from `evalshift.yaml` at bundle time, so the bundle's `decision.policy` (see [Hosted EvalShift](#hosted-evalshift)) is the policy the config held when the bundle was built — editing `migration_policy` between `analyze` and `bundle` changes what the bundle carries. `push ` builds a bundle only when `run_bundle.json.gz` is missing — an existing bundle is uploaded as-is — so re-run `evalshift bundle ` after a config edit. Either way, that is what lets the hosted gate check a pull request against exactly the budgets the bundle's own verdict used, instead of a separate, web-edited policy. `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. `evalshift.yaml` is the source of truth for that policy, and the web app's project policy view is becoming a read-only display of the snapshot each run pushed. +- **`migration_policy` is the single source of truth for these budgets.** `migration_decision.json`'s `policy` field is the resolved policy `analyze` computed the verdict under — every top-level budget with its default applied, plus `slices`. `bundle`/`push` do not read `migration_decision.json`; `bundle` re-resolves the verdict from `evalshift.yaml` at bundle time, so the bundle's `decision.policy` (see [Hosted EvalShift](#hosted-evalshift)) is the policy the config held when the bundle was built — editing `migration_policy` between `analyze` and `bundle` changes what the bundle carries. `push ` builds a bundle only when `run_bundle.json.gz` is missing — an existing bundle is uploaded as-is — so re-run `evalshift bundle ` after a config edit. Either way, that is what lets the hosted gate check a pull request against exactly the budgets the bundle's own verdict used, instead of a separate, web-edited policy. `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. `evalshift.yaml` is the source of truth for that policy, and the web app's project policy view is becoming a read-only display of the snapshot each run pushed. Because every budget is resolved, the snapshot carries `max_invariant_violations` (default `0`) even for a config with no trace rules, so a hosted server must accept that field to take the push. - **A slice budget gates the run exactly like an overall one.** The budgets under `migration_policy.slices` are evaluated on the same terms as the top-level ones: a conclusively breached slice budget **fails** the run, and an unconfirmed breach makes it `inconclusive` — the same Wilson rule, counted over that slice's own denominator. Since the overall rows can all be green in a run a slice budget fails, `recommendations` names the one that blocked (`The 'security' slice breached its overall regression rate budget (the share of scored comparisons where the target did worse): 20% over n=20 vs the 0% limit.`) and the `inconclusive` `reason` scope-qualifies it the same way. Per-slice verdicts under `slices[*].verdict` are unchanged, and a slice that fails on *comparison severity* rather than a budget still only downgrades an overall `pass` to `conditional_pass`. - Semantic drift that stays above `min_similarity` counts as equivalent, not regression. diff --git a/docs/agents.md b/docs/agents.md index 8aa5cbeb..130ea738 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -123,7 +123,11 @@ inherited. See Hand edits inside the markers are regenerated away on the next sync. To keep them — a `severity_floor: high` on `routing`, a per-field strategy — set `managed: false` on that suite's entry; sync then prints what it would have -written instead of writing it. +written instead of writing it. The one exception is a hand-written +`evaluators.trace_invariants` block: no capture can derive a team's rules, so +sync carries it into the regenerated entry verbatim, even though the marker +comment says hand edits are overwritten. See +[Configuration → `managed`](configuration.md#managed). > **Note:** `structural.length` is intentionally **not** in the > scaffolded config. Agent runs frequently produce empty `final_text` @@ -437,6 +441,7 @@ together and `run` replays each one with its recorded history prefix. See | `tool_selection` | You care about *which* tools fire (most common). | | `tool_arguments` | You care about *what* the model passes to each tool. | | `tool_trace_structure` | You care about call counts, parallelism, or refusals. | +| `trace_invariants` | A rule must hold whatever the source did (auth before a charge, at most one charge, argument bounds). | You can run multiple at once. Each becomes an independent comparison in `analysis.json`, with the existing Benjamini-Hochberg correction @@ -460,6 +465,18 @@ It scores the same recall, counting duplicates (two expected `archive_project` calls need two actual ones) and ignoring extra calls, without penalising a permutation. Call counts are `tool_trace_structure`'s job, not this one's. +When an order genuinely matters — authenticate before charging — don't rely on +the recording's order to say so. Write it as an `order` rule under +[`trace_invariants`](configuration.md#evaluatorstrace_invariants), which checks +both models against the rule rather than against each other, and keep +`expected_set` for the fan-out. + +`trace_invariants` is also the rule-based counterpart to `agent_trace`'s +`dangerous_tools`: that one flags a target that made *more* dangerous calls +than the source, so a dangerous call both made passes; a `forbidden` or +`call_count` rule fails the target whatever the source did. It reads replayed +traces by default and imported ones with `traces: imported`. + ### Scoring free-text arguments Free-text tool arguments — search queries, titles, descriptions — will never diff --git a/docs/configuration.md b/docs/configuration.md index 1844b2d6..2fb13ac1 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -126,6 +126,7 @@ migration_policy: min_equivalence_rate: 0.75 max_tool_argument_drift: 0.20 max_tool_divergence: 0.20 # share of pairs where the target routed elsewhere + max_invariant_violations: 0 # examples where the target broke a blocking trace rule tool_argument_drift_floor: 0.9 # below this a call counts as drifted max_cost_increase: 0.30 # target may cost up to 30% more max_latency_increase: 0.30 # …and be up to 30% slower @@ -196,6 +197,25 @@ means the target called a tool the source did not, or skipped one it did. On a teacher-forced multi-round replay the row's score is the mean over the replayed rounds, so an example counts as diverged if **any** round diverged. +`max_invariant_violations` is a **count**, not a rate: the number of examples +on which the target broke at least one rule of a blocking +[`trace_invariants`](#evaluatorstrace_invariants) evaluator, judged against the +rules alone — a rule the source broke too still counts against the target. The +default `0` fails on the first broken rule. Examples are counted distinctly by +`(prompt_id, example_id)`: every `trace_invariants` entry writes its own row, and +an example that broke rules in two entries counts once, over a denominator of +the distinct examples with a blocking row. A row averaged over +`samples_per_example` repeats counts when any one sample broke a rule. Because +a rule is an assertion the team wrote down rather than a statistic, a breach is +always conclusive: no interval softens it, it carries no `1/n` granularity +warning, and it turns `inconclusive` and `conditional_pass` into `fail` even +when the suite is too small for the paired tests. The same holds per slice: a +slice whose own `max_invariant_violations` breaches reads `fail` however few +comparisons it has. The budget's row appears only in scopes that scored at least +one blocking `trace_invariants` row, so a project with no trace rules (or only +advisory ones) sees no new budget row. It is overridable per slice like the +others. + A `tool_selection.conformance` row where **both** models missed the recorded ground truth by the same margin is excluded from every policy rate: its zero delta is a shared failure, not evidence the migration is safe. Such rows are @@ -263,6 +283,16 @@ Two behaviours to know: `conclusive: false` on a scope that scored zero records — their `0/0` default looks clean but measures nothing. +* **`max_invariant_violations` is conclusive by construction.** It counts + distinct examples, so it has no interval and no "too small to be sure": a breach + fails the run (or the slice) outright, overriding `inconclusive` and + `conditional_pass`, and the verdict's `reason` names the scope and the + count. Only blocking `trace_invariants` evaluators count toward it. When the + source broke the same rule on the same example, `recommendations` adds a + line saying so — not an exemption, but a hint that either both models break + the rule or the rule no longer matches the toolset, which its owner should + review. + * **`fail_on_dropped_params` gates constraints, not scores.** Default `false`. A promoted capture can pin generation parameters (`response_format`, `tool_choice`, `parallel_tool_calls`, `top_p`, a @@ -312,6 +342,8 @@ the snapshot each run pushed. `policy` is `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. See [What `push` sends](hosted.md#what-push-sends-block-by-block) for the upload contract and the notices `push` prints around a policy-less run. +Because every budget is resolved, the snapshot carries `max_invariant_violations` +(at its default `0`) even for a config that never mentions trace rules. ## `prompts` @@ -404,8 +436,9 @@ bad generations fall back to deterministic templated prose. ## `evaluators` -Seven sub-keys, all optional — `structural`, `semantic`, `tool_selection`, -`tool_arguments`, `tool_trace_structure`, `agent_trace` and `llm_judge`, each +Eight sub-keys, all optional — `structural`, `semantic`, `tool_selection`, +`tool_arguments`, `tool_trace_structure`, `agent_trace`, `trace_invariants` and +`llm_judge`, each documented below. **At least one evaluator must be configured for `evalshift evaluate` to do anything.** @@ -610,6 +643,122 @@ evaluators: dangerous_tools: ["issue_refund"] ``` +### `evaluators.trace_invariants` + +A list. Each entry is a set of hand-written rules that every in-scope tool-call +trace must satisfy, checked on **both** sides of every pair and judged against +the rules, not against the source: a rule the source also broke still counts +against the target. See [Evaluators → Trace invariants](evaluators.md#trace-invariants) +for when to reach for it. + +Write it in the top-level `evaluators:` block (outside the `capture sync` +managed region) or in a suite's own `evaluators:` override. `capture sync` +keeps a managed suite entry's `trace_invariants` when it regenerates that +entry — see [`managed`](#managed). + +```yaml +evaluators: + trace_invariants: + - name: payments_contract + owner: "@payments-team" # optional; echoed onto every record and the report + applies_to: ["checkout-*"] # prompt-id globs; default ["*"] + blocking: true # default; false = reported, never gates + traces: replayed # default; `imported` checks `evalshift traces import` traces + rules: + - id: auth-before-charge + type: order + before: authenticate + after: charge_card + - id: no-legacy-refund + type: forbidden + tools: [refund_v1] + - id: one-charge + type: call_count + tool: charge_card + max_calls: 1 # min_calls, max_calls or both; both equal = exact + - id: must-book + type: required + tools: [create_booking] + - id: sane-amount + type: arguments + tool: charge_card + json_schema: + type: object + required: [amount] + properties: + amount: {type: number, minimum: 0.01, maximum: 5000} + +migration_policy: + max_invariant_violations: 0 # default; examples where the target broke any blocking rule +``` + +| Field | Type | Default | Description | +| ------------ | ------ | ------------ | ----------- | +| `name` | string | (required) | Identifier surfaced in scores and reports. | +| `applies_to` | list | `["*"]` | Glob list of prompt ids. **Enforced here**, unlike on the other families: a prompt outside these globs is never checked, gets no row and is never counted. If the globs match no prompt in a run (with `traces: imported`, no imported trace pair), `evaluate` prints a warning naming the entry and its `applies_to`, because none of its rules were checked (`compare` scores quietly and does not print it); matching at least one prompt stays silent. | +| `blocking` | bool | `true` | `true` counts target violations toward [`max_invariant_violations`](#migration_policy); `false` still checks and reports them, but never gates. | +| `traces` | string | `replayed` | `replayed` checks the tool calls `evalshift run` replayed (a pair whose replay recorded no tool trace gets no row). `imported` checks the traces brought in with [`evalshift traces import`](traces.md); `evaluate` fails on a run with no `traces.jsonl`, naming `trace_invariants (traces: imported)` in the error. | +| `owner` | string | `null` | Free text (a team or a handle). Echoed onto every record and into the report, so a violation names who to ask. Nothing enforces it. | +| `rules` | list | (required) | At least one rule. Every rule has an `id` (unique within the entry; it names violations in reports) and a `type`. | + +The unit of checking is a **response**: one model turn, plus the tool calls +already in its **context** before it answered. On a replayed trace there is one +response per replayed round, and round *k*'s context is the tool calls on the +example's `history` assistant turns plus the *recorded* calls of the rounds the +teacher-forced replay fed back before it; the model's own earlier rounds are +never context, because the replay never feeds them back. An imported trace is +one response with an empty context — every call in it was the agent's own. +Context calls can never *be* violations, since the model under test did not +make them, but they can satisfy or count toward a rule: + +| `type` | Fields | Violated when | Context role | +| ------------ | ------ | ------------- | ------------ | +| `forbidden` | `tools` (list) | the model calls any tool in `tools` (one violation per call) | none | +| `required` | `tools` (list) | some tool in `tools` is called in no response by the model | none: a call only in the context does not satisfy it. Use `call_count` `min_calls` when the conversation may already contain the call | +| `order` | `before`, `after` (two different tools) | the model calls `after` and no `before` precedes it, either in the context or earlier in the same response | a context `before` satisfies it | +| `call_count` | `tool`, `min_calls` and/or `max_calls` (at least one; `max_calls` ≥ `min_calls`) | `max_calls`: a model call to `tool` takes the running count past it, counting context calls first (one violation per excess call). `min_calls`: at the end of the last response, context calls plus the model's calls to `tool` number fewer than it (one violation). Both set and equal is exact cardinality | context calls count, toward both bounds | +| `arguments` | `tool`, `json_schema` | a model call to `tool` has arguments that fail `json_schema` (Draft 2020-12, itself validated when the config loads) | none | + +"Earlier in the same response" means emitted earlier; imported traces are +ordered by `sequence_index`, so the two modes agree. + +**Scoring.** Each side scores the share of rules it broke zero times — +`(rules − rules broken) / rules` — so a rule broken three times costs the same +as one broken once, and `delta = target − source`. The record's kind is +`trace_invariants`; its metadata carries `rules_checked`, `source_violations` +and `target_violations` (each a list of `{rule_id, rule_type, tool, +round_index, detail}`), `owner` when set, and the failure category +`INVARIANT_VIOLATION` when the target broke anything. With +`samples_per_example` above 1 the example's row keeps every scored sample's +violations, each tagged with its `sample` ordinal. The HTML report's +**Trace rules broken** panel and `report.json`'s `invariant_violations` list +every rule the target broke, whatever the delta — including those of advisory +(`blocking: false`) entries, which never count toward the budget — with the +owner and whether the source broke it too. Each row carries the entry's +`blocking` flag, and the panel tags `blocking: false` rows **advisory**. + +Four things to know: + +- **`required` and `min_calls` are not vacuous.** `order`, `arguments` and a + `call_count` with only `max_calls` say nothing about a trace that never calls + the tool. `required` and `call_count` with `min_calls` fail on every in-scope + trace that lacks the calls. A single-shot replay sees only the first + response, so scope them with `applies_to`, or use them with + `capture sync --rounds all` or with imported traces. +- **`required` counts the model's calls; `call_count` counts the + conversation's.** `required` asks whether the model under test made the + call. `call_count` asks whether the conversation contains it, so a recorded + history that already called `charge_card` satisfies `min_calls: 1` and + leaves `max_calls: 1` no room. +- **Teacher-forced rounds see recorded context.** Round *k* is judged with the + recorded rounds before it as context, not the model's own earlier rounds. + That applies to `min_calls` too: it is judged on the last response's context + plus that response's own calls. +- **Malformed arguments fail any schema that requires a field.** Arguments the + model emitted as unparseable JSON are recorded as + `{"_raw": …, "_parse_error": true}`, which fails any `json_schema` that + requires a field. + ### `evaluators.llm_judge` A list of pairwise judges. Each entry has: @@ -769,8 +918,8 @@ suites: The block takes the same families as the top-level `evaluators:` — `structural`, `semantic`, `llm_judge`, `tool_selection`, `tool_arguments`, -`tool_trace_structure`, `agent_trace` — and resolution is **family-level -replacement**: +`tool_trace_structure`, `agent_trace`, `trace_invariants` — and resolution is +**family-level replacement**: - A family the suite does **not** mention is inherited from the top level. - A family it **does** mention replaces the top-level one wholesale. There is no @@ -787,9 +936,15 @@ a name that has no `suites:` entry, resolves to the top-level block. ### `managed` -`capture sync` regenerates a managed suite's entire entry — `path` and +`capture sync` regenerates a managed suite's entry — `path` and `evaluators` — from what that suite's captures contain, so hand edits inside the -marker-delimited region are overwritten. Set `managed: false` to freeze an entry: +marker-delimited region are overwritten. The one exception is +`evaluators.trace_invariants`: no capture can derive a team's rules, so sync +copies a managed entry's existing `trace_invariants` block into the entry it +regenerates, verbatim. The marker comment above the region still says hand edits +are overwritten; that text cannot change (sync finds the region by matching it +exactly), so this paragraph is where the exception is written down. Every other +hand edit is overwritten. Set `managed: false` to freeze an entry: ```yaml suites: @@ -883,7 +1038,9 @@ key on evaluator names across runs, so regenerating a suite must not rename what it already wired. `structural` is deliberately not derived: nothing in a capture says what shape an answer must have. Sync regenerates only the suites it just promoted and carries every other entry in the region forward verbatim, and -`managed: false` freezes an entry entirely (see [`suites`](#suites)). +`managed: false` freezes an entry entirely (see [`suites`](#suites)). A +regenerated entry keeps its hand-written `evaluators.trace_invariants` (see +[`managed`](#managed)). Captures with no recorded events are skipped, and captures whose replayed content duplicates an already-promoted case (or an earlier capture in the same run) are skipped too — duplicate examples inflate *n* and corrupt the diff --git a/docs/evaluators.md b/docs/evaluators.md index b98b1e5d..f65c3090 100644 --- a/docs/evaluators.md +++ b/docs/evaluators.md @@ -22,7 +22,8 @@ inspection and excluded from slicing, the paired tests and the policy rates. The run always completes. EvalShift ships four families (plus `agent_trace` for -[imported external traces](traces.md)): +[imported external traces](traces.md), and `trace_invariants` for +[hand-written trace rules](#trace-invariants)): ## Structural (deterministic, free) @@ -198,6 +199,32 @@ tool evaluators at all. See [Agent migrations](agents.md#configuration) and [Configuration → per-suite evaluators](configuration.md#per-suite-evaluators). +### Trace invariants + +Every other tool evaluator treats "the source did it" as correct: +`tool_selection` divergence, `tool_arguments` against the source and +`agent_trace`'s `dangerous_tools` all measure how far the target moved from +the source, so a wrong call both models make scores as agreement. +`trace_invariants` judges **both** sides against rules your team wrote — +`forbidden` tools, `required` tools, an `order` between two tools, a +`call_count` (min, max or exact per tool) and `arguments` that must satisfy a +JSON Schema. A rule the target breaks counts against it whatever the source +did, and `migration_policy.max_invariant_violations` (default `0`) fails the +run on it, even on a suite too small for statistics. + +Write rules only for invariants where a silent miss is expensive: auth before +a write, a deprecated endpoint that must stay unused, at most one charge, +argument bounds. They are a short contract next to the config, not a second +copy of the suite; production traces stay the broad corpus that the other +evaluators compare. + +Keep the split clean. Captured `expected_tools` are the *recording*, and +`conformance: expected` grades their order as recorded — including order that +was incidental. An ordering that *matters* belongs in an `order` rule, where it +is stated once, owned, and checked on every in-scope trace, not in the +recording. Configuration, rule semantics and scoring: +[Configuration → `evaluators.trace_invariants`](configuration.md#evaluatorstrace_invariants). + ## Mixing evaluators You can configure several at once. They all run on every (prompt, @@ -221,6 +248,7 @@ Per (prompt, example) pair, each evaluator means: | llm_judge | 1 judge model completion | | tool_selection | $0 (compares parsed traces only) | | tool_trace_structure | $0 (compares parsed traces only) | +| trace_invariants | $0 (checks parsed traces only) | | tool_arguments | $0 without an `evaluators.semantic` block (free text uses `difflib`). With one, embedding calls per free-text or `semantic`-strategy field (cached) | A 100-example suite with 1 prompt and 4 evaluators (2 structural + diff --git a/docs/getting-started.md b/docs/getting-started.md index a7b5ebce..b9ef6ec3 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -203,8 +203,9 @@ evalshift report --open ``` Configure `evaluators.agent_trace` to compare tool order, argument -drift, extra dangerous actions, and missing verification steps. See -[Agent traces](traces.md) for the JSONL schema. +drift, extra dangerous actions, and missing verification steps, or +`evaluators.trace_invariants` with `traces: imported` to hold both sides to +rules you write. See [Agent traces](traces.md) for the JSONL schema. ## 9. Optional: push to hosted EvalShift diff --git a/docs/hosted.md b/docs/hosted.md index 9792704e..bc6cf723 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -296,10 +296,10 @@ The bundle itself contains: | --- | --- | | `manifest` | Run id, `org/project` slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the **local suite file path** as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. | | `examples[]` — one row per prompt × example | The example's template variables (`inputs`) **verbatim**; its `expected` reference output **verbatim**; both models' **full output text**; tool-call traces from the replay: each side's tool calls (names, arguments, call ids) with round markers, any final text and refusal messages, capped at 256 KB per side. Imported agent traces (`traces import`) are not uploaded; per-evaluator scores and error strings; per-side cost and latency; tags and slice names. | -| `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. | +| `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It therefore carries `max_invariant_violations` (default `0`) even when the config has no trace rules, which the server must accept. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. | | `methodology_notes` | The model ids and the statistical-contract sentences shown in every report. | | `insights` | The machine-written run narrative, when one was generated. It is prose *about* your run and can paraphrase or quote the regressions it summarizes. | -| `evaluator_config` | Config version; the prompt list **metadata only** — prompt names, file paths, and variable names, with every prompt body replaced by a `content_hash`; the whole `defaults` block (model ids, concurrency, cache flag, cost ceiling, max_tokens, samples_per_example); an always-empty `slices` list (a legacy key, kept so `eval_config_hash` does not change — the top-level `slices:` config key was removed); and the full evaluators block — which includes each `llm_judge` entry's `criterion_prompt` text, so keep judge criteria free of secrets. | +| `evaluator_config` | Config version; the prompt list **metadata only** — prompt names, file paths, and variable names, with every prompt body replaced by a `content_hash`; the whole `defaults` block (model ids, concurrency, cache flag, cost ceiling, max_tokens, samples_per_example); an always-empty `slices` list (a legacy key, kept so `eval_config_hash` does not change — the top-level `slices:` config key was removed); and the full evaluators block — which includes each `llm_judge` entry's `criterion_prompt` text, so keep judge criteria free of secrets. An empty `trace_invariants` list is left out of it, so a config without trace rules keeps its `eval_config_hash`; adding rules changes the hash like any other evaluator edit. | | `dataset_snapshot` | Suite path, example count, slice names, and one `examples_hash`. **No example content.** | ### What never leaves your machine diff --git a/docs/methodology.md b/docs/methodology.md index ef01f1eb..74920324 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -116,10 +116,12 @@ After BH correction, each comparison gets a severity tag based on the `migration_policy` budgets are computed over subsets of the scored records — `max_tool_argument_drift` only looks at tool-argument rows, -`max_tool_divergence` only at `tool_selection.divergence` rows, and the +`max_tool_divergence` only at `tool_selection.divergence` rows, +`max_invariant_violations` only at `trace_invariants` rows, and the semantic regression rule only applies to semantic ones. That selection is made on each record's **evaluator kind** (`tool_arguments`, `tool_selection.conformance`, -`tool_selection.divergence`, `semantic`, `llm_judge`, `structural`, …), which is +`tool_selection.divergence`, `trace_invariants`, `semantic`, `llm_judge`, +`structural`, …), which is a property of the evaluator's type, not of what you called it. An evaluator that measures more than one thing gets a slug per measurement — `tool_selection` scores ground-truth conformance and target-vs-source divergence, and they are @@ -244,9 +246,9 @@ zero-tolerance choice, not a mistake. A denominator of `0` measured nothing at all, which `BudgetResult.conclusive` already reports. And `min_equivalence_rate` is excluded because it is a *floor*: below one row's granularity it collapses to maximally lax — only a 0% rate could fail it — so "effective tolerance is zero" -would be the opposite of the truth. The count budget -(`max_critical_regressions`) and the cost/latency ratios have no row denominator -for `1/n` to describe. +would be the opposite of the truth. The count budgets +(`max_critical_regressions`, `max_invariant_violations`) are not rates, and the +cost/latency ratios have no row denominator, so `1/n` describes none of them. Widening the budget is not the fix, and neither is a confidence interval: at 1 material drift in 10 the 95% Wilson lower bound is `0.0179`, still above `0.01`, @@ -311,6 +313,7 @@ last from the first two, but it does not say *how much* was counted. Each | `max_critical_regressions` | the same records | | `max_tool_argument_drift` | the scope's `tool_arguments` rows only | | `max_tool_divergence` | the scope's `tool_selection.divergence` rows only | +| `max_invariant_violations` | the distinct examples (`prompt_id`, `example_id`) with a blocking `trace_invariants` row in the scope — one per example however many entries scored it; the row is emitted only when there is at least one | | `max_cost_increase` / `max_latency_increase` | the error-free calls both averages were taken over, across both roles | The first three are counted over *measurements*, not over examples. An @@ -353,9 +356,10 @@ binomial proportion, and only those get a confidence interval. There are four: | `max_tool_argument_drift` | materially drifted rows over the scope's `tool_arguments` rows | | `max_tool_divergence` | diverged rows over the scope's `tool_selection.divergence` rows | -`max_critical_regressions` is a raw count and the cost/latency budgets are -ratios of two averages; neither describes a proportion, so neither is given an -interval and both report `ci_low`/`ci_high` as `null`. +`max_critical_regressions` and `max_invariant_violations` are raw counts and +the cost/latency budgets are ratios of two averages; none describes a +proportion, so none is given an interval and all report `ci_low`/`ci_high` as +`null`. Each of the four carries a 95% Wilson score interval — Wilson rather than the normal approximation because these samples are small and these rates sit near 0 @@ -390,6 +394,35 @@ is reported `inconclusive` — which is what it always was statistically — and `1/n` granularity warning above still fires to say the sample is too coarse to express the budget. +### A broken trace rule is a finding, not a statistic + +`max_invariant_violations` counts the distinct examples on which the target +broke a blocking [`trace_invariants`](configuration.md#evaluatorstrace_invariants) +rule — any `(prompt_id, example_id)` with a blocking `trace_invariants` row whose +target score is below `1.0`, which includes a row averaged over repeated samples +where one sample broke a rule. Each entry writes its own row, so an example that +broke rules in two entries is still one example, over a denominator of distinct +examples rather than rows. It +is judged against the rules alone, so a rule the source broke on the same +example still counts. + +Every other budget estimates a rate the next run could land on either side of; +this one reports an assertion the team wrote down and the target falsified. +There is nothing to be uncertain about, so the budget is **conclusive by +construction**: no interval, no `1/n` warning, and a breach is a `fail` that +overrides `inconclusive` and `conditional_pass` — including the +"every comparison is `insufficient`" path a small suite otherwise takes, and in +a slice exactly as overall. The decision's `reason` names the worst scope and +its count. When the source broke the same rule on the same example, +`recommendations` adds a line naming the rule: either both models break it or +it no longer matches the toolset, and only its owner can say which. That line +is advice, never an exemption. + +The row exists only in scopes that scored at least one blocking +`trace_invariants` row. A project with no trace rules, or only advisory ones, +sees no new budget row, and a slice that holds no in-scope example has no row +to breach. + ### Non-applicable measurements are absent, not scored A turn where **both** models produced only tool calls has no text to compare. diff --git a/docs/traces.md b/docs/traces.md index a99d437b..10f7f72b 100644 --- a/docs/traces.md +++ b/docs/traces.md @@ -112,6 +112,33 @@ The evaluator emits normal `scores.jsonl` records with failure categories such as `TOOL_ORDER_DRIFT`, `ARGUMENT_VALUE_DRIFT`, `DANGEROUS_ACTION_DRIFT`, and `MISSING_VERIFICATION_STEP`. +`agent_trace` compares the target with the source: `dangerous_tools` flags a +target that made more dangerous calls than the source did, so a dangerous call +both sides made goes unflagged. To hold both sides to rules instead, add a +[`trace_invariants`](configuration.md#evaluatorstrace_invariants) entry with +`traces: imported`: + +```yaml +evaluators: + trace_invariants: + - name: refund_contract + traces: imported + rules: + - id: policy-before-refund + type: order + before: check_refund_policy + after: issue_refund + - id: one-refund + type: call_count + tool: issue_refund + max_calls: 1 +``` + +An imported trace is checked as one response with no prior context: every call +in it, in `sequence_index` order, was the agent's own. A rule the target breaks +counts toward `migration_policy.max_invariant_violations` whatever the source +did. + Debug commands become trace-aware: ```bash @@ -124,8 +151,8 @@ evalshift replay case --model target --trace Imported traces are not uploaded. They stay in `.evalshift/runs//traces.jsonl`, where `evaluate` (the `agent_trace` -evaluator), the local report and `diff case` / `inspect case` / `replay case` -read them. +evaluator and `trace_invariants` entries with `traces: imported`), the local +report and `diff case` / `inspect case` / `replay case` read them. `evalshift bundle` / `evalshift push` carry only the replay's own tool-call trace — one stream per model side with the tool calls (names, arguments, call diff --git a/llms-full.txt b/llms-full.txt index e8e6c208..108ce906 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -85,7 +85,7 @@ as one section under the pipeline block (also on stage failure); errors are neve | scores.jsonl | evaluate | one EvalRecord per (pair x evaluator): source/target scores in [0,1], delta | | analysis.json | analyze | per-(prompt,evaluator,slice) statistics + severity | | migration_decision.json | analyze | policy verdict (only when migration_policy configured) | -| report.json + report.html | report | payload + single-file HTML (no external assets, dark-only) | +| report.json + report.html | report | payload + single-file HTML (no external assets, dark-only). report.json top-level invariant_violations: one row per trace rule the target broke (see evaluators.trace_invariants) | | insights.json | report | cached machine-written narrative (optional; --no-insights skips) | | traces.jsonl | traces import | optional BYO agent traces | | run_bundle.json.gz | bundle/push | optional hosted upload bundle | @@ -323,6 +323,10 @@ evalshift capture sync [--suite S] [--input-var NAME=input] [--tag T]... [--stri carried forward VERBATIM (raw mappings, not revalidated -- a malformed hand edit is never silently deleted). Partition guarantee: syncing one suite cannot disturb another. managed: false on an entry -> left alone; sync prints the entry it WOULD have written. + EXCEPTION for regenerated managed entries: a hand-written evaluators.trace_invariants block + is copied VERBATIM into the regenerated entry (no capture can derive a team's rules). The + marker comment still says hand edits are overwritten -- its text is matched exactly to find + the region, so it cannot change; this is the documented exception. Sync never touches config OUTSIDE the markers, so a stale top-level tool evaluator block is yours to delete by hand. Mapping: first model input -> inputs (bare string lands under --input-var), recorded tool @@ -608,7 +612,41 @@ evaluators: # every evaluator config also takes blocking check_arguments: bool = true check_missing_verification: bool = true verification_tools: [str] - dangerous_tools: [str] + dangerous_tools: [str] # flags target calls IN EXCESS of the source's (relative) + trace_invariants: # list; hand-written rules, BOTH sides judged vs the RULES + - name: str # (not vs the source): a rule the source also broke still + applies_to: ["*"] # counts. applies_to IS ENFORCED here (other families + # accept it unenforced): an out-of-scope prompt gets no row and is never counted. + # Globs matching NO prompt (imported: no trace pair) -> evaluate prints a warning + # naming the entry + applies_to (not under quiet, so not in `compare`); >=1 match + # stays silent. + # traces: imported with no traces.jsonl -> evaluate errors naming the family. + blocking: bool = true # false = checked + reported, never counts toward the budget + traces: replayed|imported = replayed # imported = traces from `traces import` + owner: str|null = null # free text, echoed onto records + report; unenforced + rules: # >=1; id unique within the entry; discriminated on type + - {id: str, type: forbidden, tools: [str]} # any model call to one -> 1 per call + - {id: str, type: required, tools: [str]} # a tool the MODEL called in no response + - {id: str, type: order, before: str, after: str} # model calls `after` with no `before` + # earlier (context or earlier in same response); before != after + - {id: str, type: call_count, tool: str, min_calls: int>=0|null, max_calls: int>=0|null} + # >=1 bound; max>=min; equal = exact. max: running count (context first) past it, + # 1 per excess call. min: context + model calls at end of LAST response < it (1) + - {id: str, type: arguments, tool: str, json_schema: {...}} # Draft 2020-12, checked at + # load; {"_parse_error": true} args fail any schema that requires a field + # Unit = RESPONSE (one model turn) + CONTEXT (tool calls already in it). Replayed: one + # response per round; round k context = history assistant tool_calls + RECORDED rounds + # 0..k-1 the teacher-forced replay fed back (never the model's own earlier rounds). + # Imported: one response, empty context. Context calls are never violations; they + # satisfy `order` and count toward both call_count bounds; `required` ignores them. + # required / min_calls are NOT vacuous (fail every in-scope trace lacking the call); on a + # single-shot replay they see only the first response -> scope with applies_to, or use + # --rounds all / imported traces. order/arguments/max-only call_count are vacuous when + # the tool is never called. Replayed pairs with no tool trace get no row. + # Score per side = (rules - rules broken) / rules; kind slug trace_invariants; metadata + # rules_checked, source_violations, target_violations (each [{rule_id, rule_type, tool, + # round_index, detail}], with sample ordinal when samples_per_example > 1), owner; + # failure_categories [INVARIANT_VIOLATION] when the target broke anything. suites: # managed block; capture sync rewrites between : # ">>> evalshift suites" markers source: captured|jsonl = captured @@ -622,6 +660,7 @@ suites: # managed block; capture sync rewrites betwe tool_arguments: [...]|null # [] or null = REMOVE the inherited family tool_trace_structure: [...]|null # absent vs null are DIFFERENT instructions agent_trace: [...]|null # (resolution reads model_fields_set) + trace_invariants: [...]|null # kept verbatim by capture sync on managed entries # Resolved by EvalShiftConfig.evaluators_for(suite_name) -- the single resolution point; # evaluate, report and the hosted bundle all route through it, so scored set == reported set. # RunState.suite_name carries the name (set by run/compare from --suite-name; None for a raw @@ -633,6 +672,8 @@ migration_policy: # optional; fractions not percents max_tool_argument_drift: float = 0.20 (0..1) max_tool_divergence: float = 0.20 (0..1) # share of tool_selection.divergence rows that regressed # (multi-round replay: any diverged round counts) + max_invariant_violations: int = 0 (>=0) # COUNT of examples where the target broke a + # blocking trace_invariants rule; always conclusive tool_argument_drift_floor: float = 0.9 (0..1) # target arg score below which a call counts as drifted max_cost_increase: float = 0.30 (0..10) max_latency_increase: float = 0.30 (0..10) @@ -807,6 +848,18 @@ actually wrote scores.jsonl rows (kind == "llm_judge", or the legacy name prefix resolved provider equals an arm's renders a third "The judge graded its own relatives" banner after the two above (self-preference bias; advisory, no policy knob). Empty when no config is loadable, no judge contributed, or no judge overlaps. Provider "other" never matches. +Trace-rule violations (report.json invariant_violations: list of {prompt_id, example_id, +evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, +sample}): +one row per violation in a trace_invariants record's target_violations, WHATEVER the delta (a +rule both sides broke scores delta 0 and never reaches the top regressions, yet fails the run); +errored records list nothing. source_also = the source broke the same rule_id on that example. +sample = null on a single-sample run, else the zero-based ordinal among the samples that +scored successfully (NOT the raw sample_index). Advisory (blocking: false) entries' violations +ARE listed; only blocking ones count toward max_invariant_violations. blocking = the record's +flag (false = advisory entry). report.html renders the same rows as a "Trace rules broken" +panel (rule, owner, detail, "Source too?"), tagging blocking=false rows with a muted +"advisory" badge. ## Migration policy verdict algorithm @@ -837,6 +890,19 @@ Verdicts: pass | conditional_pass | fail | inconclusive. Written to migration_de tool divergence) report conclusive: false when the scope scored 0 records - their 0/0 default measures nothing. The two per-axis budgets also need one of THEIR OWN rows: a scope with no tool_arguments evaluator, or with divergence: off, is unmeasured however much else it scored. +- max_invariant_violations: a COUNT (not a rate) of DISTINCT examples (prompt_id, example_id) + with any blocking trace_invariants row whose target_score < 1.0 (each entry writes its own + row, so an example breaking two entries counts ONCE; a row averaged over samples counts if + ANY sample broke a rule); + judged vs the rules alone, so a rule the source broke too still counts. CONCLUSIVE BY + CONSTRUCTION: no CI (ci_low/ci_high null), no 1/n warning; a breach -> fail, overriding + inconclusive and conditional_pass, even when every comparison is insufficient, and in a + SLICE exactly as overall (a slice whose own max_invariant_violations breaches reads fail + however few comparisons it has). reason names the worst scope + count. Source broke the same + rule on the same example -> one recommendations line (both models break it, or the rule is + stale: review with its owner; its example count is distinct examples too) -- advice, not an + exemption. The budget row is emitted ONLY in scopes that scored >= 1 BLOCKING + trace_invariants row; denominator = distinct examples with such a row. - A SHARED GROUND-TRUTH MISS is not equivalence. A tool_selection.conformance row grades each side absolutely, so both can miss at the same height (0.0/0.0 where both models called a tool the recording never made). delta == 0 read as "equivalent" and is the whole of the @@ -882,7 +948,9 @@ Verdicts: pass | conditional_pass | fail | inconclusive. Written to migration_de complement, same rows) / max_critical_regressions - counted over MEASUREMENTS, so an evaluator scoring two axes contributes two rows per example and that is correct; tool_arguments rows for max_tool_argument_drift; tool_selection.divergence rows for - max_tool_divergence; the error-free calls behind both averages (both roles summed, + max_tool_divergence; distinct examples with a blocking trace_invariants row for + max_invariant_violations; the + error-free calls behind both averages (both roles summed, 0 when either role has none) for max_cost_increase / max_latency_increase. Slices report their own counts. Three readings, all distinct: 0 = counted and the sample was empty (observed is a default, passed is vacuous); positive int = that many units counted; @@ -908,7 +976,8 @@ Verdicts: pass | conditional_pass | fail | inconclusive. Written to migration_de Advisory (blocking: false) evaluators never count as blind gates: advisory-ness is read from the scored rows OR from the "advisory:" note on a zero-row comparison, via one shared helper, so a silent advisory evaluator neither demotes the verdict nor gets named. - inconclusive: all comparisons insufficient, or unconfirmed rate-budget breach. + inconclusive: all comparisons insufficient, or unconfirmed rate-budget breach (never when + max_invariant_violations is breached -- that is fail). pass: otherwise. - Semantic drift above min_similarity counts as equivalent, not regression. min_equivalence_rate floors the NON-REGRESSION rate: equivalent and improved both count toward it. @@ -928,7 +997,9 @@ migration_policy is configured; inconclusive exits 0). as-is, so re-run `bundle RUN_ID` after a config edit. Either way this is what lets the hosted gate check a PR against the exact budgets the bundle's own verdict used, instead of a separate, web-edited policy -- see `evalshift push` above and - docs/hosted.md. null on the inconclusive_decision path (no migration_policy configured) and + docs/hosted.md. decision.policy always carries max_invariant_violations (default 0), even + for a config with no trace rules, so the hosted server must accept that field. + null on the inconclusive_decision path (no migration_policy configured) and on a migration_decision.json written before this field existed. evalshift.yaml is the source of truth for the policy; the web app's per-project policy view is becoming a read-only display of the snapshot each run pushed. @@ -1018,7 +1089,8 @@ content leaves the machine. concurrency, cache, max_cost_usd, max_tokens, samples_per_example); slices (always [] -- a legacy key kept so eval_config_hash stays stable; the top-level config key is removed); full evaluators block INCLUDING each llm_judge criterion_prompt text (keep judge criteria - free of secrets). + free of secrets). An EMPTY trace_invariants list is dropped from it, so configs without + trace rules keep their eval_config_hash; adding rules changes it. - dataset_snapshot: suite path, size, slice names, examples_hash. No example content. - NEVER uploads: provider API keys; the hosted token (never inside a bundle); prompt bodies / system prompts (manual content -> content_hash; python_string bodies never enter the config); @@ -1192,7 +1264,7 @@ evaluators: blocking: false migration_policy: {max_overall_regression_rate: 0.30, max_critical_regressions: 1, min_equivalence_rate: 0.75, max_tool_argument_drift: 0.20, max_tool_divergence: 0.20, - max_cost_increase: 0.30, max_latency_increase: 0.30} + max_invariant_violations: 0, max_cost_increase: 0.30, max_latency_increase: 0.30} # ^ the model-upgrade profile = the MigrationPolicy field defaults. init does NOT # scaffold tool_argument_drift_floor; it is inherited at 0.9. Tighter presets: # --profile cost-reduction|quantization|provider-switch|local-model. @@ -1226,7 +1298,8 @@ suites: - {name: routing_args, against: expected} # <<< evalshift suites <<< # Keeping a hand edit (severity_floor: high, a per-field strategy)? Set `managed: false` on -# that entry -- otherwise the next sync regenerates it. +# that entry -- otherwise the next sync regenerates it. (evaluators.trace_invariants is the +# one hand edit sync keeps on a managed entry.) # golden.jsonl rows {"id": "ex_security_01", "inputs": {"query": "User account_42 had 5 failed login attempts in the last hour"}, "tags": ["security"], "expected_tools": [{"tool_name": "notify_security_team", "match_strategy": "subset"}], "toolset_ref": "sha256:1a2b3c..."} diff --git a/src/evalshift_cli/analysis/policy.py b/src/evalshift_cli/analysis/policy.py index 2e58c215..e8a498b7 100644 --- a/src/evalshift_cli/analysis/policy.py +++ b/src/evalshift_cli/analysis/policy.py @@ -42,6 +42,9 @@ # a tool-selection row. _TOOL_CONFORMANCE_KIND = "tool_selection.conformance" _TOOL_DIVERGENCE_KIND = "tool_selection.divergence" +# Hand-written trace rules. Selected by slug like every other budget; no +# legacy name prefix exists, because no row predating the slug can be one. +_INVARIANTS_KIND = "trace_invariants" # Legacy selection, kept only for records checkpointed before ``EvalRecord.kind`` # existed. Never add a new metric on top of these: an evaluator's ``name`` is # whatever the user typed in evalshift.yaml, so a name-prefix filter silently @@ -82,6 +85,7 @@ "min_equivalence_rate": "Equivalent-or-better rate", "max_tool_argument_drift": "Tool-argument drift", "max_tool_divergence": "Tool-selection divergence", + "max_invariant_violations": "Trace-rule violations", "max_cost_increase": "Cost increase", "max_latency_increase": "Latency increase", } @@ -97,13 +101,16 @@ ), "max_tool_argument_drift": ("how often the target filled tool arguments differently"), "max_tool_divergence": "how often the target called different tools than the source", + "max_invariant_violations": ( + "the number of examples where the target broke a hand-written trace rule" + ), "max_cost_increase": "how much more the target model costs to run", "max_latency_increase": "how much slower the target model answers", } # The budgets whose values are whole counts, not rates — rendered bare where # every other budget renders as a percentage. -_COUNT_BUDGETS = frozenset({"max_critical_regressions"}) +_COUNT_BUDGETS = frozenset({"max_critical_regressions", "max_invariant_violations"}) # Two-sided 95% normal quantile — the confidence level of every Wilson interval # below. The exact quantile rather than the textbook ``1.96`` (nominal coverage @@ -390,10 +397,12 @@ def evaluate_migration_policy( floor = policy.tool_argument_drift_floor overall = _metrics(records=blocking_records, calls=calls, drift_floor=floor) overall_divergence, overall_divergence_n = _tool_divergence_counts(blocking_records) + overall_invariants, overall_invariants_n = _invariant_violation_counts(blocking_records) overall_denominators = _budget_denominators( metrics=overall, tool_argument_records=_tool_argument_record_count(blocking_records), tool_divergence_records=overall_divergence_n, + invariant_records=overall_invariants_n, calls=calls, ) overall_budgets = _budget_results( @@ -402,6 +411,7 @@ def evaluate_migration_policy( scope="overall", denominators=overall_denominators, divergence_count=overall_divergence, + invariant_violations=overall_invariants, cost_measured=_has_measured_ratio(calls, field="cost_usd"), latency_measured=_has_measured_ratio(calls, field="latency_ms"), ) @@ -434,10 +444,12 @@ def evaluate_migration_policy( drift_floor=slice_policy.tool_argument_drift_floor, ) scoped_divergence, scoped_divergence_n = _tool_divergence_counts(scoped_records) + scoped_invariants, scoped_invariants_n = _invariant_violation_counts(scoped_records) scoped_denominators = _budget_denominators( metrics=scoped_metrics, tool_argument_records=_tool_argument_record_count(scoped_records), tool_divergence_records=scoped_divergence_n, + invariant_records=scoped_invariants_n, calls=calls, ) budgets = _budget_results( @@ -446,6 +458,7 @@ def evaluate_migration_policy( scope=name, denominators=scoped_denominators, divergence_count=scoped_divergence, + invariant_violations=scoped_invariants, cost_measured=_has_measured_ratio(calls, field="cost_usd"), latency_measured=_has_measured_ratio(calls, field="latency_ms"), ) @@ -520,6 +533,14 @@ def evaluate_migration_policy( else None ) + invariant_breaches = _invariant_breaches(gating_budgets) + if invariant_breaches: + # `_verdict_for` already fails on these; repeated here because the + # slice rules above can still demote its answer to conditional_pass, + # and a broken trace rule is not softened by any of them. + verdict = "fail" + reason = _invariant_reason(invariant_breaches) + if policy.fail_on_dropped_params and dropped_params: # An explicit opt-in, so it overrides every other verdict including a # clean pass: the user has said the constraint *is* the contract, and @@ -557,6 +578,9 @@ def evaluate_migration_policy( # run this fails, so without this line the verdict names no # number the reader can go and look at. *_slice_budget_notes(gating_budgets), + # A rule both models broke is still a failure, but it is also the + # one signal that the rule itself may be what needs a look. + *_shared_invariant_notes(blocking_records), # Appended, never substituted: a sub-granular budget does not # change what the run measured, so it must not displace the advice # the verdict earned — it explains how the gate was really set. @@ -747,6 +771,24 @@ def _tool_divergence_counts(records: list[EvalRecord]) -> tuple[int, int]: return sum(1 for r in rows if r.delta < 0), len(rows) +def _invariant_violation_counts(records: list[EvalRecord]) -> tuple[int, int]: + """``(violated, total)`` distinct examples over this scope's ``trace_invariants`` rows. + + Counted per ``(prompt_id, example_id)``, not per row: each + ``trace_invariants`` entry writes its own row for an example, so two rule + sets would otherwise count one example twice and double the denominator. + An example is violated when the target broke at least one rule in any + entry: any target score below 1.0, which also catches a row averaged over + repeated samples where one sample broke a rule. Judged against the rules + alone -- a rule the source broke too is still broken, which is the reason + the evaluator exists. + """ + rows = [r for r in _evidence(records) if _is_kind(r, _INVARIANTS_KIND)] + examples = {(r.prompt_id, r.example_id) for r in rows} + violated = {(r.prompt_id, r.example_id) for r in rows if r.target_score < 1.0} + return len(violated), len(examples) + + def _metrics( *, records: list[EvalRecord], @@ -806,6 +848,7 @@ def _budget_results( scope: str, denominators: Mapping[str, int], divergence_count: int, + invariant_violations: int, cost_measured: bool, latency_measured: bool, ) -> list[BudgetResult]: @@ -819,7 +862,10 @@ def _budget_results( ``denominators["max_tool_divergence"]`` (see :func:`_tool_divergence_counts`); it arrives as a count rather than a rate because the Wilson interval needs the count either way, and rebuilding one - from the other rounds. ``cost_measured`` / + from the other rounds. ``invariant_violations`` is the numerator of + ``max_invariant_violations`` over ``denominators["max_invariant_violations"]`` + (see :func:`_invariant_violation_counts`); that budget's row is emitted only + when the denominator is positive. ``cost_measured`` / ``latency_measured`` are the call-derived budgets' conclusiveness flags and stay separate from their denominators: those ratios also default to 0.0 with a perfectly good call count behind them, when both roles average zero @@ -864,7 +910,7 @@ def _budget_results( divergence_rate = _rate(divergence_count, divergence_n) divergence_low, divergence_high = _wilson_interval(divergence_count, divergence_n) divergence_breached = divergence_rate > policy.max_tool_divergence - return [ + results = [ BudgetResult( name="max_overall_regression_rate", observed=metrics.regression_rate, @@ -947,6 +993,24 @@ def _budget_results( denominator=denominators["max_latency_increase"], ), ] + invariant_n = denominators["max_invariant_violations"] + if invariant_n > 0: + # Emitted only where a trace rule was checked: a "measured nothing" + # row on every project that never wrote one would be noise in the + # report and the insights. A count, so no interval: a breach is a + # breach, and conclusive however small the sample. + results.append( + BudgetResult( + name="max_invariant_violations", + observed=float(invariant_violations), + allowed=float(policy.max_invariant_violations), + passed=invariant_violations <= policy.max_invariant_violations, + scope=scope, + conclusive=True, + denominator=invariant_n, + ), + ) + return results def _budget_denominators( @@ -954,6 +1018,7 @@ def _budget_denominators( metrics: PolicyMetricSummary, tool_argument_records: int, tool_divergence_records: int, + invariant_records: int, calls: list[Call], ) -> dict[str, int]: """Every budget's own denominator for one scope, keyed by budget name. @@ -966,7 +1031,7 @@ def _budget_denominators( the one it was *judged* on. The companion of :data:`_RATE_CEILING_DENOMINATORS`, which names the rate - ceilings' rows in prose; this covers all seven budgets, because + ceilings' rows in prose; this covers every budget, because ``BUNDLE_SPEC.md`` asks every emitted budget for its sample size. """ return { @@ -989,6 +1054,9 @@ def _budget_denominators( # axis is configured on or off. "max_tool_argument_drift": tool_argument_records, "max_tool_divergence": tool_divergence_records, + # The exception: distinct examples, not rows -- the budget is a count + # of examples, and each rule set writes its own row per example. + "max_invariant_violations": invariant_records, **{ name: _call_ratio_denominator(calls, field=call_field) for name, call_field in _CALL_RATIO_FIELDS.items() @@ -1199,6 +1267,11 @@ def _verdict_for( comparisons: list[ComparisonResult], budgets: list[BudgetResult], ) -> MigrationVerdict: + if _invariant_breaches(budgets): + # Before the small-sample early return: a broken trace rule is an + # assertion the team wrote down, not a statistic, so it needs no + # sample size to mean something -- in a slice exactly as overall. + return "fail" if comparisons and all(c.severity == "insufficient" for c in comparisons): return "inconclusive" if any(not b.passed and b.conclusive for b in budgets): @@ -1267,6 +1340,60 @@ def _slice_budget_notes(budgets: list[BudgetResult]) -> list[str]: ] +def _invariant_breaches(budgets: list[BudgetResult]) -> list[BudgetResult]: + """The breached ``max_invariant_violations`` rows -- always conclusive.""" + return [b for b in budgets if b.name == "max_invariant_violations" and not b.passed] + + +def _invariant_reason(breaches: list[BudgetResult]) -> str: + """Why a broken trace rule failed the run, naming the worst scope.""" + worst = max(breaches, key=lambda b: b.observed) + where = "" if worst.scope == "overall" else f" in the '{worst.scope}' slice" + return ( + f"the target broke a hand-written trace rule on {int(worst.observed)} of " + f"{worst.denominator} examples{where}, over the {int(worst.allowed)} allowed by " + "migration_policy.max_invariant_violations. Trace rules are assertions, so no " + "sample size softens them." + ) + + +def _violated_rule_ids(raw: Any) -> set[str]: + """The ``rule_id`` of every violation in one side's metadata list.""" + if not isinstance(raw, list): + return set() + return {rule for v in raw if isinstance(v, dict) and isinstance(rule := v.get("rule_id"), str)} + + +def _shared_invariant_notes(records: list[EvalRecord]) -> list[str]: + """One line when the source broke the same rule on the same example as the target. + + Not an exemption -- the target still fails -- but a shared break is the + signal that either both models get this wrong or the rule no longer + matches the toolset, and only the rule's owner can say which. + """ + shared_rules: set[str] = set() + # Distinct examples, not rows: two rule sets sharing a break on one + # example are still one example. + examples: set[tuple[str, str]] = set() + for r in _evidence(records): + if not _is_kind(r, _INVARIANTS_KIND): + continue + common = _violated_rule_ids(r.metadata.get("source_violations")) & _violated_rule_ids( + r.metadata.get("target_violations") + ) + if common: + shared_rules |= common + examples.add((r.prompt_id, r.example_id)) + if not examples: + return [] + return [ + f"The source model also broke {', '.join(sorted(shared_rules))} on {len(examples)} " + "example(s) where the target did. Either both models break the rule or it no " + "longer matches the toolset: review it with its owner rather than raising " + "max_invariant_violations." + ] + + def _pct(value: float) -> str: """``0.375`` → ``"37.5%"`` — prose never shows a reader raw fractions.""" return f"{round(value * 100, 1):g}%" @@ -1414,6 +1541,11 @@ def _slice_policy( if override.max_tool_divergence is None else override.max_tool_divergence ), + max_invariant_violations=( + base.max_invariant_violations + if override.max_invariant_violations is None + else override.max_invariant_violations + ), tool_argument_drift_floor=( base.tool_argument_drift_floor if override.tool_argument_drift_floor is None diff --git a/src/evalshift_cli/cli/commands/_scaffold.py b/src/evalshift_cli/cli/commands/_scaffold.py index a1e47944..8d59d4ce 100644 --- a/src/evalshift_cli/cli/commands/_scaffold.py +++ b/src/evalshift_cli/cli/commands/_scaffold.py @@ -40,6 +40,7 @@ min_equivalence_rate: 0.95 max_tool_argument_drift: 0.01 max_tool_divergence: 0.03 + max_invariant_violations: 0 max_cost_increase: 0.20 max_latency_increase: 0.30 """, @@ -50,6 +51,7 @@ min_equivalence_rate: 0.97 max_tool_argument_drift: 0.01 max_tool_divergence: 0.02 + max_invariant_violations: 0 max_cost_increase: 0.05 max_latency_increase: 0.30 """, @@ -60,6 +62,7 @@ min_equivalence_rate: 0.90 max_tool_argument_drift: 0.02 max_tool_divergence: 0.05 + max_invariant_violations: 0 max_cost_increase: 0.00 max_latency_increase: 0.50 """, @@ -70,6 +73,7 @@ min_equivalence_rate: 0.97 max_tool_argument_drift: 0.005 max_tool_divergence: 0.02 + max_invariant_violations: 0 max_cost_increase: 0.00 max_latency_increase: 0.20 """, @@ -80,6 +84,7 @@ min_equivalence_rate: 0.95 max_tool_argument_drift: 0.01 max_tool_divergence: 0.03 + max_invariant_violations: 0 max_cost_increase: 0.20 max_latency_increase: 0.40 """, @@ -100,6 +105,7 @@ min_equivalence_rate: 0.75 max_tool_argument_drift: 0.20 max_tool_divergence: 0.20 + max_invariant_violations: 0 max_cost_increase: 0.30 max_latency_increase: 0.30 """, diff --git a/src/evalshift_cli/cli/commands/capture.py b/src/evalshift_cli/cli/commands/capture.py index 09fe34d4..e9fe4cff 100644 --- a/src/evalshift_cli/cli/commands/capture.py +++ b/src/evalshift_cli/cli/commands/capture.py @@ -661,8 +661,11 @@ def _build_suite_entries( frozen: dict[str, Any] = {} for name in sorted(suite_paths): rel = Path(os.path.relpath(suite_paths[name].resolve(), config_dir)).as_posix() - fresh = suite_entry_payload(path=rel, evaluators=evaluators.get(name)) prior = existing.get(name) + fresh = _keep_trace_invariants( + suite_entry_payload(path=rel, evaluators=evaluators.get(name)), + prior, + ) if isinstance(prior, Mapping) and prior.get("managed") is False: frozen[name] = fresh # `entries[name]` already holds the frozen entry. continue @@ -670,6 +673,33 @@ def _build_suite_entries( return entries, frozen +def _keep_trace_invariants(fresh: dict[str, Any], prior: object) -> dict[str, Any]: + """Carry a managed entry's hand-written ``trace_invariants`` through a sync. + + Everything else in a managed entry is derived from captures and is + regenerated. Trace rules are the exception: nobody can derive them, a team + wrote and owns them, and silently dropping a contract on the next sync is + exactly the "auto-update the baseline" failure they exist to prevent. + + Args: + fresh: The entry this sync regenerated from the suite's rows. + prior: The entry the managed region held before (any shape -- it is + raw YAML, carried forward unvalidated like every other entry). + + Returns: + ``fresh``, with the prior ``evaluators.trace_invariants`` value copied in + verbatim when there was one; otherwise ``fresh`` unchanged. + """ + if not isinstance(prior, Mapping): + return fresh + prior_evaluators = prior.get("evaluators") + if not isinstance(prior_evaluators, Mapping) or "trace_invariants" not in prior_evaluators: + return fresh + evaluators = dict(fresh.get("evaluators") or {}) + evaluators["trace_invariants"] = prior_evaluators["trace_invariants"] + return {**fresh, "evaluators": evaluators} + + @capture_app.command(name="sync") def capture_sync( suite: Annotated[ diff --git a/src/evalshift_cli/cli/commands/evaluate.py b/src/evalshift_cli/cli/commands/evaluate.py index 66594758..2a6b6803 100644 --- a/src/evalshift_cli/cli/commands/evaluate.py +++ b/src/evalshift_cli/cli/commands/evaluate.py @@ -44,6 +44,7 @@ import typer from rich.console import Console +from rich.markup import escape from rich.progress import ( BarColumn, MofNCompleteColumn, @@ -65,6 +66,7 @@ from evalshift_cli.config.models import EvalShiftConfig, EvaluatorsConfig from evalshift_cli.evaluators.agent_trace import AgentTraceEvaluator from evalshift_cli.evaluators.base import EvalRecord, Evaluator, EvaluatorError, PairedScore +from evalshift_cli.evaluators.failures import INVARIANT_VIOLATION from evalshift_cli.evaluators.llm_judge import PairwiseJudgeEvaluator from evalshift_cli.evaluators.semantic import CosineSimilarityEvaluator, _cosine from evalshift_cli.evaluators.structural import ( @@ -80,6 +82,8 @@ from evalshift_cli.evaluators.tool_models import ToolSpec from evalshift_cli.evaluators.tool_selection import ToolSelectionEvaluator from evalshift_cli.evaluators.tool_trace_structure import ToolTraceStructureEvaluator +from evalshift_cli.evaluators.trace_invariants import KIND as TRACE_INVARIANTS_KIND +from evalshift_cli.evaluators.trace_invariants import build_trace_invariants_evaluator from evalshift_cli.models.capabilities import honors_temperature from evalshift_cli.models.client import ModelClient from evalshift_cli.models.registry import resolve_model @@ -240,6 +244,10 @@ def run_evaluate( ), ) + if not quiet: + for evaluator in _unchecked_trace_invariants(evaluators, coverage): + console.print(_unchecked_trace_invariants_warning(evaluator)) + output_path = run_dir / SCORES_FILENAME with output_path.open("w", encoding="utf-8") as fh: for record in records: @@ -443,6 +451,11 @@ def _add(evaluator: Evaluator, *, blocking: bool) -> None: _add(ToolTraceStructureEvaluator(tts), blocking=tts.blocking) # type: ignore[arg-type] for agent_trace in evaluators_cfg.agent_trace: _add(AgentTraceEvaluator(agent_trace), blocking=agent_trace.blocking) # type: ignore[arg-type] + # ``traces: replayed`` builds a ``score_pair`` evaluator and ``imported`` + # a ``score_trace_pair`` one, so the method-based dispatch below routes + # each to its own path without knowing about the family. + for ti in evaluators_cfg.trace_invariants: + _add(build_trace_invariants_evaluator(ti), blocking=ti.blocking) # type: ignore[arg-type] return out @@ -555,6 +568,68 @@ def _is_agent_trace_evaluator(evaluator: Evaluator) -> bool: return hasattr(evaluator, "score_trace_pair") +def _applies(evaluator: Evaluator, prompt_id: str) -> bool: + """Whether ``evaluator`` is scoped to ``prompt_id`` at all. + + An evaluator outside its scope was never asked, so it gets no cell: no + row, and no coverage attempt that would later read as "unmeasured". + Only evaluators that define ``applies`` are scoped; every other family + keeps today's behaviour, where ``applies_to`` is accepted but unenforced. + + Args: + evaluator: Any built evaluator. + prompt_id: The prompt a pair was produced from. + + Returns: + ``False`` only when the evaluator defines ``applies`` and it rejects + ``prompt_id``. + """ + applies = getattr(evaluator, "applies", None) + return True if applies is None else bool(applies(prompt_id)) + + +def _unchecked_trace_invariants( + evaluators: list[Evaluator], + coverage: list[EvaluatorCoverage], +) -> list[Evaluator]: + """The ``trace_invariants`` evaluators that were handed no pair at all. + + :func:`_applies` gives an out-of-scope pair no cell, so an ``applies_to`` + that matches nothing leaves no row *and* no coverage entry: the run + would read clean without a word about the rules. Any cell -- recorded, + errored or unmeasured -- books a coverage entry, so a missing entry is + exactly "checked nothing". Partial scoping (one prompt matched) has an + entry and stays silent. + + Args: + evaluators: Every evaluator built for the run. + coverage: The run's per-axis coverage from :func:`_coverage_for`. + + Returns: + The ``trace_invariants`` evaluators with no coverage entry, in + configured order. + """ + attempted = {c.evaluator_name for c in coverage if c.kind == TRACE_INVARIANTS_KIND} + return [ + e + for e in evaluators + if _evaluator_kind(e) == TRACE_INVARIANTS_KIND and e.name not in attempted + ] + + +def _unchecked_trace_invariants_warning(evaluator: Evaluator) -> str: + """The console line for a ``trace_invariants`` evaluator that checked nothing.""" + config = getattr(evaluator, "config", None) + applies_to = list(getattr(config, "applies_to", ["*"])) + target = "imported trace pair" if _is_agent_trace_evaluator(evaluator) else "pair" + return ( + f"[yellow]⚠[/yellow] trace_invariants evaluator {escape(repr(evaluator.name))} " + f"checked nothing: applies_to {escape(repr(applies_to))} matched no {target} " + "in this run, so none of its rules were checked. Check the globs against " + "the run's prompt ids." + ) + + def _pair_calls(run_dir: Path) -> list[_PairedCalls]: """Pair each source row with the target row of the same sample. @@ -702,7 +777,12 @@ async def _score_all( if not evaluators: return [] - work = [(pair, evaluator) for pair in pairs for evaluator in evaluators] + work = [ + (pair, evaluator) + for pair in pairs + for evaluator in evaluators + if _applies(evaluator, pair.prompt_id) + ] sem = asyncio.Semaphore(concurrency) progress: Progress | None = None task_id: TaskID | None = None @@ -810,6 +890,7 @@ def _reduce_one_group(group: list[_ScoredCell]) -> _ScoredCell: "explanation": explanation, "metadata": { **lead.metadata, + **_merged_violations(successful), "samples": _samples_metadata(len(group), successful), }, }, @@ -817,6 +898,47 @@ def _reduce_one_group(group: list[_ScoredCell]) -> _ScoredCell: return replace(first, record=reduced, sample_index=0) +_VIOLATION_KEYS = ("source_violations", "target_violations") + + +def _merged_violations(records: list[EvalRecord]) -> dict[str, Any]: + """Every scored sample's trace-rule violations, not only the lead's. + + A reduced row's scores are means over samples, so one broken sample in + three pulls ``target_score`` below 1.0 and the gate counts it -- but the + row's metadata is the lead sample's, which may be the clean one. Without + this the report would show a failed example and no violation to read. + ``sample`` is the ordinal among the scored samples. Empty for rows that + carry no violation lists, which is every family but ``trace_invariants``. + + Args: + records: The successfully scored samples of one example, lead first. + + Returns: + Metadata keys to overlay on the lead's: the merged violation lists, + and ``failure_categories`` with ``INVARIANT_VIOLATION`` when any + sample's target broke a rule. + """ + merged: dict[str, Any] = {} + for key in _VIOLATION_KEYS: + if not any(key in record.metadata for record in records): + continue + merged[key] = [ + {**violation, "sample": ordinal} + for ordinal, record in enumerate(records) + for violation in record.metadata.get(key) or [] + if isinstance(violation, dict) + ] + if merged.get("target_violations"): + categories = [ + c for c in records[0].metadata.get("failure_categories") or [] if isinstance(c, str) + ] + if INVARIANT_VIOLATION not in categories: + categories.append(INVARIANT_VIOLATION) + merged["failure_categories"] = categories + return merged + + def _samples_metadata(n: int, successful: list[EvalRecord]) -> dict[str, Any]: deltas = [r.delta for r in successful] return { @@ -879,6 +1001,20 @@ def _cells_for( ] +def _imported_trace_families(evaluators: list[Evaluator]) -> str: + """Name the configured families that read ``traces.jsonl``, in a stable order. + + Both ``agent_trace`` and ``trace_invariants`` with ``traces: imported`` + land here, and an error that named only the first sent a user who + configured only the second looking for an evaluator they never wrote. + """ + labels = { + TRACE_INVARIANTS_KIND: "trace_invariants (traces: imported)", + } + kinds = sorted({_evaluator_kind(e) for e in evaluators}) + return " and ".join(labels.get(kind, kind) for kind in kinds) + + async def _score_agent_traces( *, run_dir: Path, @@ -889,7 +1025,7 @@ async def _score_agent_traces( traces_path = run_dir / TRACES_FILENAME if not traces_path.exists(): raise EvaluatorError( - "agent_trace evaluators require imported traces. " + f"{_imported_trace_families(evaluators)} evaluators require imported traces. " f"Run: evalshift traces import {run_id} --source ... --target ...", ) try: @@ -905,6 +1041,8 @@ async def _score_agent_traces( cells: list[_ScoredCell] = [] for trace_pair in trace_pairs: for evaluator in evaluators: + if not _applies(evaluator, trace_pair.prompt_id): + continue try: record = await evaluator.score_trace_pair( # type: ignore[attr-defined] run_id=run_id, diff --git a/src/evalshift_cli/config/models.py b/src/evalshift_cli/config/models.py index d1aef5fe..a46de0a9 100644 --- a/src/evalshift_cli/config/models.py +++ b/src/evalshift_cli/config/models.py @@ -15,8 +15,10 @@ from __future__ import annotations -from typing import Any, Literal, Self +from typing import Annotated, Any, Literal, Self +from jsonschema import Draft202012Validator +from jsonschema.exceptions import SchemaError from pydantic import BaseModel, ConfigDict, Field, model_validator from evalshift_cli.suite.tags import RESERVED_SLICE_NAME @@ -385,6 +387,161 @@ class AgentTraceEvaluatorConfig(_StrictModel): dangerous_tools: list[str] = Field(default_factory=list) +# --------------------------------------------------------------------------- +# Trace invariants — hand-written rules every in-scope trace must satisfy +# --------------------------------------------------------------------------- + + +class ForbiddenToolsRule(_StrictModel): + """The model must never call any tool in ``tools``.""" + + type: Literal["forbidden"] + id: str = Field(min_length=1) + tools: list[str] = Field(min_length=1) + + +class RequiredToolsRule(_StrictModel): + """Every tool in ``tools`` must be called at least once by the model. + + The only rule that is not vacuous on a trace that never mentions its + tools, so scope it with ``applies_to``: on a single-shot replay it sees + the first response only. + """ + + type: Literal["required"] + id: str = Field(min_length=1) + tools: list[str] = Field(min_length=1) + + +class ToolOrderRule(_StrictModel): + """Every call to ``after`` must be preceded by a call to ``before``. + + "Preceded" counts the calls already in the response's context (recorded + history, and on a teacher-forced replay the recorded earlier rounds) and + calls emitted earlier in the same response. + """ + + type: Literal["order"] + id: str = Field(min_length=1) + before: str = Field(min_length=1) + after: str = Field(min_length=1) + + @model_validator(mode="after") + def _distinct_tools(self) -> Self: + """A tool cannot be required to precede itself.""" + if self.before == self.after: + raise ValueError(f"order rule {self.id!r}: before and after name the same tool") + return self + + +class CallCountRule(_StrictModel): + """Between ``min_calls`` and ``max_calls`` calls to ``tool`` in the conversation. + + Both bounds count the calls already in a response's context plus the + model's own, so a recorded history that already charged leaves the model + one charge fewer. ``max_calls`` is checked at every call; ``min_calls`` at + the end of the last response. Exact cardinality is both bounds equal. + Mirrors ``min_chars``/``max_chars`` on the length evaluator: at least one + bound is required. + """ + + type: Literal["call_count"] + id: str = Field(min_length=1) + tool: str = Field(min_length=1) + min_calls: int | None = Field(default=None, ge=0) + max_calls: int | None = Field(default=None, ge=0) + + @model_validator(mode="after") + def _bounded(self) -> Self: + """Require at least one bound, and a non-empty range when both are set.""" + if self.min_calls is None and self.max_calls is None: + raise ValueError( + f"call_count rule {self.id!r}: at least one of 'min_calls' or 'max_calls' " + "is required", + ) + if ( + self.min_calls is not None + and self.max_calls is not None + and self.max_calls < self.min_calls + ): + raise ValueError( + f"call_count rule {self.id!r}: max_calls ({self.max_calls}) is below " + f"min_calls ({self.min_calls})", + ) + return self + + +class ArgumentsRule(_StrictModel): + """Every call to ``tool`` must have arguments valid under ``json_schema``. + + JSON Schema Draft 2020-12, checked at config load so a typo in the schema + fails ``evalshift.yaml`` rather than every call it was meant to guard. + """ + + type: Literal["arguments"] + id: str = Field(min_length=1) + tool: str = Field(min_length=1) + json_schema: dict[str, Any] + + @model_validator(mode="after") + def _schema_is_valid(self) -> Self: + """Reject a schema that is not itself valid Draft 2020-12.""" + try: + Draft202012Validator.check_schema(self.json_schema) + except SchemaError as exc: + raise ValueError( + f"arguments rule {self.id!r}: json_schema is not valid JSON Schema: {exc.message}", + ) from exc + return self + + +InvariantRule = Annotated[ + ForbiddenToolsRule | RequiredToolsRule | ToolOrderRule | CallCountRule | ArgumentsRule, + Field(discriminator="type"), +] +"""One hand-written trace rule; ``type`` selects which.""" + + +class TraceInvariantsEvaluatorConfig(_StrictModel): + """Configuration for the trace-invariants evaluator. + + Deterministic rules every in-scope trace must satisfy, checked on both + sides of every pair independently of each other: a rule the source also + broke is still broken. ``migration_policy.max_invariant_violations`` + turns a target violation into a failed verdict. + + Attributes: + name: Stable identifier surfaced in reports. + applies_to: Glob list of prompt ids this evaluator applies to. Unlike + the other families (where the field is not yet enforced), a prompt + outside these globs is never checked and never counted. + blocking: Whether violations can fail the migration verdict. + traces: ``replayed`` (default) checks the traces ``evalshift run`` + replayed; ``imported`` checks traces brought in with + ``evalshift traces import``. + owner: Free-text owner of these rules (a team or handle), echoed onto + every record and the report so a violation names who to ask. + rules: The rules, at least one, ids unique within this evaluator. + """ + + name: str = Field(min_length=1) + applies_to: list[str] = Field(default_factory=lambda: ["*"]) + blocking: bool = True + traces: Literal["replayed", "imported"] = "replayed" + owner: str | None = None + rules: list[InvariantRule] = Field(min_length=1) + + @model_validator(mode="after") + def _unique_rule_ids(self) -> Self: + """Rule ids name violations in reports, so they must not collide.""" + seen: set[str] = set() + for rule in self.rules: + if rule.id in seen: + raise ValueError(f"trace_invariants {self.name!r}: duplicate rule id {rule.id!r}") + seen.add(rule.id) + return self + + class EvaluatorsConfig(_StrictModel): """Container for all evaluator configurations attached to a run.""" @@ -399,6 +556,8 @@ class EvaluatorsConfig(_StrictModel): ) # CLI Phase 2 — imported bring-your-own-agent trace evaluators. agent_trace: list[AgentTraceEvaluatorConfig] = Field(default_factory=list) + # Hand-written trace rules, checked on both sides regardless of the source. + trace_invariants: list[TraceInvariantsEvaluatorConfig] = Field(default_factory=list) @property def tool_evaluator_names(self) -> frozenset[str]: @@ -441,6 +600,7 @@ class SliceMigrationPolicy(_StrictModel): min_equivalence_rate: float | None = Field(default=None, ge=0.0, le=1.0) max_tool_argument_drift: float | None = Field(default=None, ge=0.0, le=1.0) max_tool_divergence: float | None = Field(default=None, ge=0.0, le=1.0) + max_invariant_violations: int | None = Field(default=None, ge=0) tool_argument_drift_floor: float | None = Field(default=None, ge=0.0, le=1.0) max_cost_increase: float | None = Field(default=None, ge=0.0, le=10.0) max_latency_increase: float | None = Field(default=None, ge=0.0, le=10.0) @@ -479,6 +639,9 @@ class MigrationPolicy(_StrictModel): those scores would be worse than failing. Not available per slice: a model either accepts a parameter or does not, which is not a property any subset of examples can vary. + max_invariant_violations: Most examples on which the target may break + a blocking ``trace_invariants`` rule before the policy fails. A + count, not a rate; ``0`` (default) fails on the first violation. """ max_overall_regression_rate: float = Field(default=0.30, ge=0.0, le=1.0) @@ -496,6 +659,10 @@ class MigrationPolicy(_StrictModel): # divergence score below 1.0 means the target called a tool the source did # not, or skipped one it did — a behavioural difference at any magnitude. max_tool_divergence: float = Field(default=0.20, ge=0.0, le=1.0) + # Examples on which the target broke a blocking trace rule. A count, not a + # rate, and zero by default: a rule is an assertion the team wrote down, + # so a single break is a finding rather than noise to budget for. + max_invariant_violations: int = Field(default=0, ge=0) # Argument scoring is continuous: a reworded search query or an omitted # optional filter lands below 1.0 without being wrong. Counting every # negative delta would weigh a 0.98 the same as a 0.0 and burn the drift @@ -553,6 +720,7 @@ class SuiteEvaluatorsOverride(_StrictModel): tool_arguments: list[ToolArgumentsEvaluatorConfig] | None = None tool_trace_structure: list[ToolTraceStructureEvaluatorConfig] | None = None agent_trace: list[AgentTraceEvaluatorConfig] | None = None + trace_invariants: list[TraceInvariantsEvaluatorConfig] | None = None class SuiteSource(_StrictModel): @@ -740,16 +908,23 @@ def _check_unique_prompt_ids(self) -> Self: __all__ = [ "DEFAULT_JUDGE_MODEL", "AgentTraceEvaluatorConfig", + "ArgumentsRule", + "CallCountRule", "Defaults", "EvalShiftConfig", "EvaluatorsConfig", + "ForbiddenToolsRule", + "InvariantRule", "LLMJudgeConfig", "PromptDefinition", + "RequiredToolsRule", "SemanticEvaluatorConfig", "StructuralEvaluatorConfig", "SuiteEvaluatorsOverride", "SuiteSource", "ToolArgumentsEvaluatorConfig", + "ToolOrderRule", "ToolSelectionEvaluatorConfig", "ToolTraceStructureEvaluatorConfig", + "TraceInvariantsEvaluatorConfig", ] diff --git a/src/evalshift_cli/evaluators/failures.py b/src/evalshift_cli/evaluators/failures.py index 80f9b311..d2381941 100644 --- a/src/evalshift_cli/evaluators/failures.py +++ b/src/evalshift_cli/evaluators/failures.py @@ -19,6 +19,10 @@ MISSING_VERIFICATION_STEP: str = "MISSING_VERIFICATION_STEP" UNNECESSARY_TOOL_CALL: str = "UNNECESSARY_TOOL_CALL" REFUSAL_REGRESSION: str = "REFUSAL_REGRESSION" +#: The target broke a rule the team wrote down in ``trace_invariants`` -- +#: judged against the rule, never against the source, so a rule both models +#: broke still carries it. +INVARIANT_VIOLATION: str = "INVARIANT_VIOLATION" #: Why a model failing ground truth recorded from itself indicts the harness. #: Defined once and imported by every surface that reports it — ``evaluate`` @@ -46,6 +50,7 @@ MISSING_VERIFICATION_STEP: "Verification step skipped", UNNECESSARY_TOOL_CALL: "Unnecessary extra tool call", REFUSAL_REGRESSION: "New refusal", + INVARIANT_VIOLATION: "Broke a hand-written trace rule", } @@ -75,6 +80,7 @@ def category_label(category: str) -> str: "CATEGORY_LABELS", "DANGEROUS_ACTION_DRIFT", "FORMAT_FAILURE", + "INVARIANT_VIOLATION", "MISSING_VERIFICATION_STEP", "REFUSAL_REGRESSION", "SEMANTIC_REGRESSION", diff --git a/src/evalshift_cli/evaluators/invariants.py b/src/evalshift_cli/evaluators/invariants.py new file mode 100644 index 00000000..8319e857 --- /dev/null +++ b/src/evalshift_cli/evaluators/invariants.py @@ -0,0 +1,240 @@ +"""Deterministic trace rules: what a tool-calling model must and must not do. + +The checker behind the ``trace_invariants`` evaluator, kept free of traces, +suites and records so the rule semantics can be read and tested in one place. + +The unit of checking is a :class:`Response` -- one model turn -- carrying the +tool calls that were already in its context when it answered. Context calls +were not made by the model under test, so no rule is ever *violated* by one, +but they satisfy an ``order`` rule's ``before`` and count toward both +``call_count`` bounds: on a teacher-forced replay, round *k* was answered after the +recorded rounds before it, and those calls really happened in that +conversation. +""" + +from __future__ import annotations + +from collections.abc import Sequence +from dataclasses import asdict, dataclass +from typing import Any + +from jsonschema import Draft202012Validator + +from evalshift_cli.config.models import ( + ArgumentsRule, + CallCountRule, + ForbiddenToolsRule, + InvariantRule, + RequiredToolsRule, + ToolOrderRule, +) + + +@dataclass(frozen=True, slots=True) +class ObservedCall: + """One tool call the model under test made.""" + + name: str + arguments: dict[str, Any] + + +@dataclass(frozen=True, slots=True) +class Response: + """One model response and the tool calls already in its context. + + Attributes: + round_index: Which replayed round this response answered; ``0`` for a + single-shot replay and for an imported trace. + context: Tool names called before this response by someone other than + the model under test -- recorded history, and the recorded earlier + rounds a teacher-forced replay fed back -- in order. + calls: The model's own calls, in the order it emitted them. + """ + + round_index: int + context: tuple[str, ...] + calls: tuple[ObservedCall, ...] + + +@dataclass(frozen=True, slots=True) +class Violation: + """One broken rule, attributed to one call (or one missing tool).""" + + rule_id: str + rule_type: str + tool: str + round_index: int + detail: str + + def to_dict(self) -> dict[str, Any]: + """The JSON shape stored under a record's ``*_violations`` metadata.""" + return asdict(self) + + +def check_rules(rules: Sequence[InvariantRule], responses: Sequence[Response]) -> list[Violation]: + """Every violation of every rule across ``responses``, rule by rule. + + Args: + rules: The rules to check, in configured order. + responses: One side's model responses, in round order. + + Returns: + The violations grouped by rule (in ``rules`` order), each rule's in + the order they occur in the trace; empty when every rule holds. + """ + violations: list[Violation] = [] + for rule in rules: + violations.extend(_check(rule, responses)) + return violations + + +def side_score(rules: Sequence[InvariantRule], violations: Sequence[Violation]) -> float: + """Share of ``rules`` broken zero times; ``1.0`` when there are no rules. + + Args: + rules: The rules that were checked. + violations: What :func:`check_rules` returned for them. + + Returns: + A score in ``[0.0, 1.0]``. A rule broken many times costs the same + as a rule broken once; a violation naming no rule in ``rules`` costs + nothing. + """ + if not rules: + return 1.0 + broken = {violation.rule_id for violation in violations} + kept = sum(1 for rule in rules if rule.id not in broken) + return kept / len(rules) + + +def _check(rule: InvariantRule, responses: Sequence[Response]) -> list[Violation]: + if isinstance(rule, ForbiddenToolsRule): + return _forbidden(rule, responses) + if isinstance(rule, RequiredToolsRule): + return _required(rule, responses) + if isinstance(rule, ToolOrderRule): + return _order(rule, responses) + if isinstance(rule, CallCountRule): + return _call_count(rule, responses) + return _arguments(rule, responses) + + +def _forbidden(rule: ForbiddenToolsRule, responses: Sequence[Response]) -> list[Violation]: + banned = set(rule.tools) + return [ + Violation( + rule_id=rule.id, + rule_type=rule.type, + tool=call.name, + round_index=response.round_index, + detail=f"called forbidden tool {call.name!r}", + ) + for response in responses + for call in response.calls + if call.name in banned + ] + + +def _required(rule: RequiredToolsRule, responses: Sequence[Response]) -> list[Violation]: + called = {call.name for response in responses for call in response.calls} + last_round = responses[-1].round_index if responses else 0 + return [ + Violation( + rule_id=rule.id, + rule_type=rule.type, + tool=tool, + round_index=last_round, + detail=f"never called required tool {tool!r}", + ) + for tool in rule.tools + if tool not in called + ] + + +def _order(rule: ToolOrderRule, responses: Sequence[Response]) -> list[Violation]: + violations: list[Violation] = [] + for response in responses: + seen_before = rule.before in response.context + for call in response.calls: + if call.name == rule.before: + seen_before = True + elif call.name == rule.after and not seen_before: + violations.append( + Violation( + rule_id=rule.id, + rule_type=rule.type, + tool=call.name, + round_index=response.round_index, + detail=f"called {rule.after!r} before any {rule.before!r}", + ), + ) + return violations + + +def _call_count(rule: CallCountRule, responses: Sequence[Response]) -> list[Violation]: + violations: list[Violation] = [] + final_count = 0 + for response in responses: + count = response.context.count(rule.tool) + for call in response.calls: + if call.name != rule.tool: + continue + count += 1 + if rule.max_calls is not None and count > rule.max_calls: + violations.append( + Violation( + rule_id=rule.id, + rule_type=rule.type, + tool=call.name, + round_index=response.round_index, + detail=f"call #{count} to {rule.tool!r}; at most {rule.max_calls} allowed", + ), + ) + # What the conversation holds once this response is done. Only the + # last response's value is judged: an earlier round cannot know what + # later rounds will add, and its own calls were never fed forward. + final_count = count + if rule.min_calls is not None and final_count < rule.min_calls: + violations.append( + Violation( + rule_id=rule.id, + rule_type=rule.type, + tool=rule.tool, + round_index=responses[-1].round_index if responses else 0, + detail=( + f"{final_count} call(s) to {rule.tool!r} by the end; " + f"at least {rule.min_calls} required" + ), + ), + ) + return violations + + +def _arguments(rule: ArgumentsRule, responses: Sequence[Response]) -> list[Violation]: + validator = Draft202012Validator(rule.json_schema) + violations: list[Violation] = [] + for response in responses: + for call in response.calls: + if call.name != rule.tool: + continue + errors = sorted( + validator.iter_errors(call.arguments), + key=lambda error: [str(part) for part in error.absolute_path], + ) + if not errors: + continue + first = errors[0] + where = "/".join(str(part) for part in first.absolute_path) or "(arguments)" + violations.append( + Violation( + rule_id=rule.id, + rule_type=rule.type, + tool=call.name, + round_index=response.round_index, + detail=f"{where}: {first.message}", + ), + ) + return violations + + +__all__ = ["ObservedCall", "Response", "Violation", "check_rules", "side_score"] diff --git a/src/evalshift_cli/evaluators/trace_invariants.py b/src/evalshift_cli/evaluators/trace_invariants.py new file mode 100644 index 00000000..3ccece7f --- /dev/null +++ b/src/evalshift_cli/evaluators/trace_invariants.py @@ -0,0 +1,236 @@ +"""Evaluator: hand-written trace rules, checked on both sides of every pair. + +Every other tool evaluator asks "did the target do what the source did?", +which lets a wrong call the source also made pass as correct. This one asks +"did each side keep the rules?" -- the source is not the yardstick, the +rules are. See :mod:`evalshift_cli.evaluators.invariants` for what each rule +means, and ``migration_policy.max_invariant_violations`` for how a broken +rule fails the run. +""" + +from __future__ import annotations + +from fnmatch import fnmatchcase +from typing import Any + +from evalshift_cli.config.models import TraceInvariantsEvaluatorConfig +from evalshift_cli.evaluators.base import EvalRecord +from evalshift_cli.evaluators.failures import INVARIANT_VIOLATION +from evalshift_cli.evaluators.invariants import ( + ObservedCall, + Response, + Violation, + check_rules, + side_score, +) +from evalshift_cli.evaluators.tool_models import ToolTrace +from evalshift_cli.suite.models import SuiteExample +from evalshift_cli.traces.models import AgentTrace + +#: Stable evaluator-type slug stamped onto every record. The analysis +#: layer selects the ``max_invariant_violations`` budget on this. +KIND = "trace_invariants" + + +def replayed_responses(trace: ToolTrace, example: SuiteExample) -> list[Response]: + """One :class:`Response` per replayed round, with what that round was shown. + + Round *k* was asked after the example's ``history`` and the recorded + rounds ``0..k-1`` the replay fed back -- exactly the prefix + ``runner/orchestrator.py:build_round_messages`` builds -- never after the + model's own earlier rounds. + + Args: + trace: One side's replayed trace; a single-round trace is one response. + example: The suite example that was replayed. Its ``history``, + ``expected_tool_rounds`` and ``tool_result_fixtures`` decide each + round's context. + + Returns: + One response per round of ``trace``, in round order. + """ + history = tuple( + call.name for message in example.history or [] for call in message.tool_calls or [] + ) + recorded = example.expected_tool_rounds or [] + fed_back = min(len(recorded), len(example.tool_result_fixtures or [])) + responses: list[Response] = [] + for index, round_trace in enumerate(trace.rounds()): + earlier = tuple( + call.tool_name + for round_calls in recorded[: min(index, fed_back)] + for call in round_calls + ) + responses.append( + Response( + round_index=index, + context=history + earlier, + calls=tuple(ObservedCall(c.tool_name, c.arguments) for c in round_trace.calls), + ), + ) + return responses + + +def imported_responses(trace: AgentTrace) -> list[Response]: + """An imported trace is the agent's own whole run: one response, no context. + + Args: + trace: One side's imported trace. + + Returns: + A single response holding every tool call in sequence order. + """ + return [ + Response( + round_index=0, + context=(), + calls=tuple(ObservedCall(c.name, c.arguments) for c in trace.tool_calls), + ), + ] + + +class _TraceInvariantsBase: + kind = KIND + + def __init__(self, config: TraceInvariantsEvaluatorConfig) -> None: + self.config = config + self.name = config.name + + def applies(self, prompt_id: str) -> bool: + """Whether ``prompt_id`` matches one of the configured ``applies_to`` globs.""" + return any(fnmatchcase(prompt_id, pattern) for pattern in self.config.applies_to) + + def _record( + self, + *, + run_id: str, + prompt_id: str, + example_id: str, + source: list[Response], + target: list[Response], + ) -> EvalRecord: + rules = self.config.rules + source_violations = check_rules(rules, source) + target_violations = check_rules(rules, target) + source_score = side_score(rules, source_violations) + target_score = side_score(rules, target_violations) + metadata: dict[str, Any] = { + "rules_checked": [rule.id for rule in rules], + "source_violations": [v.to_dict() for v in source_violations], + "target_violations": [v.to_dict() for v in target_violations], + } + if self.config.owner: + metadata["owner"] = self.config.owner + if target_violations: + metadata["failure_categories"] = [INVARIANT_VIOLATION] + return EvalRecord( + run_id=run_id, + prompt_id=prompt_id, + example_id=example_id, + evaluator_name=self.name, + kind=KIND, + source_score=source_score, + target_score=target_score, + delta=target_score - source_score, + explanation=_explanation(target_violations), + metadata=metadata, + ) + + +class TraceInvariantsEvaluator(_TraceInvariantsBase): + """Rules over a replayed (source, target) :class:`ToolTrace` pair.""" + + async def score_pair( + self, + *, + run_id: str, + prompt_id: str, + example: SuiteExample, + source_trace: ToolTrace, + target_trace: ToolTrace, + ) -> EvalRecord: + """Check every rule on both sides of one replayed pair. + + Args: + run_id: The run the pair belongs to. + prompt_id: The prompt the example was replayed against. + example: The replayed suite example (supplies round context). + source_trace: The source model's replayed trace. + target_trace: The target model's replayed trace. + + Returns: + One record: each side scored as the share of rules it kept, with + both sides' violations in ``metadata``. + """ + return self._record( + run_id=run_id, + prompt_id=prompt_id, + example_id=example.id, + source=replayed_responses(source_trace, example), + target=replayed_responses(target_trace, example), + ) + + +class ImportedTraceInvariantsEvaluator(_TraceInvariantsBase): + """Rules over an imported (source, target) :class:`AgentTrace` pair.""" + + async def score_trace_pair( + self, + *, + run_id: str, + source_trace: AgentTrace, + target_trace: AgentTrace, + ) -> EvalRecord: + """Check every rule on both sides of one imported pair. + + Args: + run_id: The run the pair belongs to. + source_trace: The source side's imported trace; its + ``prompt_id`` / ``example_id`` key the record. + target_trace: The target side's imported trace. + + Returns: + One record: each side scored as the share of rules it kept, with + both sides' violations in ``metadata``. + """ + return self._record( + run_id=run_id, + prompt_id=source_trace.prompt_id, + example_id=source_trace.example_id, + source=imported_responses(source_trace), + target=imported_responses(target_trace), + ) + + +def build_trace_invariants_evaluator( + config: TraceInvariantsEvaluatorConfig, +) -> TraceInvariantsEvaluator | ImportedTraceInvariantsEvaluator: + """The evaluator for ``config.traces``; ``evaluate`` routes on its method. + + Args: + config: The configured ``trace_invariants`` evaluator. + + Returns: + A :class:`TraceInvariantsEvaluator` for ``traces: replayed``, an + :class:`ImportedTraceInvariantsEvaluator` for ``traces: imported``. + """ + if config.traces == "imported": + return ImportedTraceInvariantsEvaluator(config) + return TraceInvariantsEvaluator(config) + + +def _explanation(violations: list[Violation]) -> str: + if not violations: + return "target kept every trace rule" + broken = sorted({v.rule_id for v in violations}) + return f"target broke {len(broken)} trace rule(s): {', '.join(broken)}" + + +__all__ = [ + "KIND", + "ImportedTraceInvariantsEvaluator", + "TraceInvariantsEvaluator", + "build_trace_invariants_evaluator", + "imported_responses", + "replayed_responses", +] diff --git a/src/evalshift_cli/hosted/bundle.py b/src/evalshift_cli/hosted/bundle.py index 8461e895..5faa86ef 100644 --- a/src/evalshift_cli/hosted/bundle.py +++ b/src/evalshift_cli/hosted/bundle.py @@ -264,11 +264,17 @@ def _evaluator_config_snapshot( dict, so two suites of the same project are meant to hash differently. """ dumped = cfg.model_dump(mode="json") + evaluators_dumped = evaluators.model_dump(mode="json") + # Added in 1.3 with an empty default. Hashing the empty list would move + # ``eval_config_hash`` for every config that never wrote the key and + # orphan every hosted baseline -- the same reason ``slices`` below is kept. + if not evaluators_dumped.get("trace_invariants"): + evaluators_dumped.pop("trace_invariants", None) return { "version": dumped["version"], "prompts": dumped["prompts"], "defaults": dumped["defaults"], - "evaluators": evaluators.model_dump(mode="json"), + "evaluators": evaluators_dumped, # Legacy constant. This key used to carry the top-level ``slices:`` # block, which was removed from evalshift.yaml because nothing ever # read it; a config that still sets it no longer loads. The key stays, diff --git a/src/evalshift_cli/hosted/bundle_manifest.schema.json b/src/evalshift_cli/hosted/bundle_manifest.schema.json index d27e986a..8893a55d 100644 --- a/src/evalshift_cli/hosted/bundle_manifest.schema.json +++ b/src/evalshift_cli/hosted/bundle_manifest.schema.json @@ -1209,6 +1209,19 @@ "title": "Max Critical Regressions", "type": "integer" }, + "max_invariant_violations": { + "anyOf": [ + { + "minimum": 0, + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Max Invariant Violations" + }, "max_latency_increase": { "maximum": 10.0, "minimum": 0.0, @@ -1638,6 +1651,19 @@ "default": null, "title": "Max Critical Regressions" }, + "max_invariant_violations": { + "anyOf": [ + { + "minimum": 0, + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Max Invariant Violations" + }, "max_latency_increase": { "anyOf": [ { diff --git a/src/evalshift_cli/insights/facts.py b/src/evalshift_cli/insights/facts.py index a81ff29b..2d8ea832 100644 --- a/src/evalshift_cli/insights/facts.py +++ b/src/evalshift_cli/insights/facts.py @@ -77,8 +77,8 @@ #: The two policy budgets that are ratios of increase rather than rates. _INCREASE_BUDGETS: frozenset[str] = frozenset({"max_cost_increase", "max_latency_increase"}) -#: The one budget that is a plain count. -_COUNT_BUDGETS: frozenset[str] = frozenset({"max_critical_regressions"}) +#: The budgets that are plain counts. +_COUNT_BUDGETS: frozenset[str] = frozenset({"max_critical_regressions", "max_invariant_violations"}) @dataclass(frozen=True, slots=True) diff --git a/src/evalshift_cli/reports/html.py b/src/evalshift_cli/reports/html.py index f38c920d..ef07d34f 100644 --- a/src/evalshift_cli/reports/html.py +++ b/src/evalshift_cli/reports/html.py @@ -139,6 +139,7 @@ def _confidence_label(severity: str, p_corrected: float) -> str: "tool_trace_structure": "Tool sequence", "semantic": "Semantic", "structural": "Structure", + "trace_invariants": "Trace rules", } # Axis labels, keyed on the record's ``kind`` slug and never on the @@ -514,6 +515,7 @@ def render_html(report: ReportData, *, insight: Insight | None = None) -> str: dropped_params=report.dropped_params, samples_per_example=report.samples_per_example, judge_family_overlap=report.judge_family_overlap, + invariant_violations=report.invariant_violations, suite_name=_suite_name(report.suite_path), source_cost_usd=role_totals["source"]["cost"], target_cost_usd=role_totals["target"]["cost"], diff --git a/src/evalshift_cli/reports/json.py b/src/evalshift_cli/reports/json.py index 035ee56b..c3022b68 100644 --- a/src/evalshift_cli/reports/json.py +++ b/src/evalshift_cli/reports/json.py @@ -16,7 +16,7 @@ import json from collections.abc import Mapping -from dataclasses import dataclass, field +from dataclasses import asdict, dataclass, field from pathlib import Path from typing import Any @@ -216,6 +216,34 @@ class ExampleRow: turn_index: int | None = None +@dataclass(frozen=True, slots=True) +class InvariantViolationRow: + """One trace rule the target broke on one example. + + Listed whatever the delta: a rule both models broke scores ``delta == 0`` + and would never surface among the top regressions, yet a *blocking* entry + still counts against ``max_invariant_violations`` under a policy. + ``blocking`` is the record's flag: ``False`` rows come from a + ``blocking: false`` entry and are reported but never gate. + """ + + prompt_id: str + example_id: str + evaluator_name: str + owner: str | None + rule_id: str + rule_type: str + tool: str + round_index: int + detail: str + source_also: bool + blocking: bool + # Zero-based ordinal among the scored samples, set only on a row reduced + # over ``samples_per_example > 1``: the same rule then appears once per + # sample that broke it. ``None`` on a single-sample run. + sample: int | None = None + + @dataclass(frozen=True, slots=True) class PromptSection: """Per-prompt slice of the report payload.""" @@ -265,6 +293,9 @@ class ReportData: # self-preference note under the banners. Empty when no judge overlaps, # no judge contributed a score, or the report ran without a config. judge_family_overlap: list[dict[str, Any]] = field(default_factory=list) + # Every trace rule the target broke, one row per violation, whatever the + # delta. Empty when no ``trace_invariants`` evaluator scored the run. + invariant_violations: list[InvariantViolationRow] = field(default_factory=list) def build_report_payload( @@ -342,6 +373,57 @@ def build_report_payload( source_model=state.models.source, target_model=state.models.target, ), + invariant_violations=_invariant_violations(scores), + ) + + +def _invariant_violations(scores: list[EvalRecord]) -> list[InvariantViolationRow]: + """One row per trace rule the target broke, read off ``trace_invariants`` records. + + ``source_also`` is whether the source broke the same rule (by ``rule_id``, + on any sample) on the same example: a rule both sides break is more + likely a rule to review than a migration regression. Errored records + carry no verdict and list nothing. + """ + rows: list[InvariantViolationRow] = [] + for r in scores: + if r.kind != "trace_invariants" or r.error is not None: + continue + source_rules = { + v.get("rule_id") + for v in r.metadata.get("source_violations") or [] + if isinstance(v, dict) + } + owner = r.metadata.get("owner") + for v in r.metadata.get("target_violations") or []: + if not isinstance(v, dict): + continue + sample = v.get("sample") + rows.append( + InvariantViolationRow( + prompt_id=r.prompt_id, + example_id=r.example_id, + evaluator_name=r.evaluator_name, + owner=owner if isinstance(owner, str) else None, + rule_id=str(v.get("rule_id", "")), + rule_type=str(v.get("rule_type", "")), + tool=str(v.get("tool", "")), + round_index=int(v.get("round_index", 0)), + detail=str(v.get("detail", "")), + source_also=v.get("rule_id") in source_rules, + blocking=r.blocking, + sample=sample if isinstance(sample, int) else None, + ), + ) + return sorted( + rows, + key=lambda row: ( + row.prompt_id, + row.example_id, + row.rule_id, + row.round_index, + -1 if row.sample is None else row.sample, + ), ) @@ -1088,12 +1170,14 @@ def _to_jsonable(report: ReportData) -> dict[str, Any]: "dropped_params": report.dropped_params, "samples_per_example": report.samples_per_example, "judge_family_overlap": report.judge_family_overlap, + "invariant_violations": [asdict(v) for v in report.invariant_violations], } __all__ = [ "REPORT_JSON_FILENAME", "SHARED_GROUND_TRUTH_NOTE_PREFIX", + "InvariantViolationRow", "PromptEconomics", "PromptSection", "ReportData", diff --git a/src/evalshift_cli/reports/templates/report.css b/src/evalshift_cli/reports/templates/report.css index ed363db9..cf7d81fd 100644 --- a/src/evalshift_cli/reports/templates/report.css +++ b/src/evalshift_cli/reports/templates/report.css @@ -144,6 +144,8 @@ pre { .badge-ok { color: var(--ok); background: var(--ok-tint); border-color: var(--ok-edge); } .badge-warn { color: var(--warn); background: var(--warn-tint); border-color: var(--warn-edge); } .badge-bad { color: var(--bad); background: var(--bad-tint); border-color: var(--bad-edge); } +/* A broken rule from a `blocking: false` entry: reported, never gates. */ +.advisory-tag { color: var(--dim2); } .notice { padding: 14px 16px; diff --git a/src/evalshift_cli/reports/templates/report.html.j2 b/src/evalshift_cli/reports/templates/report.html.j2 index 4b50ebd1..a0bd0c01 100644 --- a/src/evalshift_cli/reports/templates/report.html.j2 +++ b/src/evalshift_cli/reports/templates/report.html.j2 @@ -269,7 +269,7 @@ {% endif %} {% if failed_budgets %} -{% set count_budgets = ["max_critical_regressions"] %} +{% set count_budgets = ["max_critical_regressions", "max_invariant_violations"] %} {% set increase_budgets = ["max_cost_increase", "max_latency_increase"] %} {%- macro fmt_budget(name, value) -%} {%- if name in count_budgets -%} @@ -406,6 +406,34 @@ {% endif %} +{% if invariant_violations %} +
+
+
+

Trace rules broken

+ {{ invariant_violations|length }} by the target +
+

Every rule the target broke, whatever the score change. A rule both models broke scores no change, so it never shows among the regressions below.

+
+ + + + {% for v in invariant_violations %} + + + + + + + + {% endfor %} + +
ExampleRuleOwnerWhat happenedSource too?
{{ v.prompt_id }} · {{ v.example_id }}{% if v.round_index %} · round {{ v.round_index + 1 }}{% endif %}{% if v.sample is not none %} · sample {{ v.sample + 1 }}{% endif %}{{ v.rule_id }} {{ v.rule_type }}{% if not v.blocking %} advisory{% endif %}{{ v.owner or "—" }}{{ v.detail }}{{ "yes — review the rule" if v.source_also else "no" }}
+
+
+
+{% endif %} + {% for section in prompt_sections %}
diff --git a/tests/integration/test_tool_pipeline.py b/tests/integration/test_tool_pipeline.py index a7688629..84f637b4 100644 --- a/tests/integration/test_tool_pipeline.py +++ b/tests/integration/test_tool_pipeline.py @@ -392,3 +392,96 @@ def test_all_prints_the_finding_above_the_verdict( assert "broken eval harness" in flat assert "20 of 20 tool-selection conformance rows (100%)" in flat assert flat.index("broken eval harness") < flat.index("Migration verdict") + + +class TestTraceInvariantsGate: + """A rule the target breaks fails the run, whether or not the source broke it too.""" + + _RULES = """\ + trace_invariants: + - name: security_contract + owner: "@secops" + rules: + - id: never-email + type: forbidden + tools: [send_email] +""" + + def _decision(self, tmp_path: Path, run_id: str) -> dict[str, Any]: + import json + + result = runner.invoke(app, ["analyze", run_id]) + assert result.exit_code == 0, result.stdout + path = tmp_path / ".evalshift" / "runs" / run_id / "migration_decision.json" + return json.loads(path.read_text(encoding="utf-8")) + + def test_target_breaking_a_rule_fails( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold_agent_project(tmp_path, extra_evaluators=self._RULES) + config = tmp_path / "evalshift.yaml" + config.write_text( + config.read_text(encoding="utf-8") + + "\nmigration_policy:\n max_invariant_violations: 0\n", + encoding="utf-8", + ) + _patch_with_tools( + monkeypatch, + tool_for_role={"source": "notify_security_team", "target": "send_email"}, + ) + run_id = _run_pipeline(monkeypatch, tmp_path) + + decision = self._decision(tmp_path, run_id) + budgets = {b["name"]: b for b in decision["budget_results"]} + assert decision["verdict"] == "fail" + assert budgets["max_invariant_violations"]["observed"] == 20.0 + assert budgets["max_invariant_violations"]["denominator"] == 20 + assert budgets["max_invariant_violations"]["passed"] is False + + def test_a_rule_both_models_break_still_fails( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + # Both models call ``send_email``: tool_selection sees equivalent + # traces, so only the trace rule's budget can fail the run. + _scaffold_agent_project(tmp_path, extra_evaluators=self._RULES) + config = tmp_path / "evalshift.yaml" + config.write_text( + config.read_text(encoding="utf-8") + + "\nmigration_policy:\n max_invariant_violations: 0\n", + encoding="utf-8", + ) + _patch_with_tools( + monkeypatch, + tool_for_role={"source": "send_email", "target": "send_email"}, + ) + run_id = _run_pipeline(monkeypatch, tmp_path) + + decision = self._decision(tmp_path, run_id) + budgets = {b["name"]: b for b in decision["budget_results"]} + assert decision["verdict"] == "fail" + assert budgets["max_invariant_violations"]["observed"] == 20.0 + assert budgets["max_invariant_violations"]["denominator"] == 20 + assert budgets["max_invariant_violations"]["passed"] is False + failed = [b["name"] for b in decision["budget_results"] if not b["passed"]] + assert failed == ["max_invariant_violations"] + + def test_clean_target_passes_the_rule( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _scaffold_agent_project(tmp_path, extra_evaluators=self._RULES) + config = tmp_path / "evalshift.yaml" + config.write_text( + config.read_text(encoding="utf-8") + + "\nmigration_policy:\n max_invariant_violations: 0\n", + encoding="utf-8", + ) + _patch_with_tools( + monkeypatch, + tool_for_role={"source": "notify_security_team", "target": "notify_security_team"}, + ) + run_id = _run_pipeline(monkeypatch, tmp_path) + + budgets = {b["name"]: b for b in self._decision(tmp_path, run_id)["budget_results"]} + assert budgets["max_invariant_violations"]["observed"] == 0.0 + assert budgets["max_invariant_violations"]["denominator"] == 20 + assert budgets["max_invariant_violations"]["passed"] is True diff --git a/tests/unit/test_bundle_shape.py b/tests/unit/test_bundle_shape.py index b34163ad..beac3b2c 100644 --- a/tests/unit/test_bundle_shape.py +++ b/tests/unit/test_bundle_shape.py @@ -299,6 +299,7 @@ def test_every_emitted_budget_carries_an_integer_denominator(run_fixture: RunFix "min_equivalence_rate", "max_tool_argument_drift", "max_tool_divergence", + "max_invariant_violations", "tool_argument_drift_floor", "max_cost_increase", "max_latency_increase", @@ -306,7 +307,7 @@ def test_every_emitted_budget_carries_an_integer_denominator(run_fixture: RunFix "slices", } -# The eight per-slice ratio/count budgets: the nine top-level fields minus +# The nine per-slice ratio/count budgets: the ten top-level fields minus # ``fail_on_dropped_params`` (not overridable per slice — see # ``MigrationPolicy.fail_on_dropped_params``'s docstring) and minus ``slices`` # itself (a slice cannot nest another slice). @@ -654,6 +655,21 @@ def _spy(**kwargs: Any) -> Any: bundle_module._build_examples = real # type: ignore[assignment] assert seen["names"] == frozenset({"routing"}) + #: ``manifest.eval_config_hash`` for :attr:`_CONFIG` resolved for + #: ``main_chat``, computed by the CLI at 5b608f2 — before + #: ``trace_invariants`` existed. A suite override that never mentions the + #: family must not start hashing its empty default. + _HASH_BEFORE_TRACE_INVARIANTS = ( + "sha256:18b43295c4f2d1343c322522348565762fffd670a009dfc4bda861de15e6896f" + ) + + def test_eval_config_hash_of_a_suite_resolved_run_is_unchanged( + self, run_fixture: RunFixture + ) -> None: + manifest = self._bundle(run_fixture, suite_name="main_chat")["manifest"] + assert isinstance(manifest, dict) + assert manifest["eval_config_hash"] == self._HASH_BEFORE_TRACE_INVARIANTS + class TestBundleShipsNoSuiteContent: """Suite content stays local; only content hashes go on the wire. @@ -774,6 +790,73 @@ def test_evaluator_config_still_ships_an_empty_slices_list( assert config["slices"] == [] +class TestEvaluatorConfigHashesTraceInvariantsOnlyWhenSet: + """``trace_invariants`` enters ``eval_config_hash`` only when configured. + + An empty family is dropped from the snapshot so configs that never wrote + the key keep the hash their hosted baselines were recorded under (pinned + by :class:`TestEvaluatorConfigKeepsTheLegacySlicesKey`). Configured rules + are methodology and must move the hash like any other evaluator. + """ + + _BASE = TestEvaluatorConfigKeepsTheLegacySlicesKey._CONFIG + _WITH_RULES = """ + version: 1 + project: acme/model-migration + prompts: + - id: greet + detection: manual + content: "Hello {name}" + variables: [name] + defaults: + source_model: gemini/gemini-2.5-flash + target_model: gemini/gemini-3.1-flash-lite-preview + evaluators: + structural: + - type: length + min_chars: 1 + llm_judge: + - criterion_name: tone + criterion_prompt: Which reply is friendlier? + trace_invariants: + - name: payments_contract + rules: + - id: no-refund + type: forbidden + tools: [refund] + migration_policy: + max_overall_regression_rate: 0.10 + slices: + checkout: + max_overall_regression_rate: 0.05 + """ + + def _bundle(self, run_fixture: RunFixture, text: str) -> dict[str, Any]: + run_fixture.config.write_text(text, encoding="utf-8") + return _load(run_fixture.build().path) + + def test_an_unconfigured_family_is_absent_from_the_snapshot( + self, run_fixture: RunFixture + ) -> None: + config = self._bundle(run_fixture, self._BASE)["evaluator_config"] + assert isinstance(config, dict) + assert "trace_invariants" not in config["evaluators"] + + def test_configured_rules_ship_and_move_the_hash(self, run_fixture: RunFixture) -> None: + bundle = self._bundle(run_fixture, self._WITH_RULES) + config = bundle["evaluator_config"] + assert isinstance(config, dict) + [family] = config["evaluators"]["trace_invariants"] + assert family["name"] == "payments_contract" + assert [rule["id"] for rule in family["rules"]] == ["no-refund"] + manifest = bundle["manifest"] + assert isinstance(manifest, dict) + assert ( + manifest["eval_config_hash"] + != TestEvaluatorConfigKeepsTheLegacySlicesKey._HASH_BEFORE_THE_REMOVAL + ) + + def _rewrite_suite_history(suite_path: Path, system_prompt: str) -> None: """Give every suite example a conversation prefix carrying ``system_prompt``.""" rows = [ diff --git a/tests/unit/test_cli_capture.py b/tests/unit/test_cli_capture.py index 288c0c64..8f1c4702 100644 --- a/tests/unit/test_cli_capture.py +++ b/tests/unit/test_cli_capture.py @@ -11,7 +11,7 @@ from evalshift_cli.captures.models import CaptureEnvelope, PromotedCase from evalshift_cli.captures.toolset import EMPTY_TOOLSET_FINGERPRINT, fingerprint_tools -from evalshift_cli.cli.commands.capture import _declared_tool_properties +from evalshift_cli.cli.commands.capture import _build_suite_entries, _declared_tool_properties from evalshift_cli.cli.commands.init import render_minimal_config from evalshift_cli.cli.main import app from evalshift_cli.config.loader import load_config @@ -2010,3 +2010,125 @@ def test_sync_is_silent_when_no_workflow_uses_the_action(tmp_path: Path) -> None assert result.exit_code == 0, result.stdout assert "CI installs" not in result.stdout + + +# --------------------------------------------------------------------------- +# sync — hand-written trace_invariants survive regeneration +# --------------------------------------------------------------------------- + + +def test_sync_keeps_a_managed_suites_trace_invariants(tmp_path: Path) -> None: + """Review Focus 5: a contract is human-owned; regeneration must not drop it.""" + config_path = tmp_path / "evalshift.yaml" + suite_path = tmp_path / "suites" / "checkout" / "golden.jsonl" + suite_path.parent.mkdir(parents=True) + suite_path.write_text("", encoding="utf-8") + rules = [ + { + "name": "payments", + "rules": [{"id": "no-v1", "type": "forbidden", "tools": ["refund_v1"]}], + } + ] + existing = { + "checkout": { + "source": "captured", + "path": "suites/checkout/golden.jsonl", + "evaluators": {"tool_selection": [{"name": "old"}], "trace_invariants": rules}, + }, + } + + entries, frozen = _build_suite_entries( + {"checkout": suite_path}, + {"checkout": None}, + config_path=config_path, + existing=existing, + ) + + assert frozen == {} + assert entries["checkout"]["evaluators"] == {"trace_invariants": rules} + + +def test_sync_without_prior_rules_is_unchanged(tmp_path: Path) -> None: + config_path = tmp_path / "evalshift.yaml" + suite_path = tmp_path / "golden.jsonl" + suite_path.write_text("", encoding="utf-8") + entries, _ = _build_suite_entries( + {"s": suite_path}, + {"s": None}, + config_path=config_path, + existing={}, + ) + assert "evaluators" not in entries["s"] + + +def test_sync_preview_of_a_frozen_suite_shows_its_trace_invariants_kept(tmp_path: Path) -> None: + """The ``managed: false`` preview is what sync *would* write -- rules included.""" + config_path = tmp_path / "evalshift.yaml" + suite_path = tmp_path / "golden.jsonl" + suite_path.write_text("", encoding="utf-8") + rules = [{"name": "p", "rules": [{"id": "r", "type": "forbidden", "tools": ["t"]}]}] + prior = { + "managed": False, + "source": "captured", + "path": "golden.jsonl", + "evaluators": {"trace_invariants": rules}, + } + + entries, frozen = _build_suite_entries( + {"s": suite_path}, {"s": None}, config_path=config_path, existing={"s": prior} + ) + + assert entries["s"] is prior # the frozen entry itself is written back untouched + assert frozen["s"]["evaluators"] == {"trace_invariants": rules} + + +_HAND_WRITTEN_RULES = """\ + trace_invariants: + - name: payments + owner: payments-team + rules: + - id: no-v1 + type: forbidden + tools: + - refund_v1 + - id: refund-args + type: arguments + tool: refund + json_schema: + type: object + required: + - order_id +""" + + +def test_sync_round_trips_hand_written_trace_invariants_through_the_config( + tmp_path: Path, +) -> None: + """End to end: rules hand-added to a managed entry survive a re-sync on disk.""" + _write_capture(tmp_path, capture_id="cap_1", suite="alpha") + config = tmp_path / "evalshift.yaml" + _write_min_config(config) + assert _invoke(["sync", "--config", str(config)], tmp_path).exit_code == 0 + text = config.read_text(encoding="utf-8") + assert " evaluators:\n" in text + config.write_text( + text.replace(" evaluators:\n", " evaluators:\n" + _HAND_WRITTEN_RULES, 1), + encoding="utf-8", + ) + expected = load_config(config).suites["alpha"].evaluators + assert expected is not None + assert expected.trace_invariants is not None + + result = _invoke(["sync", "--config", str(config)], tmp_path) + + assert result.exit_code == 0, result.stdout + override = load_config(config).suites["alpha"].evaluators + assert override is not None + assert override.trace_invariants == expected.trace_invariants + # Every other key is still regenerated. + assert override.tool_selection is not None + assert [e.name for e in override.tool_selection] == ["routing"] + # ...and the rewritten region is stable from here on. + settled = config.read_text(encoding="utf-8") + assert _invoke(["sync", "--config", str(config)], tmp_path).exit_code == 0 + assert config.read_text(encoding="utf-8") == settled diff --git a/tests/unit/test_config_models.py b/tests/unit/test_config_models.py index b3cd3643..03797a8d 100644 --- a/tests/unit/test_config_models.py +++ b/tests/unit/test_config_models.py @@ -12,6 +12,8 @@ from evalshift_cli.config.models import ( DEFAULT_JUDGE_MODEL, AgentTraceEvaluatorConfig, + ArgumentsRule, + CallCountRule, Defaults, EvalShiftConfig, EvaluatorsConfig, @@ -22,7 +24,9 @@ SliceMigrationPolicy, StructuralEvaluatorConfig, ToolArgumentsEvaluatorConfig, + ToolOrderRule, ToolSelectionEvaluatorConfig, + TraceInvariantsEvaluatorConfig, ) # --------------------------------------------------------------------------- @@ -672,3 +676,107 @@ def test_accepts_repeated_sampling(self) -> None: def test_bounds(self, bad: int) -> None: with pytest.raises(ValidationError): Defaults(samples_per_example=bad) + + +class TestTraceInvariantsConfig: + def _cfg(self, **overrides: object) -> TraceInvariantsEvaluatorConfig: + payload: dict[str, object] = { + "name": "payments_contract", + "rules": [ + {"id": "auth-first", "type": "order", "before": "authenticate", "after": "charge"}, + {"id": "one-charge", "type": "call_count", "tool": "charge", "max_calls": 1}, + ], + **overrides, + } + return TraceInvariantsEvaluatorConfig.model_validate(payload) + + def test_rules_parse_into_their_own_types(self) -> None: + cfg = self._cfg() + assert isinstance(cfg.rules[0], ToolOrderRule) + assert isinstance(cfg.rules[1], CallCountRule) + assert cfg.traces == "replayed" + assert cfg.applies_to == ["*"] + assert cfg.blocking is True + assert cfg.owner is None + + def test_an_unknown_rule_type_is_rejected(self) -> None: + with pytest.raises(ValidationError): + self._cfg(rules=[{"id": "x", "type": "sometimes", "tools": ["a"]}]) + + def test_rule_ids_must_be_unique(self) -> None: + with pytest.raises(ValidationError, match="duplicate rule id"): + self._cfg( + rules=[ + {"id": "dup", "type": "forbidden", "tools": ["a"]}, + {"id": "dup", "type": "forbidden", "tools": ["b"]}, + ], + ) + + def test_an_empty_rule_list_is_rejected(self) -> None: + with pytest.raises(ValidationError): + self._cfg(rules=[]) + + def test_order_rejects_a_tool_ordered_before_itself(self) -> None: + with pytest.raises(ValidationError, match="before and after"): + self._cfg(rules=[{"id": "x", "type": "order", "before": "a", "after": "a"}]) + + def test_an_invalid_json_schema_is_rejected_at_load(self) -> None: + with pytest.raises(ValidationError, match="json_schema"): + self._cfg( + rules=[ + { + "id": "amount", + "type": "arguments", + "tool": "charge", + "json_schema": {"type": "not-a-type"}, + }, + ], + ) + + def test_a_valid_json_schema_is_kept_verbatim(self) -> None: + schema = {"type": "object", "properties": {"amount": {"type": "number", "maximum": 5000}}} + cfg = self._cfg( + rules=[{"id": "amount", "type": "arguments", "tool": "charge", "json_schema": schema}], + ) + rule = cfg.rules[0] + assert isinstance(rule, ArgumentsRule) + assert rule.json_schema == schema + + def test_call_count_rejects_a_negative_bound(self) -> None: + with pytest.raises(ValidationError): + self._cfg(rules=[{"id": "x", "type": "call_count", "tool": "a", "max_calls": -1}]) + + def test_call_count_needs_at_least_one_bound(self) -> None: + with pytest.raises(ValidationError, match="min_calls"): + self._cfg(rules=[{"id": "x", "type": "call_count", "tool": "a"}]) + + def test_call_count_rejects_max_below_min(self) -> None: + with pytest.raises(ValidationError, match="below"): + self._cfg( + rules=[ + {"id": "x", "type": "call_count", "tool": "a", "min_calls": 2, "max_calls": 1}, + ], + ) + + def test_call_count_exact_is_both_bounds_equal(self) -> None: + cfg = self._cfg( + rules=[{"id": "x", "type": "call_count", "tool": "a", "min_calls": 1, "max_calls": 1}], + ) + rule = cfg.rules[0] + assert isinstance(rule, CallCountRule) + assert (rule.min_calls, rule.max_calls) == (1, 1) + + def test_the_family_defaults_to_empty(self) -> None: + assert EvaluatorsConfig().trace_invariants == [] + + +class TestMaxInvariantViolationsPolicy: + def test_defaults_to_zero(self) -> None: + assert MigrationPolicy().max_invariant_violations == 0 + + def test_slice_override_defaults_to_inherit(self) -> None: + assert SliceMigrationPolicy().max_invariant_violations is None + + def test_rejects_negative(self) -> None: + with pytest.raises(ValidationError): + MigrationPolicy(max_invariant_violations=-1) diff --git a/tests/unit/test_evaluate_command.py b/tests/unit/test_evaluate_command.py index d761654f..614b3989 100644 --- a/tests/unit/test_evaluate_command.py +++ b/tests/unit/test_evaluate_command.py @@ -21,22 +21,37 @@ SCORES_FILENAME, _build_evaluators, _coverage_for, + _is_agent_trace_evaluator, + _is_tool_evaluator, _pair_calls, _PairedCalls, + _reduce_sample_cells, + _score_agent_traces, _score_all, _score_one, + _ScoredCell, ) from evalshift_cli.cli.main import app -from evalshift_cli.config.models import ToolSelectionEvaluatorConfig -from evalshift_cli.evaluators.base import PairedScore +from evalshift_cli.config.models import ( + EvaluatorsConfig, + ToolSelectionEvaluatorConfig, + TraceInvariantsEvaluatorConfig, +) +from evalshift_cli.evaluators.base import EvalRecord, PairedScore +from evalshift_cli.evaluators.failures import INVARIANT_VIOLATION from evalshift_cli.evaluators.llm_judge import PairwiseJudgeEvaluator from evalshift_cli.evaluators.semantic import CosineSimilarityEvaluator from evalshift_cli.evaluators.tool_models import ToolCall, ToolTrace from evalshift_cli.evaluators.tool_selection import ToolSelectionEvaluator +from evalshift_cli.evaluators.trace_invariants import ( + ImportedTraceInvariantsEvaluator, + TraceInvariantsEvaluator, +) from evalshift_cli.models.client import ModelClient from evalshift_cli.runner.checkpoint import append_call, read_state, write_state from evalshift_cli.runner.models import Call, EvaluatorCoverage, RunModels, RunState from evalshift_cli.suite.models import ChatMessage, Suite, SuiteExample +from evalshift_cli.traces.loader import TRACES_FILENAME from tests.unit.suite_examples import suite_example runner = CliRunner() @@ -462,6 +477,7 @@ def test_agent_trace_evaluator_requires_imported_traces( assert result.exit_code == 1 assert "agent_trace evaluators require imported traces" in result.stdout + assert "trace_invariants" not in result.stdout # --------------------------------------------------------------------------- @@ -1468,3 +1484,371 @@ def test_scores_jsonl_has_one_row_per_example_and_evaluator( assert row["metadata"]["samples"]["scored"] == 2 coverage = read_state(run_dir).evaluator_coverage assert [(c.attempted, c.recorded) for c in coverage] == [(2, 2), (2, 2)] + + +# --------------------------------------------------------------------------- +# trace_invariants — build, applies_to scoping, per-sample violations +# --------------------------------------------------------------------------- + +_INVARIANTS = { + "name": "payments", + "rules": [{"id": "no-v1", "type": "forbidden", "tools": ["refund_v1"]}], +} + + +def _tool_pair(prompt_id: str, example_id: str, target_tool: str) -> _PairedCalls: + def call(role: str, tool: str) -> Call: + return Call( + run_id="r1", + prompt_id=prompt_id, + example_id=example_id, + model_id=f"m/{role}", + role=role, # type: ignore[arg-type] + trace=ToolTrace(calls=[ToolCall(tool_name=tool, sequence_index=0)]), + ) + + return _PairedCalls( + prompt_id=prompt_id, + example_id=example_id, + source=call("source", "lookup"), + target=call("target", target_tool), + ) + + +def _imported_trace_line(prompt_id: str, role: str, tool: str) -> str: + return json.dumps( + { + "run_id": "r1", + "prompt_id": prompt_id, + "example_id": "e1", + "role": role, + "events": [ + { + "type": "tool_call", + "sequence_index": 0, + "timestamp": "2026-06-09T12:00:00Z", + "metadata": {}, + "name": tool, + "arguments": {}, + }, + ], + }, + ) + + +def _write_imported_traces(run_dir: Path, prompt_ids: list[str]) -> list[_PairedCalls]: + run_dir.mkdir(parents=True, exist_ok=True) + lines = [ + _imported_trace_line(prompt_id, role, tool) + for prompt_id in prompt_ids + for role, tool in (("source", "lookup"), ("target", "refund_v1")) + ] + (run_dir / TRACES_FILENAME).write_text("\n".join(lines) + "\n", encoding="utf-8") + return [_tool_pair(prompt_id, "e1", "refund_v1") for prompt_id in prompt_ids] + + +class TestTraceInvariantsWiring: + def test_build_routes_on_traces(self, tmp_path: Path) -> None: + cfg = EvaluatorsConfig( + trace_invariants=[ + TraceInvariantsEvaluatorConfig.model_validate(_INVARIANTS), + TraceInvariantsEvaluatorConfig.model_validate( + {**_INVARIANTS, "name": "imported", "traces": "imported"}, + ), + ], + ) + built = _build_evaluators(cfg, tmp_path, judge_client=ModelClient()) + assert isinstance(built[0], TraceInvariantsEvaluator) + assert isinstance(built[1], ImportedTraceInvariantsEvaluator) + + def test_each_variant_lands_on_its_own_scoring_path(self, tmp_path: Path) -> None: + cfg = EvaluatorsConfig( + trace_invariants=[ + TraceInvariantsEvaluatorConfig.model_validate(_INVARIANTS), + TraceInvariantsEvaluatorConfig.model_validate( + {**_INVARIANTS, "name": "imported", "traces": "imported"}, + ), + ], + ) + replayed, imported = _build_evaluators(cfg, tmp_path, judge_client=ModelClient()) + assert _is_tool_evaluator(replayed) + assert not _is_agent_trace_evaluator(replayed) + assert _is_agent_trace_evaluator(imported) + assert not _is_tool_evaluator(imported) + + def test_the_config_blocking_flag_rides_on_the_evaluator(self, tmp_path: Path) -> None: + cfg = EvaluatorsConfig( + trace_invariants=[ + TraceInvariantsEvaluatorConfig.model_validate({**_INVARIANTS, "blocking": False}), + ], + ) + (built,) = _build_evaluators(cfg, tmp_path, judge_client=ModelClient()) + assert built.blocking is False # type: ignore[attr-defined] + + async def test_a_prompt_outside_applies_to_is_never_attempted(self) -> None: + evaluator = TraceInvariantsEvaluator( + TraceInvariantsEvaluatorConfig.model_validate({**_INVARIANTS, "applies_to": ["pay*"]}), + ) + cells = await _score_all( + Console(), + [evaluator], # type: ignore[list-item] + [_tool_pair("payments", "e1", "refund_v1"), _tool_pair("support", "e2", "refund_v1")], + "r1", + {}, + concurrency=2, + quiet=True, + ) + assert [(c.prompt_id, c.record is not None) for c in cells] == [("payments", True)] + coverage = _coverage_for(cells) + assert (coverage[0].attempted, coverage[0].unmeasured) == (1, []) + + async def test_imported_prompt_outside_applies_to_is_skipped(self, tmp_path: Path) -> None: + evaluator = ImportedTraceInvariantsEvaluator( + TraceInvariantsEvaluatorConfig.model_validate( + {**_INVARIANTS, "traces": "imported", "applies_to": ["pay*"]}, + ), + ) + run_dir = tmp_path / "r1" + pairs = _write_imported_traces(run_dir, ["support"]) + + cells = await _score_agent_traces( + run_dir=run_dir, + evaluators=[evaluator], # type: ignore[list-item] + pairs=pairs, + run_id="r1", + ) + + assert cells == [] + + async def test_imported_prompt_inside_applies_to_is_scored(self, tmp_path: Path) -> None: + evaluator = ImportedTraceInvariantsEvaluator( + TraceInvariantsEvaluatorConfig.model_validate( + {**_INVARIANTS, "traces": "imported", "applies_to": ["pay*"]}, + ), + ) + run_dir = tmp_path / "r1" + pairs = _write_imported_traces(run_dir, ["payments", "support"]) + + cells = await _score_agent_traces( + run_dir=run_dir, + evaluators=[evaluator], # type: ignore[list-item] + pairs=pairs, + run_id="r1", + ) + + assert [(c.prompt_id, c.kind) for c in cells] == [("payments", "trace_invariants")] + record = cells[0].record + assert record is not None + assert record.target_score < 1.0 + assert record.metadata["failure_categories"] == [INVARIANT_VIOLATION] + + def test_sample_reduction_keeps_every_samples_violations(self) -> None: + """Review Focus 1: the lead sample was clean, sample 2 broke the rule.""" + + def cell(index: int, target_violations: list[dict[str, Any]]) -> _ScoredCell: + target_score = 0.0 if target_violations else 1.0 + metadata: dict[str, Any] = { + "rules_checked": ["no-v1"], + "source_violations": [], + "target_violations": target_violations, + } + if target_violations: + metadata["failure_categories"] = [INVARIANT_VIOLATION] + return _ScoredCell( + prompt_id="p", + example_id="e1", + evaluator_name="payments", + kind="trace_invariants", + record=EvalRecord( + run_id="r1", + prompt_id="p", + example_id="e1", + evaluator_name="payments", + kind="trace_invariants", + source_score=1.0, + target_score=target_score, + delta=target_score - 1.0, + metadata=metadata, + ), + sample_index=index, + ) + + broken = { + "rule_id": "no-v1", + "rule_type": "forbidden", + "tool": "refund_v1", + "round_index": 0, + "detail": "called forbidden tool 'refund_v1'", + } + (reduced,) = _reduce_sample_cells([cell(0, []), cell(1, []), cell(2, [broken])]) + assert reduced.record is not None + assert reduced.record.target_score < 1.0 + assert reduced.record.metadata["target_violations"] == [{**broken, "sample": 2}] + assert INVARIANT_VIOLATION in reduced.record.metadata["failure_categories"] + + async def test_other_families_reduced_metadata_gains_no_violation_keys(self) -> None: + cells = await _cells({"ex1": ["0.2", "0.4", "0.9"]}) + (cell,) = cells + assert cell.record is not None + assert set(cell.record.metadata) == {"raw", "samples"} + + +# --------------------------------------------------------------------------- +# trace_invariants that checked nothing -- an applies_to typo must not be silent +# --------------------------------------------------------------------------- + + +def _write_invariants_config( + tmp_path: Path, + *, + applies_to: list[str], + traces: str = "replayed", + with_agent_trace: bool = False, +) -> Path: + agent_trace_block = ( + """ + agent_trace: + - name: trace_safety + dangerous_tools: [issue_refund]""" + if with_agent_trace + else "" + ) + cfg_yaml = f""" + version: 1 + prompts: + - id: greet + detection: manual + content: "Hi {{name}}" + variables: [name] + defaults: + source_model: gemini-2.5-flash + target_model: gemini-2.5-pro + evaluators: + trace_invariants: + - name: payments_contract + applies_to: {json.dumps(applies_to)} + traces: {traces} + rules: + - id: no-v1 + type: forbidden + tools: [refund_v1]{agent_trace_block} + """ + path = tmp_path / "evalshift.yaml" + path.write_text(cfg_yaml, encoding="utf-8") + return path + + +def _evaluate_output(tmp_path: Path, *, quiet: bool = False) -> str: + from evalshift_cli.cli.commands.evaluate import run_evaluate + + console = Console(record=True, width=400) + run_evaluate( + run_id="r_20260601_aaaaaa", + config_path=tmp_path / "evalshift.yaml", + runs_base=tmp_path / ".evalshift" / "runs", + console=console, + quiet=quiet, + ) + return console.export_text() + + +def _write_greet_traces(tmp_path: Path) -> None: + run_dir = tmp_path / ".evalshift" / "runs" / "r_20260601_aaaaaa" + lines = [ + _imported_trace_line("greet", role, tool).replace('"e1"', f'"{example_id}"') + for example_id in ("ex1", "ex2") + for role, tool in (("source", "lookup"), ("target", "lookup")) + ] + (run_dir / TRACES_FILENAME).write_text("\n".join(lines) + "\n", encoding="utf-8") + + +class TestTraceInvariantsThatCheckedNothing: + def test_a_replayed_evaluator_matching_no_prompt_warns(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["checkout-*"]) + _scaffold_run(tmp_path) + + output = _evaluate_output(tmp_path) + + assert "payments_contract" in output + assert "checked nothing" in output + assert "['checkout-*']" in output + + def test_an_imported_evaluator_matching_no_trace_pair_warns(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["pay*"], traces="imported") + _scaffold_run(tmp_path) + _write_greet_traces(tmp_path) + + output = _evaluate_output(tmp_path) + + assert "payments_contract" in output + assert "checked nothing" in output + assert "['pay*']" in output + assert "imported trace pair" in output + + def test_partial_scoping_stays_silent(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["gre*", "checkout-*"]) + _scaffold_run(tmp_path) + + assert "checked nothing" not in _evaluate_output(tmp_path) + state = read_state(tmp_path / ".evalshift" / "runs" / "r_20260601_aaaaaa") + assert [(c.evaluator_name, c.attempted) for c in state.evaluator_coverage] == [ + ("payments_contract", 2) + ] + + def test_an_imported_evaluator_that_scored_stays_silent(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["gre*"], traces="imported") + _scaffold_run(tmp_path) + _write_greet_traces(tmp_path) + + assert "checked nothing" not in _evaluate_output(tmp_path) + scores = tmp_path / ".evalshift" / "runs" / "r_20260601_aaaaaa" / SCORES_FILENAME + assert len(scores.read_text(encoding="utf-8").splitlines()) == 2 + + def test_quiet_suppresses_the_warning(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["checkout-*"]) + _scaffold_run(tmp_path) + + assert "checked nothing" not in _evaluate_output(tmp_path, quiet=True) + + def test_the_cli_shows_the_warning( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _write_invariants_config(tmp_path, applies_to=["checkout-*"]) + run_id = _scaffold_run(tmp_path) + monkeypatch.chdir(tmp_path) + + result = runner.invoke(app, ["evaluate", run_id]) + + assert result.exit_code == 0 + assert "checked nothing" in " ".join(result.stdout.split()) + + +class TestMissingImportedTracesNamesTheFamily: + def test_imported_trace_invariants_alone_is_named( + self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path + ) -> None: + _write_invariants_config(tmp_path, applies_to=["*"], traces="imported") + run_id = _scaffold_run(tmp_path) + monkeypatch.chdir(tmp_path) + + result = runner.invoke(app, ["evaluate", run_id]) + + assert result.exit_code == 1 + stdout = " ".join(result.stdout.split()) + assert "trace_invariants (traces: imported) evaluators require imported traces" in stdout + assert "agent_trace" not in stdout + + def test_both_families_are_named(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + _write_invariants_config( + tmp_path, applies_to=["*"], traces="imported", with_agent_trace=True + ) + run_id = _scaffold_run(tmp_path) + monkeypatch.chdir(tmp_path) + + result = runner.invoke(app, ["evaluate", run_id]) + + assert result.exit_code == 1 + stdout = " ".join(result.stdout.split()) + assert ( + "agent_trace and trace_invariants (traces: imported) evaluators require imported traces" + ) in stdout diff --git a/tests/unit/test_init.py b/tests/unit/test_init.py index 9ac113ab..9eb7c02e 100644 --- a/tests/unit/test_init.py +++ b/tests/unit/test_init.py @@ -17,6 +17,7 @@ from typer.testing import CliRunner import evalshift_cli +from evalshift_cli.analysis.policy import BUDGET_LABELS from evalshift_cli.cli.commands._agents import ( AGENT_INSTRUCTIONS_FILENAME, DEFAULT_AGENT_CONTEXT_FILE, @@ -176,11 +177,25 @@ def test_default_profile_budgets_leave_room_for_a_first_migration(self, in_tmp: "min_equivalence_rate: 0.75", "max_tool_argument_drift: 0.20", "max_tool_divergence: 0.20", + "max_invariant_violations: 0", "max_cost_increase: 0.30", "max_latency_increase: 0.30", ): assert f" {line}" in body + @pytest.mark.parametrize("profile", sorted(INIT_PROFILE_POLICIES)) + def test_every_profile_scaffolds_every_budget(self, profile: str) -> None: + """A budget a profile leaves out is one its reader never learns exists. + + ``BUDGET_LABELS`` names every budget the policy emits a row for, the + CLI-only ones (``max_tool_divergence``, ``max_invariant_violations``) + included. ``tool_argument_drift_floor`` and ``fail_on_dropped_params`` + are not budgets, so they are absent from both by design. + """ + block = INIT_PROFILE_POLICIES[profile] + missing = [name for name in BUDGET_LABELS if f"\n {name}: " not in block] + assert not missing, f"profile {profile!r} omits budget(s) {missing}" + def test_prints_capture_first_next_steps(self, in_tmp: Path) -> None: result = runner.invoke(app, ["init"]) assert "evalshift capture sync" in result.stdout @@ -207,7 +222,7 @@ def _policy_pairs() -> list[str]: def test_every_budget_is_parsed(self) -> None: """Guard the parser itself: an empty list would pass every doc check.""" pairs = self._policy_pairs() - assert len(pairs) == 7, pairs + assert len(pairs) == 8, pairs assert all(pair.count(": ") == 1 for pair in pairs), pairs @pytest.mark.parametrize("doc_name", POLICY_DOC_FILENAMES) diff --git a/tests/unit/test_insight_facts.py b/tests/unit/test_insight_facts.py index 8862d23c..db4afe66 100644 --- a/tests/unit/test_insight_facts.py +++ b/tests/unit/test_insight_facts.py @@ -8,7 +8,7 @@ from typing import Any -from evalshift_cli.analysis.policy import BlockingRegression, FailureCategoryCount +from evalshift_cli.analysis.policy import BlockingRegression, BudgetResult, FailureCategoryCount from evalshift_cli.insights.facts import build_facts from evalshift_cli.insights.templates import fallback_insight from evalshift_cli.reports.economics import PromptEconomics @@ -153,6 +153,30 @@ def test_budget_limits_and_failure_categories_are_carried( assert "15" in facts.allowed_numbers +def test_the_trace_rule_budget_limit_is_a_count_not_a_rate( + sample_run: dict[str, Any], +) -> None: + """``max_invariant_violations`` counts examples, so its limit renders bare. + + Rendered as a rate, an allowance of three violations would read "300%". + """ + sample_run["decision"] = decision( + budget_results=[ + *budgets(), + BudgetResult( + name="max_invariant_violations", + observed=1.0, + allowed=3.0, + passed=True, + conclusive=True, + denominator=4, + ), + ], + ) + facts = build_facts(**sample_run) + assert facts.budget_limits["max_invariant_violations"] == "3" + + def test_regression_samples_are_capped_and_truncated(sample_run: dict[str, Any]) -> None: facts = build_facts(**sample_run) assert len(facts.regression_samples) <= 8 diff --git a/tests/unit/test_invariants.py b/tests/unit/test_invariants.py new file mode 100644 index 00000000..239b4e66 --- /dev/null +++ b/tests/unit/test_invariants.py @@ -0,0 +1,218 @@ +"""Tests for the pure trace-rule checker.""" + +from __future__ import annotations + +from typing import Any, ClassVar + +import pytest +from pydantic import TypeAdapter + +from evalshift_cli.config.models import InvariantRule +from evalshift_cli.evaluators.invariants import ( + ObservedCall, + Response, + check_rules, + side_score, +) + +_RULE = TypeAdapter(InvariantRule) + + +def _rule(**payload: Any) -> InvariantRule: + return _RULE.validate_python(payload) + + +def _response( + *calls: str | tuple[str, dict[str, Any]], + context: tuple[str, ...] = (), + round_index: int = 0, +) -> Response: + observed = tuple( + ObservedCall(name=c, arguments={}) if isinstance(c, str) else ObservedCall(*c) + for c in calls + ) + return Response(round_index=round_index, context=context, calls=observed) + + +class TestForbidden: + def test_each_forbidden_call_is_one_violation(self) -> None: + rule = _rule(id="no-v1", type="forbidden", tools=["refund_v1"]) + violations = check_rules([rule], [_response("lookup", "refund_v1", "refund_v1")]) + assert [v.rule_id for v in violations] == ["no-v1", "no-v1"] + assert violations[0].tool == "refund_v1" + + def test_a_forbidden_tool_in_context_is_not_the_models_violation(self) -> None: + rule = _rule(id="no-v1", type="forbidden", tools=["refund_v1"]) + assert check_rules([rule], [_response("lookup", context=("refund_v1",))]) == [] + + +class TestRequired: + def test_missing_tool_is_one_violation_per_tool(self) -> None: + rule = _rule(id="must", type="required", tools=["book", "confirm"]) + violations = check_rules([rule], [_response("book")]) + assert [(v.rule_id, v.tool) for v in violations] == [("must", "confirm")] + + def test_a_call_in_any_round_satisfies_it(self) -> None: + rule = _rule(id="must", type="required", tools=["book"]) + responses = [_response("search"), _response("book", round_index=1)] + assert check_rules([rule], responses) == [] + + def test_a_call_only_in_context_does_not_satisfy_it(self) -> None: + rule = _rule(id="must", type="required", tools=["book"]) + assert len(check_rules([rule], [_response("search", context=("book",))])) == 1 + + +class TestOrder: + _RULE_ARGS: ClassVar[dict[str, Any]] = { + "id": "auth-first", + "type": "order", + "before": "authenticate", + "after": "charge", + } + + def test_after_without_before_is_a_violation(self) -> None: + violations = check_rules([_rule(**self._RULE_ARGS)], [_response("charge")]) + assert [v.tool for v in violations] == ["charge"] + + def test_before_earlier_in_the_same_response_satisfies_it(self) -> None: + assert check_rules([_rule(**self._RULE_ARGS)], [_response("authenticate", "charge")]) == [] + + def test_before_later_in_the_same_response_does_not(self) -> None: + violations = check_rules([_rule(**self._RULE_ARGS)], [_response("charge", "authenticate")]) + assert len(violations) == 1 + + def test_before_in_context_satisfies_it(self) -> None: + responses = [_response("charge", context=("authenticate",))] + assert check_rules([_rule(**self._RULE_ARGS)], responses) == [] + + def test_a_trace_that_never_calls_after_is_vacuously_fine(self) -> None: + assert check_rules([_rule(**self._RULE_ARGS)], [_response("lookup")]) == [] + + def test_the_models_own_earlier_round_is_not_context(self) -> None: + """Teacher-forced: round 1 never saw round 0's own calls, only the recorded ones.""" + responses = [_response("authenticate"), _response("charge", round_index=1)] + violations = check_rules([_rule(**self._RULE_ARGS)], responses) + assert [v.round_index for v in violations] == [1] + + +class TestCallCount: + _MAX_ONE: ClassVar[dict[str, Any]] = { + "id": "one", + "type": "call_count", + "tool": "charge", + "max_calls": 1, + } + _MIN_ONE: ClassVar[dict[str, Any]] = { + "id": "some", + "type": "call_count", + "tool": "charge", + "min_calls": 1, + } + + def test_the_call_past_the_maximum_is_the_violation(self) -> None: + violations = check_rules([_rule(**self._MAX_ONE)], [_response("charge", "charge")]) + assert len(violations) == 1 + assert "call #2" in violations[0].detail + + def test_context_calls_count_toward_the_maximum(self) -> None: + responses = [_response("charge", context=("charge",))] + assert len(check_rules([_rule(**self._MAX_ONE)], responses)) == 1 + + def test_context_alone_over_the_maximum_is_not_the_models_violation(self) -> None: + responses = [_response("lookup", context=("charge", "charge"))] + assert check_rules([_rule(**self._MAX_ONE)], responses) == [] + + def test_each_round_counts_from_its_own_context(self) -> None: + """Round 0's own call is an alternative never fed back to round 1.""" + responses = [_response("charge"), _response("charge", round_index=1)] + assert check_rules([_rule(**self._MAX_ONE)], responses) == [] + + def test_max_zero_forbids_the_tool(self) -> None: + rule = _rule(id="none", type="call_count", tool="charge", max_calls=0) + assert len(check_rules([rule], [_response("charge")])) == 1 + + def test_fewer_than_the_minimum_is_one_violation(self) -> None: + rule = _rule(id="two", type="call_count", tool="charge", min_calls=2) + violations = check_rules([rule], [_response("charge")]) + assert len(violations) == 1 + assert "1 call(s)" in violations[0].detail + assert "at least 2" in violations[0].detail + + def test_context_calls_count_toward_the_minimum(self) -> None: + """Unlike ``required``: the conversation already made the call.""" + responses = [_response("lookup", context=("charge",))] + assert check_rules([_rule(**self._MIN_ONE)], responses) == [] + + def test_the_minimum_is_judged_on_the_last_response(self) -> None: + """The model's own round 0 was never fed back; the recorded round was.""" + own_round_zero_only = [ + _response("charge"), + _response("lookup", context=("authenticate",), round_index=1), + ] + violations = check_rules([_rule(**self._MIN_ONE)], own_round_zero_only) + assert [v.round_index for v in violations] == [1] + recorded = [_response("lookup"), _response("lookup", context=("charge",), round_index=1)] + assert check_rules([_rule(**self._MIN_ONE)], recorded) == [] + + def test_exact_is_both_bounds(self) -> None: + rule = _rule(id="exactly-one", type="call_count", tool="charge", min_calls=1, max_calls=1) + assert check_rules([rule], [_response("charge")]) == [] + assert len(check_rules([rule], [_response()])) == 1 + assert len(check_rules([rule], [_response("charge", "charge")])) == 1 + + +class TestArguments: + _SCHEMA: ClassVar[dict[str, Any]] = { + "type": "object", + "required": ["amount"], + "properties": {"amount": {"type": "number", "minimum": 0.01, "maximum": 5000}}, + } + + def _rule(self) -> InvariantRule: + return _rule(id="sane", type="arguments", tool="charge", json_schema=self._SCHEMA) + + def test_valid_arguments_pass(self) -> None: + assert check_rules([self._rule()], [_response(("charge", {"amount": 12.5}))]) == [] + + def test_out_of_range_names_the_field(self) -> None: + violations = check_rules([self._rule()], [_response(("charge", {"amount": 9000}))]) + assert len(violations) == 1 + assert "amount" in violations[0].detail + assert "5000" in violations[0].detail + + def test_missing_required_field(self) -> None: + violations = check_rules([self._rule()], [_response(("charge", {}))]) + assert "amount" in violations[0].detail + + def test_unparseable_arguments_fail_a_schema_that_requires_a_field(self) -> None: + violations = check_rules([self._rule()], [_response(("charge", {"_parse_error": True}))]) + assert len(violations) == 1 + + def test_other_tools_are_not_checked(self) -> None: + assert check_rules([self._rule()], [_response(("refund", {"amount": -1}))]) == [] + + +class TestSideScore: + def test_share_of_rules_never_broken(self) -> None: + rules = [ + _rule(id="a", type="forbidden", tools=["x"]), + _rule(id="b", type="forbidden", tools=["y"]), + ] + violations = check_rules(rules, [_response("x", "x")]) + assert side_score(rules, violations) == pytest.approx(0.5) + + def test_clean_is_one(self) -> None: + rules = [_rule(id="a", type="forbidden", tools=["x"])] + assert side_score(rules, []) == 1.0 + + +def test_violation_to_dict_is_json_ready() -> None: + rule = _rule(id="no-v1", type="forbidden", tools=["refund_v1"]) + (violation,) = check_rules([rule], [_response("refund_v1", round_index=2)]) + assert violation.to_dict() == { + "rule_id": "no-v1", + "rule_type": "forbidden", + "tool": "refund_v1", + "round_index": 2, + "detail": "called forbidden tool 'refund_v1'", + } diff --git a/tests/unit/test_policy.py b/tests/unit/test_policy.py index 5f085b18..aec1f397 100644 --- a/tests/unit/test_policy.py +++ b/tests/unit/test_policy.py @@ -22,7 +22,11 @@ ) from evalshift_cli.config.models import MigrationPolicy, SliceMigrationPolicy from evalshift_cli.evaluators.base import EvalRecord -from evalshift_cli.evaluators.failures import TOOL_GROUND_TRUTH_MISS, TOOL_SELECTION_DRIFT +from evalshift_cli.evaluators.failures import ( + INVARIANT_VIOLATION, + TOOL_GROUND_TRUTH_MISS, + TOOL_SELECTION_DRIFT, +) from evalshift_cli.evaluators.tool_arguments import KIND as KIND_ARGUMENTS from evalshift_cli.evaluators.tool_selection import KIND_CONFORMANCE, KIND_DIVERGENCE from evalshift_cli.runner.models import Call @@ -569,6 +573,7 @@ def test_zero_blocking_records_recommendation_says_enable_blocking(self) -> None "min_equivalence_rate", "max_tool_argument_drift", "max_tool_divergence", + "max_invariant_violations", "tool_argument_drift_floor", "max_cost_increase", "max_latency_increase", @@ -2745,3 +2750,228 @@ def test_a_partially_diverged_row_is_counted(self) -> None: ] assert budget.denominator == 2 assert budget.observed == pytest.approx(0.5) + + +# --------------------------------------------------------------------------- +# Hand-written trace rules: the ``max_invariant_violations`` budget +# --------------------------------------------------------------------------- + + +def _invariant_record( + example_id: str, + source_score: float, + target_score: float, + *, + blocking: bool = True, + rule_id: str = "no-v1", +) -> EvalRecord: + def broken(score: float) -> list[dict[str, Any]]: + return ( + [] + if score == 1.0 + else [ + { + "rule_id": rule_id, + "rule_type": "forbidden", + "tool": "refund_v1", + "round_index": 0, + "detail": "x", + } + ] + ) + + metadata: dict[str, Any] = { + "rules_checked": [rule_id], + "source_violations": broken(source_score), + "target_violations": broken(target_score), + } + if target_score < 1.0: + metadata["failure_categories"] = [INVARIANT_VIOLATION] + return EvalRecord( + run_id="r", + prompt_id="p", + example_id=example_id, + evaluator_name="payments", + kind="trace_invariants", + source_score=source_score, + target_score=target_score, + delta=target_score - source_score, + metadata=metadata, + blocking=blocking, + ) + + +def _insufficient(evaluator_name: str = "payments") -> ComparisonResult: + """One comparison too small for a paired test -- what n < 5 produces.""" + return _comparison(evaluator_name=evaluator_name, severity="insufficient") + + +class TestInvariantViolationBudget: + def test_counts_examples_whose_target_broke_a_rule(self) -> None: + decision = _decide( + [ + _invariant_record("e1", 1.0, 0.0), + _invariant_record("e2", 1.0, 1.0), + _invariant_record("e3", 1.0, 0.5), + ], + policy=MigrationPolicy(max_invariant_violations=5), + ) + budget = _budgets(decision)["max_invariant_violations"] + assert (budget.observed, budget.allowed, budget.denominator) == (2.0, 5.0, 3) + assert budget.passed and budget.conclusive + + def test_a_violation_the_source_shares_still_counts(self) -> None: + decision = _decide([_invariant_record("e1", 0.0, 0.0)]) + assert _budgets(decision)["max_invariant_violations"].observed == 1.0 + assert decision.verdict == "fail" + + def test_a_broken_rule_fails_a_suite_too_small_for_statistics(self) -> None: + """Review Focus 3: every comparison insufficient must not soften a broken rule.""" + decision = _decide( + [_invariant_record("e1", 1.0, 0.0)], + comparisons=[_insufficient()], + ) + assert decision.verdict == "fail" + assert decision.reason is not None + assert "max_invariant_violations" in decision.reason + + def test_no_invariant_rows_emits_no_budget_row(self) -> None: + """Review Focus 4: projects without trace rules see no new row.""" + decision = _decide([_divergence_record("e1", 1.0)]) + assert "max_invariant_violations" not in _budgets(decision) + + def test_advisory_rules_never_gate(self) -> None: + decision = _decide([_invariant_record("e1", 1.0, 0.0, blocking=False)]) + assert "max_invariant_violations" not in _budgets(decision) + assert decision.verdict != "fail" + + def test_selected_by_kind_not_name(self) -> None: + record = _invariant_record("e1", 1.0, 0.0).model_copy(update={"evaluator_name": "anything"}) + assert _budgets(_decide([record]))["max_invariant_violations"].observed == 1.0 + + def test_slice_override_applies(self) -> None: + records = [_invariant_record("e1", 1.0, 0.0)] + comparisons = [ + _comparison(evaluator_name="payments", slice_name="checkout", severity="none"), + _comparison(evaluator_name="payments", slice_name="all", severity="none"), + ] + decision = _decide( + records, + policy=MigrationPolicy( + max_invariant_violations=0, + slices={"checkout": SliceMigrationPolicy(max_invariant_violations=3)}, + ), + comparisons=comparisons, + ) + checkout = {b.name: b for b in decision.slices["checkout"].budget_results} + assert checkout["max_invariant_violations"].passed + assert decision.verdict == "fail" # the overall budget of 0 still breaches + + def test_shared_violation_recommends_reviewing_the_rule(self) -> None: + decision = _decide([_invariant_record("e1", 0.0, 0.0)]) + assert any("no-v1" in line and "owner" in line for line in decision.recommendations) + + def test_the_budget_has_a_label_and_meaning(self) -> None: + from evalshift_cli.analysis.policy import BUDGET_LABELS, BUDGET_MEANINGS + + assert BUDGET_LABELS["max_invariant_violations"] == "Trace-rule violations" + assert "trace rule" in BUDGET_MEANINGS["max_invariant_violations"] + + def test_a_slice_that_broke_a_rule_fails_however_small(self) -> None: + """Review Focus 3, one scope down: an insufficient slice must not soften a break. + + The slice verdict ships in the bundle and is what ``_recommendations`` + reads to name the unsafe slices, so ``inconclusive`` here would turn + "keep checkout on the source model" into "do not migrate globally". + """ + records = [ + _invariant_record("e1", 1.0, 0.0), + *[_record(example_id=f"s{i}", delta=0.0) for i in range(6)], + ] + comparisons = [ + _insufficient(), + _comparison(severity="insufficient", evaluator_name="payments", slice_name="checkout"), + _comparison(severity="none", slice_name="all", delta_avg_score=0.0), + _comparison(severity="none", slice_name="support", delta_avg_score=0.0), + ] + decision = _decide( + records, + policy=MigrationPolicy( + max_invariant_violations=1, + slices={"checkout": SliceMigrationPolicy(max_invariant_violations=0)}, + ), + comparisons=comparisons, + ) + checkout = decision.slices["checkout"] + breach = {b.name: b for b in checkout.budget_results}["max_invariant_violations"] + assert not breach.passed and breach.conclusive + assert checkout.verdict == "fail" + assert decision.slices["support"].verdict == "pass" + assert decision.verdict == "fail" + assert "Safe slices: support. Keep checkout on the source model." in ( + decision.recommendations + ) + + def test_counts_distinct_examples_not_rows_across_rule_sets(self) -> None: + """Two ``trace_invariants`` entries write two rows per example. + + One example breaking both must count once, over a denominator of the + suite's two examples -- the budget and every doc page promise a count + of examples, and rows would double it. + """ + records = [ + _invariant_record("e1", 1.0, 0.0), + _invariant_record("e1", 1.0, 0.0, rule_id="no-v2").model_copy( + update={"evaluator_name": "refunds"} + ), + _invariant_record("e2", 1.0, 1.0), + _invariant_record("e2", 1.0, 1.0, rule_id="no-v2").model_copy( + update={"evaluator_name": "refunds"} + ), + ] + decision = _decide(records, policy=MigrationPolicy(max_invariant_violations=1)) + budget = _budgets(decision)["max_invariant_violations"] + assert (budget.observed, budget.denominator) == (1.0, 2) + assert budget.passed + + breached = _decide(records, policy=MigrationPolicy(max_invariant_violations=0)) + assert breached.reason is not None + assert "on 1 of 2 examples" in breached.reason + + def test_the_same_example_id_under_two_prompts_is_two_examples(self) -> None: + records = [ + _invariant_record("e1", 1.0, 0.0), + _invariant_record("e1", 1.0, 0.0).model_copy(update={"prompt_id": "q"}), + ] + budget = _budgets(_decide(records))["max_invariant_violations"] + assert (budget.observed, budget.denominator) == (2.0, 2) + + def test_shared_note_counts_distinct_examples(self) -> None: + records = [ + _invariant_record("e1", 0.0, 0.0), + _invariant_record("e1", 0.0, 0.0, rule_id="no-v2").model_copy( + update={"evaluator_name": "refunds"} + ), + ] + notes = [line for line in _decide(records).recommendations if "owner" in line] + assert len(notes) == 1 + assert "no-v1, no-v2 on 1 example(s)" in notes[0] + + def test_a_breached_budget_overrides_the_failed_slice_demotion(self) -> None: + """No overall comparison plus a failed slice demotes to conditional_pass. + + The run-level override must still turn a breached overall + trace-rule budget into ``fail``: a broken rule is not softened by + the slice rules. + """ + records = [ + _invariant_record("e1", 1.0, 0.0), + *[_record(example_id=f"s{i}", delta=0.0) for i in range(6)], + ] + comparisons = [_comparison(severity="critical", slice_name="checkout")] + decision = _decide(records, comparisons=comparisons) + assert decision.slices["checkout"].verdict == "fail" + assert not _budgets(decision)["max_invariant_violations"].passed + assert decision.verdict == "fail" + assert decision.reason is not None + assert "max_invariant_violations" in decision.reason diff --git a/tests/unit/test_reports.py b/tests/unit/test_reports.py index 6bd2525b..18fbf766 100644 --- a/tests/unit/test_reports.py +++ b/tests/unit/test_reports.py @@ -3,8 +3,10 @@ from __future__ import annotations import json +import re from datetime import UTC, datetime from pathlib import Path +from typing import Any import pytest from typer.testing import CliRunner @@ -800,6 +802,33 @@ def test_renders_migration_verdict_when_present(self, tmp_path: Path) -> None: assert "≤ 0" in html # count budget stays an integer, no percent assert "0.030" not in html # no bare fractions anymore + def test_a_failed_trace_rule_budget_renders_as_a_count(self, tmp_path: Path) -> None: + """``max_invariant_violations`` is a count of examples, never a percentage.""" + cwd, run_id = _scaffold_full_run(tmp_path) + run_dir = cwd / ".evalshift" / "runs" / run_id + _write_migration_decision(run_dir, run_id) + path = run_dir / "migration_decision.json" + decision = json.loads(path.read_text(encoding="utf-8")) + decision["budget_results"] = [ + { + "name": "max_invariant_violations", + "observed": 2.0, + "allowed": 1.0, + "passed": False, + "scope": "overall", + "conclusive": True, + "denominator": 2, + }, + ] + path.write_text(json.dumps(decision), encoding="utf-8") + + html = render_html(build_report_payload(run_dir)) + + assert "Trace-rule violations" in html + assert '2' in html # observed: two examples + assert "≤ 1<" in html # allowed: one example + assert "200.0%" not in html + def test_sub_granular_budget_warning_reaches_the_report(self, tmp_path: Path) -> None: # The report describes the *persisted* decision, so the warning has to # survive `to_dict()` → migration_decision.json → payload → HTML. A @@ -2457,3 +2486,132 @@ def call(role: str, example_id: str, latency: int, cached_rounds: int = 0) -> Ca assert by_id["live"].delta_latency_ms == 50 assert not by_id["mixed"].latency_comparable assert by_id["mixed"].delta_latency_ms == 0 + + +def _run_dir_with_scores(tmp_path: Path, records: list[dict[str, Any]]) -> Path: + """A completed run (``_scaffold_full_run``) plus extra ``scores.jsonl`` rows.""" + cwd, run_id = _scaffold_full_run(tmp_path) + run_dir = cwd / ".evalshift" / "runs" / run_id + with (run_dir / SCORES_FILENAME).open("a", encoding="utf-8") as fh: + for record in records: + fh.write(EvalRecord.model_validate(record).model_dump_json() + "\n") + return run_dir + + +class TestInvariantViolationsPanel: + def _record(self, example_id: str, *, source_broke: bool) -> dict[str, Any]: + violation = { + "rule_id": "no-v1", + "rule_type": "forbidden", + "tool": "refund_v1", + "round_index": 0, + "detail": "called forbidden tool 'refund_v1'", + } + return { + "run_id": "r1", + "prompt_id": "p", + "example_id": example_id, + "evaluator_name": "payments", + "kind": "trace_invariants", + "source_score": 0.0 if source_broke else 1.0, + "target_score": 0.0, + "delta": 0.0 if source_broke else -1.0, + "metadata": { + "owner": "@payments", + "rules_checked": ["no-v1"], + "source_violations": [violation] if source_broke else [], + "target_violations": [violation], + "failure_categories": ["INVARIANT_VIOLATION"], + }, + } + + def test_payload_lists_every_target_violation_even_at_zero_delta(self, tmp_path: Path) -> None: + run_dir = _run_dir_with_scores( + tmp_path, + [self._record("e1", source_broke=True), self._record("e2", source_broke=False)], + ) + report = build_report_payload(run_dir) + rows = [ + (v.example_id, v.rule_id, v.source_also, v.owner) for v in report.invariant_violations + ] + assert rows == [("e1", "no-v1", True, "@payments"), ("e2", "no-v1", False, "@payments")] + + def test_html_renders_the_panel_with_owner_and_detail(self, tmp_path: Path) -> None: + run_dir = _run_dir_with_scores(tmp_path, [self._record("e1", source_broke=False)]) + html = render_html(build_report_payload(run_dir)) + assert "Trace rules broken" in html + assert "no-v1" in html and "@payments" in html + assert "called forbidden tool" in html + + def test_no_panel_without_trace_rules(self, tmp_path: Path) -> None: + run_dir = _run_dir_with_scores(tmp_path, []) + assert "Trace rules broken" not in render_html(build_report_payload(run_dir)) + + def test_report_json_carries_the_violations(self, tmp_path: Path) -> None: + run_dir = _run_dir_with_scores(tmp_path, [self._record("e1", source_broke=False)]) + write_report_json(build_report_payload(run_dir), run_dir) + data = json.loads((run_dir / REPORT_JSON_FILENAME).read_text(encoding="utf-8")) + (row,) = data["invariant_violations"] + assert row["rule_id"] == "no-v1" + assert row["owner"] == "@payments" + assert row["source_also"] is False + assert row["sample"] is None + + def test_report_json_key_is_empty_without_trace_rules(self, tmp_path: Path) -> None: + run_dir = _run_dir_with_scores(tmp_path, []) + write_report_json(build_report_payload(run_dir), run_dir) + data = json.loads((run_dir / REPORT_JSON_FILENAME).read_text(encoding="utf-8")) + assert data["invariant_violations"] == [] + + def test_a_reduced_row_lists_each_samples_violation_once(self, tmp_path: Path) -> None: + # samples_per_example > 1: evaluate merges every scored sample's + # violations onto the reduced row, each tagged with its ``sample``. + record = self._record("e1", source_broke=False) + (violation,) = record["metadata"]["target_violations"] + record["metadata"]["target_violations"] = [ + {**violation, "sample": 2}, + {**violation, "sample": 0}, + ] + run_dir = _run_dir_with_scores(tmp_path, [record]) + report = build_report_payload(run_dir) + assert [(v.rule_id, v.sample) for v in report.invariant_violations] == [ + ("no-v1", 0), + ("no-v1", 2), + ] + html = render_html(report) + assert "sample 1" in html and "sample 3" in html + + def test_an_errored_row_lists_nothing(self, tmp_path: Path) -> None: + record = self._record("e1", source_broke=False) + record["error"] = "trace missing" + run_dir = _run_dir_with_scores(tmp_path, [record]) + assert build_report_payload(run_dir).invariant_violations == [] + + def test_rows_carry_the_records_blocking_flag(self, tmp_path: Path) -> None: + advisory = {**self._record("e2", source_broke=False), "blocking": False} + run_dir = _run_dir_with_scores(tmp_path, [self._record("e1", source_broke=False), advisory]) + report = build_report_payload(run_dir) + assert [(v.example_id, v.blocking) for v in report.invariant_violations] == [ + ("e1", True), + ("e2", False), + ] + write_report_json(report, run_dir) + data = json.loads((run_dir / REPORT_JSON_FILENAME).read_text(encoding="utf-8")) + assert [row["blocking"] for row in data["invariant_violations"]] == [True, False] + + def test_html_tags_an_advisory_violation_without_bad_styling(self, tmp_path: Path) -> None: + advisory = {**self._record("e1", source_broke=False), "blocking": False} + html = render_html(build_report_payload(_run_dir_with_scores(tmp_path, [advisory]))) + tag = re.search(r']*>advisory', html) + assert tag is not None + assert "badge-bad" not in tag.group(1) + + def test_html_leaves_a_blocking_violation_untagged(self, tmp_path: Path) -> None: + run_dir = _run_dir_with_scores(tmp_path, [self._record("e1", source_broke=False)]) + html = render_html(build_report_payload(run_dir)) + assert re.search(r'class="[^"]*advisory-tag', html) is None + + def test_family_label_reads_as_words(self) -> None: + from evalshift_cli.reports.html import _kind_label + + assert _kind_label("trace_invariants") == "Trace rules" diff --git a/tests/unit/test_trace_invariants.py b/tests/unit/test_trace_invariants.py new file mode 100644 index 00000000..ca556b51 --- /dev/null +++ b/tests/unit/test_trace_invariants.py @@ -0,0 +1,200 @@ +"""Tests for the trace-invariants evaluators.""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from evalshift_cli.config.models import TraceInvariantsEvaluatorConfig +from evalshift_cli.evaluators.failures import CATEGORY_LABELS, INVARIANT_VIOLATION +from evalshift_cli.evaluators.tool_models import ToolCall, ToolTrace +from evalshift_cli.evaluators.trace_invariants import ( + KIND, + ImportedTraceInvariantsEvaluator, + TraceInvariantsEvaluator, + build_trace_invariants_evaluator, + replayed_responses, +) +from evalshift_cli.suite.models import ChatMessage +from evalshift_cli.traces.models import AgentTrace +from tests.unit.suite_examples import suite_example + +_RULES: list[dict[str, Any]] = [ + {"id": "auth-first", "type": "order", "before": "authenticate", "after": "charge"}, + {"id": "one-charge", "type": "call_count", "tool": "charge", "max_calls": 1}, +] + + +def _config(**overrides: Any) -> TraceInvariantsEvaluatorConfig: + return TraceInvariantsEvaluatorConfig.model_validate( + {"name": "payments", "rules": _RULES, **overrides}, + ) + + +def _trace(*calls: str | tuple[str, int], round_count: int = 1) -> ToolTrace: + """Calls as names, or (name, round_index) pairs for a multi-round replay.""" + built = [ + ToolCall( + tool_name=c if isinstance(c, str) else c[0], + arguments={}, + sequence_index=i, + round_index=0 if isinstance(c, str) else c[1], + ) + for i, c in enumerate(calls) + ] + return ToolTrace(calls=built, round_count=round_count) + + +async def _score( + source: ToolTrace, + target: ToolTrace, + *, + config: TraceInvariantsEvaluatorConfig | None = None, + **example: Any, +) -> Any: + evaluator = TraceInvariantsEvaluator(config or _config()) + return await evaluator.score_pair( + run_id="r", + prompt_id="checkout", + example=suite_example(id="ex1", **example), + source_trace=source, + target_trace=target, + ) + + +class TestRecordShape: + async def test_clean_pair_is_equivalent(self) -> None: + record = await _score(_trace("authenticate", "charge"), _trace("authenticate", "charge")) + assert record.kind == KIND + assert (record.source_score, record.target_score, record.delta) == (1.0, 1.0, 0.0) + assert record.metadata["target_violations"] == [] + assert "failure_categories" not in record.metadata + + async def test_target_violation_is_categorised_and_scored(self) -> None: + record = await _score(_trace("authenticate", "charge"), _trace("charge", "charge")) + assert record.target_score == pytest.approx(0.0) # both rules broken + assert record.delta == pytest.approx(-1.0) + assert record.metadata["failure_categories"] == [INVARIANT_VIOLATION] + rule_ids = {v["rule_id"] for v in record.metadata["target_violations"]} + assert rule_ids == {"auth-first", "one-charge"} + assert "auth-first" in record.explanation + + async def test_a_violation_the_source_shares_still_marks_the_target(self) -> None: + record = await _score(_trace("charge"), _trace("charge")) + assert record.delta == 0.0 + assert record.target_score < 1.0 + assert record.metadata["failure_categories"] == [INVARIANT_VIOLATION] + assert record.metadata["source_violations"] + + async def test_owner_and_rules_checked_ride_on_the_record(self) -> None: + record = await _score(_trace(), _trace(), config=_config(owner="@payments")) + assert record.metadata["owner"] == "@payments" + assert record.metadata["rules_checked"] == ["auth-first", "one-charge"] + + +class TestAppliesTo: + def test_globs_select_prompts(self) -> None: + evaluator = TraceInvariantsEvaluator(_config(applies_to=["check*"])) + assert evaluator.applies("checkout") + assert not evaluator.applies("support") + + def test_default_applies_everywhere(self) -> None: + assert TraceInvariantsEvaluator(_config()).applies("anything") + + +class TestTeacherForcedContext: + """Review Focus 2: round k's context is the *recorded* rounds, not the model's own.""" + + def _example(self) -> dict[str, Any]: + return { + "expected_tool_rounds": [ + [{"tool_name": "authenticate"}], + [{"tool_name": "charge"}], + ], + "tool_result_fixtures": [ + [{"tool_name": "authenticate", "result": "ok"}], + [{"tool_name": "charge", "result": "ok"}], + ], + } + + async def test_recorded_round_zero_satisfies_order_in_round_one(self) -> None: + # Target skipped auth in its own round 0 (called lookup) but round 1 was + # answered after the *recorded* authenticate. + target = _trace(("lookup", 0), ("charge", 1), round_count=3) + record = await _score(target, target, **self._example()) + assert record.metadata["target_violations"] == [] + + async def test_own_round_zero_alternative_does_not_count_toward_round_one(self) -> None: + target = _trace(("charge", 0), ("charge", 1), round_count=3) + record = await _score(target, target, **self._example()) + violations = record.metadata["target_violations"] + # Round 0: charge without auth (order). Round 1: context has the recorded + # authenticate only, so its single charge is the first -- no call_count break. + assert [(v["rule_id"], v["round_index"]) for v in violations] == [("auth-first", 0)] + + async def test_round_two_counts_the_recorded_charge(self) -> None: + target = _trace(("authenticate", 0), ("charge", 2), round_count=3) + record = await _score(target, target, **self._example()) + violations = record.metadata["target_violations"] + assert [(v["rule_id"], v["round_index"]) for v in violations] == [("one-charge", 2)] + + def test_history_tool_calls_are_context(self) -> None: + example = suite_example( + id="ex1", + history=[ + ChatMessage(role="user", content="pay"), + ChatMessage( + role="assistant", + tool_calls=[{"id": "c1", "name": "authenticate", "arguments": {}}], + ), + ChatMessage(role="tool", tool_call_id="c1", content="ok"), + ], + ) + (response,) = replayed_responses(_trace("charge"), example) + assert response.context == ("authenticate",) + + +class TestImported: + def _trace(self, role: str, *names: str) -> AgentTrace: + return AgentTrace.model_validate( + { + "run_id": "r", + "prompt_id": "checkout", + "example_id": "ex1", + "role": role, + "events": [ + { + "type": "tool_call", + "sequence_index": i, + "timestamp": "2026-10-02T12:00:00Z", + "name": name, + "arguments": {}, + } + for i, name in enumerate(names) + ], + }, + ) + + async def test_imported_traces_are_one_response_in_sequence_order(self) -> None: + evaluator = ImportedTraceInvariantsEvaluator(_config(traces="imported")) + record = await evaluator.score_trace_pair( + run_id="r", + source_trace=self._trace("source", "authenticate", "charge"), + target_trace=self._trace("target", "charge", "authenticate", "charge"), + ) + assert (record.prompt_id, record.example_id, record.kind) == ("checkout", "ex1", KIND) + rule_ids = sorted(v["rule_id"] for v in record.metadata["target_violations"]) + assert rule_ids == ["auth-first", "one-charge"] + + +def test_the_builder_routes_on_traces() -> None: + assert isinstance(build_trace_invariants_evaluator(_config()), TraceInvariantsEvaluator) + assert isinstance( + build_trace_invariants_evaluator(_config(traces="imported")), + ImportedTraceInvariantsEvaluator, + ) + + +def test_the_category_has_a_display_label() -> None: + assert CATEGORY_LABELS[INVARIANT_VIOLATION] == "Broke a hand-written trace rule"