From 6c24c19939860f8535bbbc26b656eeca36f8cc3e Mon Sep 17 00:00:00 2001 From: Lukas Babaliauskas Date: Thu, 8 Oct 2026 14:06:10 +0200 Subject: [PATCH] docs: cover trace invariants on the summary surfaces; fix advisory-tag scope and hosted-gate wording Cross-repo docs audit after #35/#36. The reference sections (DOCS.md Trace invariants, docs/configuration.md, llms-full.txt) were right; the summary surfaces around them were not updated: - Profile tables (DOCS.md, llms-full.txt) gain the trace-rule-violations column (0 in every profile). Report-layout lists (DOCS.md, llms-full.txt, README) gain the "Trace rules broken" panel and say the executive summary and the strip's mean score delta come from blocking evaluators. - README agent section and docs/agents.md ("four new evaluators", the top-level placement carve-out, "Reading a tool run's report") mention trace_invariants; the "tool evaluators are not written at top level" comments in DOCS.md and llms-full.txt carry the same carve-out. - Hosted wording (docs/hosted.md, DOCS.md, llms-full.txt, methodology.md, CHANGELOG): the server accepts and stores max_invariant_violations since 2026-10-08 and never re-evaluates it; a run that carries its own policy is answered with the CLI's decision, only policy-less runs get the six-budget re-check. The stale "waits on that server deploy" sentence is gone. - Inconclusive FAQs (docs/faq.md, DOCS.md, llms-full.txt): a breached max_invariant_violations is never inconclusive. Resume FAQs: a CLI upgrade that added config fields moves config_hash. - Corrections: the "Overall, by evaluator" advisory tag applies to every blocking: false evaluator, not only trace-rule entries (CHANGELOG, DOCS.md, llms-full.txt, configuration.md); call_count has min_calls/max_calls, not an "exact" key (CHANGELOG, evaluators.md); "they all run on every pair" and the blocking qualifier in evaluators.md/traces.md; DOCS.md applies_to and the four other families' applies_to rows in configuration.md now say it is accepted but enforced only by trace_invariants; round numbering in llms-full.txt uses the replay's 1-based convention; CHANGELOG mentions the analyze recommendation for source-shared breaks. - push.py docstring: six of ten budgets, other four. - Re-vendored schemas/bundle_manifest.schema.json from evalshift-server PR #30 (one description string). Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 16 +++++--- DOCS.md | 29 +++++++------- README.md | 11 +++++- docs/agents.md | 16 ++++++-- docs/configuration.md | 15 ++++---- docs/evaluators.md | 15 +++++--- docs/faq.md | 9 ++++- docs/hosted.md | 2 +- docs/methodology.md | 9 +++-- docs/traces.md | 5 ++- llms-full.txt | 38 ++++++++++++------- .../hosted/bundle_manifest.schema.json | 2 +- src/evalshift_cli/hosted/push.py | 2 +- 13 files changed, 108 insertions(+), 61 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4091f91..3bd1a43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - `evaluators.trace_invariants`: hand-written rules over tool-call traces — - `forbidden`, `required`, `order`, `call_count` (min/max/exact per tool) and + `forbidden`, `required`, `order`, `call_count` (`min_calls` and/or + `max_calls` per tool; equal bounds = an exact count) and `arguments` (JSON Schema) — checked on both sides of every pair, on replayed traces or (`traces: imported`) on imported ones. A rule the target breaks counts whatever the source did. Its `applies_to` is enforced: a prompt @@ -23,16 +24,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 rule, so an example that broke rules in two entries counts once. A breach fails the run even when the suite is too small for statistics, and a slice whose own budget breaches reads `fail` however few comparisons it has. - Pushed bundles' `decision.policy` now always carries the field, so pushing - needs an evalshift-server that accepts `max_invariant_violations` in policy - snapshots; this CLI release waits on that server deploy. + `analyze` adds a recommendation when the source broke the same rules the + target did: review the rule with its owner rather than raising the budget. + Pushed bundles' `decision.policy` now always carries the field; hosted + EvalShift accepts and stores it since 2026-10-08 (it never re-evaluates it — + a run that carries its own policy is answered with the CLI's decision). - The HTML report lists every broken trace rule in a "Trace rules broken" panel, with its owner and whether the source broke it too — advisory (`blocking: false`) entries included, though only blocking ones count toward the budget; those rows are tagged "advisory". `report.json` carries the same rows under a new top-level `invariant_violations` key, each with - the entry's `blocking` flag. The "Overall, by evaluator" table tags such an - entry's row too and says what a trace-rule row measures, and the "Top + the entry's `blocking` flag. The "Overall, by evaluator" table tags every + advisory (`blocking: false`) evaluator's row the same way and says what a + trace-rule row measures, and the "Top regression causes" panel counts `INVARIANT_VIOLATION` as the budget does — distinct examples with a blocking target violation — so the narrative never quotes a figure the verdict did not use. diff --git a/DOCS.md b/DOCS.md index d82df50..619d086 100644 --- a/DOCS.md +++ b/DOCS.md @@ -137,13 +137,13 @@ See [Project setup](#project-setup) and [Capturing from production](#capturing-f - `--provider gemini|openai|anthropic|deepseek` — which provider's model ids the scaffold uses (prompted on a TTY; defaults to `gemini` otherwise). Gemini and OpenAI scaffolds include an embedding-based semantic evaluator; the Anthropic and DeepSeek scaffolds comment it out (no embedding endpoint). - `--profile` — pre-tuned migration-policy budgets: -| Profile | regression ≤ | critical ≤ | equivalence ≥ | arg drift ≤ | tool divergence ≤ | cost Δ ≤ | latency Δ ≤ | -|---|---|---|---|---|---|---|---| -| `model-upgrade` (default) | 30% | 1 | 75% | 20% | 20% | +30% | +30% | -| `cost-reduction` | 2% | 0 | 97% | 1% | 2% | +5% | +30% | -| `local-model` | 5% | 0 | 90% | 2% | 5% | +0% | +50% | -| `quantization` | 2% | 0 | 97% | 0.5% | 2% | +0% | +20% | -| `provider-switch` | 3% | 0 | 95% | 1% | 3% | +20% | +40% | +| Profile | regression ≤ | critical ≤ | equivalence ≥ | arg drift ≤ | tool divergence ≤ | trace-rule violations ≤ | cost Δ ≤ | latency Δ ≤ | +|---|---|---|---|---|---|---|---|---| +| `model-upgrade` (default) | 30% | 1 | 75% | 20% | 20% | 0 | +30% | +30% | +| `cost-reduction` | 2% | 0 | 97% | 1% | 2% | 0 | +5% | +30% | +| `local-model` | 5% | 0 | 90% | 2% | 5% | 0 | +0% | +50% | +| `quantization` | 2% | 0 | 97% | 0.5% | 2% | 0 | +0% | +20% | +| `provider-switch` | 3% | 0 | 95% | 1% | 3% | 0 | +20% | +40% | The `model-upgrade` row is the loose first-migration starting point — the `migration_policy` field defaults (see [Migration policy and CI gating](#migration-policy-and-ci-gating)); the other four are tighter presets to move to as the suite grows. @@ -211,7 +211,7 @@ init → doctor → run → evaluate → analy - **`run`** parses prompts, validates every example against every prompt, estimates cost, then dispatches `(prompt × example × {source, target})` calls through an async orchestrator under a concurrency semaphore. Responses are cached — per call, and per replayed round for tool-calling examples (see [Response cache](#response-cache)); progress is checkpointed every 50 completions. - **`evaluate`** scores each (source, target) pair with the configured evaluators, one `EvalRecord` per pair × evaluator. Scoring runs under the same `defaults.concurrency` semaphore as `run`, and the embedding/judge calls it makes go through the same response cache. - **`analyze`** runs paired statistics per `(prompt, evaluator, slice)`, applies Benjamini–Hochberg FDR correction, classifies severities, and — when a `migration_policy` is configured — computes a pass/fail verdict. -- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary, the narrative, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. Top regressions are collapsed cards — expand one for the trace diff, the tool diffs and the conversation context. The report is dark-only. +- **`report`** renders the single-file HTML report (no external assets; works offline and attaches cleanly to a PR or email), and writes the machine-written [run insights](#run-insights) narrative unless `--no-insights` is passed. The page opens on a verdict / advisory-signal / economics panel row and a six-cell run strip (examples, calls, failed-or-truncated, spend, latency Δ, mean score Δ), then the executive summary (one row per prompt, drawn from blocking evaluators — advisory ones only on a prompt nothing else scored; the strip's mean score Δ averages those rows), the narrative, the **Trace rules broken** panel when the target broke a hand-written trace rule, one section per prompt, and the methodology. Every figure on it is derived from the run's own artefacts; the deltas in the header are the run-level rollup of the per-prompt economics. Top regressions are collapsed cards — expand one for the trace diff, the tool diffs and the conversation context. The report is dark-only. - **`compare`** chains everything end to end over one suite and adds `--gate`, `--policy-gate`, `--push`, `--open`. Warnings raised while the pipeline runs (LiteLLM deprecation notices, insights-retry notes) are deferred and printed as one `⚠` section directly under the pipeline block — also when a stage fails, since a held-back warning may explain the failure. Errors are never deferred. ### Artefacts @@ -286,7 +286,8 @@ evaluators: # Tool evaluators are not written here: `capture sync` derives them per suite, # inside the managed region at the end of the file, from what each suite's own -# captures contain. +# captures contain. Hand-written `trace_invariants` are the exception: top +# level is fine, and sync keeps them on the managed entries it rewrites. migration_policy: max_overall_regression_rate: 0.30 @@ -485,7 +486,7 @@ same run. See [Agent evaluation](#agent-evaluation). Evaluators score each (source, target) output pair. Scores live in `[0, 1]`; the **delta = target − source**, so negative deltas are regressions. Every evaluator config accepts: - `blocking: true|false` (default `true`) — **blocking** evaluators feed the migration-policy verdict and CI gates; **advisory** (`blocking: false`) evaluators are computed, reported, and summarised separately but can never fail a run on their own. The `init` scaffold ships semantic and judge as advisory deliberately: at small suite sizes their noise would gate the verdict. -- `applies_to: ["*"]` — restrict to specific prompt ids (where supported). +- `applies_to: ["*"]` — prompt-id globs. Accepted by every family, enforced only by [`trace_invariants`](#trace-invariants); a no-op on the others. When an evaluator's own measurement breaks (judge call fails, embedding call fails), the record is stored as **errored and excluded from the statistics** — not silently scored neutral. Upstream failures are recorded the same way: if a *model call* failed or was truncated, the pair gets an errored row (a 0.5/0.5 placeholder with the error attached), kept in `scores.jsonl` and excluded from the statistics. The run always completes. @@ -608,7 +609,7 @@ Rules are checked per **response** (one model turn) against the tool calls alrea | `call_count` | `max_calls`: a model call takes the running count (context first) past it, one violation per excess call. `min_calls`: context plus model calls at the end of the last response fall short (one violation). Both equal = exact | context calls count toward both bounds | | `arguments` | a model call to `tool` fails `json_schema` (Draft 2020-12, validated at config load); unparseable arguments fail any schema that requires a field | none | -`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). The **Overall, by evaluator** table tags an advisory entry's row the same way and says what a trace-rule row measures (each side judged against the entry's rules, not against the other side); the **Top regression causes** panel (`failure_categories` in the decision) counts `INVARIANT_VIOLATION` as the budget does — distinct examples with a blocking target violation, never per row and never for advisory entries; and the per-prompt executive summary (with the header's mean score Δ, the mean of its rows) is drawn from blocking evaluators only — worst severity first, ties to the most negative effect size — falling back to advisory ones on a prompt that has nothing else. Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants). +`required` and `min_calls` are the two rules that are not vacuous on a trace that never mentions the tool, so on a single-shot replay (which sees only the first response) scope them with `applies_to`, or use `--rounds all` or imported traces. An entry whose `applies_to` matches no prompt in the run (with `traces: imported`, no imported trace pair) checks nothing and leaves no row, so `evaluate` prints a warning naming the entry and its `applies_to` (suppressed under `compare`'s quiet scoring); an entry that matched at least one prompt stays silent. A `traces: imported` entry on a run without `traces.jsonl` fails `evaluate` with an error naming that family. Each side scores the share of rules it broke zero times; the record's kind is `trace_invariants`, its metadata carries `rules_checked`, `source_violations`, `target_violations` (each `{rule_id, rule_type, tool, round_index, detail}`) and `owner`, and the target's violations stamp `INVARIANT_VIOLATION`. [`max_invariant_violations`](#migration-policy-and-ci-gating) turns a target violation of a blocking entry into a failed verdict. The HTML report's **Trace rules broken** panel and `report.json`'s `invariant_violations` list every rule the target broke, whatever the delta — advisory entries' violations included, though only blocking ones count toward the budget. Each `invariant_violations` row is `{prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}`; `source_also` says whether the source broke the same rule on that example, `blocking` is the entry's flag (`false` rows are reported but never gate, and the panel tags them **advisory**), and `sample` is `null` on a single-sample run and otherwise the violating sample's zero-based ordinal among the samples that scored successfully (not the raw `sample_index`). The **Overall, by evaluator** table tags every advisory (`blocking: false`) evaluator's row the same way and says what a trace-rule row measures (each side judged against the entry's rules, not against the other side); the **Top regression causes** panel (`failure_categories` in the decision) counts `INVARIANT_VIOLATION` as the budget does — distinct examples with a blocking target violation, never per row and never for advisory entries; and the per-prompt executive summary (with the header's mean score Δ, the mean of its rows) is drawn from blocking evaluators only — worst severity first, ties to the most negative effect size — falling back to advisory ones on a prompt that has nothing else. Full reference: [docs/configuration.md](docs/configuration.md#evaluatorstrace_invariants). --- @@ -731,7 +732,7 @@ How the verdict is computed: - **A cost/latency ratio measured only from zeroes says so.** When the calls exist but every `cost_usd` (or `latency_ms`) is `0` on both models, `analyze` adds a recommendations line beside the `conclusive: false`: `The cost increase budget could not be measured: all 4 error-free calls across both models recorded a cost of 0, so its observed 0.00 is a default, not a measurement.` A run with no calls gets no line — the empty `raw.jsonl` already explains itself. Emitted once per run, since every scope reads the same calls. - **Every budget reports its own `denominator`** — the sample `observed` was computed over. Scored records for the regression rate, the equivalence rate and the critical count — counted over *measurements*, so an evaluator scoring two axes contributes two rows per example; `tool_arguments` rows for tool drift; `tool_selection.divergence` rows for tool divergence; the distinct examples with a blocking `trace_invariants` row for the trace-rule count; the error-free calls behind both averages for the cost and latency ratios. Slices report their own counts. `0` means "counted, and the sample was empty", so `observed` is a default; a *missing* `denominator` means "no sample size reported" and is **not** zero — only bundles written before the field say that, and the hosted gate falls back to `conclusive` for them. It is the same number the `1/n` granularity warning is judged on, and it is orthogonal to `conclusive`: an all-zero cost ratio counted every call it averaged and still measured nothing, so it reports a positive denominator beside `conclusive: false`. The hosted gate derives its own Wilson interval from these denominators, over the same three rate budgets and with the same confidence constant the CLI uses, so a local verdict and a hosted one now agree on whether a breach was confirmed — see [methodology](docs/methodology.md). - Any conclusive budget failure, or any blocking critical/high-severity comparison → **`fail`**. Any lower-severity blocking regression → **`conditional_pass`**. Otherwise → **`pass`**. -- **`migration_policy` is the single source of truth for these budgets.** `migration_decision.json`'s `policy` field is the resolved policy `analyze` computed the verdict under — every top-level budget with its default applied, plus `slices`. `bundle`/`push` do not read `migration_decision.json`; `bundle` re-resolves the verdict from `evalshift.yaml` at bundle time, so the bundle's `decision.policy` (see [Hosted EvalShift](#hosted-evalshift)) is the policy the config held when the bundle was built — editing `migration_policy` between `analyze` and `bundle` changes what the bundle carries. `push ` builds a bundle only when `run_bundle.json.gz` is missing — an existing bundle is uploaded as-is — so re-run `evalshift bundle ` after a config edit. Either way, that is what lets the hosted gate check a pull request against exactly the budgets the bundle's own verdict used, instead of a separate, web-edited policy. `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. `evalshift.yaml` is the source of truth for that policy, and the web app's project policy view is becoming a read-only display of the snapshot each run pushed. Because every budget is resolved, the snapshot carries `max_invariant_violations` (default `0`) even for a config with no trace rules, so a hosted server must accept that field to take the push. +- **`migration_policy` is the single source of truth for these budgets.** `migration_decision.json`'s `policy` field is the resolved policy `analyze` computed the verdict under — every top-level budget with its default applied, plus `slices`. `bundle`/`push` do not read `migration_decision.json`; `bundle` re-resolves the verdict from `evalshift.yaml` at bundle time, so the bundle's `decision.policy` (see [Hosted EvalShift](#hosted-evalshift)) is the policy the config held when the bundle was built — editing `migration_policy` between `analyze` and `bundle` changes what the bundle carries. `push ` builds a bundle only when `run_bundle.json.gz` is missing — an existing bundle is uploaded as-is — so re-run `evalshift bundle ` after a config edit. Either way, that is what lets the hosted gate check a pull request against exactly the budgets the bundle's own verdict used, instead of a separate, web-edited policy. `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. `evalshift.yaml` is the source of truth for that policy, and the web app's project policy view is becoming a read-only display of the snapshot each run pushed. Because every budget is resolved, the snapshot carries `max_invariant_violations` (default `0`) even for a config with no trace rules; hosted EvalShift accepts and stores it (since 2026-10-08) but never re-evaluates it — a run that carries its own policy is answered with the CLI's decision, which did check it, and only runs pushed without a policy fall back to the server's six-budget re-check. - **A slice budget gates the run exactly like an overall one.** The budgets under `migration_policy.slices` are evaluated on the same terms as the top-level ones: a conclusively breached slice budget **fails** the run, and an unconfirmed breach makes it `inconclusive` — the same Wilson rule, counted over that slice's own denominator. Since the overall rows can all be green in a run a slice budget fails, `recommendations` names the one that blocked (`The 'security' slice breached its overall regression rate budget (the share of scored comparisons where the target did worse): 20% over n=20 vs the 0% limit.`) and the `inconclusive` `reason` scope-qualifies it the same way. Per-slice verdicts under `slices[*].verdict` are unchanged, and a slice that fails on *comparison severity* rather than a budget still only downgrades an overall `pass` to `conditional_pass`. - Semantic drift that stays above `min_similarity` counts as equivalent, not regression. @@ -967,7 +968,7 @@ The error is recorded on that call in `raw.jsonl`; the run completes. At evaluat ### `--resume` aborts with a config-hash mismatch -Resume requires the config and the suite path to match the original run — a changed config would corrupt the pairing. Start a fresh run. The suite's *contents* are not part of the hash, so an edited suite at the same path resumes without this error; start a fresh run after editing examples. +Resume requires the config and the suite path to match the original run — a changed config would corrupt the pairing. Start a fresh run. Upgrading the CLI counts as a config change when the release added config fields (they join the hashed canonical dump), so an older CLI's in-progress run cannot be resumed after upgrading. The suite's *contents* are not part of the hash, so an edited suite at the same path resumes without this error; start a fresh run after editing examples. ### Everything comes back severity `none` @@ -975,7 +976,7 @@ Usually correct behaviour: no statistically significant difference. Check n (n < ### The policy verdict is `inconclusive` -Three common causes, all by design: (1) every configured evaluator is advisory (`blocking: false` — the fresh `init` state) so nothing gates quality — set `blocking: true` on at least one trusted evaluator; (2) a rate budget was breached but the Wilson CI can't confirm it at this suite size — grow the suite; (3) all comparisons were `insufficient` (n < 5). Note that case (1) still reads `fail` when the cost or latency budget is breached — those are computed from the run's calls and hold with no blocking evaluator at all. `analyze` and `compare` print the specific reason and the recommended fix under the verdict line (also in `migration_decision.json` as `reason` / `recommendations`). +Three common causes, all by design: (1) every configured evaluator is advisory (`blocking: false` — the fresh `init` state) so nothing gates quality — set `blocking: true` on at least one trusted evaluator; (2) a rate budget was breached but the Wilson CI can't confirm it at this suite size — grow the suite; (3) all comparisons were `insufficient` (n < 5). Note that case (1) still reads `fail` when the cost or latency budget is breached — those are computed from the run's calls and hold with no blocking evaluator at all — and that a breached `max_invariant_violations` is the exception to all three: a count, conclusive however small the sample, so a blocking trace rule the target broke reads `fail`, never `inconclusive`. `analyze` and `compare` print the specific reason and the recommended fix under the verdict line (also in `migration_decision.json` as `reason` / `recommendations`). ### Does EvalShift evaluate multi-turn conversations? diff --git a/README.md b/README.md index a8ffd3b..6d8bbc0 100644 --- a/README.md +++ b/README.md @@ -263,7 +263,11 @@ shape, `fail-on` modes, and baseline behavior. Migrating an agent (a prompt that uses tools)? EvalShift detects regressions in *which* tools the new model calls, *what* arguments it -passes, and in what order and parallelism within a response. By default +passes, and in what order and parallelism within a response. Hand-written +`trace_invariants` rules (auth before a charge, a tool that must never be +called, at most one charge, argument bounds) hold *both* models to a +contract whatever the source did, gated by +`migration_policy.max_invariant_violations`. By default each example is one model call scored against the first recorded round; promote with `--rounds all` to replay every recorded round teacher-forced, with the recorded tool results fed back and each round scored on its own. @@ -289,7 +293,10 @@ assets, works offline) has: * **Migration verdict** — the policy decision up top: which budgets failed, the top regression causes, and the recommendation. -* **Executive summary** — one row per prompt with a severity badge. +* **Executive summary** — one row per prompt with a severity badge, drawn from + blocking evaluators (advisory ones only when nothing else scored the prompt). +* **Trace rules broken** — every hand-written trace rule the target broke, + with its owner and whether the source broke it too; advisory rows are tagged. * **What changed, in plain language** — the verdict, the economics and the behavioural drift explained by `defaults.insights_model`. Every figure in it is copied from the computed statistics, never generated. Needs a provider diff --git a/docs/agents.md b/docs/agents.md index 130ea73..1a2b593 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -22,8 +22,9 @@ The killer scenario it catches: * **`ToolTrace` data model**: provider-agnostic, populated from Anthropic / OpenAI / Gemini responses. -* **Three new evaluators**: `tool_selection`, `tool_arguments`, - `tool_trace_structure`. +* **Four new evaluators**: `tool_selection`, `tool_arguments`, + `tool_trace_structure`, and hand-written `trace_invariants` (both sides + judged against your rules, not against each other). * **Suite extension**: optional `expected_tools`, `expected_tool_count`, `expected_no_tools`, `expected_parallel` per example, plus a required `toolset_ref` or inline `tools` — see [Suite ground truth](#suite-ground-truth). @@ -85,7 +86,9 @@ defaults: evaluators: # Top-level: what every suite is scored with. Tool evaluators do not belong - # here — `capture sync` writes them per suite (see below). + # here — `capture sync` writes them per suite (see below). Hand-written + # `trace_invariants` are the exception: top level is fine, and sync keeps + # them on the managed entries it rewrites. semantic: embedding_model: gemini/gemini-embedding-001 blocking: false @@ -600,6 +603,13 @@ identically is not a ✗ — it is the same pair before and after, and what is wrong is the ground truth. Look for **Ground truth missed by both** in the evaluator table when that happens. +A **Trace rules broken** panel appears when the target broke a hand-written +[`trace_invariants`](configuration.md#evaluatorstrace_invariants) rule: every +broken rule with its owner and whether the source broke it too, whatever the +delta, advisory (`blocking: false`) rows tagged **advisory**. `report.json` +carries the same rows under `invariant_violations`, and the evaluator table +tags an advisory entry's row the same way. + ## Troubleshooting * **A suite reports tool evaluator scores you didn't expect on some rows** — diff --git a/docs/configuration.md b/docs/configuration.md index 91637fc..6676607 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -504,7 +504,7 @@ A list. Each entry has: | `name` | string | (required) | Identifier surfaced in reports. | | `conformance` | enum | `expected` | Ground-truth axis: `expected` / `expected_set` / `off`. `expected_set` is `expected` made order-insensitive: multiset recall of `example.expected_tools` names. Use it when the expected calls are a parallel fan-out whose order carries no meaning. | | `divergence` | enum | `set` | Target-vs-source axis: `set` (Jaccard on tool names) / `exact` (sequence equality) / `first` (first call only) / `off`. | -| `applies_to` | list | `["*"]` | Glob list of prompt ids. | +| `applies_to` | list | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). | | `severity_floor` | enum | `null` | If set, surfaces in metadata so the analysis layer can floor severity. | The two axes are independent and each writes **its own record**, under @@ -535,7 +535,7 @@ deprecated, and a config still carrying one fails validation. | Field | Type | Default | Description | | ----------------------- | ------ | ------- | ----------- | | `name` | string | (required) | Identifier. | -| `applies_to` | list | `["*"]` | Glob list. | +| `applies_to` | list | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). | | `against` | enum | `source` | What arguments are compared to: `source` (drift from the source model) or `expected` (correctness against `expected_tools[].arguments`, scored on both sides). | | `strategies` | dict | `{}` | Per-field strategy overrides (`exact`/`subset`/`numeric`/`semantic`/`auto`). | | `default_strategy` | enum | `auto` | Strategy for fields `strategies` does not name. `auto` is the ladder below; `exact` restores byte-equality scoring. | @@ -611,7 +611,7 @@ unaffected: it counts calls whose *target* score fell below | Field | Type | Default | Description | | --------------------- | ------ | ------- | ----------- | | `name` | string | (required) | Identifier. | -| `applies_to` | list | `["*"]` | Glob list. | +| `applies_to` | list | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). | | `check_call_count` | bool | `true` | Score the number of tool calls. | | `check_parallelism` | bool | `true` | Score parallel-vs-sequential alignment. | | `check_refusals` | bool | `true` | Score refusal alignment; mismatches force `severity_floor: high`. | @@ -626,7 +626,7 @@ the normal `raw.jsonl` model-call artifact. | Field | Type | Default | Description | | ---------------------------- | ------ | ------- | ----------- | | `name` | string | required | Identifier surfaced in scores and reports. | -| `applies_to` | list | `["*"]` | Glob list of prompt ids. | +| `applies_to` | list | `["*"]` | Glob list of prompt ids (accepted, not yet enforced — only [`trace_invariants`](#evaluatorstrace_invariants) enforces it). | | `check_tool_order` | bool | `true` | Compare source and target tool-call order. | | `check_arguments` | bool | `true` | Compare arguments for same-name matched tool calls. | | `check_missing_verification` | bool | `true` | Check dangerous tools have an earlier verification tool. | @@ -740,9 +740,10 @@ example's row keeps every scored sample's violations, each tagged with its whatever the delta — including those of advisory (`blocking: false`) entries, which never count toward the budget — with the owner and whether the source broke it too. Each row carries the entry's `blocking` flag, and the panel tags -`blocking: false` rows **advisory**. So does the **Overall, by evaluator** -table, where every `trace_invariants` row also says what it measures: each -side judged against the entry's rules, not against the other side. The +`blocking: false` rows **advisory**. The **Overall, by evaluator** table tags +every advisory (`blocking: false`) evaluator's row the same way, and every +`trace_invariants` row there also says what it measures: each side judged +against the entry's rules, not against the other side. The per-prompt **Executive summary** — and the header's mean score Δ, which averages its rows — is drawn from blocking evaluators only (worst severity first, ties to the most negative effect size), falling back to advisory ones diff --git a/docs/evaluators.md b/docs/evaluators.md index f65c309..d2826cc 100644 --- a/docs/evaluators.md +++ b/docs/evaluators.md @@ -207,10 +207,12 @@ Every other tool evaluator treats "the source did it" as correct: the source, so a wrong call both models make scores as agreement. `trace_invariants` judges **both** sides against rules your team wrote — `forbidden` tools, `required` tools, an `order` between two tools, a -`call_count` (min, max or exact per tool) and `arguments` that must satisfy a -JSON Schema. A rule the target breaks counts against it whatever the source -did, and `migration_policy.max_invariant_violations` (default `0`) fails the -run on it, even on a suite too small for statistics. +`call_count` (`min_calls` and/or `max_calls` per tool; equal bounds = an exact +count) and `arguments` that must satisfy a JSON Schema. A rule the target +breaks counts against it whatever the source did, and for a blocking entry +(the default) `migration_policy.max_invariant_violations` (default `0`) fails +the run on it, even on a suite too small for statistics; `blocking: false` +entries are reported but never gate. Write rules only for invariants where a silent miss is expensive: auth before a write, a deprecated endpoint that must stay unused, at most one charge, @@ -227,8 +229,9 @@ recording. Configuration, rule semantics and scoring: ## Mixing evaluators -You can configure several at once. They all run on every (prompt, -example) pair, and the analysis layer treats each as a separate +You can configure several at once. Most run on every (prompt, example) +pair — `trace_invariants` only on the prompts its `applies_to` matches, +and only on pairs with a trace to check — and the analysis layer treats each as a separate comparison (so BH correction adjusts for the multiple-test count correctly). diff --git a/docs/faq.md b/docs/faq.md index be0e6d3..23aeb1b 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -98,7 +98,9 @@ since, and continues from where it left off. Already-completed calls (including ones that errored at the LLM layer) are skipped. A config change or a different suite path between attempts aborts the -resume — start a fresh run instead. The suite's *contents* are not checked, +resume — start a fresh run instead. Upgrading the CLI counts when the release +added config fields (they join the hashed canonical dump), so an older CLI's +in-progress run cannot be resumed after upgrading. The suite's *contents* are not checked, so after editing examples, start a fresh run yourself. ## How do I push a run to hosted EvalShift? @@ -180,6 +182,11 @@ Three common causes, all by design: confirm it** at this suite size — grow the suite. 3. **All comparisons were `insufficient`** (n < 5). +A breached `max_invariant_violations` is the exception to all three: it is a +count, conclusive however small the sample, so a blocking hand-written trace +rule the target broke reads `fail`, never `inconclusive` — even when every +comparison is insufficient. + `analyze` and `compare` print the specific reason and the recommended fix under the verdict line, and record them in `migration_decision.json` (`reason` / `recommendations`). diff --git a/docs/hosted.md b/docs/hosted.md index bc6cf72..7e1f85e 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -296,7 +296,7 @@ The bundle itself contains: | --- | --- | | `manifest` | Run id, `org/project` slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the **local suite file path** as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. | | `examples[]` — one row per prompt × example | The example's template variables (`inputs`) **verbatim**; its `expected` reference output **verbatim**; both models' **full output text**; tool-call traces from the replay: each side's tool calls (names, arguments, call ids) with round markers, any final text and refusal messages, capped at 256 KB per side. Imported agent traces (`traces import`) are not uploaded; per-evaluator scores and error strings; per-side cost and latency; tags and slice names. | -| `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It therefore carries `max_invariant_violations` (default `0`) even when the config has no trace rules, which the server must accept. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. | +| `aggregate`, `analysis`, `decision`, `economics` | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. `decision.policy` is the resolved `migration_policy` this run's verdict was computed under — every top-level budget with its default applied, plus `slices` — or `null` when no `migration_policy` is configured. It therefore carries `max_invariant_violations` (default `0`) even when the config has no trace rules; hosted EvalShift accepts and stores it (since 2026-10-08) but never re-evaluates it — a run that carries its own policy is answered with the CLI's decision, which did check it, and only runs pushed without a policy fall back to the server's six-budget re-check. It is what lets the hosted gate check a pull request against the exact budgets the verdict used, instead of a separate policy configured elsewhere. | | `methodology_notes` | The model ids and the statistical-contract sentences shown in every report. | | `insights` | The machine-written run narrative, when one was generated. It is prose *about* your run and can paraphrase or quote the regressions it summarizes. | | `evaluator_config` | Config version; the prompt list **metadata only** — prompt names, file paths, and variable names, with every prompt body replaced by a `content_hash`; the whole `defaults` block (model ids, concurrency, cache flag, cost ceiling, max_tokens, samples_per_example); an always-empty `slices` list (a legacy key, kept so `eval_config_hash` does not change — the top-level `slices:` config key was removed); and the full evaluators block — which includes each `llm_judge` entry's `criterion_prompt` text, so keep judge criteria free of secrets. An empty `trace_invariants` list is left out of it, so a config without trace rules keeps its `eval_config_hash`; adding rules changes the hash like any other evaluator edit. | diff --git a/docs/methodology.md b/docs/methodology.md index 7492032..8be806f 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -336,10 +336,11 @@ counted every one of them and still measured nothing, so it reports a positive denominator beside `conclusive: false`. Reading either field as a stand-in for the other loses that. -The field exists for the hosted gate. `GET /runs/{run_id}/policy-check` -re-decides an uploaded run against the project's *current* policy and never saw -the underlying records, so without a sample size it cannot tell a clean run from -an empty one. Bundles written before the field omit it entirely, and a missing +The field exists for the hosted gate. For a run pushed without a policy, +`GET /runs/{run_id}/policy-check` re-decides it against the project's *current* +policy (a run that carries its own `decision.policy` is answered with the CLI's +stored decision instead), and the server never saw the underlying records, so +without a sample size it cannot tell a clean run from an empty one. Bundles written before the field omit it entirely, and a missing `denominator` means **unknown, not zero** — the server falls back to `conclusive` for those. This CLI always knows its own denominators, so every budget it emits carries an integer. diff --git a/docs/traces.md b/docs/traces.md index 10f7f72..127459c 100644 --- a/docs/traces.md +++ b/docs/traces.md @@ -137,7 +137,10 @@ evaluators: An imported trace is checked as one response with no prior context: every call in it, in `sequence_index` order, was the agent's own. A rule the target breaks counts toward `migration_policy.max_invariant_violations` whatever the source -did. +did, when the entry is blocking (the default); `blocking: false` entries are +reported but never gate. The entry's `applies_to` scopes imported pairs by +`prompt_id`, and `evaluate` fails on a run with no `traces.jsonl`, naming +`trace_invariants (traces: imported)` in the error. Debug commands become trace-aware: diff --git a/llms-full.txt b/llms-full.txt index 00dfb6a..4481a93 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -143,10 +143,11 @@ evalshift init [-f/--force] [-d/--directory DIR] [--ci] [--wire-agents/--no-wire Without --ci, warns after writing when an existing .github/workflows/*.yml pins an older or newer evalshift-version than this CLI, or none (advisory, exit 0; see CI pin drift). --ci writes the pin itself and does not warn about the file it just wrote. - PROFILE budgets (regression/critical/equivalence/arg-drift/tool-divergence/cost/latency): - model-upgrade (default): .30/1/.75/.20/.20/.30/.30 (= MigrationPolicy defaults) - cost-reduction: .02/0/.97/.01/.02/.05/.30 | local-model: .05/0/.90/.02/.05/.00/.50 - quantization: .02/0/.97/.005/.02/.00/.20 | provider-switch: .03/0/.95/.01/.03/.20/.40 + PROFILE budgets (regression/critical/equivalence/arg-drift/tool-divergence/ + trace-rule-violations/cost/latency; every profile pins trace-rule violations at 0): + model-upgrade (default): .30/1/.75/.20/.20/0/.30/.30 (= MigrationPolicy defaults) + cost-reduction: .02/0/.97/.01/.02/0/.05/.30 | local-model: .05/0/.90/.02/.05/0/.00/.50 + quantization: .02/0/.97/.005/.02/0/.00/.20 | provider-switch: .03/0/.95/.01/.03/0/.20/.40 evalshift doctor Env/config check. Exit 1 ONLY when an existing evalshift.yaml fails validation (the row shows the problem count + "run `evalshift validate` for details"; validate prints each problem); @@ -191,9 +192,11 @@ evalshift report RUN_ID [-c CONFIG] [--open] [--insights/--no-insights (default report.html layout: header (run id, source->target, suite, examples/calls, spend) -> verdict / advisory-signal / economics panels -> six-cell run strip (examples, calls, failed/truncated, total cost, latency delta, mean score delta) -> recommendations + - top regression causes -> executive summary -> insights narrative -> one section per - prompt (economics, per-example, by-evaluator, slices, top regressions as collapsible - cards) -> methodology. Dark-only, no external assets, no JS. Rendered surfaces show + top regression causes -> executive summary (one row per prompt, drawn from blocking + evaluators; advisory ones only on a prompt nothing else scored) -> insights narrative -> + "Trace rules broken" panel (only when the target broke a hand-written trace rule; + blocking: false rows tagged advisory) -> one section per prompt (economics, per-example, + by-evaluator, slices, top regressions as collapsible cards) -> methodology. Dark-only, no external assets, no JS. Rendered surfaces show display names, never internal identifiers: failure categories as plain language (TOOL_SELECTION_DRIFT -> "Different tools chosen"; map in evaluators/failures.py), budgets as their labels (max_tool_divergence -> "Tool-selection divergence"; map in @@ -636,7 +639,8 @@ evaluators: # every evaluator config also takes blocking # load; {"_parse_error": true} args fail any schema that requires a field # Unit = RESPONSE (one model turn) + CONTEXT (tool calls already in it). Replayed: one # response per round; round k context = history assistant tool_calls + RECORDED rounds - # 0..k-1 the teacher-forced replay fed back (never the model's own earlier rounds). + # 1..k-1 the teacher-forced replay fed back (never the model's own earlier rounds; + # replay numbering as in "run" above -- a violation's round_index is 0-based). # Imported: one response, empty context. Context calls are never violations; they # satisfy `order` and count toward both call_count bounds; `required` ignores them. # required / min_calls are NOT vacuous (fail every in-scope trace lacking the call); on a @@ -861,8 +865,8 @@ scored successfully (NOT the raw sample_index). Advisory (blocking: false) entri ARE listed; only blocking ones count toward max_invariant_violations. blocking = the record's flag (false = advisory entry). report.html renders the same rows as a "Trace rules broken" panel (rule, owner, detail, "Source too?"), tagging blocking=false rows with a muted -"advisory" badge. The "Overall, by evaluator" table tags an advisory entry's row the same way -and says what a trace-rule row measures (each side judged against the entry's rules). The +"advisory" badge. The "Overall, by evaluator" table tags every advisory (blocking: false) +evaluator's row the same way and says what a trace-rule row measures (each side judged against the entry's rules). The per-prompt executive summary (and the header's mean score Δ, the mean of its rows) is drawn from blocking evaluators only -- worst severity first, ties to the most negative effect size, the order the hero panel and the narrative use -- falling back to advisory ones on a prompt @@ -1005,7 +1009,9 @@ migration_policy is configured; inconclusive exits 0). what lets the hosted gate check a PR against the exact budgets the bundle's own verdict used, instead of a separate, web-edited policy -- see `evalshift push` above and docs/hosted.md. decision.policy always carries max_invariant_violations (default 0), even - for a config with no trace rules, so the hosted server must accept that field. + for a config with no trace rules; the hosted server stores it (accepted since 2026-10-08) + and never re-evaluates it -- a run that carries its own policy is answered with the CLI's + decision, only policy-less runs get the server's six-budget re-check. null on the inconclusive_decision path (no migration_policy configured) and on a migration_decision.json written before this field existed. evalshift.yaml is the source of truth for the policy; the web app's per-project policy view is becoming a @@ -1282,7 +1288,8 @@ suites: {} # init writes the managed region LAST, after # evalshift.yaml — agent prompt (tool-calling additions). Tool evaluators live in the # PER-SUITE block `capture sync` generates, not at the top level: a tool-free suite that -# inherits a tool evaluator scores an empty denominator = an inconclusive gate. +# inherits a tool evaluator scores an empty denominator = an inconclusive gate. Hand-written +# trace_invariants are the exception (top level is fine; sync keeps them on managed entries). prompts: - id: customer_routing detection: python_string @@ -1404,14 +1411,17 @@ All severities "none" -> usually genuinely no significant difference; check n (< Verdict "inconclusive" -> (1) all evaluators advisory (fresh init: flip blocking: true as the suite grows), (2) rate-budget breach unconfirmed by Wilson CI (grow the suite), (3) all n<5, (4) every blocking row excluded as a shared ground-truth miss -> fix the harness, not the -sample size; more pairs from the same setup are more excluded rows. +sample size; more pairs from the same setup are more excluded rows. Exception: a breached +max_invariant_violations is never inconclusive -- a count, it fails however small the sample. "broken eval harness" row -> the SOURCE model failed ground truth captured from itself. The suite does not describe the model under test: re-capture it against the agent actually running, or set conformance: off. Any verdict printed beside it is arithmetic over the wrong suite. analyze/compare print the specific reason + recommended fix under the verdict line (also in migration_decision.json as reason/recommendations). --resume aborts -> config or suite path changed since the run started (config_hash mismatch); start - fresh. Suite contents are not hashed: an edited suite at the same path resumes silently. + fresh. A CLI upgrade that added config fields moves the hash too (an older CLI's in-progress + run cannot be resumed). Suite contents are not hashed: an edited suite at the same path + resumes silently. Failed calls -> recorded in raw.jsonl with error, pair gets an errored 0.5/0.5 row excluded from the statistics; fresh run retries them (cache serves the successes; for a multi-round tool example, the rounds before the failed one). diff --git a/src/evalshift_cli/hosted/bundle_manifest.schema.json b/src/evalshift_cli/hosted/bundle_manifest.schema.json index 8893a55..de73e3e 100644 --- a/src/evalshift_cli/hosted/bundle_manifest.schema.json +++ b/src/evalshift_cli/hosted/bundle_manifest.schema.json @@ -218,7 +218,7 @@ } ], "default": null, - "description": "How many units `observed` was computed over \u2014 records for the rate budgets, tool-bearing records for tool drift, paired calls for the cost and latency ratios. 0 means the budget was evaluated against an empty sample, so `observed` is a default and not a measurement. Null means no sample size was reported, which is NOT the same as zero: bundles written before this field existed omit it, and consumers must treat its absence as 'unknown' and fall back to `conclusive`.", + "description": "How many units `observed` was computed over \u2014 records for the rate budgets, tool-bearing records for tool drift, the distinct examples a blocking trace rule was checked on for `max_invariant_violations`, and the error-free calls behind the per-role means for the cost and latency ratios. 0 means the budget was evaluated against an empty sample, so `observed` is a default and not a measurement. Null means no sample size was reported, which is NOT the same as zero: bundles written before this field existed omit it, and consumers must treat its absence as 'unknown' and fall back to `conclusive`.", "title": "Denominator" }, "name": { diff --git a/src/evalshift_cli/hosted/push.py b/src/evalshift_cli/hosted/push.py index 70e2730..0aa467a 100644 --- a/src/evalshift_cli/hosted/push.py +++ b/src/evalshift_cli/hosted/push.py @@ -633,7 +633,7 @@ def _policy_yaml_block(policy: dict[str, Any]) -> str | None: CLI's model rejects is a block that would break the file it is pasted into. Only the keys the server actually sent survive (``exclude_unset``). A - web-app policy has six of the nine budgets; writing out the other three + web-app policy has six of the ten budgets; writing out the other four would pin today's CLI defaults into the user's file as if they had chosen them, and they are meant to move with the CLI. """