Repository navigation
feat: trace_invariants evaluator and max_invariant_violations budget - #35
Merged
Merged
Conversation
…lations budget Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d traces Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…very sample's violations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ils the run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… shared Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Document the trace_invariants evaluator family and the max_invariant_violations budget across DOCS.md, llms-full.txt and the docs/ pages, and add the CHANGELOG entries under [Unreleased]. Sites enumerated mechanically: - `max_tool_divergence` grep: 25 hits; 13 edited in place or extended beside (every budget list/table/schema/init block), 12 reviewed and left as-is (prose examples, or lists of the four rate budgets that correctly exclude a count budget). - `agent_trace\b` grep: 15 hits; 13 edited (every evaluator-family list, plus imported-trace sections now pointing at `traces: imported`), 2 reviewed and left (README examples table, agents.md flattening note). - `max_critical_regressions` grep: 10 hits; all 10 budget blocks covered (9 edited in place, 1 covered by an adjacent new bullet). Beyond the greps: the managed-region exception (sync keeps trace_invariants despite the marker comment), report.json's invariant_violations key, the INVARIANT_VIOLATION failure category, decision.policy always carrying the field, and the eval_config_hash note. init now scaffolds `max_invariant_violations: 0` in every profile, matching how every other BUDGET_LABELS budget (CLI-only max_tool_divergence included) is scaffolded; a test pins every profile to every budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r row Each trace_invariants entry writes its own row per example, so two rule sets with one example breaking both reported 2 of 4 examples on a 2-example suite. max_invariant_violations and the shared-break note now count distinct (prompt_id, example_id) pairs. Also pins the run-level override that turns a slice-demoted conditional_pass into fail on a breached trace-rule budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An applies_to that matches no prompt (replayed) or no imported trace pair left no row and no coverage entry, so the run read clean without a word about the rules. evaluate now prints a warning naming the evaluator and its applies_to (suppressed by quiet, like the module's other warnings); partial scoping stays silent. The missing-traces.jsonl error now names the family that needs it, trace_invariants (traces: imported) as well as agent_trace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
invariant_violations rows in report.json gain a `blocking` field from the record, and the "Trace rules broken" panel tags rows from a `blocking: false` entry with a muted "advisory" badge, so a reader can tell which breaks gate the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The class now also covers a rule both sides break, so "though the source broke nothing" no longer describes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y-scope warning max_invariant_violations counts distinct (prompt_id, example_id) examples over a denominator of distinct examples, and its row appears only where a blocking trace_invariants row was scored. Documents the evaluate warning for an entry whose applies_to matched nothing, the family-naming missing-traces error, and the `blocking` field / advisory tag on invariant_violations rows. CHANGELOG: the capture sync bullet moves to Added, and the init bullet warns that older CLIs reject the scaffolded field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hand-written contracts over tool-call traces, checked independently of the source side.
evaluators.trace_invariants: rulesforbidden,required,order,call_count(min/max/exact per tool),arguments(JSON Schema);traces: replayed|imported;owner;applies_toenforced;blocking: falsefor advisory entriesmigration_policy.max_invariant_violations(default0, per-slice overridable): counts distinct examples where the target broke a blocking rule. A breach fails the run even when the suite is too small for statistics, and a slice that breaches fails however small it isreport.jsongets a top-levelinvariant_violationskeycapture synckeeps a managed entry's hand-writtentrace_invariantsblockevaluatewarns when an entry'sapplies_tomatched nothinginitscaffolds the new budgetMerge order: evalshift/evalshift-server#23 must be deployed first, because pushed bundles now always carry the field. Release as 1.3.0 (minor). Upgrading moves the resume
config_hash(see CHANGELOG).Tests:
make cipasses with 2573 tests and 94.7% coverage.test_the_vendored_schema_matches_the_server_exportfails locally against servermainuntil the server PR merges; CI skips it.🤖 Generated with Claude Code