Skip to content

feat(push): send score explanations and trace-rule violations - #38

Merged
babaliauskas merged 10 commits into
mainfrom
feat/push-score-violations
Oct 8, 2026
Merged

babaliauskas merged 10 commits into
mainfrom
feat/push-score-violations

Conversation

@babaliauskas

Copy link
Copy Markdown
Collaborator

Summary

  • each uploaded score row now carries the record's one-line explanation and, for trace_invariants rows that measured, violations ({source, target} of {rule_id, rule_type, tool, round_index, detail, sample}), filtered to the server's contract keys; errored rows send null
  • rules_checked, owner and failure_categories are not repeated on the row (they already ship in evaluator_config / decision)
  • re-vendored bundle_manifest.schema.json from the server's export (byte-identical)
  • docs: DOCS.md, docs/hosted.md, docs/traces.md, llms-full.txt and the changelog say what uploads — including that trace-rule verdicts computed on imported traces upload tool names and, for arguments rules, the quoted offending value, even though the timelines stay local

Depends on

evalshift/evalshift-server PR "feat: carry score explanations and trace-rule violations" being deployed, not just merged: the server's bundle models are extra="forbid", so pushes from this branch to the current server 422 at finalize. Do not release the CLI before that deploy is live.

Until the server PR merges, tests/unit/test_bundle_shape.py::test_the_vendored_schema_matches_the_server_export fails locally when ../evalshift-server is on main (it compares against the sibling checkout); it skips in CI. Re-run it un-deselected after the server merge.

Verification

  • ruff check, ruff format --check, mypy --strict, pytest -m "not integration": 2565 passed (the one environmental deselect above)
  • a populated violations row validates against the vendored schema (new drift-guard test)
  • pushed testApp's planner run to a local server running the server branch: the example-detail route returned the rule verdicts and the web panel rendered them

🤖 Generated with Claude Code

babaliauskas and others added 10 commits October 8, 2026 23:32
Each uploaded score row now carries the record's one-line explanation and, for trace_invariants rows that measured, the rules each side broke (rule id, type, tool, round, detail, sample). rules_checked, owner and failure_categories stay local; they already ship in evaluator_config and decision. The traces the verdicts were computed from upload already, so this adds no new exposure. Re-vendors the server schema that accepts the fields.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d schema

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@babaliauskas
babaliauskas merged commit 503836c into main Oct 8, 2026
4 checks passed
@babaliauskas
babaliauskas deleted the feat/push-score-violations branch October 9, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant