Skip to content

fix(report): advisory evaluators out of the summary; trace-rule causes counted as the budget does - #36

Merged
babaliauskas merged 1 commit into
mainfrom
fix/report-advisory-counts
Oct 8, 2026
Merged

babaliauskas merged 1 commit into
mainfrom
fix/report-advisory-counts

Conversation

@babaliauskas

Copy link
Copy Markdown
Collaborator

Follow-up to #35, found while rendering a real report with an advisory (blocking: false) trace_invariants entry. Three summary surfaces disagreed with the verdict; the rule checks themselves were correct.

What was wrong

Surface Before After
Executive summary row / header Avg score Δ notes_watch +0.091 (advisory; the target "improved" by calling no tools) routing −0.091 (the gating evaluator, same sign as the hero tile's −0.30)
Top regression causes → "Broke a hand-written trace rule" 8 — every record tagged, advisory and source-shared included; all 8 had delta 0.0 5 — distinct examples with a blocking target violation, i.e. the budget's own number
AI narrative "violated a trace rule in 8 instances, which triggered a failure" built on 5, the figure the verdict used
Overall, by evaluator notes_watch looked like a gating evaluator; trace-rule rows had no "what this measures" line tagged advisory; "each side judged against the entry's hand-written rules"

Changes

  • reports/json.py: _build_executive_summary orders by (severity, effect_size, p_corrected) — the ordering _headline_comparison and the insights already use — over gating evaluators, falling back to advisory ones only for a prompt nothing else scored. Aggregate rows carry advisory so the template can tag them.
  • analysis/policy.py: _failure_categories counts INVARIANT_VIOLATION via _invariant_violation_counts over blocking rows, so it can never disagree with max_invariant_violations.
  • reports/html.py + template: advisory tag and a kind_blurb for trace_invariants in the evaluator table.
  • Docs (docs/configuration.md, DOCS.md, llms-full.txt) and CHANGELOG (Unreleased: amended Added bullet + a Changed entry for the tie-break).

Tests

8 new tests (TDD, each watched red first): TestInvariantFailureCategory (2) in test_policy.py, TestAdvisoryEvaluatorsInSummary (6) in test_reports.py.

pytest: 2581 passed, 1 failed — test_bundle_shape.py::test_the_vendored_schema_matches_the_server_export. That failure is pre-existing and environmental (reproduced with this branch's changes stashed): it diffs the vendored bundle schema against the sibling evalshift-server checkout, which is on main without server #23's max_invariant_violations. It passes once #23 is merged. ruff check, ruff format --check, mypy --strict, pre-commit run --all-files all green.

Release note: still blocked on server #23 deploying first (see #35).

🤖 Generated with Claude Code

…-rule causes as the budget does

- The per-prompt Executive summary (and the header's mean score Δ, which
  averages its rows) picks the worst severity with ties toward the most
  negative effect size -- the order the hero panel and the narrative already
  used -- over gating evaluators only. A `blocking: false` evaluator speaks
  for a prompt only when nothing else scored it. On a real run the advisory
  `notes_watch` entry, satisfied by a target that called no tools, read as
  the prompt's "+9% improved" directly above a failed verdict.
- `failure_categories` counts INVARIANT_VIOLATION the way
  max_invariant_violations does -- distinct examples with a blocking target
  violation -- instead of per row, advisory and shared breaks included. The
  "Top regression causes" panel and the insights built on it read 8 where
  the verdict read 5, and none of the 8 rows was a regression.
- "Overall, by evaluator" tags an advisory entry's row the way the broken
  rules panel does, and says what a trace_invariants row measures.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@babaliauskas
babaliauskas merged commit 98acc3f into main Oct 8, 2026
4 checks passed
@babaliauskas
babaliauskas deleted the fix/report-advisory-counts branch October 9, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant