[napetrov]

Summary

Part of #220.

eval redteam QUESTIONS.json checks that each question's grading block rejects the answers it should. It grades four synthetic answers per question:

• canary: the right answer, built from one literal check (match, fuzzy, exact, choice, json). It must pass.
• wrong: a planted wrong answer that carries no positive check value. It must fail.
• empty: the empty answer. It must fail.
• hedge: a non-committal sentence with no check value. It must fail.

A negate check does not switch the other attacks off: wrong, empty and hedge grade the block's non-negated checks. A block of only negated checks gets none of the three and is reported as negated_only.

Statuses: pass, blocked (an attack was accepted or a canary was rejected), inconclusive (a graded question was not measured, or nothing was graded). The stuffed answer was dropped as an attack (it could not be a violation for a positive block, and a negated block is unmeasured either way); stuffed_for stays as a grade_check self-check with unit tests. Exit 0 on pass, 1 on blocked or inconclusive, 2 on an unreadable question set or an unwritable --out. --out writes redteam.v1. Judge checks are unscored unless --judge is given, so the default run makes no model call. No score changes.

Evidence

• Red then green: each new test failed before its fix and passes after. The regressions cover: a planted wrong answer that contains a check value (match 31, match 1, fuzzy on) no longer blocks a correct grader; negated checks no longer silence the attacks on a positive check; a single negated value builds its self-check; a negated whole-answer check builds none; a check value in the hedge skips the hedge probe; the canary for match at the end with ignore_punctuation: false has no full stop; a run that probed nothing is inconclusive; an unwritable --out exits 2; the summary counts unprobed questions.
• Fresh-context review (FRESH-REVIEW-A2, diff e2147b1...04f2142; its quoted line numbers match 04f2142, not 71cc8e6) found three blockers, all fixed in 6e16c6b: a wrong answer that contained a check value blocked correct graders; stuffing a negated exact, json or location: exact check blocked correct graders; a block with one negated value was never probed. In-session review found the negated-check silencing (fixed in 04f2142) and the hedge clash (fixed in 068abc8).
• Local CI on 6e16c6b (make ci-local --offline): ruff, mypy (273 files), deptry, bandit, schema and registry checks, mkdocs and the CLI-surface job (which reports the added eval redteam) pass. Pytest: 9735 passed, 5 skipped, 1 failed: test_container_agent.py::test_probe_reports_a_writable_docker_socket (AF_UNIX path too long). That test depends on the path length of the worktree: it failed in these /localdisk/napetrov/scratch worktrees and on a clean intel/main checkout (earlier run), and passed in the shorter base worktree.
• Negated mode: all fix on e881ab5: a negated match with mode: all and the default location: end got the stuffed answer Paris Lyon. That answer ends with one value only, so the all-match is false and the negation passes it: the real grader accepted it and a correct grader was blocked. The stuffed probe is now skipped for negated mode: all, as the positive rule already skips mode: all. Red then green: the two new cases failed on 6e16c6b (2 failed, 41 passed) and pass on e881ab5 (43 passed). ruff check and mypy pass. A fresh-context claude -p review of 6e16c6bb..e881ab57 plus stuffed_for found no blocker; it confirmed the grader rejects every stuffed answer that is still built, and its two minor points (the skip is wider than the one failing case; the docstrings overstated) are addressed.
• The redteam tests: 112 cases at 92830e84 (tests/test_redteam.py), all passing.

Merge Danger

• New command eval redteam and new persisted format redteam.v1 (schema, fixture, registry entry). Both are new in this PR.
• Exit 1 on a blocked or inconclusive run. A CI job that runs eval redteam now fails when a grader accepts a wrong answer, and when a question set has nothing gradeable (for example, judge checks only, without --judge).
• Known limits, each confirmed by a probe on the final code (the threshold-block pass and the negated-check false block listed in earlier rounds are fixed: c09f9b6, 3a4cf5b):
  • A regex that matches the hedge text is not skipped; only literal values are checked for a clash with the hedge.
  • A single negated check with threshold 0 or below used to be blocked by its stuffed probe (the grader passes everything); with the probe gone it is inconclusive, exit 1 either way.
  • A choice check with several letters ("AB"): the canary ANSWER: AB is rejected by the grader, so the run blocks. This is a grader issue in validate_grading, not in the red team.
  • Exit 1 when any graded question is not measured, even if others are probed. The table and the summary name the unmeasured questions.
  • With --judge, judge checks call the model. The wrong answer is a fixed sentinel string, not one written for each question.

🤖 Generated with Claude Code