[napetrov]

Follow-up to #55 (review: the shared verdict parser).

Summary

The wrong-figure cap reads each numeric_claims entry's verdict with one regex. A qualifier list ending in the verdict was invisible to it, so a figure the judge called wrong did not cap correctness.

 Judge._VERDICT            unchanged, tried first: every claim main could read keeps its reading
+Judge._QUALIFIED_VERDICT  fallback, only for claims _VERDICT cannot read
+  - the verdict word ends the claim (only whitespace, "." and ")" may follow)
+  - it follows a separator (":" " -" "(") or a ",", directly or behind
+    single-token "/" qualifiers that start right at the separator
+  - "not" flips it; only intensifiers (partially, mostly, largely, completely,
+    totally, entirely) may qualify the word; other -ly adverbs leave it unread
+  - "right/wrong", "correct, wrong" (alternatives) stay unread
 Judge._is_wrong_figure(claim)
-  "one execution port: misleading/wrong"   -> unread   (no cap)
+  "one execution port: misleading/wrong"   -> wrong    (cap at 40)
-  "4.5x threshold: reasonable, not wrong"  -> unread
+  "4.5x threshold: reasonable, not wrong"  -> right
-  "8 ms: not right"                        -> unread
+  "8 ms: not right"                        -> wrong    (cap at 40; `not correct` too)
   "2x: correctly marked as WRONG to claim" -> unread   (unchanged, on purpose)
   "x: see docs/wrong"                      -> unread   (a path after a word is not a verdict)

A verdict word with no separator in front stays unread: there the judge is saying the answer flagged a bad claim, not that its figure is wrong. not right and not correct now cap correctness at 40.

Evidence

Old vs new Judge._is_wrong_figure (head 308a23e, the real method; main's reading is _VERDICT alone) over 6,424 real numeric_claims entries from Haiku single-judge runs on the perf golden set. Re-parse only, no LLM call (rerun.py):

change	n	entries
unread -> wrong	6	all <qualifier>/wrong (misleading/wrong x3, ambiguous/wrong x2, speculative/wrong)
unread -> right	6	conditional/right, reasonable/right, reasonable, not wrong, (not wrong) x2, …, correct
right <-> wrong	0	none

12 entries change in total. typically correct is no longer in the list: it stays unread.

• Before: test_a_wrong_figure_caps_correctness_at_forty[.../wrong] and the other qualified cases fail on main (correctness == 95, expected 40). After: pass. The unread cases (x: see docs/wrong, x: unverified, right/wrong, x: unverified, wrong, right, x: dubious, hardly wrong, correctly marked as WRONG to claim) are asserted through unread_claims == [0], so a read as right (also 95) cannot hide a regression. wrong -> corrected: right keeps reading as wrong.
• pytest -q: 5205 passed, 2 skipped, 4 failed in test_model_defaults.py (litellm price lock, unrelated to this change); make ci-local: all other jobs pass (ruff, mypy, schema, changed-line coverage). Fresh CI run pending on the PR.

Merge Danger

Door: two-way. One regex; revert restores the old reading.

Blast Radius: few scores

On the measured set, 6 of 6,424 entries now cap correctness to 40 where they did not. Affected judgements drop on the single judge, and on the panel once #55 lands (it shares the parser). 6 entries leave unread_claims as right. No prompt, schema or flag change. Judge prompts and the cache are untouched; the difference is applied when replies are read, so cached replies re-read with the new rule.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — N/A, documented rule ("a wrong figure caps at 40") unchanged
• No ad-hoc run artifacts committed

🤖 Generated with Claude Code

[vchamarthi]

Thanks, this is a nice targeted fix. The parametrized cases come straight from real Haiku output, and the must-not-cap cases (correctly marked as WRONG) pin down the boundary well. One thing on the new separators and a small comment tidy-up inline.

[vchamarthi]

Since / and , now count as separators anywhere in the string, a figure that itself contains one can win the leftmost match: left/right shift by 3: wrong and 4 KiB, correct alignment assumed: wrong both come out as right now, so the cap gets skipped where main capped them. Could we anchor the search after the first :/ -/( (or try the old pattern first and only fall back to / and ,)? A test for one of those would lock it in.

[napetrov]

The old _VERDICT is tried first so nothing main could read changes. The fallback only searches after the first separator. The cases you mentioned (left/right shift by 3: wrong, 4 KiB, correct alignment assumed: wrong) are pinned at test_eval_judge.py:279-280 and remain capped at 40. fa1d016 tries the old _VERDICT first, so these keep main's reading; e5b0dff additionally requires the fallback's verdict word to end the claim.

[napetrov]

No change on this branch: fa1d016 tries the old _VERDICT first and the fallback only searches after the first separator, so the left/right shift and 4 KiB claims stay capped at 40 (pinned at test_eval_judge.py:278-279).

[vchamarthi]

Nit: the old two-line comment just above still describes the colon/dash/paren-only version, so there are now two descriptions of _VERDICT back to back. Worth folding it into the new one?

[napetrov]

Folded the duplicate comment into the updated _QUALIFIED_VERDICT documentation. The comment now explains that the search starts at the first separator and the verdict word must end the claim. Commit e5b0dff.

[napetrov]

No further change: the duplicate description was folded into the _QUALIFIED_VERDICT comment in e5b0dff.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.31%. Comparing base (9753cc7) to head (7388adf).
⚠️ Report is 26 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #67      +/-   ##
==========================================
+ Coverage   90.06%   90.31%   +0.24%     
==========================================
  Files         205      206       +1     
  Lines       24193    24432     +239     
  Branches     3892     3944      +52     
==========================================
+ Hits        21790    22066     +276     
+ Misses       1724     1699      -25     
+ Partials      679      667      -12     

Files with missing lines	Coverage Δ
dokimi/dokimi/eval/judge.py	95.18% <100.00%> (+1.41%)	⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

The qualifier-list reading is right. One regression, matching the open review: / and , now count anywhere in the claim, so a figure that contains one wins the match. Over a 91-string corpus (single judge and panel, which share _cap_for_wrong_figures), left/right shift by 3: wrong, 4 KiB, correct alignment assumed: wrong and TILE 64 (not wrong): wrong capped on main and now read as right, so a wrong figure passes; note, wrong figure: right now caps; and not entirely wrong and not necessarily wrong become silently "right" instead of staying unread.

Pushed fa1d016: the old pattern is tried first, so nothing main could read changes; only for the rest / and , are searched after the first :, - or (; negated hedges stay unread; the duplicated comment is folded. Versus main that is 18 changes, all unread to read, with zero right/wrong flips; 7 new cases are red on this head. not wrong, wrong still reads as right and sort of right/wrong as wrong, as described.

[antonwolfy]

Review at fa1d016. The fix does what it says for the qualifier lists, and trying the old _VERDICT first means nothing main could read changes: over a 900-string corpus, including dict claims, there are 0 right/wrong flips and 0 claims that capped on main and no longer cap. The problem is the fallback. It starts at the first separator in the claim and then accepts a , or / anywhere after it, so a figure or a clause of prose that contains a separator can still supply the "verdict". Two inline comments on that; I'd fix the first before merge.

What I ran, on a Linux review host at fa1d016:

• make ci-local: 18 passed, 1 failed, 4 skipped. 4,318 tests passed and 1 skipped; ruff and mypy are clean; changed-line coverage is 100%. The one failure is schema_compat, which reports tool_preference.v1.json and tool_preference_groups.v1.json as removed. The PR does not touch them: the branch predates main's commit that added them. It will clear on a rebase.
• Old vs new _is_wrong_figure on the PR's examples and on the cases in the inline comments, with the LLM transport mocked.
• A mutation run on the new code: 44 mutants, 17 of which pass the whole of test_eval_judge.py. Details in the last inline comment.
• The new tests against main's judge.py: 8 fail (the three .../wrong caps, 8 ms: not right, the four is_read cases), so those pin the fix. The "must not cap" and "figure with / or ," groups already pass on main and only guard against regression.

Other points

1. not right and not correct now cap correctness at 40, and the description does not say so. 8 ms: not right was unread on main and is now wrong (test_eval_judge.py:285 pins it). That is a reasonable reading, but the Summary lists only not wrong flipping to right. Please add it to the description and to the "unread -> wrong" row of the evidence table, since that table says every such change is a <qualifier>/wrong entry.
2. The Summary diff block is stale against fa1d016. It shows / and , added to _VERDICT's separators. The code now keeps _VERDICT as it was and searches / and , only in a second pattern, after the first separator.

Checked, no issues

• Old readings kept: _VERDICT runs first, so x: not wrong - wrong, x: wrong -> right and x: incorrect/not wrong read as on main.
• Hedges: not entirely right, not really correct, neither right nor wrong and wrong, but right in context stay as on main.
• Dict claims: {"verdict": "misleading/wrong"} is wrong, {"verdict": "reasonable, not wrong"} is right.
• ReDoS: inputs up to 1.2 MB of letter runs, spaces and repeated :a, (, - finish in under 0.2 s.
• Repo rules: no schema change, no generated files, tests offline, no doc describes the verdict grammar. methodology.md still matches the cap and unread_claims.

[antonwolfy]

Review at e5b0dff. The end-anchor fixes what I raised at fa1d016: every claim from my two comments now reads as it should, and nothing main could read changes. One thing is worth fixing before merge: the anchored fallback still reads a trailing token as the verdict in a few cases (inline).

What I ran, on a Linux review host at e5b0dff:

• make ci-local: 19 passed, 0 failed, 4 skipped. 4,345 tests passed and 1 skipped; ruff and mypy are clean; changed-line coverage is 100% (13 lines). The schema_compat failure from fa1d016 is gone.
• Old vs new _is_wrong_figure over about 390,000 generated claims (separators in the figure, trailing punctuation and notes, dashes, dict claims), with the LLM transport mocked: 0 right/wrong flips, 0 claims that capped on main and no longer cap. The fallback only moves claims from unread to read.
• All the claims from my two earlier comments: the seven that must stay unread return None, the four misleading/wrong ones return True.
• A mutation run on the new code: 77 mutants, 51 killed. The survivors that matter are listed under the open test thread below.
• ReDoS: 28 adversarial inputs of about 1 MB (runs of separators, letters, spaces, commas) finish in 0.25 s or less.

Other points

1. Nit: the evidence in the description predates the end-anchor. The 6 unread -> wrong and 7 unread -> right figures were measured on the fa1d016 parser. e5b0dff can only move entries back to unread (misleading/wrong;, misleading/wrong (see note), reasonable, not wrong! are unread now). Please re-run the 6,424 entries and update the table.

Still open from my earlier review:

• dokimi/tests/test_eval_judge.py:365 (nit): most of the requested cases are in. Four mutants still pass the whole file: dropping .lower() on the negated branch (x: Not Right then reads as not wrong, 95 instead of 40), dropping . from the trailing [\s.)]* (x: misleading/wrong. goes unread), dropping re.I from _VERDICT (64 B: WRONG (expected 128) goes unread, because the fallback now reads it), and dropping the dashes from the fallback (x — not wrong; x — misleading/wrong is read through the /, so the new em-dash case does not test the dash). Add x: Not Right as 95/40, x: misleading/wrong. as 95/40, 64 B: WRONG (expected 128) as 95/40 and x — not wrong to the is-read list.

Checked, no issues

• Old readings kept: _VERDICT runs first, so x: wrong -> corrected: right, x: not wrong - wrong and x: wrong (see note) read as on main.
• Multi-line claims: no re.M, so $ does not match at a line break. Adding re.M fails TILE (default, wrong\non ARM): unverified, so the test pins it.
• Repo rules: no schema change, no generated files, tests offline, docs/methodology.md still matches the cap and unread_claims.

[napetrov]

On the review-body points:

• antonwolfy still-open tests (test_eval_judge.py:365): Added the four test cases from mutation coverage feedback: x: Not Right (95/40), x: misleading/wrong. (95/40), 64 B: WRONG (expected 128) (95/40), and x — not wrong to the is-read list. All four already behave correctly and their mutants fail when the relevant regex part is dropped. Commit 27580e1.

[antonwolfy]

Review at 58208b1. The e5b0dff problems are mostly fixed: x: see /wrong, x: unverified, right/wrong and x: dubious, hardly wrong stay unread, and nothing main could read changes. The slash case is still open, though, and the hedge handling and its tests have gaps. Two of the inline comments I'd fix before merge.

What I ran, on a Linux review host at 58208b1:

• make ci-local: 19 passed, 0 failed, 4 skipped. 4,354 tests passed and 1 skipped; ruff and mypy are clean; changed-line coverage is 100% (13 lines).
• Old vs new _is_wrong_figure over about 700,000 generated claims plus 250 hand-written ones (separators in the figure and in the verdict part, qualifiers, not, adverbs, hedges, dict claims, unicode dashes, multi-line, case), with the LLM transport mocked: 0 right/wrong flips, 0 claims that capped on main and no longer cap. Every change is unread -> read.
• A mutation run on the new code: 79 mutants, 25 pass the whole of test_eval_judge.py; 5 of those are equivalent. Details in the test comment below.
• The PR's test file against main's judge.py: 26 fail, so they pin the fix.
• ReDoS: 29 adversarial inputs of 100 KB to 1 MB finish in 0.2 s or less.

Still open from my earlier review:

• dokimi/dokimi/eval/judge.py:488: a / after any word character is still read as a verdict. x: see docs/wrong is True (capped to 40), x: unverified (see docs/wrong) and 64 B: unverified, see section 4.2/wrong the same, and x: see docs/correct, x: unverified (see docs/right) and x: unverified, matches the spec/correct read as right. All are None on main. The see /wrong fix does not cover a path whose directory is a word.
• dokimi/tests/test_eval_judge.py:278 and :296: read as on main and still do and Additional mutations: ... describe history, not behaviour.
• The description is not updated: the diff block still shows / and , added to _VERDICT; the evidence table was measured before the end-anchor; not right and not correct now cap at 40 (8 ms: not right) and the description does not say so.

Checked, no issues

• Old readings kept: _VERDICT runs first and is unchanged, so every claim main could read keeps its reading.
• Hedges after not: not entirely wrong, not hardly wrong, not necessarily right, not quite right stay unread.
• Lookbehinds on verdict words: x: unverified, right/wrong, right, wrong, wrong, wrong stay unread; x: incorrect/wrong reads wrong through _VERDICT.
• Search offset: search(text) and search(text, part.start()) give identical groups and spans on 148,000 generated and 400,000 random strings, so the comment that the offset changes nothing is correct.
• Repo rules: no schema change, no generated files, tests offline.

[xaleryb]

Re-ran our 91-string corpus at 2110cce for the single judge and the panel (both call _cap_for_wrong_figures): 16 claims change versus main, all unread to read, with no right/wrong flip, and sort of right/wrong and not wrong, wrong are now unread; our earlier cases stay red on the pre-fix parser.

The open slash case reproduces on this head: x: see docs/wrong and 64 B: unverified, see section 4.2/wrong now cap 95 to 40, and x: see docs/correct and x: cf. AVX/correct read as right, all unread on main, so a path-like tail decides a figure the judge did not judge. Separately, the description still shows / and , added to _VERDICT, and its evidence table was measured on the earlier parser (it lists typically correct as unread to right, which 2110cce pins as unread).

[antonwolfy]

Review at 45a61b8. The slash and hedge problems from my last review are fixed: x: see /wrong, x: see docs/wrong, x: unverified, right/wrong and x: dubious, hardly wrong stay unread, x: dubious, entirely wrong caps, and nothing main could read changes. One thing to fix before merge: the tests leave several branches of the new regex unpinned (inline, and the first item below).

What I ran, on a Linux review host at 45a61b8:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,100 tests passed and 1 skipped; ruff, mypy, schema checks and the changed-line coverage check pass.
• Old vs new _is_wrong_figure over 400,000 generated claims (separators, dashes, / and ,, not, adverbs, trailing punctuation, newlines, dict and non-string claims), with the LLM transport mocked: 0 right/wrong flips, 0 claims that capped on main and no longer cap. Every change is unread -> read.
• ReDoS: 36 adversarial inputs of about 1 MB finish in 0.35 s or less, and the time scales linearly.
• A mutation run on the new code: 100 mutants, 30 of which pass the whole of test_eval_judge.py; 5 of those are equivalent or artificial.
• The PR's tests against main's judge.py: 33 fail, so they pin the fix.

Still open from my earlier review:

• dokimi/tests/test_eval_judge.py:322: the alternatives that read as right (x: unverified, wrong/right, x: unverified, wrong, right) are asserted as 95/95, and a claim read as right also scores 95, so they do not pin "unread". Dropping (?<!wrong) before the comma turns x: unverified, wrong, right from None into False, and all 149 tests still pass. Put them in test_a_figure_with_a_slash_or_a_hedge_is_not_read_as_right, which checks unread_claims == [0].
• The description, as in my earlier nit and @xaleryb's comment: the diff block and the evidence table still describe an earlier parser. It also does not say that not right and not correct now cap at 40.

Checked, no issues

• Old readings kept: _VERDICT runs first and is unchanged, so every claim main could read keeps its reading.
• Hedges: not entirely wrong, not necessarily right, x: dubious, hardly wrong, x: unverified, possibly wrong stay unread; entirely, completely, partially after a comma cap.
• Dict and non-string claims: {"verdict": "misleading/wrong"} is wrong, {"verdict": "reasonable, not wrong"} is right, other types return None.
• Repo rules: no schema change, no generated files, tests offline, changelog fragment present, docs/methodology.md still matches the cap and unread_claims.

[napetrov]

On the review-body points:

• antonwolfy point 1 (test :322): Fixed in 308a23e: the alternatives and hedges (x: unverified, wrong/right, x: unverified, wrong, right and the rest) are now in the unread_claims test, so dropping (?<!wrong) fails it.

• antonwolfy point 2 (description): Rewrote the description for the current parser: _VERDICT unchanged plus the _QUALIFIED_VERDICT fallback, and it now says not right and not correct cap at 40. I re-ran the 6,424-claim comparison with the real _is_wrong_figure at 308a23e: 6 unread to wrong, 6 unread to right, 0 flips, and typically correct stays unread.

• xaleryb point 1 (slash case at 2110cce): Fixed after 2110cce: a / qualifier must start right at the separator, so see docs/wrong and see section 4.2/wrong stay unread, pinned in the unread test. x: see docs/correct and cf. AVX/correct are unread for the same reason.

• xaleryb point 2 (description): The description now shows _VERDICT unchanged plus the fallback, and the evidence table is the re-run on the current parser (typically correct is gone from it).

[antonwolfy]

Review at 1dee08b. Four of my five threads are fixed, the parser holds up, and nothing main could read changes. One thread stays open, and two more test gaps let real regressions through; I'd close them before merge.

What I ran, on a Linux review host at 1dee08b:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,210 tests passed and 1 skipped; ruff, mypy, schema checks and the changed-line coverage check pass.
• Old vs new _is_wrong_figure over about 63,000 generated claims plus dict claims, with the LLM transport mocked: 0 right/wrong flips, 0 claims that capped on main and no longer cap. Every change is unread -> read (2,244 to right, 2,244 to wrong).
• A mutation run on the new code: 105 mutants, 73 killed by test_eval_judge.py. The survivors that matter are below and in the two test comments; the rest are equivalent.
• ReDoS: inputs of up to 100,000 repetitions of separators, slashes, commas and not finish in under 0.2 s.

Still open from my earlier review:

• test_eval_judge.py:312: the six cases are in and the ( branch is pinned, but the dash and incorrect branches of _VERDICT still are not, and neither is right/correct. Every dash test ends at the verdict, so the fallback reads it when _VERDICT loses the branch. Mutants that pass all 156 tests: dropping the \s[-\u2013\u2014] alternative (64 B - wrong (expected 128) goes True -> None), dropping incorrect (64 B: incorrect (expected 128) the same), dropping right (64 B: right (expected 64) goes False -> None). Adding those three, plus 64 B — incorrect, expected 128, as cap rows would close it.

Checked, no issues

• Old readings kept: _VERDICT runs first and is unchanged, so every claim main could read keeps its reading.
• Unread cases: the unread_claims == [0] list now pins the alternatives, hedges, non-intensifier adverbs, prose and path tails; x: see docs/wrong, x: unverified, possibly wrong, x: unverified, right/wrong are all unread.
• Docs: docs/methodology.md still matches the cap and unread_claims; no schema change, no generated files, tests offline, changelog fragment present.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

• The fallback regex is now complex enough that future changes are error-prone (multiple… · New
• The docstring says the gate looks for - specifically, but the implementation matches any… · New

What changed in this PR

This PR fixes verdict parsing in Judge._is_wrong_figure() so the “wrong figure caps correctness at 40” rule also applies when verdicts are qualified (e.g., misleading/wrong) or negated (e.g., not right).

Changes:

• Add a fallback regex-based parser to read end-anchored qualified/negated verdicts when the primary _VERDICT pattern can’t.
• Expand unit tests to cover qualified verdict lists, negations, separators, and false-positive cases.
• Add a changelog entry describing the new parsing behavior and preserved non-matches.

File	Description
dokimi/​dokimi/​eval/​judge.py	Adds fallback verdict parsing (_QUALIFIED_VERDICT) and supporting logic for qualified/negated verdicts.
dokimi/​tests/​test_eval_judge.py	Adds extensive test cases for qualified/negated verdict parsing and non-verdict false positives.
dokimi/​changelog.d/​20261016_122711_napetrov_verdict-parser.md	Documents the bugfix and the intended parsing boundaries.

_{Review effort: Lite}

[napetrov]

Not changed: named groups and re.VERBOSE would be a refactor beyond this PR; the cap table tests already exercise each group position.

[Copilot]

🟡 Changes recommended

Qualified entirely wrong verdicts remain unread and bypass the correctness cap.

3 open issues

• Recognize “entirely wrong” as an intensifier for correctness capping · New
• The fallback regex is now complex enough that future changes are error-prone (multiple…
• The docstring says the gate looks for - specifically, but the implementation matches any…

_{Review effort: Balanced}

[antonwolfy]

Review at 0fb7df0. Three of the five threads are closed and the parser still reads everything main could read as before. Two things are worth fixing before merge: dropping entirely from the intensifiers brings back a missed cap, and the _VERDICT test gap from the last round is still open.

What I ran, on a Linux review host at 0fb7df0:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,216 tests passed and 1 skipped; ruff, mypy, schema checks and the changed-line coverage check pass (15 lines, 100%).
• Reproductions of _is_wrong_figure with the LLM transport mocked, on the claims below.
• A mutation run on the new code: about 75 mutants, 17 of which pass the whole of test_eval_judge.py. The ones that matter are below.

Other points

1. x: not completely wrong is unread only because of one untested condition. _is_wrong_figure returns None for a negated intensifier (judge.py:481, if adverb and (negated or ...)). The only negated-adverb test uses not entirely wrong, and entirely is no longer an intensifier, so the negated or part is not exercised. Mutation check: replacing the condition with adverb and adverb.lower() not in cls._INTENSIFIERS passes all 162 tests, and x: not completely wrong goes None -> False (read as right). Suggestion: add x: not completely wrong and x: not mostly wrong to the unread_claims == [0] list.

Still open from my earlier review:

• dokimi/tests/test_eval_judge.py:322: the dash, incorrect and right branches of _VERDICT are still not pinned, and the 0fb7df0 diff adds none of the four cases. Mutants that pass all 162 tests: dropping the \s[-\u2013\u2014] alternative (64 B - wrong (expected 128) and 64 B — incorrect, expected 128 go True -> None), dropping incorrect (64 B: incorrect (expected 128) the same), dropping right (64 B: right (expected 64) goes False -> None). Add those four as cap rows (95/40, 95/40, 95/95 for the right one).
• dokimi/tests/test_eval_judge.py:326: the .lower() on the adverb is no longer pinned. x: dubious, Entirely wrong was my case for it, and with entirely out of _INTENSIFIERS it expects 95/95, so it passes with or without .lower(). Dropping .lower() on adverb in _is_wrong_figure passes all 162 tests; x: dubious, MOSTLY wrong and x: dubious, Completely wrong go True -> None. Use one of those as the 95/40 row.

Checked, no issues

• Old readings kept: _VERDICT runs first and is unchanged, so every claim main could read keeps its reading.
• Mutants killed: every separator alternative of _QUALIFIED_VERDICT, the three lookbehinds, the $ anchor, the . and ) in the tail class, the path-qualifier lookahead words, re.I and each of the five intensifiers.
• Docs: docs/methodology.md still matches the cap and unread_claims; the changelog fragment does not list entirely and its examples match the code; no schema change, no generated files, tests offline.

[xaleryb]

Re-checked at 385a1d6 on Linux: full suite, ruff, mypy and bandit, earlier review cases against main, and mutants of the _VERDICT branches.

• Every reading from our earlier comments is as fixed, and our fa1d016 rows are still red on the pre-fix parser.
• Was open (from the 1dee08b/0fb7df0 review): the fallback now reads the plain dash, incorrect and right claims, so dropping any of those branches from _VERDICT passes all judge and panel tests (main's tests catch each one). With such a mutant, 64 B - wrong (expected 128) and 64 B: incorrect (expected 128) stop capping (True -> None), and 64 B: right (expected 64) becomes unread. Expected: each mutant fails a test.
• Pushed 16b1e70: 3 cap rows and 1 is-read row, each with text after the verdict. Each mutant is red with them; the suite, ruff, mypy and bandit are clean.

Nothing else is open.

[xaleryb]

Approved at 16b1e70. Every earlier item of ours and every reviewer thread is fixed or answered at this head; our commits pin what was left, each red without its code. Full suite, ruff and mypy clean on Linux; CI green.

[napetrov]

Checked at 16b1e70 (the fork head, your pin commit): pytest tests/test_eval_judge.py 170 passed, ruff clean. I also applied each of the three _VERDICT mutants (dash separator removed, incorrect removed, right removed) on a scratch copy: each one fails the judge tests (2, 2 and 1 failures). Nothing further from this item is open on our side.

[antonwolfy]

Review at 16b1e70. The last commits fix what I raised at 0fb7df0: entirely is an intensifier again, Entirely pins the .lower(), and the dash, incorrect and right branches of _VERDICT are pinned. The parser still reads everything main could read as before. What is left is the new negation rule and the tests around it, plus one comment line.

What I ran, on a Linux review host at 16b1e70:

• make ci-local: 19 passed, 0 failed, 4 skipped. 5,224 tests passed and 1 skipped; ruff, mypy and the other checks pass.
• Reproductions of _is_wrong_figure with the LLM transport mocked, on the claims in the inline comments.
• A mutation run on the new code: 32 mutants, 30 killed by test_eval_judge.py. The two survivors are in the inline comments. The three _VERDICT mutants from the last round (dash, incorrect, right) are now killed.
• The PR's test file against the base judge.py: 44 fail, so the tests pin the fix.

Checked, no issues

• Old readings kept: _VERDICT runs first and is unchanged, so every claim main could read keeps its reading.
• Intensifiers: x: dubious, entirely wrong, x: dubious, Entirely wrong and x: dubious, MOSTLY wrong cap at 40; x: dubious, entirely correct does not. Dropping .lower() on the adverb or removing any of the six intensifiers fails the file.
• Docs: docs/methodology.md still matches the cap and unread_claims; no schema change, no generated files, tests offline, changelog fragment present.

[antonwolfy]

not entirely correct and its siblings now cap correctness at 40, and not entirely wrong does not, so the two halves of the same hedge read opposite ways.

385a1d6 resolves negation before the intensifier. On this head Judge._is_wrong_figure returns:

• x: not entirely correct -> True, x: not partially correct -> True, x: not mostly correct -> True, x: not completely right -> True (cap to 40).
• x: not entirely wrong -> None, x: not completely wrong -> None.

All of the first group are None on main. "Not entirely correct" is the usual way to say a figure is mostly right with a flaw, the same as "not entirely wrong" says the reverse. A judge that writes 8 ms: not entirely correct for a figure that is close now loses 55 points of correctness. The comment at judge.py:456 states the rule but not why correct should cap and wrong should not, and the changelog fragment does not mention it.

Suggestion: either treat a negated intensified verdict as a hedge in both directions (unread; drop (['x: dubious, not entirely correct'], 95, 40) and not partially correct from the cap table and move them to the unread_claims == [0] list), or keep the cap and say in the comment and the fragment that not <intensifier> correct is wrong on purpose, with the reason.

[napetrov]

Took the first option: a negated intensifier now hedges in both directions. not entirely correct, not partially correct, not mostly correct and not completely right stay unread, same as not entirely wrong. A bare not correct / not right still caps at 40. Pinned in the unread test and noted in the changelog fragment. Fixed in 7388adf.

[antonwolfy]

The negated-hedge guard is pinned for entirely only.

Mutation check: replacing if adverb and negated and is_wrong_word: (judge.py:485) with if adverb and adverb.lower() == "entirely" and negated and is_wrong_word: passes all 170 tests. Then x: not completely wrong goes None -> False (read as right). x: not mostly wrong, x: not partially wrong, x: not largely wrong and x: not totally wrong go the same way. Only 8 ms: not entirely wrong (:283, :393) is tested, so a later change that makes the hedge per-adverb passes, and a partly wrong figure reads as right.

Suggestion: add x: not completely wrong, x: not mostly wrong and x: not partially wrong to the unread_claims == [0] list (the one with 8 ms: not entirely wrong).

[napetrov]

Added not completely wrong, not mostly wrong, not partially wrong, not largely wrong and not totally wrong to the unread list, plus the correct-side negated intensifiers. Each one now fails under the per-adverb mutant. Fixed in 7388adf.

[antonwolfy]

The correct branch of _VERDICT is still not pinned.

16b1e70 adds the right row here, but correct has none. Mutation check: dropping correct from _VERDICT (judge.py:438) passes all 170 tests, because the fallback still reads a bare x: correct. A verdict followed by text tells the two paths apart: x: correct, see note and x - correct (see note) return False on this head and None with the mutant, so they would land in unread_claims.

Suggestion: add x: correct, see note next to 64 B: right (expected 64) in the is-read list.

[napetrov]

Added x: correct, see note and x - correct (see note) to the is-read list, so dropping correct from _VERDICT now fails the suite. Fixed in 7388adf.

[antonwolfy]

Nit: this comment line is 138 characters (the project line length is 100; ruff does not select E501, so CI does not catch it). The re-wrap in 385a1d6 and b7aa61e left it merged. Wrap it after "the figure)." like the lines around it. _INTENSIFIERS at :467 is 102 characters and could go on three lines.

[napetrov]

Rewrapped the gate comment to 100 columns and split _INTENSIFIERS over three lines. Fixed in 7388adf.