feat(criteria): make async the primary criterion-checking surface

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 <!-- version list -->
 
+## v0.9.0 (2026-07-28)
+
+### Features
+
+- **criteria**: Async-primary BaseCriterion contract with CheckerMisuseError escalation
+  ([#60](https://github.com/UiPath/coder_eval/pull/60))
+
 ## v0.8.10 (2026-07-24)
 
 ### Bug Fixes

CLAUDE.md

@@ -46,7 +46,7 @@ coder_eval/
 │
 ├── criteria/                      # Criterion checker plugins (one file per type)
 │   ├── __init__.py                # CriterionRegistry with auto-discovery
-│   ├── base.py                    # BaseCriterion (incl. default aggregate()) + @handle_criterion_errors
+│   ├── base.py                    # BaseCriterion (async _check_impl_async is primary; sync _check_impl derives from it, or vice versa) + @handle_criterion_errors(_async)
 │   ├── _classification_aggregate.py  # Shared overlay: accuracy / P/R/F1 / confusion matrix
 │   ├── classification_match.py    # File-based label matcher
 │   ├── command_executed.py
@@ -141,7 +141,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -182,7 +182,7 @@ Per-task (single iteration; simulation mode runs a multi-turn dialog):
         agent.communicate with execute_with_retry, per-attempt
         turn_timeout, and on_attempt_error → preserves crashed=True
         partial TurnRecords on AgentCrashError / TurnTimeoutError)
-  2. SuccessChecker.check_all() → List[CriterionResult]
+  2. SuccessChecker.check_all_async() → List[CriterionResult]
 
 Cleanup: Stop agent, save EvaluationResult, generate reports
 ```

action.yml

@@ -33,7 +33,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.8.10"   # <-- kept in sync with releases by release.yml
+    default: "0.9.0"   # <-- kept in sync with releases by release.yml
   run-dir:
     description: Run directory (--run-dir)
     required: false

docs/EXTENDING.md

@@ -174,8 +174,35 @@ class MyChecker(BaseCriterion[MyCriterion]):
 
 Notes:
 
-- **Do not override `check()`** — it's final and wraps `_check_impl` with error
-  handling (an exception becomes a score-0.0 result with the error captured).
+- **Do not override `check()` / `check_async()`** — both are `@final` and wrap
+  `_check_impl` / `_check_impl_async` with error handling (an exception becomes
+  a score-0.0 result with the error captured; a `JudgeInfrastructureError`
+  escalates instead).
+- Implement exactly ONE of `_check_impl` (plain sync — the common case, shown
+  above) or `_check_impl_async` (genuine async I/O — an async HTTP client or
+  subprocess bridge; see `llm_judge`/`agent_judge`). Whichever you implement,
+  `BaseCriterion` derives the other for free (`asyncio.to_thread` / `asyncio.run`),
+  so there is no need to hand-maintain both. Overriding neither, or overriding
+  BOTH, raises `TypeError` immediately at class-definition time (a shared
+  abstract base for a family of checkers that intentionally implements neither
+  can opt out with the `abstract=True` class keyword — every one of ITS
+  subclasses is still checked normally). `SuccessChecker.check_all_async`
+  — the orchestrator's entry point — awaits every `_check_impl_async`-native
+  checker directly on the event loop instead of pinning a thread, and offloads
+  everything else to `asyncio.to_thread`. Criteria currently run SEQUENTIALLY
+  (strictly in declaration order, matching the sync `check_all`); running
+  multiple judge criteria concurrently is a follow-up.
+- If your checker overrides `_check_impl_async`, it MUST NOT do blocking work
+  (file I/O, subprocess calls) directly on the event loop — that would stall
+  the orchestrator's own loop for the duration of the call. Offload blocking
+  calls with `await asyncio.to_thread(...)` (see `llm_judge`/`agent_judge`,
+  which do this for their sandbox/reference file reads).
+- The derived sync bridge (`_check_impl`'s default `asyncio.run(...)` call) can
+  only run when no event loop is already running — calling the public sync
+  `check()`/`check_all()` on an async-only checker from inside a running loop
+  raises `CheckerMisuseError` (escalates, like `JudgeInfrastructureError`)
+  rather than returning a wrong score. Always reach for `check_async()` /
+  `check_all_async()` from async code.
 - Return `score` in `[0.0, 1.0]` — binary criteria use `0.0`/`1.0`; fractional ones
   anything in between.
 - For **suite-level metrics** on dataset-backed tasks, override

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.8.10"
+version = "0.9.0"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"

src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.8.10"
+__version__ = "0.9.0"

src/coder_eval/criteria/agent_judge.py

@@ -12,6 +12,7 @@
 
 from __future__ import annotations
 
+import asyncio
 import logging
 from typing import TYPE_CHECKING
 
@@ -96,7 +97,7 @@ class AgentJudgeChecker(BaseCriterion[AgentJudgeCriterion]):
 
     criterion_type = "agent_judge"
 
-    def _check_impl(
+    async def _check_impl_async(
         self,
         criterion: AgentJudgeCriterion,
         sandbox: Sandbox,
@@ -136,15 +137,22 @@ def _check_impl(
         # to the prompt envelope (fast verdict, narrow tool surface); when empty, the
         # judge inspects the sandbox copy via its tools. Both modes compose — the judge
         # can ``Read`` anything else even when files are pre-attached.
-        judge_ctx = JudgeContextBuilder(
-            files=criterion.files,
-            include_reference=criterion.include_reference,
-            include_agent_output=criterion.include_agent_output,
-            include_tool_calls=criterion.include_tool_calls,
-            include_dialog=criterion.include_dialog,
-            max_dialog_chars=criterion.max_dialog_chars,
-            max_file_chars=criterion.max_file_chars,
-        ).build(sandbox, reference_code, turn_records)
+        # .build() does synchronous file I/O — offload to a worker thread so it
+        # doesn't stall the event loop (see llm_judge.py's identical comment).
+        judge_ctx = await asyncio.to_thread(
+            JudgeContextBuilder(
+                files=criterion.files,
+                include_reference=criterion.include_reference,
+                include_agent_output=criterion.include_agent_output,
+                include_tool_calls=criterion.include_tool_calls,
+                include_dialog=criterion.include_dialog,
+                max_dialog_chars=criterion.max_dialog_chars,
+                max_file_chars=criterion.max_file_chars,
+            ).build,
+            sandbox,
+            reference_code,
+            turn_records,
+        )
 
         # Mount the reference directory only when the criterion opted into seeing it.
         # When include_reference=False the judge MUST NOT see the grading material, so
@@ -175,7 +183,7 @@ def _check_impl(
         )
 
         try:
-            turn = runner.run(
+            turn = await runner.run_async(
                 user_msg,
                 max_turns=criterion.max_turns,
                 turn_timeout=float(criterion.turn_timeout),