Skip to content

feat(verifiers): mandatory criteria and a binary pass (mandatory_all_pass); a verdict that won't score an ungraded run - #755

Closed
sarah-chitty-1 wants to merge 1 commit into
mainfrom
sarahchitty/fd-6216-mandatory-verdict
Closed

sarah-chitty-1 wants to merge 1 commit into
mainfrom
sarahchitty/fd-6216-mandatory-verdict

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

PSI's always-on tasks grade with ML's rule (Linear FD-6216): a task passes only if every mandatory process criterion passes, plus its end-state checks. Each criterion is marked mandatory or optional by the task writer; optional ones are for analysis. Weights 3 (mandatory) / 1 (optional) feed a reported "rubric satisfaction" average, never pass/fail. One judge call grades all criteria.

Everything here is opt-in; tasks that don't set the new fields behave exactly as before.

What it adds

  • mandatory on a criterion (rubrics_verifier, so always_on_agent_verifier too): true|false, left out = optional. It's recorded on the result row and never shown to the judge, so it can't sway grading.
  • score_aggregator: "mandatory_all_pass": the step scores 1.0 when every mandatory criterion passed (by the output format's own pass rule), else 0.0. The entry also gets passed, mandatory_failed and rubric_satisfaction (weighted share passed; mandatory_weight default 3, optional_weight default 1). A task using it with no mandatory criterion is refused at load.
  • aggregate_verifiers, two opt-in settings:
    • use_verifier_scores: scores each verifier by its own score, instead of re-scoring all their rows together (with all_pass, an optional row would otherwise fail the verdict).
    • fail_task_when_ungraded: a named verifier that has no entry, failed, or was skipped makes the task error (so it's re-run or re-graded) instead of scoring it.
  • Skip entries for a broken assistant run (prompt_error, no screenshots) carry graded: false. This is additive; it only changes behaviour with fail_task_when_ungraded on.

Example

{"type": "always_on_agent_verifier", "verifier_id": "process", "score_aggregator": "mandatory_all_pass",
 "criteria": [{"id": "asked_before_paying", "mandatory": true, "criterion": "..."},
              {"id": "told_after_each_order", "mandatory": false, "criterion": "..."}]}
{"type": "aggregate_verifiers", "verifier_ids": ["end_state", "process"], "score_aggregator": "all_pass",
 "use_verifier_scores": true, "fail_task_when_ungraded": true}

Tests

29 new tests in tst/unit/task_step/verifiers/test_mandatory_verdict.py. They cover all-pass, a mandatory fail, an optional-only fail, the config refusal, weights, and the verdict's ungraded paths. tst/unit/task_step passes, apart from the 2 known test_mcp_cli_builder failures (missing click).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

Fix the failed-verifier check before merging so indirect dependencies cannot bypass fail_task_when_ungraded.

Fix All in CursorFindings

  1. P1 Failed verifiers still get scored ▶
Fix with agent prompt
### Issue 1
src/agent_env/task_step/task_steps/verifiers/aggregate_verifiers.py:174
`fail_task_when_ungraded` misses a failed named verifier when it is only an indirect dependency. A container verifier can record a reward score of 1, time out, and let later steps continue with `fail_task_on_error=False`. If another step sits between it and this aggregate, this check ignores its failure and accepts the passing score. Check failures for the named verifiers even when they are not direct dependencies.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds opt-in mandatory criteria and a binary rubric verdict.

  • Rubric runs pass when every mandatory criterion passes.
  • Combined verifier checks can use each verifier’s own score.
  • The combined verdict stops when a verifier did not grade the run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Criterion results] --> B[Mandatory criteria all pass?]
  A --> C[Weighted satisfaction for reporting]
  B --> D[Verifier score]
  E[End-state score] --> F[Aggregate verifier scores]
  D --> F
  F --> G{All named verifiers graded?}
  G -->|No| H[Raise error]
  G -->|Yes| I[Record verdict]
Loading

Reviews (1) · Last reviewed commit: "feat(verifiers): mandatory criteria, man..." · Reviewed by Greptile

…n verdict that won't score an ungraded run

rubrics_verifier (and always_on_agent_verifier) criteria take an optional
`mandatory: true|false`. A new score_aggregator, mandatory_all_pass, passes a
run only when every mandatory criterion passes; its entry also records
`passed`, `mandatory_failed` and `rubric_satisfaction` (weights 3/1, settable
as mandatory_weight / optional_weight), which never decides the pass. The
judge never sees the flag. A mandatory_all_pass step with no mandatory
criterion is a config problem, and is refused again at run time for a run's
own criteria.

aggregate_verifiers gains two opt-in settings: use_verifier_scores combines
the named verifiers' own scores (so a mandatory_all_pass judge combines with
end-state checks by its own rule), and fail_task_when_ungraded fails the step
with UngradedVerifierError when a named verifier has no entry, skipped the run
(`graded: false`, now marked on rubric and prompt-response skip entries) or
recorded no score, or a step it depends on failed. Defaults are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 18, 2026 01:00
for failed in context.metadata.get("failed_steps") or []:
if not isinstance(failed, dict) or failed.get("retried") or failed.get("step_id") == self.id:
continue
if depends is None or failed.get("step_id") in depends:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Failed verifiers still get scored

fail_task_when_ungraded misses a failed named verifier when it is only an indirect dependency. A container verifier can record a reward score of 1, time out, and let later steps continue with fail_task_on_error=False. If another step sits between it and this aggregate, this check ignores its failure and accepts the passing score. Check failures for the named verifiers even when they are not direct dependencies.

Knowledge Base Used: Task-step context and execution

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/aggregate_verifiers.py
Line: 174

Comment:
**Failed verifiers still get scored**

`fail_task_when_ungraded` misses a failed named verifier when it is only an indirect dependency. A container verifier can record a reward score of 1, time out, and let later steps continue with `fail_task_on_error=False`. If another step sits between it and this aggregate, this check ignores its failure and accepts the passing score. Check failures for the named verifiers even when they are not direct dependencies.

**Knowledge Base Used:** [Task-step context and execution](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%2Fknowledge-base%2Fscaleapi%2Fagentenv-framework%2F-%2Fdocs%2Ftask-step-execution.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant