Repository navigation
feat(verifiers): mandatory criteria and a binary pass (mandatory_all_pass); a verdict that won't score an ungraded run - #755
Closed
sarah-chitty-1 wants to merge 1 commit into
Closed
sarah-chitty-1 wants to merge 1 commit into
sarah-chitty-1 wants to merge 1 commit into
Conversation
…n verdict that won't score an ungraded run rubrics_verifier (and always_on_agent_verifier) criteria take an optional `mandatory: true|false`. A new score_aggregator, mandatory_all_pass, passes a run only when every mandatory criterion passes; its entry also records `passed`, `mandatory_failed` and `rubric_satisfaction` (weights 3/1, settable as mandatory_weight / optional_weight), which never decides the pass. The judge never sees the flag. A mandatory_all_pass step with no mandatory criterion is a config problem, and is refused again at run time for a run's own criteria. aggregate_verifiers gains two opt-in settings: use_verifier_scores combines the named verifiers' own scores (so a mandatory_all_pass judge combines with end-state checks by its own rule), and fail_task_when_ungraded fails the step with UngradedVerifierError when a named verifier has no entry, skipped the run (`graded: false`, now marked on rubric and prompt-response skip entries) or recorded no score, or a step it depends on failed. Defaults are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| for failed in context.metadata.get("failed_steps") or []: | ||
| if not isinstance(failed, dict) or failed.get("retried") or failed.get("step_id") == self.id: | ||
| continue | ||
| if depends is None or failed.get("step_id") in depends: |
There was a problem hiding this comment.
Failed verifiers still get scored
fail_task_when_ungraded misses a failed named verifier when it is only an indirect dependency. A container verifier can record a reward score of 1, time out, and let later steps continue with fail_task_on_error=False. If another step sits between it and this aggregate, this check ignores its failure and accepts the passing score. Check failures for the named verifiers even when they are not direct dependencies.
Knowledge Base Used: Task-step context and execution
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/aggregate_verifiers.py
Line: 174
Comment:
**Failed verifiers still get scored**
`fail_task_when_ungraded` misses a failed named verifier when it is only an indirect dependency. A container verifier can record a reward score of 1, time out, and let later steps continue with `fail_task_on_error=False`. If another step sits between it and this aggregate, this check ignores its failure and accepts the passing score. Check failures for the named verifiers even when they are not direct dependencies.
**Knowledge Base Used:** [Task-step context and execution](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%2Fknowledge-base%2Fscaleapi%2Fagentenv-framework%2F-%2Fdocs%2Ftask-step-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
PSI's always-on tasks grade with ML's rule (Linear FD-6216): a task passes only if every mandatory process criterion passes, plus its end-state checks. Each criterion is marked mandatory or optional by the task writer; optional ones are for analysis. Weights 3 (mandatory) / 1 (optional) feed a reported "rubric satisfaction" average, never pass/fail. One judge call grades all criteria.
Everything here is opt-in; tasks that don't set the new fields behave exactly as before.
What it adds
mandatoryon a criterion (rubrics_verifier, soalways_on_agent_verifiertoo):true|false, left out = optional. It's recorded on the result row and never shown to the judge, so it can't sway grading.score_aggregator: "mandatory_all_pass": the step scores 1.0 when every mandatory criterion passed (by the output format's own pass rule), else 0.0. The entry also getspassed,mandatory_failedandrubric_satisfaction(weighted share passed;mandatory_weightdefault 3,optional_weightdefault 1). A task using it with no mandatory criterion is refused at load.aggregate_verifiers, two opt-in settings:use_verifier_scores: scores each verifier by its own score, instead of re-scoring all their rows together (withall_pass, an optional row would otherwise fail the verdict).fail_task_when_ungraded: a named verifier that has no entry, failed, or was skipped makes the task error (so it's re-run or re-graded) instead of scoring it.prompt_error, no screenshots) carrygraded: false. This is additive; it only changes behaviour withfail_task_when_ungradedon.Example
{"type": "always_on_agent_verifier", "verifier_id": "process", "score_aggregator": "mandatory_all_pass", "criteria": [{"id": "asked_before_paying", "mandatory": true, "criterion": "..."}, {"id": "told_after_each_order", "mandatory": false, "criterion": "..."}]} {"type": "aggregate_verifiers", "verifier_ids": ["end_state", "process"], "score_aggregator": "all_pass", "use_verifier_scores": true, "fail_task_when_ungraded": true}Tests
29 new tests in
tst/unit/task_step/verifiers/test_mandatory_verdict.py. They cover all-pass, a mandatory fail, an optional-only fail, the config refusal, weights, and the verdict's ungraded paths.tst/unit/task_steppasses, apart from the 2 knowntest_mcp_cli_builderfailures (missingclick).🤖 Generated with Claude Code
Fix the failed-verifier check before merging so indirect dependencies cannot bypass
fail_task_when_ungraded.Fix with agent prompt
Summary
Adds opt-in mandatory criteria and a binary rubric verdict.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Criterion results] --> B[Mandatory criteria all pass?] A --> C[Weighted satisfaction for reporting] B --> D[Verifier score] E[End-state score] --> F[Aggregate verifier scores] D --> F F --> G{All named verifiers graded?} G -->|No| H[Raise error] G -->|Yes| I[Record verdict]Reviews (1) · Last reviewed commit: "feat(verifiers): mandatory criteria, man..." · Reviewed by Greptile