Repository navigation
feat(verifiers): cua_verifier, a computer-use task's rubric graded on what the agent left in the computer - #772
Open
williamloo-scale wants to merge 5 commits into
Open
williamloo-scale wants to merge 5 commits into
williamloo-scale wants to merge 5 commits into
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…ugh its driver Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… what the agent left in the computer Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale
force-pushed
the
wloo/cua-verifier
branch
from
November 19, 2026 00:11
323de44 to
6bae0c0
Compare
williamloo-scale
force-pushed
the
wloo/computer-exec-step
branch
from
November 19, 2026 00:11
c1759be to
3d3ffd8
Compare
Contributor
Author
|
@greptileai review |
This comment has been minimized.
This comment has been minimized.
…output, and refuses Windows computers - An Office file is measured as it decompresses and named as not rendered when it expands past 128 MiB; a PDF is read up to 200 pages (pypdf>=6 bounds each stream); rendering runs off the event loop under a 60 s limit. - State output over the limit, or cut by the driver, is marked incomplete in the prompt and the record. - A Windows computer is refused at preflight and before any read: files and state are read through sh and POSIX tools. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…limited process, and names a damaged one A file is rendered in a child process with its address space capped at 2 GiB and killed after 60 s, so a crafted document can't exhaust or block the worker. The Office packages' expansion check runs inside render's error handling, so a document with damaged compressed data is named as unreadable, not raised. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
cua_verifier, a verifier that grades a computer-use task's rubric on what the agent left in its computer, usingrubrics_verifier's judge. It's stacked on #749, forfind_computer.What it does
cua_verifieris aRubricsVerifierTaskStepsubclass.rubricis{"criteria": [...]}or the bare list. Each criterion has anid, atitle, aweight(0 when left out) andannotations:criteria_type/criteria_categorycontainingmust-passmakes it a gate;negative, or a weight below 0, makes it a failure mode, whose weight is a penalty; anything else is regular.criteria_category: visualandtarget_result_files/target_GT_filesare passed to the judge asvisualandapplies_to.run, the control route the agent can't call.result_files: paths, absolute or under the home. Each is read 512 KiB at a time (a short chunk is an error, never an empty file), within the 1 MiB a driver returns of a command's output; a file over 32 MiB is named as too large rather than read. Each is rendered as text byjudge_utils/cua_files.py:.xlsx: cells with their formulas and calculated values, up to 6,000 cells. Both workbooks are streamed (read_only), and the walk stops after visiting 200,000 cells, so a sparse sheet can't run away..docx: paragraphs with styles, plus tables..pptx: each slide's shapes (including those inside groups), tables and notes..pdf: each page's text.pypdf>=6bounds each stream); each file is rendered in its own process, capped at 2 GiB of address space and killed after 60 s.expected_artifact_id: a file artifact universe of golden files, shown for reference.state_commands:{name, command}pairs run in the computer, their output shown as environment state, for example a mock website's state file. Output over 40,000 characters, or cut by the driver, is marked incomplete, and the record says so.instruction: the task, shown as stated requirements.guidanceis the grading instructions andgolden_notehow to treat the golden files. Both are the caller's own text; without them the step uses short defaults (DEFAULT_GUIDANCE,DEFAULT_GOLDEN_NOTE).prompt_id: the judge grades the files alone. Its stand-in prompt response is the step's own (graded_response) and never joins the context, so a parallel or later step that reads the run's latest response still reads the agent's. With one, it also sees that prompt's response, and its trajectory withuse_trajectory.gate_passed, andgates_failedwhen one failed.rubrics_verifier's convention (1 = the failure mode is absent) and keep their penalty as a negative weight.context.metadata["cua_verifier"][step_id]: each file's path, size, how it was rendered, and whether it was cut or missing; each golden file; and each state command's exit code.Base changes:
RubricsVerifierTaskStepgains agraded_responsehook (the prompt response it grades, byprompt_idby default), ascore_rowshook, defaulting toscore_aggregator's score, and_rubrics_fields, whichfrom_dictbuilds on, so a subclass can read its own fields beside them.computer_exec's_computer_problemis now public, ascomputer_problem.Other changes:
cuaextra: openpyxl, python-docx, python-pptx and pypdf. It's part ofdev, so CI tests the renderers. Without it, a file of those formats is named as not rendered.uv.lockwas regenerated with CI's uv 0.13.1; the change is additive only.cua_verifieris registered, classified for episodes likerubrics_verifier, and listed as a rubrics verifier in the explorer.Example
{"id": "grade", "type": "cua_verifier", "env_id": "desk", "instruction": "Write ~/Desktop/budget.csv ... and email the total to ana@example.com.", "result_files": ["~/Desktop/budget.csv", "~/notes.txt"], "state_commands": [{"name": "mailbox", "command": ["cat", "/tmp/mail_mock.json"]}], "rubric": {"criteria": [ {"id": "csv", "title": "Provides a CSV file at ~/Desktop/budget.csv.", "weight": 5, "annotations": {"criteria_type": "MUST-PASS"}}, {"id": "total", "title": "budget.csv has a total row of 1500.", "weight": 50}, {"id": "wrong", "title": "Any file or email states a total other than 1500.", "weight": 20, "annotations": {"criteria_type": "NEGATIVE"}}]}}Testing
tst/unit/task_step/verifiers/test_cua_verifier.py(28 tests) andtest_cua_files.py(16 tests). The computer is faked at its driver's boundary and the judge is faked.prompt_idis given; the registry round-trip; bad configs.tst/unit/task_step/verifiersandtest_field_scopes: 917 pass.tst/integration/computer/cua_verifier_test.pypasses locally (36 s). It needs no model credentials: the judge is faked at its call boundary (_prompt_llm_judge), and everything else is real.ubuntu-24.04computer with the reference driver, andcomputer_execleaves a CSV, a note and a mock mailbox's state in it, as an agent would.gates_failed: ["pdf"].Found along the way
The two verifiers in the e2e run side by side. A run's stored context never gained
gates_failed: []:build_context_update_opsemits nothing for a new key whose value is an empty list, so the key is dropped. The entry therefore writesgate_passed, andgates_failedonly when non-empty. Any step that writes a new empty list is affected the same way; I haven't changed that here.Not in this PR
shand POSIX tools, so a Windows computer is refused at preflight and before any read.visualcriteria are graded from the text rendering.🤖 Generated with Claude Code


Confidence Score: 5/5
The reviewed changes appear safe to merge.Summary
Adds
cua_verifierto grade files and state left in a computer. The latest update stops the file-reading child process when grading is cancelled.williamloo-scale deferred native Windows reads; Windows computers are explicitly refused. They also left the known empty-list persistence issue outside this PR.
Reviews (8) · Last reviewed commit: "fix(verifiers): a cancelled cua_verifier..." · Reviewed by Greptile