Skip to content

feat(verifiers): cua_verifier, a computer-use task's rubric graded on what the agent left in the computer - #772

Open
williamloo-scale wants to merge 5 commits into
mainfrom
wloo/cua-verifier
Open

williamloo-scale wants to merge 5 commits into
mainfrom
wloo/cua-verifier

Conversation

@williamloo-scale

@williamloo-scale williamloo-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Adds cua_verifier, a verifier that grades a computer-use task's rubric on what the agent left in its computer, using rubrics_verifier's judge. It's stacked on #749, for find_computer.

What it does

cua_verifier is a RubricsVerifierTaskStep subclass.

  • Rubric: rubric is {"criteria": [...]} or the bare list. Each criterion has an id, a title, a weight (0 when left out) and annotations:
    • criteria_type / criteria_category containing must-pass makes it a gate; negative, or a weight below 0, makes it a failure mode, whose weight is a penalty; anything else is regular.
    • criteria_category: visual and target_result_files / target_GT_files are passed to the judge as visual and applies_to.
  • What the judge reads: what the agent left in the computer, read through its driver's computer-exec/v1 run, the control route the agent can't call.
    • result_files: paths, absolute or under the home. Each is read 512 KiB at a time (a short chunk is an error, never an empty file), within the 1 MiB a driver returns of a command's output; a file over 32 MiB is named as too large rather than read. Each is rendered as text by judge_utils/cua_files.py:
      • Text formats as they are, up to 40,000 characters each.
      • .xlsx: cells with their formulas and calculated values, up to 6,000 cells. Both workbooks are streamed (read_only), and the walk stops after visiting 200,000 cells, so a sparse sheet can't run away.
      • .docx: paragraphs with styles, plus tables. .pptx: each slide's shapes (including those inside groups), tables and notes. .pdf: each page's text.
      • Images and other binaries by size and sha256.
      • An Office file is measured as it decompresses and isn't opened if it expands past 128 MiB; a PDF is read up to 200 pages (pypdf>=6 bounds each stream); each file is rendered in its own process, capped at 2 GiB of address space and killed after 60 s.
    • A file that isn't there is listed as not produced, for the judge to treat as unmet.
    • expected_artifact_id: a file artifact universe of golden files, shown for reference.
    • state_commands: {name, command} pairs run in the computer, their output shown as environment state, for example a mock website's state file. Output over 40,000 characters, or cut by the driver, is marked incomplete, and the record says so.
    • instruction: the task, shown as stated requirements.
  • How to grade: guidance is the grading instructions and golden_note how to treat the golden files. Both are the caller's own text; without them the step uses short defaults (DEFAULT_GUIDANCE, DEFAULT_GOLDEN_NOTE).
  • No prompt_id: the judge grades the files alone. Its stand-in prompt response is the step's own (graded_response) and never joins the context, so a parallel or later step that reads the run's latest response still reads the agent's. With one, it also sees that prompt's response, and its trajectory with use_trajectory.
  • Score:
    • Each criterion is binary: anything short of 1 counts as 0.
    • The score is the weighted average of the regular and passed must-pass criteria, less each present failure mode's penalty, clamped to [0, 1].
    • Any failed must-pass makes it 0. With only gates and no positive weights, it's 1 minus the penalties.
    • The entry records gate_passed, and gates_failed when one failed.
    • Negative criteria use rubrics_verifier's convention (1 = the failure mode is absent) and keep their penalty as a negative weight.
  • Record: what was read goes in context.metadata["cua_verifier"][step_id]: each file's path, size, how it was rendered, and whether it was cut or missing; each golden file; and each state command's exit code.

Base changes:

  • RubricsVerifierTaskStep gains a graded_response hook (the prompt response it grades, by prompt_id by default), a score_rows hook, defaulting to score_aggregator's score, and _rubrics_fields, which from_dict builds on, so a subclass can read its own fields beside them.
  • computer_exec's _computer_problem is now public, as computer_problem.

Other changes:

  • New cua extra: openpyxl, python-docx, python-pptx and pypdf. It's part of dev, so CI tests the renderers. Without it, a file of those formats is named as not rendered.
  • uv.lock was regenerated with CI's uv 0.13.1; the change is additive only.
  • cua_verifier is registered, classified for episodes like rubrics_verifier, and listed as a rubrics verifier in the explorer.

Example

{"id": "grade", "type": "cua_verifier", "env_id": "desk",
 "instruction": "Write ~/Desktop/budget.csv ... and email the total to ana@example.com.",
 "result_files": ["~/Desktop/budget.csv", "~/notes.txt"],
 "state_commands": [{"name": "mailbox", "command": ["cat", "/tmp/mail_mock.json"]}],
 "rubric": {"criteria": [
   {"id": "csv", "title": "Provides a CSV file at ~/Desktop/budget.csv.", "weight": 5, "annotations": {"criteria_type": "MUST-PASS"}},
   {"id": "total", "title": "budget.csv has a total row of 1500.", "weight": 50},
   {"id": "wrong", "title": "Any file or email states a total other than 1500.", "weight": 20, "annotations": {"criteria_type": "NEGATIVE"}}]}}

Testing

  • Unit: tst/unit/task_step/verifiers/test_cua_verifier.py (28 tests) and test_cua_files.py (16 tests). The computer is faked at its driver's boundary and the judge is faked.
    • Rubric to criteria, for each kind and annotation, and the rubrics it refuses.
    • The score for every case: all met, a miss, a penalty, the clamp, a failed gate, a partial score made binary, gates alone.
    • A run whose prompt contains the xlsx with its formula, the text file, the missing file, the golden file, the state output, the instruction and the caller's guidance, with the score and the record checked; the default guidance when none is given.
    • A failed gate; the prompt response when prompt_id is given; the registry round-trip; bad configs.
    • Chunked reads, a dash-prefixed filename read whole, a short read refused, and a file over the limit named as too large; the files-only stand-in never in the context, even while the judge runs.
    • Each renderer, grouped slide text, a sparse workbook that stops at the visit limit, cutting at the limit, a library that isn't installed, and a file the library can't read.
    • tst/unit/task_step/verifiers and test_field_scopes: 917 pass.
  • E2E: tst/integration/computer/cua_verifier_test.py passes locally (36 s). It needs no model credentials: the judge is faked at its call boundary (_prompt_llm_judge), and everything else is real.
    • It deploys an ubuntu-24.04 computer with the reference driver, and computer_exec leaves a CSV, a note and a mock mailbox's state in it, as an agent would.
    • The files and state are read through the driver into the judge's prompt.
    • The fake judge meets each criterion unless it names a file the prompt lists as not produced. A rubric with a gate, two regular criteria and a failure mode scores 1.0; a second rubric, whose must-pass PDF isn't there, scores 0 with gates_failed: ["pdf"].

Found along the way

The two verifiers in the e2e run side by side. A run's stored context never gained gates_failed: []: build_context_update_ops emits nothing for a new key whose value is an empty list, so the key is dropped. The entry therefore writes gate_passed, and gates_failed only when non-empty. Any step that writes a new empty list is affected the same way; I haven't changed that here.

Not in this PR

  • Windows computers: files and state are read through sh and POSIX tools, so a Windows computer is refused at preflight and before any read.
  • Images: the judge doesn't see images or rendered pages. visual criteria are graded from the text rendering.
  • Agent judge: it gets the same inline evidence, not the raw files staged in its sandbox.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge.

Summary

Adds cua_verifier to grade files and state left in a computer. The latest update stops the file-reading child process when grading is cancelled.

  • The rubric judge scores files and state left on a computer.
  • Office files and PDFs are shown to the judge as text.

williamloo-scale deferred native Windows reads; Windows computers are explicitly refused. They also left the known empty-list persistence issue outside this PR.

Reviews (8) · Last reviewed commit: "fix(verifiers): a cancelled cua_verifier..." · Reviewed by Greptile

@williamloo-scale
williamloo-scale requested a review from a team as a code owner November 18, 2026 11:12
@socket-security

socket-security Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​pypdf@​7.1.099100100100100

View full report

williamloo-scale and others added 2 commits November 18, 2026 16:10
…ugh its driver

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… what the agent left in the computer

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@williamloo-scale williamloo-scale changed the title feat(verifiers): cua_verifier, a CUA task's rubric graded as the in-house CUA judge grades it (FD-5492) feat(verifiers): cua_verifier, a computer-use task's rubric graded on what the agent left in the computer Nov 19, 2026
@williamloo-scale

Copy link
Copy Markdown
Contributor Author

@greptileai review

@greptile-apps

This comment has been minimized.

…output, and refuses Windows computers

- An Office file is measured as it decompresses and named as not rendered when
  it expands past 128 MiB; a PDF is read up to 200 pages (pypdf>=6 bounds each
  stream); rendering runs off the event loop under a 60 s limit.
- State output over the limit, or cut by the driver, is marked incomplete in
  the prompt and the record.
- A Windows computer is refused at preflight and before any read: files and
  state are read through sh and POSIX tools.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale and others added 2 commits November 18, 2026 17:36
…limited process, and names a damaged one

A file is rendered in a child process with its address space capped at 2 GiB
and killed after 60 s, so a crafted document can't exhaust or block the worker.
The Office packages' expansion check runs inside render's error handling, so a
document with damaged compressed data is named as unreadable, not raised.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Base automatically changed from wloo/computer-exec-step to main November 19, 2026 19:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant