Skip to content

feat(task): agent-env task lint - #819

Open
earakely-scale wants to merge 11 commits into
mainfrom
edgararakelyan/fd-5924-g6-agent-env-task-lint
Open

earakely-scale wants to merge 11 commits into
mainfrom
edgararakelyan/fd-5924-g6-agent-env-task-lint

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What

agent-env task lint checks a task for what would make it run or grade wrong before any agent runs. It takes a bundle folder, one task file, or a stored task (--id).

Errors refuse the run. They are also checked before agent-env run, bundle register, task create, task validate and eval run:

  • A field its step doesn't read (task files only; storing a task drops unknown fields). Lint hands each step's from_dict a dict that records which keys it asks for. Any other key is reported with the nearest field it does read:
    step 'tests': verify_sandbox doesn't read 'min_passing_scor' (did you mean 'min_passing_score'?), so it would be ignored
  • A step's own config_problems. Three verifiers gain checks for configs that fail or misgrade today when they grade. Each field's type is checked before it's used:
    • agent_prompt_response: a criterion that isn't an object, or has an unknown type; needles that isn't a non-empty list of strings; pattern that isn't a non-empty string or doesn't compile; flags that isn't a list of re flag names.
    • verify_sandbox: a criterion that isn't an object, or whose type isn't a string; a probe whose paths isn't a non-empty list of strings; a probe_file_contains expected that isn't a string or doesn't compile; a bash_cmd that isn't a non-empty string.
    • run_container_unit_tests: a command that isn't a string; a timeout_sec that isn't a number, or is negative.
    • Any verifier: a verifier_id that isn't a string.
  • A headline that grades nothing: headline_verifiers names one of agent-env's own steps that neither grades nor writes a verifier_id.
  • An aggregate source no verifier in the task writes, when lint sees every key the task's steps write. This allows a rubrics verifier's trajectory_mistakes key when its output format writes one.

Warnings are printed and block nothing:

  • A step's config_warnings, a new TaskStep hook that returns [] by default:
    • verify_sandbox: a criterion type it doesn't check, which a run skips; probe_file_contains with no expected text.
    • run_container_unit_tests: an empty command with no reward_path, which passes without running anything; timeout_sec: 0, meaning no limit.
  • A headline naming a step off the base, which writes no named grade: a validator, a legacy verifier such as the sdk's cua_evaluate (until G3), or any outside step.
  • An aggregate source no verifier names, when the task holds a step lint can't see into. agent-env's validators write entries under keys of their own, and an outside step may write any, as the sdk's run_lighthouse_audit writes under its verification_id. The warning names those steps.
  • A store entity read with no version. Local ids (a bundle's own and installed bundles') are skipped, and bundle plans skip this rule because a plan records each version it reads and a registration pins it.
  • No verifier at all. This is skipped when the task holds an outside step, which may grade.
  • Several verifiers and no headline_verifiers. When an aggregate covers the others, the warning names it.

Example from a bundle folder:

$ agent-env task lint ./triage
tasks/ok.json: warning: 2 verifiers and no headline_verifiers, so every grade is top-line and anything that needs one number must name one; name the one that is its reward in headline_verifiers
task lint: 1 warning

How runs say findings, and the escape hatch:

  • Grouped output. agent-env run, bundle register and eval run say each kind of finding once, errors first, with how many tasks have it and the first three. agent-env task lint still lists every one. On real bundles, the cua-driver bench goes from 1,355 warning lines to 2, and Online-Mind2Web from 600 to 1.
  • --skip-lint on run, bundle register and eval run lifts lint's own rules (a field no step reads, a headline or aggregate source that grades nothing) and prints them instead, so a lint mistake can't block a run. A step's own config_problems still refuse, as they did before lint, because they fail the run anyway. A refusal that --skip-lint would lift says so.

Checked against real task files. I linted 4,327 task files from agentenv-benchmarks, the cua-driver bench, the airops and coopbench plugins, and the hb4 and AwsBench work. There were no crashes and no false positives. The one error is a genuinely broken draft that runs already refused.

Also fixed on the way, found by fuzzing junk values through lint:

  • A step whose id or type is a JSON array or object crashed agent-env run in the resolver with a TypeError, as it does on main. The shared task-file reader now refuses it: step #1's id is a JSON array; it must be a string. An integer id, which runs today, still does.
  • A verifier_id, command or verify_sandbox criterion of the wrong type is now a step-config error, where the new checks first crashed on it.

Flake fixed: codex_computer_use_driver_test.py::test_a_repl_left_idle_is_closed_without_another_call_and_its_session_starts_afresh failed in unit (2/2) on this PR. On a busy runner, the gap between its two calls passed the test's one-second idle window, so the sweep closed the REPL before the second call read it back. The window is now 60 s for the two calls and 0.2 s once they're done. It passes 10 of 10 runs with the CPU saturated.

Flake fixed: local host ports. The slow tier's browser_clock_test failed on this PR with Docker's Bind for 127.0.0.1:27175 failed: port is already allocated; main hit the same error in run_bundle_test on v0.9.1590's bump commit.

  • Cause: the local provider's host-port cursor lived in the state root. Each xdist worker has its own state root, so each had its own cursor, and two could hand out one port in the gap before Docker binds it.
  • Fix: the cursor is now one per user per machine, in /tmp/agentenv-<uid> (0700). It sits in the machine's /tmp, not TMPDIR, which a process may set for itself. It falls back to the state root's locks when that folder can't be made or isn't the user's alone.
  • Measured: 8 processes with their own state roots, drawing 200 ports each, drew 22 ports twice before the fix and none after. That still holds with each process given its own TMPDIR too.

Design notes

  • The code is in agent_env/task/lint.py. A Finding has a severity, a rule, a message and a step id. Output is one line per finding, errors first, then a count; lint exits 1 on errors. There is no JSON output because nothing consumes it.
  • Cross-step checks are lint rules, never task_problems, which refuse a stored task at load.
  • A bundle runs lint in plan() only, not in check(). bundle init exports leave user-scoped fields empty on purpose, and check() would refuse them.
  • task validate is still lint plus the store-reading preflight.
  • Lint is sure only about agent-env's own steps. A step on the VerifierTaskStep base writes under its verifier_id. Another of agent-env's own steps writes nothing unless it's a grader, whose keys are its own. An outside step off the base may write anything, so the rules that depend on what steps write fall back to warnings.
  • A task file on its own is linted without a store; a bundle folder (planned as a run plans it) and --id read the configured stores.
  • Out of scope, moved to other issues: not-regradable (G11), lopsided pooling (G5), judge-as-solver (G13), visible tests (G12).
  • config_warnings is a new non-abstract base method, so plugin-api reports no break.
  • The two shipped trigger examples (advance-to-next-event, next-env-event) grade nothing, so they warn no step grades this task. Every other shipped bundle lints clean, and a unit test keeps it that way.

Testing

  • tst/unit/task/lint_test.py covers each rule with a task that breaks it and one that passes, plus errors-first ordering. It also covers every shipped bundle with tasks linting clean, task lint on a folder, a task file in a bundle, a standalone file and --id, task validate, and eval run refusing before any run starts.
  • tst/unit/cli/test_task_create_preflight.py: task create refuses an unknown field even with --skip-validation, refuses a lint error, and prints a warning, persisting nothing.
  • tst/integration/cli/run_bundle_test.py: agent-env run refuses a misspelled field with no task or artifact written, and runs with the warning once the field is fixed.

e2e on a Linux admin VM, from a clean venv on local stores:

  • task lint over every shipped bundle with tasks: clean, except the two trigger examples' no step grades this task.
  • A misspelled field: task lint, agent-env run and bundle register each refuse it with the did-you-mean, and neither the task nor its registry id exists afterwards.
  • With the field fixed and a second verifier added, run and register print the no-headlines warning once each and go ahead. task lint --id and task validate warn and exit 0.
  • A standalone file reading a store artifact with no version: an unpinned warning, and task create saves it.
  • A headline naming a deploy step:
    • task create refuses it with nothing saved, and saves it under --skip-validation;
    • task lint --id and task validate then exit 1;
    • eval run of an eval holding it refuses before any run, with task instances at 0 before and after;
    • the eval without it runs and passes, with its warning printed.

Chaos (43 s on the VM). Every step in the repo's 40 task files (119 steps):

  • every key misspelled three ways: 909 cases, all caught (522 flagged with the right did-you-mean, 387 refused by the step's build);
  • every key, id and type included, set to 12 junk values, standalone and in a bundle: 6,492 cases, 0 tracebacks;
  • the same junk in every nested criterion field and in every field a step reads that the files leave out: 18,108 cases, 0 tracebacks;
  • 12 malformed files through the CLI, each refused with a message and exit 1, or linted with exit 0;
  • a 5,000-step task lints in 1.5 s, alone or as a bundle;
  • every step's to_dict round trip lints clean, so task get → edit → task create raises no false unknown-field errors.

Docs

The docs site should cover agent-env task lint (inputs, the rules and their severities, where errors refuse) and the config_warnings hook for step authors.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

This PR appears safe to merge.

Summary

Adds task lint checks before tasks are saved or run, with grouped warnings and a limited --skip-lint escape hatch.

  • Task lint checks task files and stored tasks before they run or get saved.
  • Verifier settings are checked before grading starts.
  • Task files reject step IDs and types that are JSON objects or arrays.
  • Local Docker workers share one host-port cursor per user and machine.
  • The REPL idle test gives its two calls more time before checking cleanup.

Reviews (11) · Last reviewed commit: "test(local): the cursor's fallback test ..." · Reviewed by Greptile

earakely-scale and others added 3 commits November 19, 2026 09:59
A task that can't run or grade correctly is refused before any agent runs: a field its step doesn't read, a step's
own config problems, a headline that grades nothing, and an aggregate source no verifier writes. What may be wrong
is a warning that blocks nothing: a step's config_warnings, a store entity read with no version, a task with no
verifier, or several verifiers and no headline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on them

A chaos run of junk values through task lint found a verifier_id, command or criterion of the wrong type crashing
the new checks, and an array or object step id or type crashing the bundle resolver, as it does agent-env run on
main. Each is now a problem the command names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The command checks, then writes, registering twice, and each pass said the plan's warnings. The check's result
carries them, so the command says them once after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 19, 2026 18:12
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 18:12
Comment thread src/agent_env/task/lint.py Resolved
Comment thread src/agent_env/task/lint.py Resolved
Comment thread src/agent_env/cli/task/lint.py Resolved
earakely-scale and others added 3 commits November 19, 2026 10:33
A missing aggregate source, a headline that grades nothing and a task with no verifier are now sure only for
agent-env's own steps: a validator writes its entry under a key of its own, and an outside step such as
agentenvhub-sdk's run_lighthouse_audit may write any, so with one in the task each is a warning. Each verifier's
config checks read a criterion's fields only once their types are right, and say what's wrong with the rest; and
task lint's help says which inputs read the stores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… done

On a busy runner the gap between the test's two calls passed the one-second idle window, so the sweep closed the
REPL before the second call read it back. The window is long for the two calls and short once they're done.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4-g6-agent-env-task-lint

# Conflicts:
#	AGENTS.md
earakely-scale and others added 2 commits November 19, 2026 12:42
…in turn

The host-port cursor lived in the state root, so processes with their own, as each xdist worker has, kept cursors
of their own and could hand out one port in the gap before docker binds it: docker then refused it as already
allocated (the slow tier's browser clock test on this PR, run_bundle_test on main). The cursor is now one per user
per machine, in a folder of the user's own in the temp folder: 8 processes drawing 200 ports each drew 22 twice
before and none after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t runs past lint's own rules

A bundle run, a registration and an eval run said every task's lint warnings, one line each: 1,355 lines for the
cua-driver bench, 600 for Online-Mind2Web. They now say each kind once, with how many tasks have it and the first
few, which takes those to two lines and one; agent-env task lint still lists each. --skip-lint on run, bundle
register and eval run lifts lint's own rules (a field no step reads, a headline or aggregate source that grades
nothing), saying them, so a lint mistake never blocks a run; a step's own config problems still refuse it, as
they did before lint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/agent_env/task/lint.py Resolved
earakely-scale and others added 2 commits November 19, 2026 15:03
A process may set TMPDIR for itself, so processes with their own would each keep a cursor, and runs sharing a state
root, which coordinated before, would stop. The cursor is now in the user's folder of the machine's /tmp, and in the
state root's locks when that folder can't be made or isn't the user's alone. The tasks a run names for a finding
before how many more is a named constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeBuild runs the suite as root, which writes a 0500 folder all the same; a file where /tmp would be stops anyone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant