Repository navigation
feat(task): agent-env task lint - #819
Open
earakely-scale wants to merge 11 commits into
Open
earakely-scale wants to merge 11 commits into
earakely-scale wants to merge 11 commits into
Conversation
A task that can't run or grade correctly is refused before any agent runs: a field its step doesn't read, a step's own config problems, a headline that grades nothing, and an aggregate source no verifier writes. What may be wrong is a warning that blocks nothing: a step's config_warnings, a store entity read with no version, a task with no verifier, or several verifiers and no headline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on them A chaos run of junk values through task lint found a verifier_id, command or criterion of the wrong type crashing the new checks, and an array or object step id or type crashing the bundle resolver, as it does agent-env run on main. Each is now a problem the command names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The command checks, then writes, registering twice, and each pass said the plan's warnings. The check's result carries them, so the command says them once after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
marked this pull request as ready for review
November 19, 2026 18:12
A missing aggregate source, a headline that grades nothing and a task with no verifier are now sure only for agent-env's own steps: a validator writes its entry under a key of its own, and an outside step such as agentenvhub-sdk's run_lighthouse_audit may write any, so with one in the task each is a warning. Each verifier's config checks read a criterion's fields only once their types are right, and say what's wrong with the rest; and task lint's help says which inputs read the stores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… done On a busy runner the gap between the test's two calls passed the one-second idle window, so the sweep closed the REPL before the second call read it back. The window is long for the two calls and short once they're done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4-g6-agent-env-task-lint # Conflicts: # AGENTS.md
…in turn The host-port cursor lived in the state root, so processes with their own, as each xdist worker has, kept cursors of their own and could hand out one port in the gap before docker binds it: docker then refused it as already allocated (the slow tier's browser clock test on this PR, run_bundle_test on main). The cursor is now one per user per machine, in a folder of the user's own in the temp folder: 8 processes drawing 200 ports each drew 22 twice before and none after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t runs past lint's own rules A bundle run, a registration and an eval run said every task's lint warnings, one line each: 1,355 lines for the cua-driver bench, 600 for Online-Mind2Web. They now say each kind once, with how many tasks have it and the first few, which takes those to two lines and one; agent-env task lint still lists each. --skip-lint on run, bundle register and eval run lifts lint's own rules (a field no step reads, a headline or aggregate source that grades nothing), saying them, so a lint mistake never blocks a run; a step's own config problems still refuse it, as they did before lint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A process may set TMPDIR for itself, so processes with their own would each keep a cursor, and runs sharing a state root, which coordinated before, would stop. The cursor is now in the user's folder of the machine's /tmp, and in the state root's locks when that folder can't be made or isn't the user's alone. The tasks a run names for a finding before how many more is a named constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeBuild runs the suite as root, which writes a 0500 folder all the same; a file where /tmp would be stops anyone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
agent-env task lintchecks a task for what would make it run or grade wrong before any agent runs. It takes a bundle folder, one task file, or a stored task (--id).Errors refuse the run. They are also checked before
agent-env run,bundle register,task create,task validateandeval run:from_dicta dict that records which keys it asks for. Any other key is reported with the nearest field it does read:step 'tests': verify_sandbox doesn't read 'min_passing_scor' (did you mean 'min_passing_score'?), so it would be ignoredconfig_problems. Three verifiers gain checks for configs that fail or misgrade today when they grade. Each field's type is checked before it's used:agent_prompt_response: a criterion that isn't an object, or has an unknown type;needlesthat isn't a non-empty list of strings;patternthat isn't a non-empty string or doesn't compile;flagsthat isn't a list ofreflag names.verify_sandbox: a criterion that isn't an object, or whose type isn't a string; a probe whosepathsisn't a non-empty list of strings; aprobe_file_containsexpectedthat isn't a string or doesn't compile; abash_cmdthat isn't a non-empty string.run_container_unit_tests: acommandthat isn't a string; atimeout_secthat isn't a number, or is negative.verifier_idthat isn't a string.headline_verifiersnames one of agent-env's own steps that neither grades nor writes averifier_id.trajectory_mistakeskey when its output format writes one.Warnings are printed and block nothing:
config_warnings, a newTaskStephook that returns[]by default:verify_sandbox: a criterion type it doesn't check, which a run skips;probe_file_containswith no expected text.run_container_unit_tests: an empty command with noreward_path, which passes without running anything;timeout_sec: 0, meaning no limit.cua_evaluate(until G3), or any outside step.run_lighthouse_auditwrites under itsverification_id. The warning names those steps.headline_verifiers. When an aggregate covers the others, the warning names it.Example from a bundle folder:
How runs say findings, and the escape hatch:
agent-env run,bundle registerandeval runsay each kind of finding once, errors first, with how many tasks have it and the first three.agent-env task lintstill lists every one. On real bundles, the cua-driver bench goes from 1,355 warning lines to 2, and Online-Mind2Web from 600 to 1.--skip-lintonrun,bundle registerandeval runlifts lint's own rules (a field no step reads, a headline or aggregate source that grades nothing) and prints them instead, so a lint mistake can't block a run. A step's ownconfig_problemsstill refuse, as they did before lint, because they fail the run anyway. A refusal that--skip-lintwould lift says so.Checked against real task files. I linted 4,327 task files from agentenv-benchmarks, the cua-driver bench, the airops and coopbench plugins, and the hb4 and AwsBench work. There were no crashes and no false positives. The one error is a genuinely broken draft that runs already refused.
Also fixed on the way, found by fuzzing junk values through lint:
idortypeis a JSON array or object crashedagent-env runin the resolver with aTypeError, as it does onmain. The shared task-file reader now refuses it:step #1's id is a JSON array; it must be a string. An integer id, which runs today, still does.verifier_id,commandorverify_sandboxcriterion of the wrong type is now a step-config error, where the new checks first crashed on it.Flake fixed:
codex_computer_use_driver_test.py::test_a_repl_left_idle_is_closed_without_another_call_and_its_session_starts_afreshfailed inunit (2/2)on this PR. On a busy runner, the gap between its two calls passed the test's one-second idle window, so the sweep closed the REPL before the second call read it back. The window is now 60 s for the two calls and 0.2 s once they're done. It passes 10 of 10 runs with the CPU saturated.Flake fixed: local host ports. The slow tier's
browser_clock_testfailed on this PR with Docker'sBind for 127.0.0.1:27175 failed: port is already allocated;mainhit the same error inrun_bundle_teston v0.9.1590's bump commit./tmp/agentenv-<uid>(0700). It sits in the machine's/tmp, notTMPDIR, which a process may set for itself. It falls back to the state root's locks when that folder can't be made or isn't the user's alone.TMPDIRtoo.Design notes
agent_env/task/lint.py. AFindinghas a severity, a rule, a message and a step id. Output is one line per finding, errors first, then a count; lint exits 1 on errors. There is no JSON output because nothing consumes it.task_problems, which refuse a stored task at load.plan()only, not incheck().bundle initexports leave user-scoped fields empty on purpose, andcheck()would refuse them.task validateis still lint plus the store-reading preflight.verifier_id. Another of agent-env's own steps writes nothing unless it's a grader, whose keys are its own. An outside step off the base may write anything, so the rules that depend on what steps write fall back to warnings.--idread the configured stores.config_warningsis a new non-abstract base method, soplugin-apireports no break.advance-to-next-event,next-env-event) grade nothing, so they warnno step grades this task. Every other shipped bundle lints clean, and a unit test keeps it that way.Testing
tst/unit/task/lint_test.pycovers each rule with a task that breaks it and one that passes, plus errors-first ordering. It also covers every shipped bundle with tasks linting clean,task linton a folder, a task file in a bundle, a standalone file and--id,task validate, andeval runrefusing before any run starts.tst/unit/cli/test_task_create_preflight.py:task createrefuses an unknown field even with--skip-validation, refuses a lint error, and prints a warning, persisting nothing.tst/integration/cli/run_bundle_test.py:agent-env runrefuses a misspelled field with no task or artifact written, and runs with the warning once the field is fixed.e2e on a Linux admin VM, from a clean venv on local stores:
task lintover every shipped bundle with tasks: clean, except the two trigger examples'no step grades this task.task lint,agent-env runandbundle registereach refuse it with the did-you-mean, and neither the task nor its registry id exists afterwards.runandregisterprint the no-headlines warning once each and go ahead.task lint --idandtask validatewarn and exit 0.task createsaves it.task createrefuses it with nothing saved, and saves it under--skip-validation;task lint --idandtask validatethen exit 1;eval runof an eval holding it refuses before any run, with task instances at 0 before and after;Chaos (43 s on the VM). Every step in the repo's 40 task files (119 steps):
idandtypeincluded, set to 12 junk values, standalone and in a bundle: 6,492 cases, 0 tracebacks;to_dictround trip lints clean, sotask get→ edit →task createraises no false unknown-field errors.Docs
The docs site should cover
agent-env task lint(inputs, the rules and their severities, where errors refuse) and theconfig_warningshook for step authors.🤖 Generated with Claude Code


Confidence Score: 5/5
This PR appears safe to merge.Summary
Adds task lint checks before tasks are saved or run, with grouped warnings and a limited
--skip-lintescape hatch.Reviews (11) · Last reviewed commit: "test(local): the cursor's fallback test ..." · Reviewed by Greptile