Repository navigation
Record each failed step's role and cause in failed_steps - #692
Conversation
bf74480 to
26cbf52
Compare
A step that fails with fail_task_on_error=False lets its run complete, and the failed_steps entry is the only record of what the run lost. The entry now says which role the step played (grader, evidence, agent, other) and why it failed (key_budget, key_auth, transient, unknown). - StepRole on TaskStep: verifiers declare grader, snapshot and collect steps evidence, prompt_agent agent. A step overrides it in config with step_role, so a run_code step can say it grades. - task/outcome.py: FailureCause, classify_failure, FailedStep. The DAG runner builds its entry with FailedStep.from_exception. - attach_retry_config becomes attach_base_fields and hydrates step_role too. No behavior changes. Entries keep every existing key.
26cbf52 to
28de586
Compare
… errors as transient EnvOutcomeVerifierTaskStep.put passed step_role and retry_config to a constructor that rejects them; it now stores through TaskStep.put. An episode's prompt copy with a turn timeout kept no configured role. A connection failure whose message matches no pattern, such as httpx's 'All connection attempts failed', now reads as transient.
kaihackney-scale
left a comment
There was a problem hiding this comment.
Nice change. The role on the step type, step_role rather than role, writing it only when overridden, and refusing an unknown value at load all look right. Merging as-is changes no behavior. My concerns are about cause, since the next PR derives run quality from it.
I ran a few realistic messages through classify_failure on a91f0ad:
| message | got | expected |
|---|---|---|
tool get_account returned HTTP 401 Unauthorized from env MCP server |
key_auth |
unknown (the env's 401, not our key) |
rubric row 401 has no criteria |
key_auth |
unknown |
invalid token in template: {{seed} |
key_auth |
unknown |
expected status 500 to not be returned by create_order |
transient |
unknown |
…judge said: user exceeded the budget for travel |
key_budget |
unknown |
ValueError("rubric item 3 has no criteria") raised inside except TimeoutError |
transient |
unknown |
1. The key patterns match env and agent text, not just the proxy's. Error strings carry env HTTP statuses, tool output, verifier stdout and model text, so a bare \b401\b, invalid … token or error 5xx lands on things that aren't our key or our infra. Could the key patterns anchor on the proxy's own wording (LiteLLM's BudgetExceededError and Authentication Error, Invalid proxy server token) instead of bare codes?
2. _chain follows __context__ into errors that were already handled. __context__ is set for any exception raised during an except, including ones the code recovered from. Because a key cause outranks everything else, a handled budget error on a fallback path followed by a real error reads key_budget. A handled timeout followed by a real error reads transient. Suggest following __cause__, using __context__ only when not __suppress_context__, or ranking the outermost exception first.
3. A failing verdict reads like a lost grade. VerifierNotPassingError (rubrics_verifier's _raise_unless_passing, user_sim_judge) goes through the same except BaseException and records step_role=grader, cause=unknown. That's the same pair as a grader that crashed, so only error_type tells "graded, failing" apart from "grade lost". A dedicated cause such as verdict, or skipping classification for that error type, would keep the next PR from marking real failing grades as lost ones.
4. Plugin graders default to other. test_every_step_type_that_writes_verifications_is_a_grader only sees step types registered in this repo, so graders in plugin packages stay OTHER. That includes the iOS ones in agentenvhub-sdk. The source scan also misses a step that writes verifications through a helper or a constant. Maybe enforce it at registration time, or at least say so in the PR.
Nit: _EpisodeAgentStep.from_dict (task/episode/steps.py:41) rebuilds the step it wraps without attach_base_fields, so step_role and retry_config on a wrapped deploy step are lost when it's saved and reloaded. It doesn't matter today because deploy steps are other, but it's the one rebuild path the last commit didn't cover.
No objection to merging this PR on its own. I'd want 1–3 settled before the PR that derives quality from cause.
…ptions it was raised from, and give a failing verdict its own cause - Key and transient patterns match LiteLLM, gateway, HTTP-client and model-SDK wording, not bare codes that env and model text also carry. - The chain follows __cause__ and exception-group members, not __context__. - VerifierNotPassingError reads as verdict, so a failing grade is not mistaken for a lost one. - A wrapped episode deploy step keeps its configured step_role and retry_config through a round trip.
|
Thanks, these were right. I fixed 1–3 and the nit here in 179a0be rather than waiting for the next PR, since 1. Key and transient patterns. They now match only wording that LiteLLM, the gateway, the HTTP client or the model SDK writes: 2. 3. Failing verdicts. 4. Plugin graders. I didn't find a registration-time check that works: a step class doesn't say whether it writes Nit. The full unit suite passes (14,582). |
…nnection failure, as transient
kaihackney-scale
left a comment
There was a problem hiding this comment.
Thanks for fixing all of this here. I re-ran my six examples on f413ce4 and they all read unknown. A failing verdict reads verdict, and a ValueError raised during a handled timeout reads unknown. The known-gap note covers plugin graders. LGTM.
One non-blocking note for the PR that derives quality from cause: when an agent doesn't finish its turn, the A2A client raises a plain TimeoutError (a2a_agent/protocol.py:235, A2A task … did not complete within …s), and the _TRANSIENT_TYPES fallback reads that as transient. A dead agent already gets its own AgentUnreachableError, so this timeout means the agent was alive and didn't finish, which is usually its own behavior, and a retry will likely time out again. It's the human-review timeout case again, this time through the exception type. If the next PR retries transient automatically, a dedicated exception for the turn deadline that isn't read as transient would avoid paying for retries that can't succeed.
|
@andrewpark-scaleai thanks for this. On the follow-up that derives a run's quality from these: please hold it. Deciding how a run ended is already designed and planned under the eval runner work (FD-5115):
We'll take |
Why
A run whose grader, snapshot or agent step failed but was tolerated (
fail_task_on_error=False) completes and counts as a success. Nothing on the run says what it lost, so no reader can flag it, hold it, or repair it.Today a
context.metadata.failed_stepsentry records the step id, type, error text and whether it was fatal. It does not say whether the failed step was the grader or an optional snapshot, and it does not say whether the error was a model key out of budget (permanent until the budget resets) or a timeout (worth retrying). This PR adds those two facts. It changes no behavior. The next PR derives the run's quality from them.What
StepRole(grader,evidence,agent,other) onTaskStepasstep_role. Every step type that writescontext.metadata["verifications"]declaresgrader.snapshot_env,snapshot_agent_stateandcollect_artifactsdeclareevidence.prompt_agentdeclaresagent. Every other step staysother.step_role, so arun_codestep that grades can say so. It serializes only when it differs from the type's role, so existing step documents are unchanged. An unknown value is refused when the task loads.agent_env/task/outcome.pywithFailureCause(key_budget,key_auth,transient,verdict,unknown),classify_failure(exc)and theFailedStepentry withFailedStep.from_exception(...).classify_failurereturnsverdictforVerifierNotPassingError: the grader ran and the run failed its check, so the grade exists. Otherwise it reads the type names and messages of the exception and the exceptions it was raisedfrom. A key cause wins over a transient one, because a timeout that wraps an out-of-budget error still fails on every retry.failed_stepsentry withFailedStep.from_exception. The entry keeps every existing key and addsstep_roleandcause.attach_retry_configbecomesattach_base_fields, since it now hydratesstep_roleas well. All five callers are migrated.Design decisions
verificationsis not agrader, so a new verifier cannot forget.step_role, notrole.deploy_agentandmodify_env_tool_accessalready use aroleconfig field for their own meaning. A baserolefield would collide with both in config and on the instance.classify_failuremeans one place to fix when a provider changes its wording. The cost is that an unfamiliar message reads asunknown, which is the safe default.401,invalid tokenorstatus 500in that text says nothing about our key or our infra, so the patterns anchor on LiteLLM's and the gateway's own messages (Budget has been exceeded!,Authentication Error, Invalid proxy server token,API Error: 503). The patterns were checked against a week of real step failures from the worker logs: every proxy budget, auth, rate-limit and disconnect error keeps its cause, and an env script'sHTTP Error 401no longer reads as our key.__cause__, not__context__. Python sets__context__on any exception raised inside anexcept, including one the code had already recovered from. A bug raised on a fallback path after a handled timeout is not a timeout. A wrapper that forgetsfromreads asunknown, which is the safe default.step_role=grader, cause=unknownpair, and the next PR could not tell a lost grade from a real failing one.StepRolelives intask_step/task_step.pyandoutcome.pyre-exports it.task_stepcannot import fromagent_env.taskwithout an import cycle throughagent_env/task/__init__.py.Known gap
Step types registered by plugin packages default to
other, including graders such as the iOS ones in agentenvhub-sdk. The test that checks every verifier is agraderonly sees step types registered in this repo, and its source scan misses a step that writesverificationsthrough a helper. A task can mark such a step withstep_role: graderin config today. A follow-up in each plugin should declarestep_roleon its grader classes.Test
Without the change the module does not exist and collection fails. The classifier cases include real proxy, gateway and client errors from the worker logs, and env, script and model text that must read as
unknown.

Confidence Score: 5/5
The PR appears safe to merge, with no outstanding actionable findings.Summary
Adds
step_roleandcausetofailed_stepswhile keeping the existing fields and failure policy.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Step raises an error] --> B[Read the step role] A --> C[Classify the error cause] B --> D[Build FailedStep] C --> D D --> E[Append to failed_steps] E --> F[Keep existing failure policy]Reviews (6) · Last reviewed commit: "Read the model proxy's JSON-shaped 429 a..." · Reviewed by Greptile