Skip to content

Record each failed step's role and cause in failed_steps - #692

Merged
andrewpark-scaleai merged 4 commits into
mainfrom
andrew-park/rel-outcome-facts
Nov 18, 2026
Merged

andrewpark-scaleai merged 4 commits into
mainfrom
andrew-park/rel-outcome-facts

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

A run whose grader, snapshot or agent step failed but was tolerated (fail_task_on_error=False) completes and counts as a success. Nothing on the run says what it lost, so no reader can flag it, hold it, or repair it.

Today a context.metadata.failed_steps entry records the step id, type, error text and whether it was fatal. It does not say whether the failed step was the grader or an optional snapshot, and it does not say whether the error was a model key out of budget (permanent until the budget resets) or a timeout (worth retrying). This PR adds those two facts. It changes no behavior. The next PR derives the run's quality from them.

What

  • StepRole (grader, evidence, agent, other) on TaskStep as step_role. Every step type that writes context.metadata["verifications"] declares grader. snapshot_env, snapshot_agent_state and collect_artifacts declare evidence. prompt_agent declares agent. Every other step stays other.
  • A step can override its type's role in config with step_role, so a run_code step that grades can say so. It serializes only when it differs from the type's role, so existing step documents are unchanged. An unknown value is refused when the task loads.
  • New agent_env/task/outcome.py with FailureCause (key_budget, key_auth, transient, verdict, unknown), classify_failure(exc) and the FailedStep entry with FailedStep.from_exception(...).
  • classify_failure returns verdict for VerifierNotPassingError: the grader ran and the run failed its check, so the grade exists. Otherwise it reads the type names and messages of the exception and the exceptions it was raised from. A key cause wins over a transient one, because a timeout that wraps an out-of-budget error still fails on every retry.
  • The DAG runner builds its failed_steps entry with FailedStep.from_exception. The entry keeps every existing key and adds step_role and cause.
  • attach_retry_config becomes attach_base_fields, since it now hydrates step_role as well. All five callers are migrated.

Design decisions

  • Declare the role on the step, not in a central table. The step type is the one place that knows whether it grades, and a new step type declares its role when it is written. A table would need every consumer to keep a copy in sync. The cost is a one-line declaration in 31 step classes. A test scans every registered step type and fails if one that writes verifications is not a grader, so a new verifier cannot forget.
  • Name it step_role, not role. deploy_agent and modify_env_tool_access already use a role config field for their own meaning. A base role field would collide with both in config and on the instance.
  • Match error text in one function. A judge or agent error crosses A2A and the model proxy as a message string, so its original exception type is gone by the time a step raises. Text matching is unavoidable. Keeping it in classify_failure means one place to fix when a provider changes its wording. The cost is that an unfamiliar message reads as unknown, which is the safe default.
  • Match only wording the proxy, the HTTP client or the model SDK writes. An error's text also carries env responses, tool output, script output and model text. A bare 401, invalid token or status 500 in that text says nothing about our key or our infra, so the patterns anchor on LiteLLM's and the gateway's own messages (Budget has been exceeded!, Authentication Error, Invalid proxy server token, API Error: 503). The patterns were checked against a week of real step failures from the worker logs: every proxy budget, auth, rate-limit and disconnect error keeps its cause, and an env script's HTTP Error 401 no longer reads as our key.
  • Follow __cause__, not __context__. Python sets __context__ on any exception raised inside an except, including one the code had already recovered from. A bug raised on a fallback path after a handled timeout is not a timeout. A wrapper that forgets from reads as unknown, which is the safe default.
  • A failing verdict gets its own cause. Without it, a grader that crashed and a grader that graded the run as failing would record the same step_role=grader, cause=unknown pair, and the next PR could not tell a lost grade from a real failing one.
  • StepRole lives in task_step/task_step.py and outcome.py re-exports it. task_step cannot import from agent_env.task without an import cycle through agent_env/task/__init__.py.

Known gap

Step types registered by plugin packages default to other, including graders such as the iOS ones in agentenvhub-sdk. The test that checks every verifier is a grader only sees step types registered in this repo, and its source scan misses a step that writes verifications through a helper. A task can mark such a step with step_role: grader in config today. A follow-up in each plugin should declare step_role on its grader classes.

Test

uv run --extra dev pytest tst/unit/task/outcome_test.py -q
42 passed

Without the change the module does not exist and collection fails. The classifier cases include real proxy, gateway and client errors from the worker logs, and env, script and model text that must read as unknown.

uv run --all-extras pytest tst/unit -q -n 4
14582 passed, 3 skipped

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge, with no outstanding actionable findings.

Summary

Adds step_role and cause to failed_steps while keeping the existing fields and failure policy.

  • Task steps label their role, and task configs can override it.
  • Failed-step records explain each step’s role and failure cause.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Step raises an error] --> B[Read the step role]
  A --> C[Classify the error cause]
  B --> D[Build FailedStep]
  C --> D
  D --> E[Append to failed_steps]
  E --> F[Keep existing failure policy]
Loading

Reviews (6) · Last reviewed commit: "Read the model proxy's JSON-shaped 429 a..." · Reviewed by Greptile

A step that fails with fail_task_on_error=False lets its run complete, and
the failed_steps entry is the only record of what the run lost. The entry
now says which role the step played (grader, evidence, agent, other) and
why it failed (key_budget, key_auth, transient, unknown).

- StepRole on TaskStep: verifiers declare grader, snapshot and collect
  steps evidence, prompt_agent agent. A step overrides it in config with
  step_role, so a run_code step can say it grades.
- task/outcome.py: FailureCause, classify_failure, FailedStep. The DAG
  runner builds its entry with FailedStep.from_exception.
- attach_retry_config becomes attach_base_fields and hydrates step_role too.

No behavior changes. Entries keep every existing key.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-facts branch from 26cbf52 to 28de586 Compare November 17, 2026 18:09
@andrewpark-scaleai
andrewpark-scaleai marked this pull request as ready for review November 17, 2026 18:43
@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 17, 2026 18:43
Comment thread src/agent_env/cli/task/create.py Resolved
Comment thread src/agent_env/task/outcome.py Resolved
Comment thread src/agent_env/task_step/task_step.py Resolved
@andrewpark-scaleai
andrewpark-scaleai enabled auto-merge (squash) November 17, 2026 18:53
… errors as transient

EnvOutcomeVerifierTaskStep.put passed step_role and retry_config to a
constructor that rejects them; it now stores through TaskStep.put. An
episode's prompt copy with a turn timeout kept no configured role. A
connection failure whose message matches no pattern, such as httpx's
'All connection attempts failed', now reads as transient.

@kaihackney-scale kaihackney-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change. The role on the step type, step_role rather than role, writing it only when overridden, and refusing an unknown value at load all look right. Merging as-is changes no behavior. My concerns are about cause, since the next PR derives run quality from it.

I ran a few realistic messages through classify_failure on a91f0ad:

message got expected
tool get_account returned HTTP 401 Unauthorized from env MCP server key_auth unknown (the env's 401, not our key)
rubric row 401 has no criteria key_auth unknown
invalid token in template: {{seed} key_auth unknown
expected status 500 to not be returned by create_order transient unknown
…judge said: user exceeded the budget for travel key_budget unknown
ValueError("rubric item 3 has no criteria") raised inside except TimeoutError transient unknown

1. The key patterns match env and agent text, not just the proxy's. Error strings carry env HTTP statuses, tool output, verifier stdout and model text, so a bare \b401\b, invalid … token or error 5xx lands on things that aren't our key or our infra. Could the key patterns anchor on the proxy's own wording (LiteLLM's BudgetExceededError and Authentication Error, Invalid proxy server token) instead of bare codes?

2. _chain follows __context__ into errors that were already handled. __context__ is set for any exception raised during an except, including ones the code recovered from. Because a key cause outranks everything else, a handled budget error on a fallback path followed by a real error reads key_budget. A handled timeout followed by a real error reads transient. Suggest following __cause__, using __context__ only when not __suppress_context__, or ranking the outermost exception first.

3. A failing verdict reads like a lost grade. VerifierNotPassingError (rubrics_verifier's _raise_unless_passing, user_sim_judge) goes through the same except BaseException and records step_role=grader, cause=unknown. That's the same pair as a grader that crashed, so only error_type tells "graded, failing" apart from "grade lost". A dedicated cause such as verdict, or skipping classification for that error type, would keep the next PR from marking real failing grades as lost ones.

4. Plugin graders default to other. test_every_step_type_that_writes_verifications_is_a_grader only sees step types registered in this repo, so graders in plugin packages stay OTHER. That includes the iOS ones in agentenvhub-sdk. The source scan also misses a step that writes verifications through a helper or a constant. Maybe enforce it at registration time, or at least say so in the PR.

Nit: _EpisodeAgentStep.from_dict (task/episode/steps.py:41) rebuilds the step it wraps without attach_base_fields, so step_role and retry_config on a wrapped deploy step are lost when it's saved and reloaded. It doesn't matter today because deploy steps are other, but it's the one rebuild path the last commit didn't cover.

No objection to merging this PR on its own. I'd want 1–3 settled before the PR that derives quality from cause.

…ptions it was raised from, and give a failing verdict its own cause

- Key and transient patterns match LiteLLM, gateway, HTTP-client and model-SDK wording, not bare codes that env and model text also carry.
- The chain follows __cause__ and exception-group members, not __context__.
- VerifierNotPassingError reads as verdict, so a failing grade is not mistaken for a lost one.
- A wrapped episode deploy step keeps its configured step_role and retry_config through a round trip.
@andrewpark-scaleai

Copy link
Copy Markdown
Contributor Author

Thanks, these were right. I fixed 1–3 and the nit here in 179a0be rather than waiting for the next PR, since classify_failure lives in this one.

1. Key and transient patterns. They now match only wording that LiteLLM, the gateway, the HTTP client or the model SDK writes: Budget has been exceeded!, ExceededBudget:, Authentication Error, Invalid proxy server token, token_not_found_in_db, API Error: 503, Error code: 429, Server error '502 ...', and similar. Bare 401, invalid … token, status 5xx and timed out are gone. All six of your examples now read unknown. I also ran old and new against a week of real step failures from the worker logs. Every proxy budget, auth, rate-limit and disconnect error keeps its cause. Three move to unknown: an env reload script's HTTP Error 401: Unauthorized, a human review step that timed out after 10800s, and snapshot save failed with HTTP 500. The first two were wrong before. The third is a real loss, and unknown is the safe side.

2. __context__. _chain follows only __cause__ and exception-group members now. Your ValueError raised inside except TimeoutError reads unknown. A wrapper that forgets from also reads unknown. The judge retry wrapper already uses from.

3. Failing verdicts. VerifierNotPassingError now reads as a new cause, verdict, before any text matching. So a failing grade records step_role=grader, cause=verdict, and a grader that crashed records cause=unknown.

4. Plugin graders. I didn't find a registration-time check that works: a step class doesn't say whether it writes verifications until it runs. The PR description now lists this as a known gap. A task can set step_role: grader in config today, and each plugin should declare step_role on its grader classes in a follow-up.

Nit. _EpisodeAgentStep.from_dict now hydrates the wrapped step with attach_base_fields. A new test round-trips a wrapped deploy step with step_role and retry_config, and it fails without the fix.

The full unit suite passes (14,582).

Comment thread src/agent_env/task/outcome.py Resolved

@kaihackney-scale kaihackney-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing all of this here. I re-ran my six examples on f413ce4 and they all read unknown. A failing verdict reads verdict, and a ValueError raised during a handled timeout reads unknown. The known-gap note covers plugin graders. LGTM.

One non-blocking note for the PR that derives quality from cause: when an agent doesn't finish its turn, the A2A client raises a plain TimeoutError (a2a_agent/protocol.py:235, A2A task … did not complete within …s), and the _TRANSIENT_TYPES fallback reads that as transient. A dead agent already gets its own AgentUnreachableError, so this timeout means the agent was alive and didn't finish, which is usually its own behavior, and a retry will likely time out again. It's the human-review timeout case again, this time through the exception type. If the next PR retries transient automatically, a dedicated exception for the turn deadline that isn't read as transient would avoid paying for retries that can't succeed.

@andrewpark-scaleai
andrewpark-scaleai merged commit 2f0ac48 into main Nov 18, 2026
37 checks passed
@andrewpark-scaleai
andrewpark-scaleai deleted the andrew-park/rel-outcome-facts branch November 18, 2026 18:14
@earakely-scale

Copy link
Copy Markdown
Collaborator

@andrewpark-scaleai thanks for this. step_role and the failure causes are useful, and #761 (G1 of the Grades work, FD-5919) already builds on step_role: VerifierTaskStep declares grader once, so every verifier on the base, plugin ones included, gets it. That closes the plugin gap you noted once the sdk's graders move onto the base (G3).

On the follow-up that derives a run's quality from these: please hold it. Deciding how a run ended is already designed and planned under the eval runner work (FD-5115):

  • E2 (FD-5950): one shared function writes each task instance's result at the end of Task.run, versioned (result_rules_version), the same from every launcher, the hub worker included.
  • G7 (FD-5927): the result rules. A run is passed, below_threshold, graded or ungraded, with a cause (agent_timeout, agent_error, infra_environment, infra_model_provider, verifier_failed, task_error, and so on), plus is_transient(exc) for a task's own failing steps.

We'll take step_role and FailureCause as inputs there rather than build a second classifier, so nothing in #692 needs undoing. If the follow-up covers a need E2 and G7 don't (holding or repairing runs that lost their grader, say), could you add it to FD-5927 or ping me, so it lands in one place?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants