Skip to content

Stamp a completed run's quality in the same write as its status - #693

Draft
andrewpark-scaleai wants to merge 6 commits into
mainfrom
andrew-park/rel-outcome-quality
Draft

andrewpark-scaleai wants to merge 6 commits into
mainfrom
andrew-park/rel-outcome-quality

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

A run whose grader, snapshot or agent step failed but was tolerated (fail_task_on_error=False) completes and counts as a success. A reader of status cannot tell such a run from a clean one.

Stacked on #692. That PR records each failed step's role and cause in failed_steps. This PR sums them up on the run, so a reader can check one field instead of decoding the list. It changes no behavior.

What

  • RunQuality(reasons) with a derived degraded flag, and run_quality(failed_steps) in agent_env/task/outcome.py. reasons holds the distinct roles of tolerated failures that were not retried, not resolved and not a failing verdict (cause=verdict), most severe first: grader, agent, evidence, other.
  • The completion CAS in TaskInstanceStore writes quality = {"degraded": ..., "reasons": [...]} in the same update that sets status="completed". It derives it from the document as this write leaves it, so a sibling step's earlier failure counts too.
  • TaskInstance.quality reads it back. A run completed before this change has no quality and reads as clean, which is what every reader assumes today.
  • FailedStep.from_dict reads stored entries. An entry written before step_role and cause existed reads as other and unknown, so it still degrades the run.

Design decisions

  • Two fields, status and quality, instead of a new status value. status is lifecycle: did the run finish? Every existing reader branches on its four values, and a new degraded value would turn degraded runs into "not completed" for all of them at once. A separate quality lets each reader opt in. The cost is one more field a reader must check to know a run is clean.
  • Derive in the completion write. The store already holds every step's result when it sets completed. Writing quality in the same CAS means no reader ever sees a completed run without it. The cost is one copy of the context per run, at completion only.
  • degraded is derived from reasons. It is a dataclass field so asdict matches the stored form, but it is never passed in, so it cannot disagree with reasons.
  • Severity order puts grader first. A run with no grade cannot be scored at all, while a missing snapshot still leaves a graded run. A reader that keeps one reason takes the first.

Test

uv run --extra dev pytest tst/unit/task/run_quality_test.py -q
9 passed

Without RunQuality the test module fails to import. With the new outcome.py but the old store.py, 3 of 9 fail: the completed run has no quality, and TaskInstance has no quality to read.

uv run --extra dev pytest tst/unit/task tst/unit/task_step tst/unit/bundle tst/unit/cli tst/unit/runner tst/unit/explorer -q -n 4
5261 passed, 3 skipped in 652.56s

@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-facts branch from bf74480 to 26cbf52 Compare November 16, 2026 17:58
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-quality branch from f628698 to bd0fc7d Compare November 16, 2026 17:58
A step that fails with fail_task_on_error=False lets its run complete, and
the failed_steps entry is the only record of what the run lost. The entry
now says which role the step played (grader, evidence, agent, other) and
why it failed (key_budget, key_auth, transient, unknown).

- StepRole on TaskStep: verifiers declare grader, snapshot and collect
  steps evidence, prompt_agent agent. A step overrides it in config with
  step_role, so a run_code step can say it grades.
- task/outcome.py: FailureCause, classify_failure, FailedStep. The DAG
  runner builds its entry with FailedStep.from_exception.
- attach_retry_config becomes attach_base_fields and hydrates step_role too.

No behavior changes. Entries keep every existing key.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-facts branch from 26cbf52 to 28de586 Compare November 17, 2026 18:09
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-quality branch from bd0fc7d to d22d2a8 Compare November 17, 2026 18:29
… errors as transient

EnvOutcomeVerifierTaskStep.put passed step_role and retry_config to a
constructor that rejects them; it now stores through TaskStep.put. An
episode's prompt copy with a turn timeout kept no configured role. A
connection failure whose message matches no pattern, such as httpx's
'All connection attempts failed', now reads as transient.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-quality branch from d22d2a8 to 2785d7b Compare November 17, 2026 19:07
…ptions it was raised from, and give a failing verdict its own cause

- Key and transient patterns match LiteLLM, gateway, HTTP-client and model-SDK wording, not bare codes that env and model text also carry.
- The chain follows __cause__ and exception-group members, not __context__.
- VerifierNotPassingError reads as verdict, so a failing grade is not mistaken for a lost one.
- A wrapped episode deploy step keeps its configured step_role and retry_config through a round trip.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-quality branch from 2785d7b to 276e23c Compare November 17, 2026 23:49
Ubuntu added 3 commits November 18, 2026 01:21
A run that tolerated a step failure completes like any other, so a reader
of status cannot tell it lost its grade, its snapshot or its agent turn.
The completion CAS now also writes quality = {degraded, reasons}, derived
from failed_steps by run_quality. Reasons are the roles of the tolerated
failures that were not retried or resolved, most severe first.

TaskInstance.quality reads it back. A run completed before this change has
no quality and reads as clean, which is what every reader assumes today.

No behavior changes: status keeps its values and quality is a new field.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-outcome-quality branch from 276e23c to 6e862e6 Compare November 18, 2026 01:22
Base automatically changed from andrew-park/rel-outcome-facts to main November 18, 2026 18:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant