Repository navigation
Stamp a completed run's quality in the same write as its status - #693
Draft
andrewpark-scaleai wants to merge 6 commits into
Draft
andrewpark-scaleai wants to merge 6 commits into
andrewpark-scaleai wants to merge 6 commits into
Conversation
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-facts
branch
from
November 16, 2026 17:58
bf74480 to
26cbf52
Compare
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-quality
branch
from
November 16, 2026 17:58
f628698 to
bd0fc7d
Compare
A step that fails with fail_task_on_error=False lets its run complete, and the failed_steps entry is the only record of what the run lost. The entry now says which role the step played (grader, evidence, agent, other) and why it failed (key_budget, key_auth, transient, unknown). - StepRole on TaskStep: verifiers declare grader, snapshot and collect steps evidence, prompt_agent agent. A step overrides it in config with step_role, so a run_code step can say it grades. - task/outcome.py: FailureCause, classify_failure, FailedStep. The DAG runner builds its entry with FailedStep.from_exception. - attach_retry_config becomes attach_base_fields and hydrates step_role too. No behavior changes. Entries keep every existing key.
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-facts
branch
from
November 17, 2026 18:09
26cbf52 to
28de586
Compare
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-quality
branch
from
November 17, 2026 18:29
bd0fc7d to
d22d2a8
Compare
… errors as transient EnvOutcomeVerifierTaskStep.put passed step_role and retry_config to a constructor that rejects them; it now stores through TaskStep.put. An episode's prompt copy with a turn timeout kept no configured role. A connection failure whose message matches no pattern, such as httpx's 'All connection attempts failed', now reads as transient.
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-quality
branch
from
November 17, 2026 19:07
d22d2a8 to
2785d7b
Compare
…ptions it was raised from, and give a failing verdict its own cause - Key and transient patterns match LiteLLM, gateway, HTTP-client and model-SDK wording, not bare codes that env and model text also carry. - The chain follows __cause__ and exception-group members, not __context__. - VerifierNotPassingError reads as verdict, so a failing grade is not mistaken for a lost one. - A wrapped episode deploy step keeps its configured step_role and retry_config through a round trip.
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-quality
branch
from
November 17, 2026 23:49
2785d7b to
276e23c
Compare
added 3 commits
November 18, 2026 01:21
…nnection failure, as transient
A run that tolerated a step failure completes like any other, so a reader
of status cannot tell it lost its grade, its snapshot or its agent turn.
The completion CAS now also writes quality = {degraded, reasons}, derived
from failed_steps by run_quality. Reasons are the roles of the tolerated
failures that were not retried or resolved, most severe first.
TaskInstance.quality reads it back. A run completed before this change has
no quality and reads as clean, which is what every reader assumes today.
No behavior changes: status keeps its values and quality is a new field.
andrewpark-scaleai
force-pushed
the
andrew-park/rel-outcome-quality
branch
from
November 18, 2026 01:22
276e23c to
6e862e6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A run whose grader, snapshot or agent step failed but was tolerated (
fail_task_on_error=False) completes and counts as a success. A reader ofstatuscannot tell such a run from a clean one.Stacked on #692. That PR records each failed step's role and cause in
failed_steps. This PR sums them up on the run, so a reader can check one field instead of decoding the list. It changes no behavior.What
RunQuality(reasons)with a deriveddegradedflag, andrun_quality(failed_steps)inagent_env/task/outcome.py.reasonsholds the distinct roles of tolerated failures that were not retried, not resolved and not a failing verdict (cause=verdict), most severe first:grader,agent,evidence,other.TaskInstanceStorewritesquality = {"degraded": ..., "reasons": [...]}in the same update that setsstatus="completed". It derives it from the document as this write leaves it, so a sibling step's earlier failure counts too.TaskInstance.qualityreads it back. A run completed before this change has noqualityand reads as clean, which is what every reader assumes today.FailedStep.from_dictreads stored entries. An entry written beforestep_roleandcauseexisted reads asotherandunknown, so it still degrades the run.Design decisions
statusandquality, instead of a newstatusvalue.statusis lifecycle: did the run finish? Every existing reader branches on its four values, and a newdegradedvalue would turn degraded runs into "not completed" for all of them at once. A separatequalitylets each reader opt in. The cost is one more field a reader must check to know a run is clean.completed. Writingqualityin the same CAS means no reader ever sees a completed run without it. The cost is one copy of the context per run, at completion only.degradedis derived fromreasons. It is a dataclass field soasdictmatches the stored form, but it is never passed in, so it cannot disagree withreasons.graderfirst. A run with no grade cannot be scored at all, while a missing snapshot still leaves a graded run. A reader that keeps one reason takes the first.Test
Without
RunQualitythe test module fails to import. With the newoutcome.pybut the oldstore.py, 3 of 9 fail: the completed run has noquality, andTaskInstancehas noqualityto read.