Repository navigation
feat(episode): record episodes in the run index - #457
Open
polakamtejas wants to merge 7 commits into
Open
polakamtejas wants to merge 7 commits into
polakamtejas wants to merge 7 commits into
Conversation
polakamtejas
force-pushed
the
feat/episode-run-index
branch
from
November 6, 2026 16:32
172e5f1 to
728000f
Compare
An episode called Task.run directly, so the run index never saw it and the explorer's run lists, which read only the index, never showed it: training episodes were reachable only by instance id. An episode now records itself as a runner records the runs it accepts (runner `episode`): RUNNING before Task.run, then COMPLETED, FAILED with the error, or CANCELED with the abandon reason once its teardown is done. `EpisodeSpec.start(run_group_id=...)` files it in a run group, so a trainer can show a group of rollouts together. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g it from the explorer, and keep a failed record write from leaking - An episode's run index status now follows `run_outcome`, the rule every runner uses: a step that fails without failing the task fails the run, rather than leaving it COMPLETED. Abandoned or cancelled stays CANCELED. - The explorer's cancel route refuses, with 409 and where to stop it, a run another runner records: the configured runner can't cancel an episode a trainer plays, so asking it silently did nothing. The UI shows the reason. - The run record is written inside the try whose finally takes the episode out of `_live`, so a write that fails is the run's error and leaves nothing behind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The explorer reads a run's seed from `overrides.metadata.seed`, so an episode's run group listed `seed: null` for a seeded episode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polakamtejas
force-pushed
the
feat/episode-run-index
branch
from
November 11, 2026 03:46
e3a40ec to
167a782
Compare
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Keeps both sides in the two conflicts: main's clock-test and egress-test with the Tinker tier's tinker-test in the Makefile's .PHONY, and the TINKER capability beside main's `stat -c %f:%s` note in tst/util/capabilities.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…left over EpisodeDataset rounded a pass's batches down, so with 10 tasks and groups_per_batch=3 one task went unplayed each pass. A pass now rounds up, its last batch the tasks left over, as the cookbook's own RL datasets end a pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polakamtejas
enabled auto-merge (squash)
November 18, 2026 17:46
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An episode now records itself in the run index, as a runner records the runs it accepts:
Task.run: aRunRecordwith runnerepisode, statusRUNNING, the task, its version and the instance id. It's written before the first await, sostart()still returns with the run listed.mark_terminalwithCOMPLETED,FAILEDand the error, orCANCELEDand the abandon reason.EpisodeSpec.start(run_group_id=...): files the run in that run group (overrides.metadata.run_group_id), the key the explorer groups a task's runs by. Without it, each run is its own group.Why
The explorer's run lists read only the run index (
agent_env.runner.store). An episode calledTask.rundirectly, so its runs never appeared there: training episodes were reachable only by instance id. That holds for every trainer that plays through the Episode API, the OpenEnv-based helpers included. With a run group, a trainer can show a group of rollouts together, with their mean score.The terminal status is written after teardown, as the cancelled record already is. A scored episode still answers
result(teardown=False)at once (#393), and its run turnsCOMPLETEDonce its teardown is done.Known limit
An episode in a trainer process that dies stays
RUNNINGin the index.LocalRunnermarks its own interrupted runs failed on restart, and nothing does that for episodes yet.Tests
run_test.py::test_an_episode_is_in_the_run_index_running_then_ended_in_the_group_it_was_started_in. It checks running with its group, then completed, cancelled with the reason, and failed with the error.tst/unit/task,runner,explorer,openenvandtrainingpass, 532 of 532, on five runs. One run right afteruv synchad a single failure that didn't recur, likely a timing flake; the log didn't keep which test.run_group_id.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge.Summary
Episodes now appear in the explorer’s run history, and the PR adds a Tinker training adapter that files rollouts by training step.
No new actionable issues found. polakamtejas accepted using scores from non-fatal failed steps for training, matching OpenEnv, and deferred any shared policy change. The earlier finding was withdrawn.
Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Tinker rollout or Episode caller] --> B[Start episode with optional run group] B --> C[Write RUNNING record] C --> D[Run task and play agent turns] D --> E[Release proxies and tear down] E --> F[Write final run status] C --> G[Explorer run history] F --> GReviews (8) · Last reviewed commit: "Merge branch 'main' into feat/episode-ru..." · Reviewed by Greptile