Repository navigation
feat(episode): record episodes in the run index #457
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
polakamtejas
wants to merge
7
commits into
main
Choose a base branch
from
feat/episode-run-index
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,170
−15
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
6ddf1b8
feat(episode): record episodes in the run index
polakamtejas 5cdf1b1
fix(episode): record an episode's end as runners do, refuse cancellin…
polakamtejas 167a782
fix(episode): keep an episode's seed in its run record
polakamtejas 304ead5
feat(training): train with Tinker on a bundle's tasks (#458)
polakamtejas 39e421a
Merge origin/main into feat/episode-run-index
polakamtejas e6e5243
fix(training): play every task once a pass, its last batch the tasks …
polakamtejas d9e2068
Merge branch 'main' into feat/episode-run-index
polakamtejas File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| name: Tinker contract | ||
|
|
||
| # The Tinker helper's episodes on tinker-cookbook itself (tst/tinker): EpisodeEnv played as the cookbook's rollouts step | ||
| # a MessageEnv, on the Episode API's fake steps with a stand-in renderer, so nothing samples and no Tinker key is needed, | ||
| # on tinker-cookbook as released and on its main branch. tinker-cookbook and CPU torch are installed over the dev | ||
| # environment. It runs on pull requests that touch the helper, the Episode API or the tier, weekly, and on demand; it | ||
| # isn't a required check. | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "src/agent_env/training/**" | ||
| - "src/agent_env/task/episode/**" | ||
| - "src/agent_env/openenv/reward.py" | ||
| - "tst/tinker/**" | ||
| - "tst/unit/training/**" | ||
| - "pyproject.toml" | ||
| - "uv.lock" | ||
| - ".github/workflows/tinker.yml" | ||
| schedule: | ||
| - cron: "17 8 * * 1" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: tinker-${{ github.event_name == 'pull_request' && github.ref || github.sha }} | ||
| cancel-in-progress: ${{ github.event_name == 'pull_request' }} | ||
|
|
||
| env: | ||
| PIP_DISABLE_PIP_VERSION_CHECK: "1" | ||
|
|
||
| jobs: | ||
| episodes: | ||
| name: episodes (tinker-cookbook ${{ matrix.cookbook }}) | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: | ||
| - cookbook: "released" | ||
| requirement: "tinker-cookbook" | ||
| - cookbook: "main" | ||
| requirement: "tinker-cookbook@git+https://github.com/thinking-machines-lab/tinker-cookbook@main" | ||
| steps: | ||
| - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 | ||
| - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | ||
| with: | ||
| python-version: "3.12" | ||
| - name: Install from uv.lock with the tinker extra | ||
| run: | | ||
| python -m pip install --quiet -r .github/ci-requirements.txt | ||
| uv sync --locked --extra dev --extra tinker --python "$(command -v python)" | ||
| - name: Install tinker-cookbook and CPU torch | ||
| run: | | ||
| uv pip install --python .venv/bin/python ${{ matrix.requirement }} torch \ | ||
| --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match | ||
| .venv/bin/python -c "from importlib.metadata import version; print('tinker-cookbook', version('tinker-cookbook'), 'tinker', version('tinker'), 'torch', version('torch'))" | ||
| - name: Tinker tier | ||
| run: .venv/bin/python -m pytest tst/tinker -rA --timeout=900 --junitxml=junit-tinker.xml | ||
| - name: No skips | ||
| if: always() | ||
| run: .venv/bin/python .github/scripts/check_skip_policy.py junit-tinker.xml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| """What a trainer's adapter needs without the trainer installed: a tool call's arguments typed by its tool's schema, | ||
| a later turn's prompt as the model reads it, and the run group a trainer files its episodes in.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| from collections.abc import Mapping | ||
| from dataclasses import dataclass | ||
| from typing import Any, Literal | ||
|
|
||
|
|
||
| _JSON_TYPES = {bool: "boolean", int: "integer", float: "number", type(None): "null"} | ||
|
|
||
|
|
||
| def typed_arguments(arguments: Any, schema: Mapping[str, Any]) -> Any: | ||
| """``arguments`` with each scalar whose type the tool's schema doesn't allow, but a string, as the string it was | ||
| written as. Renderers that write parameters as untyped text (Qwen3.5's ``<parameter=zip>19122</parameter>``) parse | ||
| each with ``json.loads``, reading digits as a number; Qwen's own parser types values by the schema, as this does.""" | ||
| if not isinstance(arguments, dict): | ||
| return arguments | ||
| properties = schema.get("properties") or {} | ||
| typed = dict(arguments) | ||
| for name, value in arguments.items(): | ||
| allowed = _allowed_types(properties.get(name) or {}) | ||
| kind = _JSON_TYPES.get(type(value)) | ||
| widened = kind == "integer" and "number" in allowed | ||
| if kind is not None and "string" in allowed and kind not in allowed and not widened: | ||
| typed[name] = json.dumps(value) | ||
| return typed | ||
|
|
||
|
|
||
| def next_prompt(prompt: str, system_prompt: str | None, in_effect: str | None) -> tuple[str, str | None]: | ||
| """A later turn's prompt as every trainer's model reads it, and the system prompt in effect after it. A trainer | ||
| plays an episode as one conversation, whose system prompt is written once, at its start; so a turn that changes the | ||
| system prompt has the new one written before its prompt, as text. A turn that sets none, or the one in effect, | ||
| is its prompt alone.""" | ||
| if not system_prompt or system_prompt == in_effect: | ||
| return prompt, in_effect | ||
| return f"{system_prompt}\n\n{prompt}", system_prompt | ||
|
|
||
|
|
||
| def _allowed_types(schema: Mapping[str, Any]) -> set[str]: | ||
| declared = schema.get("type") | ||
| allowed = set(declared) if isinstance(declared, list) else {declared} if declared else set() | ||
| for option in schema.get("anyOf") or schema.get("oneOf") or []: | ||
| allowed |= _allowed_types(option) | ||
| return allowed | ||
|
|
||
|
|
||
| def run_group_id(run: str, phase: Literal["train", "eval"], step: int) -> str: | ||
| """The run group a trainer files a group of episodes in: its training run, whether they were played to train or | ||
| to evaluate, and the training step whose policy played them. Every trainer names its groups so, and the explorer | ||
| lists a task's runs by them.""" | ||
| return f"{run}/{phase}/step-{step:04d}" | ||
|
|
||
|
|
||
| class TrainingSteps: | ||
| """The training step a trainer's batches are played at: a training batch's index, and for an evaluation the step | ||
| after the last training batch handed out, for a trainer that, as ``rl.train`` does, evaluates a step's policy | ||
| before it hands out that step's batch and once more after the last.""" | ||
|
|
||
| def __init__(self) -> None: | ||
| self._last_trained: int | None = None | ||
|
|
||
| def trained(self, index: int) -> int: | ||
| self._last_trained = index | ||
| return index | ||
|
|
||
| def evaluated(self) -> int: | ||
| return 0 if self._last_trained is None else self._last_trained + 1 | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class RunGroup: | ||
| """Where a trainer files a group of episodes: under the training batch ``trained`` names, fixed when the batch is | ||
| handed out, or, for an evaluation, under the step it is played at, read each time it is played, since a trainer | ||
| may make its evaluation's groups once and play them at every evaluation, as ``rl.train`` does.""" | ||
|
|
||
| run: str | ||
| steps: TrainingSteps | ||
| trained: int | None = None | ||
|
|
||
| def id(self) -> str: | ||
| if self.trained is not None: | ||
| return run_group_id(self.run, "train", self.trained) | ||
| return run_group_id(self.run, "eval", self.steps.evaluated()) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.