Repository navigation
feat(task): tasks declare headline_verifiers - #803
Merged
earakely-scale merged 5 commits intoNov 19, 2026
Merged
earakely-scale merged 5 commits into
earakely-scale merged 5 commits into
Conversation
A task names, by step id, the grades that are its reward: one is the reward and the run's pass or fail, several a
multi-reward over them, none a multi-reward over every grade. Saved only when set, so no stored task or ledger digest
changes; a bundle task file may be {"headline_verifiers": [...], "steps": [...]}, and a bare array still loads with
none. Construction refuses a headline that names no step of the task, or one named twice. The plan, materialize,
the ledger, bundle init, episodes, task create and task get all carry it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1-g4-tasks-declare-headline_verifiers
…s reader bundle init and task get write the object form through task_file_body, which task_file_parts reads back, and the headline tests share one graded task's steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1-g4-tasks-declare-headline_verifiers
earakely-scale
deleted the
edgararakelyan/fd-5921-g4-tasks-declare-headline_verifiers
branch
November 19, 2026 16:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A task can now say which of its grades are its reward.
headline_verifiersis a list of step ids onTask:Nothing reads the field yet. G7's result rules and G9's RL rewards will. This PR declares it, stores it, versions it, and makes sure every path that rebuilds a task keeps it.
{"headline_verifiers": ["final"], "steps": [ ... ]}A bundle task file may now be this object. A bare array of steps still loads as before, with no headlines.
Decisions
POST /tasksround-trips the field, but that work lands in scaleapi after a release that includes this PR and a pin bump. It's filed under FD-5919, blocked by FD-5921, so FD-5921 closes with this PR.VerifierTaskStepin G3, so refusing a non-verifier now would block tasks that name them.[]means the same as absent. It isn't saved, so the task's document and its ledger digest match a task that never set the field.Defaults I chose
Task.headline_verifierskeeps the order given and is saved only when set, asrequiresis.from_dictreads it only when it's a list. No stored task changes shape.stepsandheadline_verifiers. Any other key is refused, naming it. The bundle parser andagent-env task createread the file through one function,bundle.parse.task_file_parts.bundle initandagent-env task getwrite it through its counterpart,task_file_body, so the shape is defined in one place.BundleEntry.headline_verifierscarries the field (compare=False, likeconfig), andconfigstays the steps list, so no code that reads the steps list changes.headline_verifiersonly when set, so every digest an earlier release recorded is unchanged (ledger_digest_test.pypins them). Changing the field writes a new version and says why, e.g.tasks/t.json: v2 (headline_verifiers final).Task, which also means a bad headline is refused before anything is written;bundle init, which writes the object form when a task has headlines;EpisodeSpec), which G9 will read;agent-env task create, which takes the object form;agent-env task get, which prints the object form, sogetandcreateround-trip.task create. A task it can't build now printsError: …and exits 1, where before it printed a traceback.Tests
Task: one, several and no headlines round-trip in order.[]and a non-list aren't saved. An unknown, non-string or repeated headline is refused. Each version keeps its own headlines.[], and the bare array; unknown keys and a non-listheadline_verifiersare refused.{"headline_verifiers": [], ...}reuse the same version;bundle init: a task with headlines exports as the object form, and the exported folder parses back to the same headlines.task createsaves them and refuses bad files with nothing written;task getprints either form.tst/integration/cli/run_bundle_test.py:agent-env runon a copy ofhellowith{"headline_verifiers": ["hello"], ...}passes, and a rerun reuses v1, whose stored task keeps the headline. Going back to a bare array writes v2 (no headline_verifiers), and v1 keeps its headline.e2e and chaos
All on an agent-environments-admin vm, at this PR's head, on that machine's local stores. No model was used.
e2e:
agent-env runon a copy ofhellowhose file nameshelloas its headline, with a second verifier beside it. v1 passes, and a rerun reuses it.bundle register: registering into the store writes v1, and registering again reuses it. Changing the headlines to["hello", "control"]writes v2 (headline_verifiers hello, control).agent-env task getprints v2 with both headlines, and v1 with its own.bundle init --from-task: the exportedtasks/g4-hello.jsonis the object form with both headlines. Run from a fresh state, the exported folder passes.task getoutput fed totask create --id g4-copiedsaves a task with the same headlines.Chaos (a few minutes):
headline_verifiersa string, a list holding a number, or nostepsheadline)headline_verifiers: nullheadline_verifiers: []nullwrote: one stored formagent-env runs of one bundle at onceCI time: no slower than main's latest green run. The fast integration job took 177 s against 185 s; the slow shards and unit tier came in at or below main's, and the gateway shards, which this PR doesn't touch, were within run-to-run variance.
Docs to cover
headline_verifiersmeans: one, several or none.task createandtask getwith the object form.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge.Summary
Adds
headline_verifiersto task definitions and carries it through storage, bundles, CLI commands, and episode copies. Reward calculation does not use it yet.No actionable issues found.
Reviews (8) · Last reviewed commit: "Merge branch 'main' into edgararakelyan/..." · Reviewed by Greptile