Skip to content

feat(task): tasks declare headline_verifiers - #803

Merged
earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-5921-g4-tasks-declare-headline_verifiers
Nov 19, 2026
Merged

earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-5921-g4-tasks-declare-headline_verifiers

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What

A task can now say which of its grades are its reward. headline_verifiers is a list of step ids on Task:

  • One names the reward, and the run's pass or fail.
  • Several make a multi-reward over the grades they name.
  • None makes a multi-reward over every grade.

Nothing reads the field yet. G7's result rules and G9's RL rewards will. This PR declares it, stores it, versions it, and makes sure every path that rebuilds a task keeps it.

{"headline_verifiers": ["final"], "steps": [ ... ]}

A bundle task file may now be this object. A bare array of steps still loads as before, with no headlines.

Decisions

  1. The hub half is its own issue. The hub's POST /tasks round-trips the field, but that work lands in scaleapi after a release that includes this PR and a pin bump. It's filed under FD-5919, blocked by FD-5921, so FD-5921 closes with this PR.
  2. What a task refuses when it's built. It refuses a headline that names no step of the task, or one named twice. Whether the named step is a verifier is left to G6's task lint: agentenvhub-sdk's own verifiers only move onto VerifierTaskStep in G3, so refusing a non-verifier now would block tasks that name them.
  3. [] means the same as absent. It isn't saved, so the task's document and its ledger digest match a task that never set the field.

Defaults I chose

  • Storage. Task.headline_verifiers keeps the order given and is saved only when set, as requires is. from_dict reads it only when it's a list. No stored task changes shape.
  • Task files. A task file is a bare array, or an object that takes only steps and headline_verifiers. Any other key is refused, naming it. The bundle parser and agent-env task create read the file through one function, bundle.parse.task_file_parts. bundle init and agent-env task get write it through its counterpart, task_file_body, so the shape is defined in one place.
  • Bundle internals. BundleEntry.headline_verifiers carries the field (compare=False, like config), and config stays the steps list, so no code that reads the steps list changes.
  • Ledger. Its digest includes headline_verifiers only when set, so every digest an earlier release recorded is unchanged (ledger_digest_test.py pins them). Changing the field writes a new version and says why, e.g. tasks/t.json: v2 (headline_verifiers final).
  • Carried everywhere a task is rebuilt:
    • the plan's validation Task, which also means a bad headline is refused before anything is written;
    • materialize;
    • bundle init, which writes the object form when a task has headlines;
    • an episode's copy (EpisodeSpec), which G9 will read;
    • agent-env task create, which takes the object form;
    • agent-env task get, which prints the object form, so get and create round-trip.
  • One small behavior change in task create. A task it can't build now prints Error: … and exits 1, where before it printed a traceback.

Tests

  • Task: one, several and no headlines round-trip in order. [] and a non-list aren't saved. An unknown, non-string or repeated headline is refused. Each version keeps its own headlines.
  • Parse: the object form, [], and the bare array; unknown keys and a non-list headline_verifiers are refused.
  • Materialize:
    • a bare file and {"headline_verifiers": [], ...} reuse the same version;
    • adding headlines writes v2 with its reason, and changing them writes the next version;
    • a headline that names no step is refused before anything is written, in a real run and in a dry run.
  • bundle init: a task with headlines exports as the object form, and the exported folder parses back to the same headlines.
  • Other paths: an episode's copy keeps them; task create saves them and refuses bad files with nothing written; task get prints either form.
  • Integration, in tst/integration/cli/run_bundle_test.py: agent-env run on a copy of hello with {"headline_verifiers": ["hello"], ...} passes, and a rerun reuses v1, whose stored task keeps the headline. Going back to a bare array writes v2 (no headline_verifiers), and v1 keeps its headline.

e2e and chaos

All on an agent-environments-admin vm, at this PR's head, on that machine's local stores. No model was used.

e2e:

  • Object-form task file: agent-env run on a copy of hello whose file names hello as its headline, with a second verifier beside it. v1 passes, and a rerun reuses it.
  • bundle register: registering into the store writes v1, and registering again reuses it. Changing the headlines to ["hello", "control"] writes v2 (headline_verifiers hello, control). agent-env task get prints v2 with both headlines, and v1 with its own.
  • bundle init --from-task: the exported tasks/g4-hello.json is the object form with both headlines. Run from a fresh state, the exported folder passes.
  • Round trip: task get output fed to task create --id g4-copied saves a task with the same headlines.

Chaos (a few minutes):

Task file Result
headline_verifiers a string, a list holding a number, or no steps refused: "a task file must be a JSON array of step objects, or an object whose steps is one…"
an unknown key (headline) refused: "unknown key 'headline'; a task file's object takes steps and headline_verifiers"
a headline named twice refused: "headline_verifiers names ['hello'] more than once"
a headline naming no step refused: "headline_verifiers names no step in this task: ['nope'] (its steps: […])"
headline_verifiers: null runs, with no headlines
headline_verifiers: [] runs, reusing the version null wrote: one stored form
5,000 steps, all of them headlines dry run in 1.6 s
two agent-env runs of one bundle at once both pass, one v2 between them, headlines kept

CI time: no slower than main's latest green run. The fast integration job took 177 s against 185 s; the slow shards and unit tier came in at or below main's, and the gateway shards, which this PR doesn't touch, were within run-to-run variance.

Docs to cover

  • The object form of a task file.
  • What headline_verifiers means: one, several or none.
  • Errors for an unknown or repeated headline.
  • task create and task get with the object form.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Adds headline_verifiers to task definitions and carries it through storage, bundles, CLI commands, and episode copies. Reward calculation does not use it yet.

  • Tasks can now mark which step grades count as their reward.
  • Bundle task files keep reward selections through runs and exports.
  • Task create and get now round-trip headline selections.

No actionable issues found.

Reviews (8) · Last reviewed commit: "Merge branch 'main' into edgararakelyan/..." · Reviewed by Greptile

earakely-scale and others added 2 commits November 18, 2026 20:16
A task names, by step id, the grades that are its reward: one is the reward and the run's pass or fail, several a
multi-reward over them, none a multi-reward over every grade. Saved only when set, so no stored task or ledger digest
changes; a bundle task file may be {"headline_verifiers": [...], "steps": [...]}, and a bare array still loads with
none. Construction refuses a headline that names no step of the task, or one named twice. The plan, materialize,
the ledger, bundle init, episodes, task create and task get all carry it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 04:17
earakely-scale and others added 3 commits November 18, 2026 23:21
…s reader

bundle init and task get write the object form through task_file_body, which task_file_parts reads back, and the
headline tests share one graded task's steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit f381f60 into main Nov 19, 2026
35 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5921-g4-tasks-declare-headline_verifiers branch November 19, 2026 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant