Skip to content

feat(config): resolve each role's model in one place, and stop freezing the judge's model into stored tasks (FD-5976) - #781

Merged
earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-5976-p2-resolve-each-role-to-a-model
Nov 18, 2026
Merged

earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-5976-p2-resolve-each-role-to-a-model

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

Each role's model now comes from one resolver, resolve_model() in config/model.py. Before, the solver's, the user simulator's and the judge's models were each resolved in their own call site. P2 of FD-5974 (FD-5976).

  • Role names. ModelRole names the roles solver, user_simulator and judge.
    • [model.roles] accepts those names and keeps agent and user as synonyms.
    • It refuses a role set under both its name and its synonym, and a role mapped to anything but a model name.
    • metadata.models keeps its keys (solver, user simulator, judge).
  • One precedence: this step's step_params, the run's override, the step's own field, the deployed agent's default, then [model.roles], then [model] default (solver and judge only). The judge falls back to claude-sonnet-4-6 last.
    • The resolver is pure: it reads no environment, key or secret. So E3 (FD-5951) can resolve every role at plan time.
    • Every call site uses it: deploy_agent, install_agent, prompt_agent (solver and user simulator), deploy_user_sim_agent, always_on, rubrics_verifier and user_sim_judge.
  • The judge's model is no longer frozen into stored tasks.
    • Today, the constructor resolves the config's judge model and to_dict writes it into the task. So a stored task carries its writer's config, and a later [model.roles].judge never applies to it.
    • Now the step keeps only its author's model and resolves the rest when it runs (_judge_model).
    • Tasks stored before keep the model frozen into them. The case fix(verifiers): judge with the configured judge model in a rubrics verifier built in code #339 fixed still holds: a verifier built in code, like the A2A validator's, judges with the configured judge model.
  • model_overrides is removed from resolve_call and Config.resolve_model_call. Nothing passes it: I checked this repo and the scaleapi org's code search, where the sdk's and a research example's callers pass only model.

Behaviour

Every existing config and override resolves to the same model as before. tst/unit/task_step/test_role_models_golden.py pins that across 164 cases, at each call site that resolves a model:

  • 7 configs, including the agentenvhub-sdk prod shape;
  • every combination of step, run and step-params layers for each role, at each call site.

It was committed first, against main's code, and passes on top of the refactor. Two of its helpers move to the new API in that commit: the deploy-time solver default and the judge's run-time model. Every other case calls the same step methods before and after.

Two things change on purpose:

  • A rubrics judge written to a store with no model of its own now resolves its model when it runs, under the config it runs with. Before, it kept whatever its writer's config named. As a result, bundle init no longer reports dropping a default_model the author never set.
  • [model.roles] refuses agent together with solver, user together with user_simulator, and non-string values. Before, a table value was silently returned as the "model".

Unchanged on purpose: the user simulator's model from deploy_user_sim_agent and always_on still ignores [model.roles], since the step names it. The golden test pins this too.

Also fixed

Preflight reported any [model] table that fails to parse as "no model endpoint is set", because model_endpoint_missing caught every ConfigError from get_litellm_base_url(). That covers an unknown key, a bad header, and now a role named both ways. It now parses the table first, so the real error surfaces: [model.roles] sets both 'solver' and 'agent', which name the same role; keep 'solver'. The e2e run below found this.

Verification

Run at the head commit, 95c44fef.

Unit tier on Linux (an EC2 host, -n 14): 16,593 passed, 3 failed.

  • The three failures fail identically on main shipped to the same host: version_lookup_benchmark_test (no git checkout in a shipped copy) and the two test_mcp_cli_builder codegen tests (that host's environment).
  • Both pass locally.

E2E and chaos on the same host. agent-env run drives a bundle: the shipped pydantic-ai agent as solver, prompted, then graded by a direct rubrics judge and a response check. It runs against a stub OpenAI-compatible endpoint that logs the model each call names.

Case Solver call Judge call Run
[model.roles] solver / judge openai/solver-a judge-a passed
agent / judge (synonyms) openai/solver-a judge-a passed
agent and solver both set — — refused: "[model.roles] sets both 'solver' and 'agent' …"
judge role changed to judge-b, same bundle and stored task openai/solver-a judge-b passed
--model openai/solver-cli --judge-model openai/judge-cli openai/solver-cli judge-cli passed
chaos: the endpoint fails the first 3 solver calls with a 500 3×500, then openai/solver-a; the span rolled back and re-ran judge-a passed on attempt 2
main, judge role changed to judge-b after a first run openai/solver-a judge-a (frozen into the stored task) passed
  • The last row is the bug: on main, the task stored by the first run keeps the first config's judge model.

E2E against a real model (Scale's LiteLLM proxy, behind a pass-through that logs the model and caller of each call; a few cents of spend):

Case What the proxy saw Run
agent judge (use_agent_judge: true, Claude Code) with [model.roles] solver / judge pydantic-ai → gpt-4.1-mini; claude-cli → claude-haiku-4-5-20251001 passed
user simulator from [model.roles].user_simulator, no user_model in the task solver → gpt-4.1-mini; user simulator → gpt-4.1-mini-2025-04-14 completed (no verifier)
chaos: task run killed with SIGKILL during the judge step, then resumed with --context-json from its stored instance and --start-step 1, under a config that changes both the solver and judge roles the resumed prompt kept gpt-4.1-mini, the default_agent_model stored at deploy; the resumed judge used the new config's claude-haiku-5-5 passed

The kill found something older than this PR: the agent judge the killed run had deployed kept running, because a judge the verifier deploys for itself isn't recorded in the run's context, so neither the resume nor a teardown can find it. I removed it by hand.

  • Every run printed the same models: line as the stub logged.

Also worth knowing

  • [model.roles] is an open map, so a config could already hold solver or user_simulator, and core ignored them. They now take effect: solver sets the solver's default, user_simulator the model of prompt_agent's user simulator.
  • The hub doesn't rely on a stored judge's default_model, checked on scaleapi master: the backend's task validation skips a null, the hub's editor sets a model on every judge it creates and saves an empty one as unset, and the sdk's Harbor export falls back to claude-sonnet-4-6. One edge: the sdk exporter's _merged_verifier_step refuses to merge judges whose default_model differs, so a task written after this PR with one judge naming claude-sonnet-4-6 and another naming none now refuses where both used to store the same model.

Docs

The user guide's [model.roles] section should name the roles solver, user_simulator and judge, with agent and user as synonyms. It should also say that a judge with no model of its own uses the run's config.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge; no new blocking issues were found.

Summary

This PR centralizes model selection for each role and lets judges without an explicit model use the run's config.

  • Model roles accept clear names and their familiar aliases.
  • Solver and user-simulator model choices follow shared resolution rules.
  • Stored judges use the current run config when their author set no model.
  • Preflight shows model config errors instead of calling them missing endpoints.
  • Model-call APIs no longer accept inline model overrides.

No new actionable issues found. The earlier unused-config finding remains fixed.

Reviews (7) · Last reviewed commit: "refactor(config): split role resolution ..." · Reviewed by Greptile

earakely-scale and others added 3 commits November 18, 2026 09:45
…lves it today

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng the judge's model into stored tasks

ModelRole names the roles solver, user_simulator and judge; [model.roles] also
takes agent and user, and refuses a role named both ways or mapped to anything
but a model name. resolve_model() is the one precedence (step_params, run,
step, agent, then [model.roles], [model] default, and claude-sonnet-4-6 for the
judge), pure so a plan can resolve every role. Every call site uses it; the
golden test shows each resolves as before.

A rubrics judge no longer writes its writer's config into a stored task: it
keeps only its author's model and resolves the rest when it runs. Tasks stored
before keep the model frozen into them. resolve_call's unused model_overrides
seam is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not that no endpoint is set

model_endpoint_missing caught every ConfigError from get_litellm_base_url, so a
[model] table that failed to parse (an unknown key, a bad header, a role named
both ways) was reported as a missing model endpoint. It now parses the table
first, so its own error surfaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 18, 2026 18:21
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 18:21
earakely-scale and others added 2 commits November 18, 2026 10:25
Config.resolve_model parsed [model] before looking at the layers, so a model a
step, run or agent named no longer protected it from an unused fallback that
can't resolve, such as default = "env:UNSET". The old call sites
short-circuited; this restores that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odel, and record each role by ModelRole

Config.resolve_model is now named_model(...) or config_model(role, config), so
the rule that [model] is read only when no layer names a model lives in one
expression rather than a second pass over an exported set. ModelRole.record_key
gives each role's metadata.models key, which every record_model call uses in
place of a literal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 6022bbc into main Nov 18, 2026
41 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5976-p2-resolve-each-role-to-a-model branch November 18, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant