Repository navigation
feat(config): resolve each role's model in one place, and stop freezing the judge's model into stored tasks (FD-5976) - #781
Merged
earakely-scale merged 5 commits intoNov 18, 2026
Conversation
…lves it today Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng the judge's model into stored tasks ModelRole names the roles solver, user_simulator and judge; [model.roles] also takes agent and user, and refuses a role named both ways or mapped to anything but a model name. resolve_model() is the one precedence (step_params, run, step, agent, then [model.roles], [model] default, and claude-sonnet-4-6 for the judge), pure so a plan can resolve every role. Every call site uses it; the golden test shows each resolves as before. A rubrics judge no longer writes its writer's config into a stored task: it keeps only its author's model and resolves the rest when it runs. Tasks stored before keep the model frozen into them. resolve_call's unused model_overrides seam is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not that no endpoint is set model_endpoint_missing caught every ConfigError from get_litellm_base_url, so a [model] table that failed to parse (an unknown key, a bad header, a role named both ways) was reported as a missing model endpoint. It now parses the table first, so its own error surfaces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
marked this pull request as ready for review
November 18, 2026 18:21
Config.resolve_model parsed [model] before looking at the layers, so a model a step, run or agent named no longer protected it from an unused fallback that can't resolve, such as default = "env:UNSET". The old call sites short-circuited; this restores that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…odel, and record each role by ModelRole Config.resolve_model is now named_model(...) or config_model(role, config), so the rule that [model] is read only when no layer names a model lives in one expression rather than a second pass over an exported set. ModelRole.record_key gives each role's metadata.models key, which every record_model call uses in place of a literal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
deleted the
edgararakelyan/fd-5976-p2-resolve-each-role-to-a-model
branch
November 18, 2026 21:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Each role's model now comes from one resolver,
resolve_model()inconfig/model.py. Before, the solver's, the user simulator's and the judge's models were each resolved in their own call site. P2 of FD-5974 (FD-5976).ModelRolenames the rolessolver,user_simulatorandjudge.[model.roles]accepts those names and keepsagentanduseras synonyms.metadata.modelskeeps its keys (solver,user simulator,judge).step_params, the run's override, the step's own field, the deployed agent's default, then[model.roles], then[model] default(solver and judge only). The judge falls back toclaude-sonnet-4-6last.deploy_agent,install_agent,prompt_agent(solver and user simulator),deploy_user_sim_agent,always_on,rubrics_verifieranduser_sim_judge.to_dictwrites it into the task. So a stored task carries its writer's config, and a later[model.roles].judgenever applies to it._judge_model).model_overridesis removed fromresolve_callandConfig.resolve_model_call. Nothing passes it: I checked this repo and the scaleapi org's code search, where the sdk's and a research example's callers pass onlymodel.Behaviour
Every existing config and override resolves to the same model as before.
tst/unit/task_step/test_role_models_golden.pypins that across 164 cases, at each call site that resolves a model:It was committed first, against main's code, and passes on top of the refactor. Two of its helpers move to the new API in that commit: the deploy-time solver default and the judge's run-time model. Every other case calls the same step methods before and after.
Two things change on purpose:
bundle initno longer reports dropping adefault_modelthe author never set.[model.roles]refusesagenttogether withsolver,usertogether withuser_simulator, and non-string values. Before, a table value was silently returned as the "model".Unchanged on purpose: the user simulator's model from
deploy_user_sim_agentandalways_onstill ignores[model.roles], since the step names it. The golden test pins this too.Also fixed
Preflight reported any
[model]table that fails to parse as "no model endpoint is set", becausemodel_endpoint_missingcaught everyConfigErrorfromget_litellm_base_url(). That covers an unknown key, a bad header, and now a role named both ways. It now parses the table first, so the real error surfaces:[model.roles] sets both 'solver' and 'agent', which name the same role; keep 'solver'. The e2e run below found this.Verification
Run at the head commit,
95c44fef.Unit tier on Linux (an EC2 host,
-n 14): 16,593 passed, 3 failed.version_lookup_benchmark_test(no git checkout in a shipped copy) and the twotest_mcp_cli_buildercodegen tests (that host's environment).E2E and chaos on the same host.
agent-env rundrives a bundle: the shipped pydantic-ai agent as solver, prompted, then graded by a direct rubrics judge and a response check. It runs against a stub OpenAI-compatible endpoint that logs the model each call names.[model.roles]solver/judgeopenai/solver-ajudge-aagent/judge(synonyms)openai/solver-ajudge-aagentandsolverboth setjudge-b, same bundle and stored taskopenai/solver-ajudge-b--model openai/solver-cli --judge-model openai/judge-cliopenai/solver-clijudge-cliopenai/solver-a; the span rolled back and re-ranjudge-ajudge-bafter a first runopenai/solver-ajudge-a(frozen into the stored task)E2E against a real model (Scale's LiteLLM proxy, behind a pass-through that logs the model and caller of each call; a few cents of spend):
use_agent_judge: true, Claude Code) with[model.roles]solver/judgegpt-4.1-mini; claude-cli →claude-haiku-4-5-20251001[model.roles].user_simulator, nouser_modelin the taskgpt-4.1-mini; user simulator →gpt-4.1-mini-2025-04-14task runkilled with SIGKILL during the judge step, then resumed with--context-jsonfrom its stored instance and--start-step 1, under a config that changes both the solver and judge rolesgpt-4.1-mini, thedefault_agent_modelstored at deploy; the resumed judge used the new config'sclaude-haiku-5-5The kill found something older than this PR: the agent judge the killed run had deployed kept running, because a judge the verifier deploys for itself isn't recorded in the run's context, so neither the resume nor a teardown can find it. I removed it by hand.
models:line as the stub logged.Also worth knowing
[model.roles]is an open map, so a config could already holdsolveroruser_simulator, and core ignored them. They now take effect:solversets the solver's default,user_simulatorthe model ofprompt_agent's user simulator.default_model, checked on scaleapi master: the backend's task validation skips a null, the hub's editor sets a model on every judge it creates and saves an empty one as unset, and the sdk's Harbor export falls back toclaude-sonnet-4-6. One edge: the sdk exporter's_merged_verifier_steprefuses to merge judges whosedefault_modeldiffers, so a task written after this PR with one judge namingclaude-sonnet-4-6and another naming none now refuses where both used to store the same model.Docs
The user guide's
[model.roles]section should name the rolessolver,user_simulatorandjudge, withagentanduseras synonyms. It should also say that a judge with no model of its own uses the run's config.🤖 Generated with Claude Code


Confidence Score: 5/5
The reviewed changes appear safe to merge; no new blocking issues were found.Summary
This PR centralizes model selection for each role and lets judges without an explicit model use the run's config.
No new actionable issues found. The earlier unused-config finding remains fixed.
Reviews (7) · Last reviewed commit: "refactor(config): split role resolution ..." · Reviewed by Greptile