Repository navigation
fix(config)!: a model key reaches only the URL it was issued for - #788
Merged
earakely-scale merged 6 commits intoNov 18, 2026
Merged
earakely-scale merged 6 commits into
earakely-scale merged 6 commits into
Conversation
…el endpoint to today Rows named leaks-* are the configured key reaching a URL it wasn't issued for; the fix that follows flips them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The configured key, from [model] api_key or LITELLM_API_KEY, now goes only to the configured endpoint. An agent, a judge or a unit-test verifier pointed at another URL with no key of its own is refused before anything starts, naming how to pass one; before, it was handed the configured key. Endpoints compare as one when they differ only in scheme or host case, a default port, a trailing slash or one trailing /v1, so the configured endpoint's headers reach it however it is spelled. Bundle preflight reports the refusal before the run writes anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts tasks name Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… takes no credentials Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…judge key, and each refusal says how to pass one The direct judge refuses a default_model_api_base, or a run override, other than the configured endpoint before it calls resolve_model_call, naming the run overrides a judge reads its key from; an agent, a unit-test verifier and preflight each name the fields theirs read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale
deleted the
edgararakelyan/fd-3938-p4-a-key-reaches-only-its-own-url
branch
November 18, 2026 22:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The configured model key, from
[model] api_keyorLITELLM_API_KEY, now goes only to the configured endpoint. Before this change, anything pointed at another URL with no key of its own was given the configured key. That covered an agent deploy, the direct judge, an agent judge and a unit-test verifier's container. Now each of them is refused before anything is called or started, and the message says how to pass a key.Part of FD-3938 (P4 of FD-5974).
What changes
litellm_base_url,env_vars.LITELLM_BASE_URL, the run'slitellm_base_url, or the agent's owndefault_env_varsLITELLM_API_KEYinenv_varsor the agent'sdefault_env_vars, orlitellm_api_keyin the run's overrides orstep_paramsjudge_litellm_base_url/litellm_base_url, or the step's storeddefault_model_api_baseLITELLM_API_KEYsent to that URL. With only[model] api_keyset, no key was sent, and litellm fell back to whatever provider key the environment heldjudge_litellm_api_key/litellm_api_keyis givenenv_vars/step_paramsLITELLM_BASE_URLorANTHROPIC_BASE_URLLITELLM_API_KEY/ANTHROPIC_API_KEY) is given/, one trailing/v1[model.headers]droppedsame_endpoint)agent-env runwrites anything, for an agent deploy and for a unit-test verifier. A storedtask runrefuses at the step, which for an agent deploy is before its sandbox is created.env_vars, itsdefault_env_vars, or the run's or step'slitellm_api_key.judge_litellm_api_keyorlitellm_api_key. A storeddefault_model_api_basehas no key field beside it.env_vars.agent-env runwith a solver endpoint, andtraining/harness.py, already pass a key with the URL.takes_credentials. An agent withtakes_credentials = false, such as the proxy, is handed no model variables, so it is checked for neither a key nor a configured endpoint. The second check was a pre-existing over-requirement.judge_litellm_base_urlwith no key fails at the judge step, after the solver has run. Bundle preflight sees only what the bundle's tasks name.fix!. A caller that relied on the configured key following its URL now fails loudly.Callers outside this repo (audited on default branches)
Break:
scaleapi/modelsgps/research/user-sim/agentenv-skyrl/src/agentenv_skyrl/rollouts.py,with_endpoints.litellm_base_urlto the trainer's capture proxy with no key.--gateway-relay, it does the same for every other agent pointed at the relay. The relay depends on the configured key reaching it, and forwards it to the gateway.env_vars.LITELLM_API_KEYnext to the URL. Any value works for the capture proxy; the relay needs the real gateway key.Break only in narrow setups:
deploy_agentnames an allowlisted non-proxy host (services/task_authoring.py) and carries noenv_vars.LITELLM_API_KEY. The platform key used to go to that host. A comment in the hub says that host doesn't accept LiteLLM keys anyway.IOS_JUDGE_API_BASE/VERIFIER_MODEL_API_BASE, run on the hub with no key.scale-cuaiOS demo script withLITELLM_BASE_URL=.../anthropicand no key env var.Safe:
litellm*override from callers.scale-cuajudge routing writes the proxy with or without/v1, which now matches the configured endpoint.Not checked: stored documents (tasks, rubrics'
default_model_api_base, agents'default_env_vars) that name another host.Tests
tst/unit/task_step/test_model_key_scope.py. The first commit pinned today's behaviour; the second flips the rows that leaked the key to refusals.step_paramsto a unit-test verifier,ANTHROPIC_BASE_URL, and the agent judge.default_model_api_basenames. The precedence test passes a run key, and checks that adefault_model_api_basewith no key is refused.tst/unit/config/test_model.py:same_endpoint, both ways round, for one endpoint written two ways and for two different endpoints.tst/unit/bundle/preflight_test.py: preflight refuses each case, still passes the keyed and configured-URL deploys, and passes a trainer's solver endpoint and an agent that takes no credentials.tst/integration/task_step/test_model_key_scope_local.py: two stand-in endpoints record the key each call brings.Bearer theirsreaches the other endpoint only, then the configured endpoint written differently gets the configured key.tst/integration/cli/bundle_init_scopes_test.py: the source machine now configures the endpoint its own tasks name.local_container_urls_test.pygive the other URL a key of their own, since they test headers and host rewriting, not the key.E2E and chaos (Linux VM, local Docker)
Runs on the VM used a pydantic-ai solver and the direct judge. Two stand-in endpoints logged the stand-in key and
X-Bill-Toof every call: A is configured ([model]withX-Bill-To: acme), B is elsewhere. No real model key was used.agent-env run, deploy names no URLconfigured-key,acmehttp://…:8765, orHTTP://…:8765/v1/configured-key,acmeconfigured-key, header droppedconfigured-keyenv_vars.LITELLM_API_KEY=theirstheirs, no header; A: judgeconfigured-keytask runwith run overridelitellm_base_url=B, no keyconfigured-key; the judge fell back to provider env keys and failed (noOPENAI_API_KEYset)--litellm-api-key theirstheirsjudge_litellm_base_url=B, no keyjudge_litellm_api_key=judgesjudges; A: solverconfigured-keyChaos:
LITELLM_BASE_URLset to B over a file that names A:configured-keywith the header;LITELLM_API_KEY=env-keysendsenv-keyto B.task run --k 3keyed: 6 calls, alltheirs.The new integration tests also passed on the VM. The agent test takes about 10 s with the echo image cached, and
bundle_init_scopes_test.pypassed too.Docs (for P13, FD-6263)
same_endpoint: which spellings match.🤖 Generated with Claude Code


Confidence Score: 5/5
The reviewed changes appear safe to merge.Summary
The PR keeps the configured model key tied to its endpoint. The latest changes make refusal messages specific to each caller and update judge tests.
No new actionable issues found.
Reviews (7) · Last reviewed commit: "fix(verifiers): a judge's own default_mo..." · Reviewed by Greptile