Skip to content

fix(config)!: a model key reaches only the URL it was issued for - #788

Merged
earakely-scale merged 6 commits into
mainfrom
edgararakelyan/fd-3938-p4-a-key-reaches-only-its-own-url
Nov 18, 2026
Merged

earakely-scale merged 6 commits into
mainfrom
edgararakelyan/fd-3938-p4-a-key-reaches-only-its-own-url

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

The configured model key, from [model] api_key or LITELLM_API_KEY, now goes only to the configured endpoint. Before this change, anything pointed at another URL with no key of its own was given the configured key. That covered an agent deploy, the direct judge, an agent judge and a unit-test verifier's container. Now each of them is refused before anything is called or started, and the message says how to pass a key.

Part of FD-3938 (P4 of FD-5974).

What changes

Where the URL comes from Before After
A deploy step's litellm_base_url, env_vars.LITELLM_BASE_URL, the run's litellm_base_url, or the agent's own default_env_vars Configured key sent to that URL Refused unless a key comes with it: LITELLM_API_KEY in env_vars or the agent's default_env_vars, or litellm_api_key in the run's overrides or step_params
Direct judge: the run's judge_litellm_base_url / litellm_base_url, or the step's stored default_model_api_base LITELLM_API_KEY sent to that URL. With only [model] api_key set, no key was sent, and litellm fell back to whatever provider key the environment held Refused unless judge_litellm_api_key / litellm_api_key is given
Agent judge, same overrides Configured key sent Refused unless a judge or run key is given
Unit-test verifier, env_vars / step_params LITELLM_BASE_URL or ANTHROPIC_BASE_URL Configured key sent with it Refused unless that URL's own key (LITELLM_API_KEY / ANTHROPIC_API_KEY) is given
The configured endpoint written another way: scheme or host case, a default port, a trailing /, one trailing /v1 Key sent; [model.headers] dropped Key and headers both sent (same_endpoint)
  • When the run is refused. Bundle preflight reports the refusal before agent-env run writes anything, for an agent deploy and for a unit-test verifier. A stored task run refuses at the step, which for an agent deploy is before its sandbox is created.
  • The refusal says how to pass a key, and the hint fits the caller.
    • An agent: its step's env_vars, its default_env_vars, or the run's or step's litellm_api_key.
    • A judge: the run's judge_litellm_api_key or litellm_api_key. A stored default_model_api_base has no key field beside it.
    • A unit-test verifier: the key in env_vars.
    • A server that needs no key accepts any non-empty value, and the message says so.
  • A trainer's solver endpoint is not affected. agent-env run with a solver endpoint, and training/harness.py, already pass a key with the URL.
  • Agents that take no credentials are skipped. Preflight reads takes_credentials. An agent with takes_credentials = false, such as the proxy, is handed no model variables, so it is checked for neither a key nor a configured endpoint. The second check was a pre-existing over-requirement.
  • Run-level overrides are checked when their step runs. A judge_litellm_base_url with no key fails at the judge step, after the solver has run. Bundle preflight sees only what the bundle's tasks name.
  • Why this is fix!. A caller that relied on the configured key following its URL now fails loudly.

Callers outside this repo (audited on default branches)

Break:

  • scaleapi/models gps/research/user-sim/agentenv-skyrl/src/agentenv_skyrl/rollouts.py, with_endpoints.
    • It sets the trained agent's litellm_base_url to the trainer's capture proxy with no key.
    • With --gateway-relay, it does the same for every other agent pointed at the relay. The relay depends on the configured key reaching it, and forwards it to the gateway.
    • Fix: set env_vars.LITELLM_API_KEY next to the URL. Any value works for the capture proxy; the relay needs the real gateway key.

Break only in narrow setups:

  • Hub runs with no key of stored tasks whose deploy_agent names an allowlisted non-proxy host (services/task_authoring.py) and carries no env_vars.LITELLM_API_KEY. The platform key used to go to that host. A comment in the hub says that host doesn't accept LiteLLM keys anyway.
  • iOS tasks baked with a non-default IOS_JUDGE_API_BASE / VERIFIER_MODEL_API_BASE, run on the hub with no key.
  • The scale-cua iOS demo script with LITELLM_BASE_URL=.../anthropic and no key env var.

Safe:

  • The hub strips every litellm* override from callers.
  • The worker, the drainer and modal-proxy pass the run key, or name no URL.
  • The scale-cua judge routing writes the proxy with or without /v1, which now matches the configured endpoint.

Not checked: stored documents (tasks, rubrics' default_model_api_base, agents' default_env_vars) that name another host.

Tests

  • tst/unit/task_step/test_model_key_scope.py. The first commit pinned today's behaviour; the second flips the rows that leaked the key to refusals.
    • Each refusal row also checks that no container was started, or no command or judge was run.
    • New rows cover a key passed beside an agent's own URL, a passed URL that beats an agent's own, a key given by step_params to a unit-test verifier, ANTHROPIC_BASE_URL, and the agent judge.
  • Rubrics judge tests. The contract tests configure the endpoint their default_model_api_base names. The precedence test passes a run key, and checks that a default_model_api_base with no key is refused.
  • tst/unit/config/test_model.py: same_endpoint, both ways round, for one endpoint written two ways and for two different endpoints.
  • tst/unit/bundle/preflight_test.py: preflight refuses each case, still passes the keyed and configured-URL deploys, and passes a trainer's solver endpoint and an agent that takes no credentials.
  • tst/integration/task_step/test_model_key_scope_local.py: two stand-in endpoints record the key each call brings.
    • The direct judge over HTTP: refused with nothing called, then Bearer theirs reaches the other endpoint only, then the configured endpoint written differently gets the configured key.
    • An echo agent deployed on local Docker: refused with no sandbox created, then its container's env holds its own key and URL and never the configured key.
  • tst/integration/cli/bundle_init_scopes_test.py: the source machine now configures the endpoint its own tasks name.
  • Updated: the header tests in local_container_urls_test.py give the other URL a key of their own, since they test headers and host rewriting, not the key.

E2E and chaos (Linux VM, local Docker)

Runs on the VM used a pydantic-ai solver and the direct judge. Two stand-in endpoints logged the stand-in key and X-Bill-To of every call: A is configured ([model] with X-Bill-To: acme), B is elsewhere. No real model key was used.

Case Branch Main at 25b862b
agent-env run, deploy names no URL A: solver and judge configured-key, acme —
Deploy names A as http://…:8765, or HTTP://…:8765/v1/ A: configured-key, acme A: configured-key, header dropped
Deploy names B, no key Refused at preflight; nothing written, 0 calls B: solver configured-key
Deploy names B, with env_vars.LITELLM_API_KEY=theirs B: solver theirs, no header; A: judge configured-key —
task run with run override litellm_base_url=B, no key Refused at the deploy; 0 calls, no sandbox B: solver configured-key; the judge fell back to provider env keys and failed (no OPENAI_API_KEY set)
Same, with --litellm-api-key theirs B: solver and judge theirs —
Run override judge_litellm_base_url=B, no key The solver ran on A, then the judge step was refused The judge fell back to provider env keys
Same, with judge_litellm_api_key=judges B: judge judges; A: solver configured-key —

Chaos:

  • B down, keyed run: the run fails at the prompt with a connection error, A gets no call, and nothing is left.
  • LITELLM_BASE_URL set to B over a file that names A:
    • A is refused at preflight;
    • B gets configured-key with the header;
    • adding LITELLM_API_KEY=env-key sends env-key to B.
  • task run --k 3 keyed: 6 calls, all theirs.
  • After every case: 0 sandbox containers and 0 sandbox folders left.

The new integration tests also passed on the VM. The agent test takes about 10 s with the echo image cached, and bundle_init_scopes_test.py passed too.

Docs (for P13, FD-6263)

  • The configured key's scope: when another URL needs its own key, and the error.
  • same_endpoint: which spellings match.
  • Passing a placeholder key for a server that needs no key.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge.

Summary

The PR keeps the configured model key tied to its endpoint. The latest changes make refusal messages specific to each caller and update judge tests.

  • Agent deploys need their own key when they name another model endpoint.
  • Judge calls need their own key when they use another endpoint.
  • Unit-test verifier commands need their own key for another URL.

No new actionable issues found.

Reviews (7) · Last reviewed commit: "fix(verifiers): a judge's own default_mo..." · Reviewed by Greptile

earakely-scale and others added 3 commits November 18, 2026 13:28
…el endpoint to today

Rows named leaks-* are the configured key reaching a URL it wasn't issued for;
the fix that follows flips them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The configured key, from [model] api_key or LITELLM_API_KEY, now goes only to the configured endpoint. An agent, a judge
or a unit-test verifier pointed at another URL with no key of its own is refused before anything starts, naming how to
pass one; before, it was handed the configured key. Endpoints compare as one when they differ only in scheme or host
case, a default port, a trailing slash or one trailing /v1, so the configured endpoint's headers reach it however it is
spelled. Bundle preflight reports the refusal before the run writes anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts tasks name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 21:44
Comment thread src/agent_env/bundle/preflight.py Resolved
earakely-scale and others added 3 commits November 18, 2026 13:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… takes no credentials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…judge key, and each refusal says how to pass one

The direct judge refuses a default_model_api_base, or a run override, other than the configured endpoint before it
calls resolve_model_call, naming the run overrides a judge reads its key from; an agent, a unit-test verifier and
preflight each name the fields theirs read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 4191685 into main Nov 18, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-3938-p4-a-key-reaches-only-its-own-url branch November 18, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant