Skip to content

fix(verifiers): grade a container verifier on the run's own LiteLLM key - #773

Open
lilyyma wants to merge 3 commits into
mainfrom
lily/verifier-run-litellm-key-v2
Open

lilyyma wants to merge 3 commits into
mainfrom
lily/verifier-run-litellm-key-v2

Conversation

@lilyyma

@lilyyma lilyyma commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

run_container_unit_tests_verifier always gives its container agent-env's own platform LiteLLM key. It ignores the key the run was launched with, which deploy_agent, deploy_env and rubrics_verifier all honour. The LiteLLM proxy rejects the platform key, so every model call made from inside a container verifier fails authentication. A check that uses an LLM judge never gets an answer, and it can never bill the launcher's key.

Seen on the Agent Env Hub: hub Audits run as this step type. Their model-calling checks (the LLM-judge Audits, and any self-serve check that calls a model) cannot be benched. The hub's contract text tells check authors to use LITELLM_API_KEY, which does not work today.

Now _merged_env takes the run's user_overrides:

  • The key is judge_litellm_api_key, else litellm_api_key, resolved like rubrics_verifier resolves it. It sets LITELLM_API_KEY and ANTHROPIC_API_KEY.
  • The URL goes with the key, as in rubrics_verifier: judge_litellm_base_url (else litellm_base_url) with a judge key, and litellm_base_url with the run's own key. It sets LITELLM_BASE_URL and ANTHROPIC_BASE_URL, and only when the run names its own key, so agent-env's key never goes to an endpoint a run chose.
  • A run key that cannot be resolved is logged as such, and agent-env's key is never used in its place.
  • A run that names neither keeps the platform key and URL, as before.
  • A key the step's own env_vars sets still wins.

Where the key comes from on the hub, so a reviewer can follow it:

  • An Audit launch encrypts the launcher's key alias and passes it to each run (agent_env_hub/routers/qc.py, start_task_run(encrypted_api_key=...)).
  • The worker decrypts it into context.metadata["user_overrides"]["litellm_api_key"] (agent-env-worker/temporal/activities/run_task.py). That is the field deploy_agent already reads, and the one this change now reads too.
  • The worker also strips any caller-set LiteLLM URL, so on the hub the URL stays the platform's.

How it was tested

  • tst/unit/task_step/verifiers/run_container_unit_tests_verifier_test.py: 73 passed. New tests cover:

    • the run's key and URL reach the container env, with the judge key first;
    • a judge key goes to the judge URL, not the agent's (and a judge URL alone never moves the run's key);
    • a key given as a reference is resolved;
    • a run without overrides keeps the platform key and URL;
    • an endpoint override moves the URL too;
    • the step's own env_vars still win;
    • the executed step hands the run's key to its container.
  • tst/unit/task_step/: the same failures on this branch as on main (109 each, all collection errors from optional extras missing in my venv: psycopg2, openenv), so nothing new breaks.

  • ruff check (0.16.10) is clean.

  • End to end, real container and real proxy. The real step ran on a local Docker container (local_docker provider). The platform key was a fake the proxy rejects, as it rejects today's platform key. The verifier's command made one real model call (anthropic/claude-haiku-4-5, 5 tokens) through litellm-proxy.ml.scale.com and wrote what it saw:

    Run main (a57b8d3) this branch
    names the caller's key in user_overrides HTTP 401, the platform key reached the container, reward 0 HTTP 200, the caller's key reached it, reward 1
    names no key 401, platform key 401, platform key (unchanged)

    ANTHROPIC_API_KEY matched LITELLM_API_KEY in every case. On that machine Docker cannot resolve host-gateway, so the harness pointed the provider's host alias at the bridge IP; the step under test was not patched.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; the earlier endpoint mismatch is fixed.

Summary

Container verifiers now use the run's model key, with judge-specific settings taking priority.

  • Container verifiers use the run's key for model calls.

No new actionable issues found.

Reviews (3) · Last reviewed commit: "fix(verifiers): a run's judge key goes t..." · Reviewed by Greptile

A run launched with a caller's key (user_overrides litellm_api_key /
litellm_base_url, as deploy_agent and deploy_env already honor) now puts
that key and URL in the verifier container's LITELLM_* / ANTHROPIC_* env.
Before, _merged_env always used agent-env's platform key, so LLM-judge
verifiers never ran on, or billed, the launcher's key, and every one
failed while the platform key was rejected by the proxy.
@lilyyma
lilyyma requested a review from a team as a code owner November 18, 2026 16:00
…ts key cannot be resolved

agent-env's key never goes to an endpoint a run chose: user_overrides.litellm_base_url applies only
when the run names its own key. A run key that cannot be resolved is logged as such and never
swapped for agent-env's key.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants