Repository navigation
Conversation
A run launched with a caller's key (user_overrides litellm_api_key / litellm_base_url, as deploy_agent and deploy_env already honor) now puts that key and URL in the verifier container's LITELLM_* / ANTHROPIC_* env. Before, _merged_env always used agent-env's platform key, so LLM-judge verifiers never ran on, or billed, the launcher's key, and every one failed while the platform key was rejected by the proxy.
…ts key cannot be resolved agent-env's key never goes to an endpoint a run chose: user_overrides.litellm_base_url applies only when the run names its own key. A run key that cannot be resolved is logged as such and never swapped for agent-env's key.
revan-aleti
approved these changes
Nov 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
run_container_unit_tests_verifieralways gives its container agent-env's own platform LiteLLM key. It ignores the key the run was launched with, whichdeploy_agent,deploy_envandrubrics_verifierall honour. The LiteLLM proxy rejects the platform key, so every model call made from inside a container verifier fails authentication. A check that uses an LLM judge never gets an answer, and it can never bill the launcher's key.Seen on the Agent Env Hub: hub Audits run as this step type. Their model-calling checks (the LLM-judge Audits, and any self-serve check that calls a model) cannot be benched. The hub's contract text tells check authors to use
LITELLM_API_KEY, which does not work today.Now
_merged_envtakes the run'suser_overrides:judge_litellm_api_key, elselitellm_api_key, resolved likerubrics_verifierresolves it. It setsLITELLM_API_KEYandANTHROPIC_API_KEY.rubrics_verifier:judge_litellm_base_url(elselitellm_base_url) with a judge key, andlitellm_base_urlwith the run's own key. It setsLITELLM_BASE_URLandANTHROPIC_BASE_URL, and only when the run names its own key, so agent-env's key never goes to an endpoint a run chose.env_varssets still wins.Where the key comes from on the hub, so a reviewer can follow it:
agent_env_hub/routers/qc.py,start_task_run(encrypted_api_key=...)).context.metadata["user_overrides"]["litellm_api_key"](agent-env-worker/temporal/activities/run_task.py). That is the fielddeploy_agentalready reads, and the one this change now reads too.How it was tested
tst/unit/task_step/verifiers/run_container_unit_tests_verifier_test.py: 73 passed. New tests cover:env_varsstill win;tst/unit/task_step/: the same failures on this branch as onmain(109 each, all collection errors from optional extras missing in my venv:psycopg2,openenv), so nothing new breaks.ruff check(0.16.10) is clean.End to end, real container and real proxy. The real step ran on a local Docker container (
local_dockerprovider). The platform key was a fake the proxy rejects, as it rejects today's platform key. The verifier's command made one real model call (anthropic/claude-haiku-4-5, 5 tokens) throughlitellm-proxy.ml.scale.comand wrote what it saw:main(a57b8d3)user_overridesANTHROPIC_API_KEYmatchedLITELLM_API_KEYin every case. On that machine Docker cannot resolvehost-gateway, so the harness pointed the provider's host alias at the bridge IP; the step under test was not patched.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; the earlier endpoint mismatch is fixed.Summary
Container verifiers now use the run's model key, with judge-specific settings taking priority.
No new actionable issues found.
Reviews (3) · Last reviewed commit: "fix(verifiers): a run's judge key goes t..." · Reviewed by Greptile