Skip to content

fix(claude-code): reach env MCP servers through the egress proxy on an armed clock - #818

Merged
earakely-scale merged 6 commits into
mainfrom
pratyush/clock-env-mcp-via-proxy
Nov 19, 2026
Merged

earakely-scale merged 6 commits into
mainfrom
pratyush/clock-env-mcp-via-proxy

Conversation

@psinghal20

@psinghal20 psinghal20 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

With the env clock armed (FD-5916), _spawn_claude put every MCP server's host in NO_PROXY. The CLI therefore reached the env's MCP server directly and checked its TLS certificate against the env's date. An env whose certificate was issued after that date failed to connect, and the agent got no env tools.

We hit this in an ST task run: virtual time 2026-11-03T09:00 America/New_York, the env's *.w.modal.host certificate valid from 2026-11-18, and the CLI listed the env MCP as failed with 0 env tools. Modal reissues these certificates every few months, so any task whose virtual date is a few weeks in the past is exposed.

On an armed clock, only loopback hosts now stay direct, the peer-agents MCP server's among them. Every other MCP host goes through the egress proxy, which runs on real time and also reaches private addresses (AGENTENV_EGRESS_PRIVATE). This removes cli_env's direct_hosts parameter, which only ever had an effect while the clock was armed. Unarmed behaviour is unchanged. The egress_env docstring and agents/README.md are updated to match.

How it was tested

  • pytest tst/unit/agents/claude_code/ tst/unit/agents/test_agent_clock_images.py: 282 passed, 30 skipped (all of the skips are in test_changelog.py). The new armed-clock test drives the real _spawn_claude with the subprocess faked: an https://…w.modal.host MCP host is not in NO_PROXY, HTTPS_PROXY is set and loopback stays direct. It fails on the old bridge.
  • ruff check is clean on the touched files.
  • A throwaway test on the egress proxy's own test harness (not committed): an MCP-style JSON-RPC POST answered as a server-sent-event stream came through, event by event, both over intercepted TLS with the clock outside the site certificate's validity and over plain HTTP.
  • End to end: the same rule, applied as a wrapper on this image's cli_env in a downstream agent built on the v0.9.1584 image, ran an ST task clone with sync_env_clock and the env's tools. That rerun is in progress and the result will be added here.

One rule for every clocked agent (added on top)

The MCP exemption came from the clock's design ("only the env's MCP hosts and a plain-HTTP model endpoint go direct"). It was copied into every clocked agent's bridge: the shared test required an identical copy of clock_armed, model_direct_hosts and egress_env in each one. Claude Code is the only clocked agent today, but each one that follows would have brought the exemption back. Two commits on top move the rule into one place:

  • src/agent_env/utils/agent_clock/agent_egress.py holds EGRESS_PROXY_ENV, clock_armed() and egress_env(environ, urls, armed=...). It reads the clock page through clockpage's own layout and check, so the format isn't copied either.

  • The rule: on an armed clock with the proxy, only these go direct:

    • loopback and host.docker.internal;
    • the deploy's NO_PROXY hosts;
    • the hosts of plain-HTTP URLs among the model endpoint and the MCP servers, which have no certificate to check.

    Every TLS host goes through the proxy, the env's MCP servers included. Unarmed, or without the proxy, nothing changes.

  • Only agent images take the folder (shared_sources.toml: agent_clock = "utils/agent_clock"). Changing the rule therefore rebuilds no computer base. source_digest is unchanged on every image under src/agent_env except claude-code.

  • claude-code: the bridge imports the rule. The Dockerfile and install.sh copy agent_egress.py and clockpage.py beside it; install_agent copies only the files it lists, so install.sh needed them.

  • Tests: test_agent_clock_images.py now requires every clocked agent to take agent_clock, import the rule, and define none of it. test_agent_egress.py covers the rule and the page reader (torn, short, foreign and missing pages, and a round trip through clockpage.write).

End-to-end and chaos, round 1 (dedicated amd64 VM, 0c4ec2a's code)

Setup:

  • Env: the shipped mail env on Modal. Its MCP URL is https://…w.modal.host/mcp, with certificate *.w.modal.host valid from 2026-11-17.
  • Agent: on local Docker.
  • Virtual times: 2026-10-27 (before the certificate) and 2026-12-01 (inside it).
  • Task: find an invoice with search_emails; a check reads the reply.
Run Result Evidence
main, Claude Code, armed before the certificate fails, check 0 MCP failed: HTTP Connection failed after 82ms: certificate is not yet valid (code: CERT_NOT_YET_VALID). The Modal host is in the CLI's NO_PROXY and never reaches the proxy.
main, unarmed passes direct
this branch, Claude Code, armed before the certificate passes proxy log: mode: intercept, decision: outside, verified: true for the Modal host
this branch, armed inside the certificate passes mode: splice, decision: inside (passed through)
this branch, unarmed passes no proxy variables
this branch, armed, local env over plain HTTP (host.docker.internal, 172.17.0.1) passes direct

Also proven:

  • The image builds from agent-env's context, and import claude_bridge works as agent.
  • tst/integration/task_step/test_agent_env_clock_local.py passes.
  • run_bundle_claude_code_install_test.py's two container installs (Debian, Alpine) pass, which covers install.sh's two new files.

Other agents (armed before the certificate; none follow the clock today, and their images are unchanged): opencode, pi, hermes, pydantic-ai, openhands, letta-code, codex and gemini all pass.

A second TLS stack: gemini built clocked in a throwaway image with the same clock pieces and this rule (not committed).

  • MCP sent direct: CERT_NOT_YET_VALID, no env tools.
  • With the rule: intercepted, passes.
  • Caveat for when gemini is put on the clock: gemini-cli builds its proxy agent from HTTPS_PROXY and ignores NO_PROXY. Its loopback calls then reach the proxy, which refuses loopback.

Chaos:

Case Outcome
Proxy killed between turns supervisord restarts it in about 3 s; the next turn passes
Proxy killed mid-turn back in about 2 s; the turn's next MCP call succeeds; the turn passes
Proxy stopped a clear failure after about 3 min (ECONNREFUSED), the same as main for the model endpoint; started again, the next turn passes
Clock jumped across the certificate's start, both ways passes; intercepted
Armed after deploy_agent the next turn goes through the proxy
Cleared mid-run the next turn goes direct
Rate 3600 and rate 0 pass
13 turns in a row all pass

Spend: about $1.80 of model spend and $0.54 of Modal.

Found, unrelated to this PR:

  • On Modal, _init_service_db_via_exec (env_gateway_provider.py) races the postgres image's temporary start-up server. pg_isready passes, then psql fails ("No such file or directory" or "the database system is shutting down") in 5 of about 38 deploys, all on the postgres:16-alpine fallback.
  • One Modal env sandbox died mid-run.

Round 2: after merging main (ddb5993)

main gained fe31f5a, which put OpenCode on the clock with its own copy of the rule. This PR's new shared-rule test caught it in CI. ddb5993 moves OpenCode onto agent_egress as well.

It also fixes Greptile's P1. NO_PROXY goes by host, so a host goes direct only when every URL for it is plain HTTP. A host also reached over HTTPS stays behind the proxy.

Build contexts: compared with main, only claude-code's and opencode's change.

Re-tested on a second dedicated VM:

  • Both images build, both bridges take the shared module, and test_agent_env_clock_local.py passes for both.
Run Result Evidence
OpenCode (this branch), armed before the certificate passes Modal MCP host intercept / outside
OpenCode, armed inside the certificate passes splice / inside
OpenCode, unarmed passes direct
OpenCode, local plain-HTTP env (host.docker.internal, 172.17.0.1) passes direct
main's OpenCode, Greptile's case: one Modal host reached over http:// and https:// fails, check 0 both URLs went direct; the HTTPS server status=failed. Shown with OpenCode's own Bun at a date before a certificate: direct gives CERT_NOT_YET_VALID, through the proxy gives HTTP 200.
OpenCode (this branch), Greptile's case passes HTTPS MCP host intercept / outside
Claude Code (this branch): armed before, unarmed, local plain-HTTP, Greptile's case all pass

Chaos on clocked OpenCode, run twice:

Case Outcome
Armed after deploy_agent the next turn is intercepted
Proxy killed between turns back in about 3 s; the next turn passes
Proxy killed mid-turn, inside a 30 s sleep serving again in about 1 s; the turn's next MCP call succeeds; the turn passes
Clock jumped across the certificate's start, both ways passes

One turn after a forward jump failed once: OpenCode dropped the MCP server 126 ms after a completed, intercepted TLS handshake. It didn't reproduce in three tries, the full chaos rerun passed 6/6, and OpenCode gives up on a remote MCP server after one failed request.

Spend, both rounds: about $4.60 of model spend at most, and $0.71 of Modal.

Found again, unrelated to this PR: the service-db start-up race on Modal hit 4 times in about 24 deploys.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Claude Code and OpenCode now share the rule for routing requests through the egress proxy when the env clock is armed.

  • Clocked Claude Code and OpenCode route TLS connections through the egress proxy.

No new actionable issues found. Tests were inspected but not run.

Reviews (7) · Last reviewed commit: "refactor(agents): OpenCode takes the sha..." · Reviewed by Greptile

…n armed clock

With the env clock armed, the CLI reached every MCP server directly and checked its TLS certificate against the
env's date, so an env whose certificate was issued after that date failed to connect and the agent got no env tools.
Only loopback hosts (the peer-agents MCP server's among them) stay direct now; every other MCP host goes through the
egress proxy, which runs on real time and reaches private addresses. Unarmed behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@psinghal20
psinghal20 requested a review from a team as a code owner November 19, 2026 17:37
earakely-scale and others added 2 commits November 19, 2026 10:41
…rvers through the proxy

The clock page check and the CLI's proxy variables move from Claude Code's bridge into
utils/faketime_clock/agent_egress.py, which reads the page by clockpage's own layout and
check. Every clocked agent's bridge imports it; its image copies it and clockpage.py
beside the bridge, and install.sh does the same for an installed agent.

On an armed clock every TLS host goes through the egress proxy, the env's MCP servers
included; only loopback, the deploy's NO_PROXY hosts and the hosts of plain-HTTP URLs
(the model endpoint's and every MCP server's) go direct. Unarmed, or without the proxy,
nothing changes. A page whose check never matches now reads as unarmed.

The AST-copy test becomes a check that every clocked bridge imports the rule, defines
none of it, and that its Dockerfile copies the module beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s take

agent_egress.py moves to utils/agent_clock, which claude-code's shared_sources.toml takes
as agent_clock; the Dockerfile and install.sh copy it beside the bridge with clock/'s
clockpage.py. The computer bases take utils/faketime_clock and utils/egress_proxy whole,
so the rule there would have changed their build contexts, and every edit to it would
rebuild them; their folders, READMEs included, are now as on main.

A page still mid-write or torn after the retries is taken at its armed flag again, as the
bridge's reader did: unarmed, the proxy relays untouched, so it is the safe side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@psinghal20

Copy link
Copy Markdown
Collaborator Author

For reference only. This doesn't need to block the refactor in progress here.

End-to-end check of the original rule from b089950c: on an armed clock only loopback MCP hosts stay direct, and env MCP goes through the egress proxy. It ran in a downstream agent built FROM the v0.9.1584 Claude Code image, with the rule applied as a wrapper on claude_bridge.cli_env. The later commits on this branch weren't exercised.

Setup: a clone of an ST task run, sync_env_clock at 2026-11-03T09:00 America/New_York, the env on *.w.modal.host (certificate notBefore 2026-11-18), and the agent under an egress allowlist (LiteLLM plus the S3 trajectory/snapshot hosts).

Without the rule With the rule
Env MCP in the CLI's init event failed, 0 env tools connected, 294 tools
Env tools used none Gmail, calendar, Notion, Duffel, get_time
date in the agent's shell Tue Nov 3 09:12 EST 2026 Tue Nov 3 09:15 EST 2026
Steps 11/11, but no env tools 11/11, including trajectory upload and agent snapshot

The runner was agent-env 0.9.1584. On 0.9.1571/0.9.1582 the same agent's clock sync fails with HTTP 401 (control token, #808).

earakely-scale and others added 2 commits November 19, 2026 11:46
…p-via-proxy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lso reached over TLS stays proxied

OpenCode's image, on the clock since main's #766, takes utils/agent_clock as agent_clock;
its Dockerfile and install.sh copy agent_egress.py and clock/'s clockpage.py beside the
bridge, which drops its own clock_armed, model_direct_hosts and egress_env and hands the
rule the model endpoint's URL and every MCP server's.

NO_PROXY matches by host, not scheme, so a host goes direct only when every URL for it is
plain HTTP; one also reached over HTTPS stays behind the proxy, which relays plain HTTP too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@andrewpark-scaleai andrewpark-scaleai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked this against my gateway and control token work (#820 to #825) and it doesn't overlap or widen anything. The proxy refuses loopback and link-local, so routing env MCP through it narrows what the agent can reach. The agent could already reach these hosts directly by dropping the proxy variables. The proxy logs only connection metadata, so MCP tokens don't land in its log. The Greptile fix holds. I ran the claude_code, opencode, test_agent_clock_images and test_agent_egress suites at ddb5993 and got 493 passed. The e2e table ran 0c4ec2a, so the Greptile fix and the OpenCode move haven't had an e2e run. The unit tests cover both. Nit: the title says claude-code, but the PR now moves OpenCode onto the shared rule too.

return {}
configured = [host.strip() for name in ("NO_PROXY", "no_proxy") for host in (environ.get(name) or "").split(",")]
parsed = [urlsplit(url) for url in urls]
tls = {parts.hostname for parts in parsed if parts.scheme != "http"}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. Every TLS MCP host now goes through the proxy, and the proxy checks upstreams only against its public roots (PublicRoots in main.go). Before this PR, Claude Code reached a TLS MCP server signed by a private CA directly, and that worked while the date sat inside the cert. Now an armed clock fails that server at any date. OpenCode already worked this way, and the Modal hosts use public certs, so I don't think anything breaks today. A line in the README would save the next person a debug session.

@earakely-scale

Copy link
Copy Markdown
Collaborator

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from the other open PRs rather than from this diff.

CLOCKED in tst/unit/agents/test_agent_clock_images.py is derived from any agents/*/shared_sources.toml that has a clock key, so an agent enrols itself just by adding that file. Five open, non-draft PRs each add one right now:

PR agent folder agent_clock key own clock_armed / model_direct_hosts / egress_env
#765 codex absent in codex_bridge.py
#768 hermes absent in hermes_bridge.py
#769 letta-code absent in letta_code_bridge.py
#770 gemini absent in gemini_bridge.py
#771 pi absent —

All five write only clock = "utils/faketime_clock" and egress = "utils/egress_proxy", so this PR's new test_the_egress_rule_is_taken_from_the_one_copy_only_agent_images_take raises on taken["agent_clock"] for each of them, and the check that replaces _clock_helpers — bridge takes clock_armed/egress_env from the shared module and defines none of it — fails for the four that carry their own copies.

The description anticipates the shape of this ("each one that follows would have brought the exemption back"); what isn't visible from here is that the five are in flight at the same time, so whichever side merges second has to carry the other's change. On the behavioural half the five are not uniform: #768 and #769 already send only plain-HTTP MCP hosts direct, which matches the rule here, while #765's description still states that MCP traffic stays direct either way.


Generated by Claude Code

@earakely-scale
earakely-scale merged commit a6e6796 into main Nov 19, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the pratyush/clock-env-mcp-via-proxy branch November 19, 2026 22:26
williamloo-scale added a commit that referenced this pull request Nov 19, 2026
The bridge imports clock_armed and egress_env from agent_clock's
agent_egress, which the image and an install copy beside it with
clockpage, and hands it the model endpoint's URL; its own copy of the
rule goes. mcp-remote, which codex hands only the trust variables, is
still not given the proxy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale added a commit that referenced this pull request Nov 19, 2026
The bridge imports clock_armed and egress_env from agent_egress, which
the image copies beside it with clockpage, and hands it the model
endpoint's URL and every MCP server's: a host also reached over HTTPS
now stays behind the proxy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale added a commit that referenced this pull request Nov 19, 2026
The bridge imports clock_armed and egress_env from agent_clock/agent_egress
and hands it every MCP server's URL, so an HTTPS MCP host goes through the
egress proxy on an armed clock (its certificate is checked on real time);
loopback, the model relay's among it, and plain-HTTP hosts stay direct.
The image and install.sh copy agent_egress.py and clockpage.py beside it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale added a commit that referenced this pull request Nov 19, 2026
The bridge imports clock_armed and egress_env from agent_egress and hands
it the model endpoint's URL and every MCP server's, so on an armed clock a
host reached over HTTPS stays behind the proxy even when it's also reached
over plain HTTP. The image copies agent_egress.py and clockpage.py beside
the bridge; an install (install/v1), which unpacks the build context with
them in agent_clock/ and clock/, puts those folders on the server's path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale added a commit that referenced this pull request Nov 19, 2026
The bridge imports clock_armed and egress_env from agent_clock/agent_egress
and hands it the model endpoint's URLs and every MCP server's, so on an
armed clock an HTTPS MCP host goes through the egress proxy like the rest.
The CLI still gets no HTTP(S)_PROXY (_CLI_PROXY_VARS): it ignores NO_PROXY.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
williamloo-scale added a commit that referenced this pull request Nov 19, 2026
…ss proxy on an armed clock

Codex hands mcp-remote only the variables its env_vars names, so on an
armed clock it went to an HTTPS MCP server directly and checked the
server's certificate against the env's date. It now gets the proxy's
variables too, and the bridge hands the egress rule every MCP server's
URL, so only plain-HTTP ones go direct, as #818 has for Claude Code and
OpenCode.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants