fix(agent): retry upstream clones so a GitHub 429 stops failing the image build (#67)

## Symptom

`docker compose build` has been failing intermittently on unrelated PRs
— most recently twice today on `cursor/admin-profile-switch-c497`. Every
other job passes; only the agent image fails.

## Root cause

Not a code defect. Both upstream clones in `agent/Dockerfile` are
**unauthenticated** requests to github.com, issued **once, with no
retry**:

```
Cloning into 'runtime'...
remote: This request was rate-limited due to too many requests.
fatal: unable to access 'https://github.com/NousResearch/hermes-agent.git/':
       The requested URL returned error: 429
exit code: 128
```

GitHub rate-limits unauthenticated traffic **per source IP**, and
Actions runners share pooled egress addresses. Under load the clone is
refused and the whole image build dies with it. Nothing about the PR
under test is involved — which is exactly why it looked random and kept
landing on innocent branches.

**Failure rate: 2 of the last 15 `docker compose build` runs.** That
flakiness pattern is itself the diagnosis; a real build break would fail
15 of 15.

## Fix

Both clones retry up to 5 times with escalating backoff (15/30/45/60s).
Retry is the right layer *because* the condition is transient — but the
loop still prints `FATAL` and **exits 1** on the last attempt, so an
upstream that is genuinely gone fails the build loudly rather than
yielding an image with no runtime in it.

This is not CI-only defensiveness: `scripts/self-host/install.sh` runs
the same build, so a self-host user behind a shared NAT hits the
identical refusal.

Also noted in a comment: the webui clone's existing `|| git clone`
fallback only ever handled a *moved ref*. Against a rate-limit it just
re-issued the same refused request, so it needed the outer retry to be
of any use at all.

## Verification

- [x] Both edited `RUN` blocks pass `sh -n` as the shell actually
receives them (continuations joined, in-`RUN` comments stripped).
Baseline check caught a bug in my own extractor first — all 6 blocks
including 3 untouched ones flagged, which is how I knew the harness was
wrong rather than the Dockerfile.
- [x] Success path: the wrapper performs a real `git clone` of
hermes-agent — exit 0, 76 entries, `pyproject.toml` present.
- [x] Give-up path: against an unreachable repo it retries 5×, prints
`FATAL`, exits **1** (not 0).
- [ ] CI `docker compose build` green on this PR — the authoritative
test.

## Follow-up, deliberately not bundled here

`AGENT_RUNTIME_REF=main` and `AGENT_API_REF=master` are **unpinned
branches**, which contradicts the pinning rule in `CLAUDE.md` (*"PINNED
for reproducibility ... Bump deliberately and re-validate"*). That's a
second, independent failure mode — an upstream commit can break this
build with no change on our side, which is precisely the class of bug
the `mcp>=1.0,<2` pin two lines below exists to prevent. Current
upstream tags are `v2026.8.18` (hermes-agent) and `v0.52.76`
(hermes-webui). Worth a separate PR that pins and re-validates.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Venkat SF <venkatesh.sakamuri@stayflexi.com>

Author: 3 people

CLAUDE.md

@@ -238,6 +238,21 @@ Admins can **View as** a sub-user from the top-right of the home layout (`Profil
    `hermes_requires <0.20.0` on a "verified" 401 that came from a hand-rolled
    `hermes serve` run rather than `hermes webui`. 0.20.0 works. Verify against the
    real start path before writing a version constraint.
+5. **The agent image build clones two third-party repos over the public internet,
+   unauthenticated.** `agent/Dockerfile` fetches `NousResearch/hermes-agent` and
+   `nesquena/hermes-webui` at build time. GitHub rate-limits unauthenticated
+   requests *per source IP*, and Actions runners share pooled egress addresses, so
+   `docker compose build` intermittently died on `fatal: unable to access ...: The
+   requested URL returned error: 429` (exit 128) — 2 of 15 runs, always on branches
+   whose diff had nothing to do with the agent. Both clones now retry 5x with
+   backoff, and still print FATAL and exit 1 on exhaustion so a genuinely dead
+   upstream cannot yield an image with no runtime in it. Two lessons worth keeping:
+   a CI failure that is *intermittent and unrelated to the diff* is a network or
+   rate-limit signature, not a code defect — read the log before bisecting the
+   branch; and the webui clone's pre-existing `|| git clone` fallback looked like
+   resilience but only ever handled a *moved ref*, re-issuing the identical refused
+   request against a 429. A fallback that fails the same way as the thing it backs
+   up is not a fallback.
 
 ### Verification Anti-Patterns (do not repeat)
 

agent/Dockerfile

@@ -52,8 +52,24 @@ ARG AGENT_RUNTIME_REF=v2026.8.18
 ARG AGENT_API_REPO=https://github.com/nesquena/hermes-webui.git
 ARG AGENT_API_REF=v0.52.76
 
-# Runtime engine (Python agent)
-RUN git clone --depth 1 --branch "${AGENT_RUNTIME_REF}" "${AGENT_RUNTIME_REPO}" runtime \
+# Runtime engine (Python agent).
+#
+# The clone retries with backoff. github.com rate-limits *unauthenticated*
+# requests per source IP, and CI runners share pooled egress addresses, so a
+# cold clone here intermittently returns HTTP 429 and took the whole image
+# build down with it (`fatal: unable to access ...: error: 429`, exit 128 --
+# 2 of 15 `docker compose build` runs). This is not CI-only defensiveness: a
+# self-host user behind a shared NAT hits the identical refusal. Retry is the
+# right layer precisely because the condition is transient; the loop still
+# exits non-zero and loudly on the last attempt rather than continuing with no
+# runtime cloned.
+RUN n=0; rm -rf runtime; \
+    until git clone --depth 1 --branch "${AGENT_RUNTIME_REF}" "${AGENT_RUNTIME_REPO}" runtime; do \
+      n=$((n+1)); \
+      if [ "$n" -ge 5 ]; then echo "FATAL: clone of ${AGENT_RUNTIME_REPO} failed after 5 attempts" >&2; exit 1; fi; \
+      echo "clone attempt $n failed (HTTP 429 / transient network); retrying in $((n*15))s" >&2; \
+      rm -rf runtime; sleep $((n*15)); \
+    done \
     && cd runtime \
     && UV_NO_CONFIG=1 uv sync \
     && if [ -d .venv ] && [ ! -d venv ]; then ln -sfn .venv venv; fi \
@@ -64,8 +80,18 @@ RUN git clone --depth 1 --branch "${AGENT_RUNTIME_REF}" "${AGENT_RUNTIME_REPO}"
 
 # HTTP API surface the frontend / backend talk to (:8787).
 # Default branch is master (not main). Fall back to HEAD if the ref moves.
-RUN (git clone --depth 1 --branch "${AGENT_API_REF}" "${AGENT_API_REPO}" api \
-      || git clone --depth 1 "${AGENT_API_REPO}" api) \
+#
+# Same 429 retry as the runtime clone above. Note the inner `||` fallback
+# handles only a *moved ref*; on a rate-limit it just re-issues the same
+# refused request, so it needed the outer retry to be any use at all.
+RUN n=0; rm -rf api; \
+    until (git clone --depth 1 --branch "${AGENT_API_REF}" "${AGENT_API_REPO}" api \
+            || git clone --depth 1 "${AGENT_API_REPO}" api); do \
+      n=$((n+1)); \
+      if [ "$n" -ge 5 ]; then echo "FATAL: clone of ${AGENT_API_REPO} failed after 5 attempts" >&2; exit 1; fi; \
+      echo "clone attempt $n failed (HTTP 429 / transient network); retrying in $((n*15))s" >&2; \
+      rm -rf api; sleep $((n*15)); \
+    done \
     && cd api \
     && if [ -f requirements.txt ]; then \
          UV_NO_CONFIG=1 uv pip install --python /opt/deepsql-agent/runtime/venv/bin/python -r requirements.txt; \