Give route-backed containers the full pinned dial budget and retry

[luke-lombardi]

Root cause (measured in production)

The CNN sandbox burst failures (20% 502s at ~1.15s, throughput collapse to ~9 req/s) trace to the pinned-container path in the pod proxy. Every sandbox on a rented machine is addressed as route://…, and that path got a 900ms dial budget with no retry, while direct addresses got 3s + one retry.

Evidence from gateway metrics for the affected stub:

• Successful tunnel dials: 100–190ms average. Failed dials: 1.12s average — i.e. the 900ms budget plus overhead. Half of all dials during bursts failed.
• Background discovery dials to the same machine fail ~40–50% continuously at a ~2s cap, so the tunnel is slow at connection setup even unloaded (raw internet RTT to the machine is 38ms; the tunnel round trip is ~160ms — relayed path).
• Observed collapse throughput of 8.9 req/s matches 16 global dial slots draining at ~1s effective dial time; the backend app itself serves 612 req/s in-container.

Fix

• Route-backed pinned requests get the same treatment as direct: 3s dial budget, one pre-connect retry, pinning. (containerPinnedRouteDialTimeout removed.)
• backendRouteDialConcurrency raised 16 → 64 so one slow machine cannot serialize dials for the whole gateway.
• A stuck-opening route now 502s after the budget plus one retry (~6s) instead of 0.9s; test updated to lock in that bound.

Test plan

• go test ./pkg/abstractions/pod/ ./pkg/network/ passes
• Post-deploy: re-run in-cluster burst (c=32) against a route-backed sandbox — expect 0% 502s

Made with Cursor

---

Summary by cubic

Route-backed container dials now get the same pinned budget and retry as direct addresses, and embedded tailscale is bumped to v1.102.2 to restore direct paths and reduce relay-only flakiness. Old: route:// used a 900ms dial with no retry and older tailscale often stuck on DERP relays. New: 3s dial with one pre-connect retry and pinning; tailscale negotiates direct connections and guards tsnet.Server.Close; stuck-opening routes now fail in ~6s instead of ~0.9s.

Review and rollout

• Unify pinned dial handling in pkg/abstractions/pod/proxy.go: always use containerPinnedDialTimeout (3s), enable retry, pin; remove containerPinnedRouteDialTimeout.
• Raise route dial concurrency 16 → 64 in pkg/network/backend_dialer.go.
• Upgrade tailscale to v1.102.2; add closeTSNetServer and use it in pkg/network/tailscale.go to tolerate tsnet.Server.Close panics on wedged servers.
• Update test to bound stuck-opening routes at ~2*3s + 1s and expect 502.
• Migration: build with Go 1.26+; monitor max request latency for route-open failures and gateway dial failure rates post-deploy.

^{Written for commit 06e351e. Summary will update on new commits.}