Report upstream circuit breaker state in /health and /metrics

README.md

@@ -705,7 +705,7 @@ Recently cached:
 | Endpoint | Description |
 |----------|-------------|
 | `GET /` | Dashboard (web UI) |
-| `GET /health` | Health check (JSON; HTTP 200 healthy, 503 unhealthy) |
+| `GET /health` | Health check and upstream circuit breaker state (JSON; HTTP 200 healthy, 503 unhealthy) |
 | `GET /stats` | Cache statistics (JSON) |
 | `GET /metrics` | Prometheus metrics |
 | `GET /npm/*` | npm registry protocol |
@@ -948,8 +948,16 @@ The proxy exposes Prometheus metrics at `GET /metrics`. All metric names are pre
 | `proxy_storage_errors_total` | counter | `operation` | Storage read/write failures |
 | `proxy_active_requests` | gauge | | In-flight requests |
 | `proxy_health_probe_failures_total` | counter | `step` | Storage health probe failures by failing step (`write`, `size`, `read`, `verify`, `delete`). |
+| `proxy_circuit_breaker_state` | gauge | `registry` | Artifact-fetch circuit breaker state per upstream registry (0 closed, 2 open). Published once that registry's breaker has tripped. |
+| `proxy_circuit_breaker_trips_total` | counter | `registry` | Circuit breaker trips per upstream registry. |
 
-Cache size and artifact count are refreshed every 60 seconds. The remaining metrics update on each request.
+Cache size and artifact count are refreshed every 60 seconds. Circuit breaker state is read from the fetcher on each scrape of `/metrics` and each `/health` request, so `proxy_circuit_breaker_trips_total` counts the trips visible between those reads — a breaker that opens and recovers entirely between two scrapes is not counted. The remaining metrics update on each request.
+
+The breaker metrics carry one series per upstream host, but only for hosts whose breaker has tripped at least once since startup. A breaker is created per host the proxy fetches artifacts from, and for some ecosystems that host comes from upstream metadata rather than from configuration (composer takes it from a package's `dist.url`, helm from the chart URLs in `index.yaml`), so publishing every host would let upstream content grow the series count for the lifetime of the process. Once a host has tripped it keeps reporting, so a recovery still shows up as a transition to 0 rather than as a series that vanishes. `/health` is not a persistent time series and lists every breaker, tripped or not.
+
+The `registry` label is the host of the URL the artifact was fetched from. Because that URL can come from upstream metadata, it is not always one a host can be read off — a signed `dist.url` that fails to parse, for instance — and such a breaker is labelled `hostless-url-<digest>` instead, where the digest is keyed by a value drawn fresh at startup. Neither `/metrics` nor `/health` requires authentication, so a fetch URL is never published as a label or a key; the digest identifies the breaker for as long as the process runs without revealing the URL behind it or letting a chosen URL be matched against it.
+
+Alert on `proxy_circuit_breaker_state == 2` sustained for more than a few minutes: while a breaker is open, artifact downloads for that upstream fail with HTTP 502 on every cache miss, and only a single probe request per backoff interval reaches the upstream. Cached artifacts keep serving, and so does metadata for the same ecosystem (metadata does not go through the circuit breaker), so installs fail in a way that looks like a partial upstream outage.
 
 ### Health Check
 
@@ -961,12 +969,20 @@ Cache size and artifact count are refreshed every 60 seconds. The remaining metr
   "checks": {
     "database": {"status": "ok"},
     "storage":  {"status": "ok"}
+  },
+  "circuit_breakers": {
+    "registry.npmjs.org": "closed",
+    "static.crates.io":   "open"
   }
 }
 ```
 
 Failing checks include an `"error"` field. Storage failures also include a `"step"` field identifying which probe step failed (`write`, `size`, `read`, `verify`, `delete`). When the database check fails, the storage entry reports `{"status": "skipped"}` so the response always carries the same key set.
 
+`circuit_breakers` reports the state of each upstream's artifact-fetch circuit breaker (`"open"` or `"closed"`), keyed by upstream host — or by the `hostless-url-<digest>` placeholder described under [Monitoring](#monitoring) where the fetch URL has no host to read. The key is omitted until the proxy has fetched an artifact from at least one upstream, and a host appears only once a breaker has been created for it. Breakers trip after repeated upstream failures and retry the upstream after an exponential backoff. While one is open, artifact downloads for that host return HTTP 502 on a cache miss without contacting the upstream; already-cached artifacts are still served from storage, since the cache is checked before the fetcher. A breaker is reported as `"open"` throughout its backoff, including the half-open window in which it admits one probe request to test recovery. Breaker state is per process and in memory, so a restart clears it, but a restart is not needed for recovery: the backoff keeps retrying for as long as the breaker is open, so it closes on its own once the upstream serves again.
+
+An open breaker does **not** set `status` to `"error"` or change the HTTP status code: it reports a specific upstream refusing to serve, not this proxy being unfit to receive traffic, and failing the readiness probe over one unhealthy upstream would pull the pod out of rotation for every other ecosystem too. Use `proxy_circuit_breaker_state` for alerting on it.
+
 Storage probe results are cached for `health.storage_probe_interval` (default 30s) to bound the cost of probing remote backends. A probe holds an internal mutex for up to 10 seconds (the hardcoded per-probe timeout), so `/health` is intended as a Kubernetes **readiness** probe rather than a liveness probe — a slow S3 round-trip should pull the pod from rotation, not restart it.
 
 Scrape config for Prometheus:

docs/architecture.md

@@ -279,7 +279,7 @@ HTTP server setup, web UI, and API handlers.
 - Web UI under `/ui`: dashboard, package browser, source browser, version comparison
 - Templates are embedded in the binary via `//go:embed`
 - Enrichment API for package metadata, vulnerability scanning, and outdated detection
-- Health, stats, and Prometheus metrics endpoints. `/health` runs an active write → size-check → read → verify → delete probe against the storage backend and returns a structured JSON response (`HealthResponse`) with `"ok"` / `"error"` status per subsystem. Probe results are cached (default 30 s, configurable via `health.storage_probe_interval`) to avoid overwhelming remote backends.
+- Health, stats, and Prometheus metrics endpoints. `/health` runs an active write → size-check → read → verify → delete probe against the storage backend and returns a structured JSON response (`HealthResponse`) with `"ok"` / `"error"` status per subsystem. Probe results are cached (default 30 s, configurable via `health.storage_probe_interval`) to avoid overwhelming remote backends. The response also carries a `circuit_breakers` map reporting each upstream's artifact-fetch breaker as `"open"` or `"closed"`, keyed by the host fetched from (or an opaque placeholder where the fetch URL has no host to read); the same state is published as the `proxy_circuit_breaker_state` gauge on each `/metrics` scrape. An open breaker leaves the overall status `"ok"` — it describes an upstream, not this proxy.
 
 ### `internal/metrics`
 

docs/swagger/docs.go

@@ -538,6 +538,13 @@ const docTemplate = `{
                         "$ref": "#/definitions/server.HealthCheck"
                     }
                 },
+                "circuit_breakers": {
+                    "description": "CircuitBreakers reports the state (\"open\" or \"closed\") of each upstream\nregistry's artifact-fetch circuit breaker, keyed by the host fetched from\nor, where the fetch URL has none to read, by an opaque placeholder\nstanding in for it. It is omitted when no breaker has been created yet.\nAn open breaker fails every artifact fetch it covers without contacting\nthe upstream, but says nothing about this proxy's own health, so it does\nnot change Status.",
+                    "type": "object",
+                    "additionalProperties": {
+                        "type": "string"
+                    }
+                },
                 "status": {
                     "type": "string"
                 }

docs/swagger/swagger.json

@@ -531,6 +531,13 @@
                         "$ref": "#/definitions/server.HealthCheck"
                     }
                 },
+                "circuit_breakers": {
+                    "description": "CircuitBreakers reports the state (\"open\" or \"closed\") of each upstream\nregistry's artifact-fetch circuit breaker, keyed by the host fetched from\nor, where the fetch URL has none to read, by an opaque placeholder\nstanding in for it. It is omitted when no breaker has been created yet.\nAn open breaker fails every artifact fetch it covers without contacting\nthe upstream, but says nothing about this proxy's own health, so it does\nnot change Status.",
+                    "type": "object",
+                    "additionalProperties": {
+                        "type": "string"
+                    }
+                },
                 "status": {
                     "type": "string"
                 }

go.mod

@@ -12,7 +12,7 @@ require (
 	github.com/git-pkgs/integrity v0.1.1
 	github.com/git-pkgs/magic v0.2.0
 	github.com/git-pkgs/purl v0.1.19
-	github.com/git-pkgs/registries v0.8.1
+	github.com/git-pkgs/registries v0.9.0
 	github.com/git-pkgs/spdx v0.3.1
 	github.com/git-pkgs/vers v0.6.0
 	github.com/git-pkgs/vulns v0.2.2
@@ -131,8 +131,9 @@ require (
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
 	github.com/fzipp/gocyclo v0.6.0 // indirect
 	github.com/ghostiam/protogetter v0.3.21 // indirect
+	github.com/git-pkgs/artifacts v0.2.0 // indirect
 	github.com/git-pkgs/packageurl-go v0.3.1 // indirect
-	github.com/git-pkgs/pom v0.1.5 // indirect
+	github.com/git-pkgs/pom v0.1.7 // indirect
 	github.com/github/go-spdx/v2 v2.7.0 // indirect
 	github.com/go-critic/go-critic v0.14.4 // indirect
 	github.com/go-logr/logr v1.4.3 // indirect
@@ -226,6 +227,7 @@ require (
 	github.com/nunnatsa/ginkgolinter v0.24.0 // indirect
 	github.com/oapi-codegen/nullable v1.2.0 // indirect
 	github.com/oapi-codegen/runtime v1.6.0 // indirect
+	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/package-url/packageurl-go v0.1.7 // indirect
 	github.com/pandatix/go-cvss v0.6.2 // indirect
 	github.com/pelletier/go-toml v1.9.5 // indirect

go.sum

@@ -259,6 +259,8 @@ github.com/ghostiam/protogetter v0.3.21 h1:EeWTGvL/Eyosp653hiWb6Byx4b69iJC4/E+za
 github.com/ghostiam/protogetter v0.3.21/go.mod h1:iAKSpyoHwYzay+OpjoWgwzRtPFthEfuUvmlomTThck0=
 github.com/git-pkgs/archives v0.5.1 h1:qwu/vsoerQZF1iysRtfcxpy1KIUSJJSpXJ5JNxzNoQw=
 github.com/git-pkgs/archives v0.5.1/go.mod h1:AKpkxnts49R9uAt1mL2ULYcHrmYujCDVu24IsFvW9so=
+github.com/git-pkgs/artifacts v0.2.0 h1:ARNBtJEnpnMtidqLipal1WGv2PspYsq3CJFcqpsZ67k=
+github.com/git-pkgs/artifacts v0.2.0/go.mod h1:qQ6XYLtENRYslxjCncLATWxyx4gtrMAXSk4UoV9XSh0=
 github.com/git-pkgs/cooldown v0.2.0 h1:0MWPHtkzZgvCR0wdiQeyvMea/dxgw9tParH1zzaFopc=
 github.com/git-pkgs/cooldown v0.2.0/go.mod h1:v7APuK/UouTiu8mWQZbdDmj7DfxxkGUeuhjaRB5gv9E=
 github.com/git-pkgs/enrichment v0.7.0 h1:LfIzlVArc2p0MONO08ybC5jiHlysfzS3YyZ5ry2d6Lw=
@@ -271,12 +273,12 @@ github.com/git-pkgs/magic v0.2.0 h1:c7HqVxnP8c88EaVMH0/KraDFVTcmiXckRiSvNZEnvMQ=
 github.com/git-pkgs/magic v0.2.0/go.mod h1:3ndidt+yvFaI1M0aEkkzkOlFnLPkeVQASIUojazcxCI=
 github.com/git-pkgs/packageurl-go v0.3.1 h1:WM3RBABQZLaRBxgKyYughc3cVBE8KyQxbSC6Jt5ak7M=
 github.com/git-pkgs/packageurl-go v0.3.1/go.mod h1:rcIxiG37BlQLB6FZfgdj9Fm7yjhRQd3l+5o7J0QPAk4=
-github.com/git-pkgs/pom v0.1.5 h1:TGT8Az2OMxGWsXnSagtUMGzZm7Oax8HrSCteA+mi0qY=
-github.com/git-pkgs/pom v0.1.5/go.mod h1:ufdMBe1lKzqOeP9IUb9NPZ458xKV8E8NvuyBMxOfwIk=
+github.com/git-pkgs/pom v0.1.7 h1:4yKdtw6eyShtjul6bcZdyz7yLQ+jdrYeYkKbskDGi4c=
+github.com/git-pkgs/pom v0.1.7/go.mod h1:ufdMBe1lKzqOeP9IUb9NPZ458xKV8E8NvuyBMxOfwIk=
 github.com/git-pkgs/purl v0.1.19 h1:AT9ReTQjZIqEr+trKhfFJ/YZoh4pwdWsyJKKZWzs23c=
 github.com/git-pkgs/purl v0.1.19/go.mod h1:hthV5mp+Q67HpQ9+LnRLLmsReu5ooyQ5EaJsCGrA8yE=
-github.com/git-pkgs/registries v0.8.1 h1:Yf2FFdARQ1HcdtZfWBYa5OZFwZHzhFYStiz7qbTDDUU=
-github.com/git-pkgs/registries v0.8.1/go.mod h1:5dc3V7rOhAI5755L/bDtjtYV4D5XV4J/4ZtyIXSEs0U=
+github.com/git-pkgs/registries v0.9.0 h1:1JRtO5Oy6ueXYvufJcTWRI1xrZ6x00QUwkKiF18GUuk=
+github.com/git-pkgs/registries v0.9.0/go.mod h1:pw9a841dR0EAGyNfhZKowCHTYN293BLCTzGjHby7cVs=
 github.com/git-pkgs/spdx v0.3.1 h1:58JPY5X9pYpXvnzzZIgehItlBykeOOw52pNc4OBcS+c=
 github.com/git-pkgs/spdx v0.3.1/go.mod h1:cqRoZcvl530s/W+oGNvwjt4ODN8T1W6D/20MUZEFdto=
 github.com/git-pkgs/vers v0.6.0 h1:droJw8+oSyl8/UoDj/96B9ZPggmxJDTl/JeixzfKzSc=
@@ -537,6 +539,8 @@ github.com/onsi/ginkgo/v2 v2.32.0 h1:Hw7s2pVrQo/8Yz5N77qdnpHaoc+c6cC9WIV1Jce+J6E
 github.com/onsi/ginkgo/v2 v2.32.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44=
 github.com/onsi/gomega v1.42.1 h1:iN1rCUX+44NZ1Dc97MPoeFYbFR0vh8zxoxMFwKdyZ6I=
 github.com/onsi/gomega v1.42.1/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg=
+github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
+github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
 github.com/otiai10/copy v1.2.0/go.mod h1:rrF5dJ5F0t/EWSYODDu4j9/vEeYHMkc8jt0zJChqQWw=
 github.com/otiai10/copy v1.14.0 h1:dCI/t1iTdYGtkvCuBG2BgR6KZa83PTclw4U5n2wAllU=
 github.com/otiai10/copy v1.14.0/go.mod h1:ECfuL02W+/FkTWZWgQqXPWZgW9oeKCSQ5qVfSc4qc4w=

internal/server/breakers.go

@@ -0,0 +1,130 @@
+package server
+
+import (
+	"log/slog"
+	"sync"
+
+	"github.com/git-pkgs/proxy/internal/metrics"
+)
+
+// Gauge values for proxy_circuit_breaker_state. The fetcher reports only open
+// or closed, so half-open (1) is never published.
+const (
+	breakerGaugeClosed = 0
+	breakerGaugeOpen   = 2
+)
+
+const (
+	breakerStateOpen   = "open"
+	breakerStateClosed = "closed"
+)
+
+// breakerStateSource reports circuit breaker state per upstream registry, keyed
+// by the identifier the fetcher derives from the fetch URL, with values
+// breakerStateOpen or breakerStateClosed. Implemented by
+// fetch.CircuitBreakerFetcher.
+type breakerStateSource interface {
+	GetBreakerState() map[string]string
+}
+
+// breakerMonitor mirrors the artifact fetcher's per-registry circuit breaker
+// state into Prometheus metrics, the health report, and the log.
+//
+// Breaker state lives only in the fetcher's memory. While a breaker is open
+// every artifact fetch it covers that misses the cache fails without reaching
+// the upstream, bar the one probe per backoff interval the breaker admits to
+// test recovery. That looks identical to an upstream outage from the outside:
+// metadata still serves (it does not go through the fetcher), other registries
+// still serve, and /health reports the database and storage as fine. Publishing
+// the state makes that distinguishable.
+type breakerMonitor struct {
+	source breakerStateSource
+	logger *slog.Logger
+
+	// mu serializes snapshots. It guards seen, which holds one entry per
+	// registry that has tripped at least once in this process — the only
+	// registries published as metrics — and keeps each state read paired with
+	// the updates it produces.
+	mu   sync.Mutex
+	seen map[string]string
+}
+
+func newBreakerMonitor(source breakerStateSource, logger *slog.Logger) *breakerMonitor {
+	if logger == nil {
+		logger = slog.Default()
+	}
+	return &breakerMonitor{
+		source: source,
+		logger: logger,
+		seen:   map[string]string{},
+	}
+}
+
+// snapshot returns the current state of every breaker the fetcher has created,
+// keyed by registry identifier, and mirrors it into the breaker metrics as a
+// side effect. It returns nil for a nil monitor so callers that build a Server
+// without a fetcher (tests) need no special case.
+//
+// The keys pass through unaltered into Prometheus labels and the /health body,
+// neither of which is authenticated, so they are only as safe to publish as the
+// fetcher makes them. It keys by the fetch URL's host, and where it cannot take
+// a host from that URL — a signed composer dist.url that fails to parse, say —
+// by an opaque keyed digest of the URL rather than the URL itself, so a
+// credential carried in one does not reach either endpoint
+// (github.com/git-pkgs/registries v0.9.0 and later).
+//
+// Only registries that have tripped at least once are published as metrics.
+// The fetcher creates a breaker per identifier it fetches under, and for some
+// ecosystems that identifier comes from upstream metadata rather than
+// configuration (composer takes it from a package's dist.url, helm from the
+// chart URLs in index.yaml), so publishing every one would let upstream content
+// grow the series count for the life of the process — the more so for URLs with
+// no host, which get an identifier apiece rather than sharing one. An
+// identifier that has never tripped carries no information a series could
+// convey; once it trips it keeps reporting, including the 0 that marks its
+// recovery. /health is a per-request response rather than a persistent series,
+// so it reports every breaker.
+//
+// Trips are counted on the closed→open transitions observed between calls,
+// because the fetcher exposes current state rather than trip events: a breaker
+// that opens and recovers entirely between two calls is not counted.
+func (m *breakerMonitor) snapshot() map[string]string {
+	if m == nil || m.source == nil {
+		return nil
+	}
+
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	// Read under the lock. Two concurrent snapshots — a /health request and a
+	// /metrics scrape landing during a transition — can otherwise apply their
+	// reads to seen in the opposite order, counting one trip twice, logging a
+	// close for a breaker that is still open, and leaving the gauge at 0 until
+	// the next call.
+	states := m.source.GetBreakerState()
+
+	for registry, state := range states {
+		previous, published := m.seen[registry]
+
+		switch {
+		case state == breakerStateOpen && previous != breakerStateOpen:
+			metrics.RecordCircuitBreakerTrip(registry)
+			m.logger.Error("circuit breaker open, artifact fetches for this registry "+
+				"fail without contacting it", "registry", registry)
+		case state == breakerStateClosed && previous == breakerStateOpen:
+			m.logger.Info("circuit breaker closed", "registry", registry)
+		case state == breakerStateClosed && !published:
+			// Never tripped: nothing to publish.
+			continue
+		}
+
+		gauge := breakerGaugeClosed
+		if state == breakerStateOpen {
+			gauge = breakerGaugeOpen
+		}
+		metrics.UpdateCircuitBreakerState(registry, gauge)
+		m.seen[registry] = state
+	}
+
+	return states
+}