diff --git a/.claude/rules/http-layer.md b/.claude/rules/http-layer.md index d51f379..da3ceef 100644 --- a/.claude/rules/http-layer.md +++ b/.claude/rules/http-layer.md @@ -14,23 +14,43 @@ paths: ## Middleware chain — order matters -1. `gin.Recovery()` — panics → 500 (handler goroutines only; lookups recover themselves) -2. `SecurityHeaders()` — X-Content-Type-Options, X-Frame-Options, `Cache-Control: +1. `Metrics()` — records latency by route template, method, status and plan after + `c.Next()` returns. **First, ahead of Recovery**: a panic unwinds past every middleware + between it and Recovery, so anywhere later a panicking request would not be counted +2. `gin.Recovery()` — panics → 500 (handler goroutines only; lookups recover themselves) +3. `SecurityHeaders()` — X-Content-Type-Options, X-Frame-Options, `Cache-Control: no-store`, HSTS, Referrer-Policy, and a strict `default-src 'none'` CSP for the JSON API. HTML pages override the CSP with `htmlPageCSP` in the handler package. -3. `BodyLimit(1MB)` -4. `ClientRequestID()` — validates/echoes `X-Client-Request-ID`, sets a context key -5. `RequestLogger()` — structured logging; runs after `c.Next()` returns -6. *(protected)* `APIKeyAuth()` — sets `customer_id`, `api_key_id`, `api_key_plan` -7. *(protected)* `RateLimit()` — per-key sliding window in Redis, **fail-open**, admin +4. `BodyLimit(1MB)` +5. `ClientRequestID()` — validates/echoes `X-Client-Request-ID`, sets a context key +6. `RequestLogger()` — structured logging; runs after `c.Next()` returns +7. *(protected)* `APIKeyAuth()` — sets `customer_id`, `api_key_id`, `api_key_plan` +8. *(protected)* `RateLimit()` — per-key sliding window in Redis, **fail-open**, admin bypass -8. *(protected + admin)* `UsageLog()` — records to `usage.Recorder` after the response -9. *(admin)* `RequireAdmin()` — `plan == "admin"` +9. *(protected + admin)* `UsageLog()` — records to `usage.Recorder` after the response +10. *(admin)* `RequireAdmin()` — `plan == "admin"` Adding middleware means choosing a position in this list, and the position is part of the change. Auth must precede rate limiting (the limit is per key); usage logging must follow auth (it needs the key) and must be after `c.Next()` so it sees the outcome. +## Metrics — `internal/metrics` + +One `*metrics.Metrics`, built in `app.go` and passed to the router, the rate limiter +(`RateLimitConfig.Metrics`) and both services. It is a concrete type, not an interface: +there is one implementation, and every recording method is a no-op on a nil pointer, so +tests pass nil or build their own (each owns its registry; `metricstest` reads one the +way a scrape does). Recording is always on. `metrics.enabled` only decides whether +`httpapi.StartMetricsServer` serves it, on **its own listener** (`metrics.listen_addr`). +`/metrics` is never a route on the API router: the endpoint is unauthenticated and the +separate port is its access control (`docs/decisions.md`). + +**Label values must be bounded.** Anything a caller chooses (path, method) goes through a +fixed fallback (`RouteUnmatched`, `MethodOther`) before it becomes a label. A new label +whose values come from a request is a way to exhaust the scraper. Metric names and labels +are also what `deploy/grafana/intelligence-api.json` queries: renaming one means editing +the dashboard and `docs/operations.md` in the same change. + ## Handler construction `handler.New` takes a single `Deps` struct. **Every field is optional**; a nil service @@ -71,7 +91,7 @@ domain.AllLookupNames)` and populated progressively in `app.go`. It feeds both t structured startup log and `GET /v1/info` — **adding a field to a struct surfaces it in both places**. Live data (runtime stats, DB pool, Redis ping, auth cache size) is computed at request time in the info handler and merged in. Response groups: `system`, `database`, -`cache`, `lookups`, `services`, `runtime`, `postgres_pool`. `./scripts/info.sh` +`cache`, `lookups`, `services` (including `services.metrics`), `runtime`, `postgres_pool`. `./scripts/info.sh` pretty-prints it (needs `SMOKE_ADMIN_KEY` from `.env`). ## Health diff --git a/.claude/rules/service-layer.md b/.claude/rules/service-layer.md index 8207d60..ba3356c 100644 --- a/.claude/rules/service-layer.md +++ b/.claude/rules/service-layer.md @@ -64,12 +64,29 @@ coalescing anywhere; `service/whole_response.go` is gone and must not come back. reads each key individually so `CACHE_TTL_WHOIS` and friends work through Viper's `AutomaticEnv`. Pulling the subtree as a map silently drops env overrides. - Expiry is absolute. There is no stale serving and no negative cache. -- `geo` and `asn` are `uncached`. `fingerprint` has no key of its own — it is computed +- `geo` is `uncached`. `fingerprint` has no key of its own — it is computed from `headers` at assembly, and its `cache_hit` mirrors the headers entry. +## Metrics + +`runLookups` records everything, so a lookup wired through a runner is instrumented with +no code of its own: fetch duration and outcome (`ok` / `error` / `timeout`) around the +fetch inside singleflight, a hit or miss per cacheable runner in the MGET pre-flight, and +a coalesced call for every caller that waited on someone else's fetch. Cache hits and +coalesced callers are deliberately absent from the duration histogram, which times +upstream work. The label is the **runner** name, so `fingerprint` is recorded as `headers` +and `risk` not at all. + +Each service's `initMetrics()` creates its lookups' series at zero at construction, +because `rate()` cannot see a series' first value. It derives the set from +`domain.AllLookupNames` / `AllIPLookupNames`, minus the lookups nothing fetches; a test +per service fails if that set and what a request actually records ever differ. + ## Adding a lookup Runner in `buildRunners()`, branch in `assemble()`, function field + setter on the service. +A lookup with no runner of its own (computed from another's data, like `fingerprint`) must +also be excluded in `initMetrics()`; the test there says so if it is not. The full multi-file checklist is the `adding-features` skill; use it rather than reconstructing the list. diff --git a/AGENTS.md b/AGENTS.md index 28bb874..6c21c71 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,10 +30,13 @@ internal/database/ shared PostgreSQL connection internal/billing/ Stripe webhooks, key provisioning, noop fallback internal/usage/ buffered usage logging, admin query internal/notifications/ Pushover alerting, throttle and spike detection +internal/metrics/ Prometheus collectors and recording methods; metricstest reads them in tests internal/safenet/ SSRF prevention at dial time internal/logging/ Zap (ECS in prod, pretty in dev) internal/config/ Viper, YAML + env overrides, StatusSnapshot migrations/ PostgreSQL schema +deploy/ Grafana dashboard (deploy/grafana/intelligence-api.json), plus the + Prometheus and Grafana config the local Compose stack mounts scripts/ docker-reset.sh, smoke-test.sh, status.sh, info.sh — dev only changelog.yaml symlink to internal/handler/changelog.yaml; edit from the root ``` @@ -110,11 +113,16 @@ open the file. `sanitizeErr()`. - Every outbound HTTP client goes through `safenet` — a new one that doesn't is a vulnerability, not a style issue. +- `/metrics` has no authentication; its own listener (`:9091`) is the access control. + Never route it on the API port, add it to a Fly service, or publish it from Compose + beyond loopback. Keep it out of the public docs, like `/v1/hello`. ## Infrastructure -Docker Compose locally (API + Redis + Postgres). Production on Fly.io (iad, auto-stop) with -Neon Postgres and Upstash Redis. One app answers for two hostnames: +Docker Compose locally (API + Redis + Postgres, plus Prometheus and Grafana for checking the +dashboard). Production on Fly.io (iad, auto-stop) with Neon Postgres and Upstash Redis; +Fly's managed Prometheus scrapes the metrics listener per `[metrics]` in `fly.toml`, and +the dashboard is imported into fly-metrics.net. One app answers for two hostnames: `securityaccelerated.com` (`/` is the company home page) and `api.securityaccelerated.com` (`/` redirects to `/docs`), told apart by `site.home_host`. A Fly volume `maxmind_data` (name predates DB-IP) at `/data/geoip` persists the GeoIP database; `docker-entrypoint.sh` chowns it as root then diff --git a/Dockerfile b/Dockerfile index 1d2e3e9..5938c02 100644 --- a/Dockerfile +++ b/Dockerfile @@ -21,5 +21,8 @@ RUN chmod +x /usr/local/bin/docker-entrypoint.sh # root-owned and there is no uid/gid option on mounts, so this is the only way a # non-root process can write to its own volume. EXPOSE 8080 +# The Prometheus listener, when metrics.enabled is set. Documentation only, as +# EXPOSE always is: never publish this port, the endpoint has no authentication. +EXPOSE 9091 ENTRYPOINT ["docker-entrypoint.sh"] CMD ["intelligence-api"] diff --git a/README.md b/README.md index 30fbf09..0416dfc 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,7 @@ client ──> Gin router ──> middleware chain ──> handler ──> servi | `internal/safenet` | SSRF prevention at dial time: every outbound HTTP client in the server goes through it | | `internal/auth`, `internal/billing`, `internal/usage` | Hashed API keys with rotation, Stripe webhooks and key provisioning, buffered usage logging | | `internal/notifications` | Pushover alerting with throttle and spike detection | +| `internal/metrics` | Prometheus instrumentation, served on a listener separate from the API's; the Grafana dashboard for it is in `deploy/grafana` | | `internal/cache`, `internal/database`, `internal/config`, `internal/logging` | Redis, PostgreSQL, Viper config with env overrides, Zap in ECS format | | `cmd/api`, `cmd/sactl` | The server, and the operator CLI that reuses the server's own packages | diff --git a/deploy/grafana/intelligence-api.json b/deploy/grafana/intelligence-api.json new file mode 100644 index 0000000..f84a94a --- /dev/null +++ b/deploy/grafana/intelligence-api.json @@ -0,0 +1,359 @@ +{ + "uid": "intelligence-api", + "title": "Intelligence API", + "description": "Service metrics for the Intelligence API: HTTP traffic, lookups, cache, rate limiting and dependencies. Source: deploy/grafana/intelligence-api.json in the intelligence-api repository.", + "tags": ["intelligence-api"], + "schemaVersion": 39, + "version": 1, + "editable": true, + "graphTooltip": 1, + "timezone": "browser", + "time": { "from": "now-1h", "to": "now" }, + "refresh": "30s", + "templating": { + "list": [ + { + "name": "datasource", + "label": "Data source", + "type": "datasource", + "query": "prometheus", + "current": {} + }, + { + "name": "app", + "label": "App", + "type": "query", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "query": "label_values(intelligence_api_build_info, app)", + "definition": "label_values(intelligence_api_build_info, app)", + "refresh": 2, + "sort": 1, + "multi": false, + "includeAll": false, + "current": {} + }, + { + "name": "instance", + "label": "Instance", + "type": "query", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "query": "label_values(intelligence_api_build_info{app=~\"$app\"}, instance)", + "definition": "label_values(intelligence_api_build_info{app=~\"$app\"}, instance)", + "refresh": 2, + "sort": 1, + "multi": true, + "includeAll": true, + "allValue": ".*", + "current": { "text": "All", "value": "$__all" } + } + ] + }, + "panels": [ + { + "id": 1, "type": "stat", "title": "Version", + "description": "The version each instance reports, from the build_info gauge. Two values during a deploy, one otherwise.", + "gridPos": { "x": 0, "y": 0, "w": 4, "h": 4 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "instant": true, "legendFormat": "{{version}}", "expr": "count by (version) (intelligence_api_build_info{app=~\"$app\",instance=~\"$instance\"})" } + ], + "options": { "textMode": "name", "colorMode": "none", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"] } } + }, + { + "id": 2, "type": "stat", "title": "Requests", + "gridPos": { "x": 4, "y": 0, "w": 4, "h": 4 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "expr": "sum(rate(intelligence_api_http_request_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "reqps", "decimals": 2 }, "overrides": [] }, + "options": { "colorMode": "none", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"] } } + }, + { + "id": 3, "type": "stat", "title": "5xx ratio", + "description": "Share of all requests answered with a 5xx status.", + "gridPos": { "x": 8, "y": 0, "w": 4, "h": 4 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "expr": "(sum(rate(intelligence_api_http_request_duration_seconds_count{app=~\"$app\",instance=~\"$instance\",status=~\"5..\"}[$__rate_interval])) or vector(0)) / sum(rate(intelligence_api_http_request_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { + "defaults": { + "unit": "percentunit", "decimals": 2, + "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 0.01 }, { "color": "red", "value": 0.05 } ] } + }, + "overrides": [] + }, + "options": { "colorMode": "value", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"] } } + }, + { + "id": 4, "type": "stat", "title": "Lookup p95", + "description": "95th percentile latency of POST /v1/domain/lookup and POST /v1/ip/lookup, as the caller sees it.", + "gridPos": { "x": 12, "y": 0, "w": 4, "h": 4 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" } + ], + "fieldConfig": { "defaults": { "unit": "s", "decimals": 2 }, "overrides": [] }, + "options": { "colorMode": "none", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"] } } + }, + { + "id": 5, "type": "stat", "title": "Cache hit ratio", + "description": "Share of cacheable lookups answered from Redis. geo is never cached and is not counted.", + "gridPos": { "x": 16, "y": 0, "w": 4, "h": 4 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "expr": "(sum(rate(intelligence_api_cache_requests_total{app=~\"$app\",instance=~\"$instance\",result=\"hit\"}[$__rate_interval])) or vector(0)) / sum(rate(intelligence_api_cache_requests_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "percentunit", "decimals": 1 }, "overrides": [] }, + "options": { "colorMode": "none", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"] } } + }, + { + "id": 6, "type": "stat", "title": "GeoIP age", + "description": "Time since the loaded DB-IP database was built. DB-IP publishes monthly and geoip.max_age (720h by default) triggers a refresh at startup, so much past five weeks means the refresh is not happening. No data means no database is loaded.", + "gridPos": { "x": 20, "y": 0, "w": 4, "h": 4 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "instant": true, "expr": "time() - min(intelligence_api_geoip_database_build_timestamp_seconds{app=~\"$app\",instance=~\"$instance\"})" } + ], + "fieldConfig": { + "defaults": { + "unit": "dtdurations", "decimals": 0, + "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 3024000 }, { "color": "red", "value": 5184000 } ] } + }, + "overrides": [] + }, + "options": { "colorMode": "value", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"] } } + }, + + { "id": 100, "type": "row", "title": "HTTP", "collapsed": false, "gridPos": { "x": 0, "y": 4, "w": 24, "h": 1 }, "panels": [] }, + { + "id": 10, "type": "timeseries", "title": "Requests by status", + "gridPos": { "x": 0, "y": 5, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{status}}", "expr": "sum by (status) (rate(intelligence_api_http_request_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 11, "type": "timeseries", "title": "Requests by route", + "description": "Routes are the registered templates. Anything that matched no route is counted under \"unmatched\".", + "gridPos": { "x": 12, "y": 5, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{method}} {{route}}", "expr": "sum by (method, route) (rate(intelligence_api_http_request_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 12, "type": "timeseries", "title": "Lookup endpoint latency", + "description": "Request latency of the two lookup endpoints, end to end. The per-lookup fetch times are in the Lookups row.", + "gridPos": { "x": 0, "y": 13, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "p50 {{route}}", "expr": "histogram_quantile(0.50, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" }, + { "refId": "B", "legendFormat": "p95 {{route}}", "expr": "histogram_quantile(0.95, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" }, + { "refId": "C", "legendFormat": "p99 {{route}}", "expr": "histogram_quantile(0.99, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" } + ], + "fieldConfig": { "defaults": { "unit": "s", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 13, "type": "timeseries", "title": "Requests by plan", + "description": "\"none\" is every request that carried no valid API key: docs pages, health checks, failed authentication.", + "gridPos": { "x": 12, "y": 13, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{plan}}", "expr": "sum by (plan) (rate(intelligence_api_http_request_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 14, "type": "timeseries", "title": "Rate-limit rejections by plan", + "description": "Requests answered 429 by the rate limiter. Empty until the first rejection.", + "gridPos": { "x": 18, "y": 13, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{plan}}", "expr": "sum by (plan) (rate(intelligence_api_rate_limit_rejections_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "reqps", "min": 0, "noValue": "0", "custom": { "fillOpacity": 20 } }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + + { "id": 101, "type": "row", "title": "Lookups", "collapsed": false, "gridPos": { "x": 0, "y": 21, "w": 24, "h": 1 }, "panels": [] }, + { + "id": 20, "type": "timeseries", "title": "Live fetches by lookup", + "description": "Lookups that actually went upstream. Cache hits and callers that shared another caller's fetch are not here. fingerprint and risk never appear: both are computed from other lookups.", + "gridPos": { "x": 0, "y": 22, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{scope}} {{lookup}}", "expr": "sum by (scope, lookup) (rate(intelligence_api_lookup_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "ops", "min": 0, "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 21, "type": "timeseries", "title": "Fetch duration p95 by lookup", + "gridPos": { "x": 12, "y": 22, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{scope}} {{lookup}}", "expr": "histogram_quantile(0.95, sum by (le, scope, lookup) (rate(intelligence_api_lookup_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval])))" } + ], + "fieldConfig": { "defaults": { "unit": "s", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 22, "type": "timeseries", "title": "Failure ratio by lookup", + "description": "Share of live fetches that ended in an error or a timeout. A fan-out lookup (reputation, threat_intel) counts as failed when any provider errored, even if it returned a partial result.", + "gridPos": { "x": 0, "y": 30, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{scope}} {{lookup}}", "expr": "(sum by (scope, lookup) (rate(intelligence_api_lookup_duration_seconds_count{app=~\"$app\",instance=~\"$instance\",outcome!=\"ok\"}[$__rate_interval])) or sum by (scope, lookup) (rate(intelligence_api_lookup_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval])) * 0) / sum by (scope, lookup) (rate(intelligence_api_lookup_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 23, "type": "timeseries", "title": "Fetch outcomes", + "gridPos": { "x": 12, "y": 30, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{outcome}}", "expr": "sum by (outcome) (rate(intelligence_api_lookup_duration_seconds_count{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { + "defaults": { "unit": "ops", "min": 0, "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, + "overrides": [ + { "matcher": { "id": "byName", "options": "ok" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "green" } } ] }, + { "matcher": { "id": "byName", "options": "error" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } } ] }, + { "matcher": { "id": "byName", "options": "timeout" }, "properties": [ { "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } } ] } + ] + }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + + { "id": 102, "type": "row", "title": "Cache", "collapsed": false, "gridPos": { "x": 0, "y": 38, "w": 24, "h": 1 }, "panels": [] }, + { + "id": 30, "type": "timeseries", "title": "Cache hit ratio by lookup", + "description": "A failed cache read and an unusable entry both count as misses, since both end in a live fetch.", + "gridPos": { "x": 0, "y": 39, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{scope}} {{lookup}}", "expr": "(sum by (scope, lookup) (rate(intelligence_api_cache_requests_total{app=~\"$app\",instance=~\"$instance\",result=\"hit\"}[$__rate_interval])) or sum by (scope, lookup) (rate(intelligence_api_cache_requests_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval])) * 0) / sum by (scope, lookup) (rate(intelligence_api_cache_requests_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull", "mean"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + { + "id": 31, "type": "timeseries", "title": "Coalesced calls by lookup", + "description": "Calls that waited on another caller's in-flight fetch of the same lookup for the same target, instead of making their own upstream call. Empty until it first happens.", + "gridPos": { "x": 12, "y": 39, "w": 12, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{scope}} {{lookup}}", "expr": "sum by (scope, lookup) (rate(intelligence_api_singleflight_coalesced_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "ops", "min": 0, "noValue": "0", "custom": { "fillOpacity": 20 } }, "overrides": [] }, + "options": { "legend": { "displayMode": "table", "placement": "right", "calcs": ["lastNotNull"] }, "tooltip": { "mode": "multi", "sort": "desc" } } + }, + + { "id": 103, "type": "row", "title": "Dependencies", "collapsed": false, "gridPos": { "x": 0, "y": 47, "w": 24, "h": 1 }, "panels": [] }, + { + "id": 40, "type": "timeseries", "title": "Postgres pool", + "description": "database/sql connection pool, summed over the selected instances. In use reaching the limit means requests are queueing for a connection.", + "gridPos": { "x": 0, "y": 48, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "in use", "expr": "sum(go_sql_in_use_connections{app=~\"$app\",instance=~\"$instance\"})" }, + { "refId": "B", "legendFormat": "idle", "expr": "sum(go_sql_idle_connections{app=~\"$app\",instance=~\"$instance\"})" }, + { "refId": "C", "legendFormat": "limit", "expr": "sum(go_sql_max_open_connections{app=~\"$app\",instance=~\"$instance\"})" } + ], + "fieldConfig": { "defaults": { "unit": "short", "min": 0, "decimals": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + { + "id": 41, "type": "timeseries", "title": "Postgres pool waits", + "description": "How often a query had to wait for a connection, and how long those waits took in total per second.", + "gridPos": { "x": 6, "y": 48, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "waits/s", "expr": "sum(rate(go_sql_wait_count_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" }, + { "refId": "B", "legendFormat": "seconds waited/s", "expr": "sum(rate(go_sql_wait_duration_seconds_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "short", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + { + "id": 42, "type": "timeseries", "title": "Redis pool", + "description": "The one go-redis pool the cache and the rate limiter share, summed over the selected instances.", + "gridPos": { "x": 12, "y": 48, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "connections", "expr": "sum(intelligence_api_redis_pool_connections{app=~\"$app\",instance=~\"$instance\"})" }, + { "refId": "B", "legendFormat": "idle", "expr": "sum(intelligence_api_redis_pool_idle_connections{app=~\"$app\",instance=~\"$instance\"})" } + ], + "fieldConfig": { "defaults": { "unit": "short", "min": 0, "decimals": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + { + "id": 43, "type": "timeseries", "title": "Redis pool events", + "description": "New connections dialed because none was free, waits for a connection that timed out, and stale connections dropped. Sustained timeouts mean the pool is too small or Redis is slow.", + "gridPos": { "x": 18, "y": 48, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "dialed", "expr": "sum(rate(intelligence_api_redis_pool_misses_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" }, + { "refId": "B", "legendFormat": "timeouts", "expr": "sum(rate(intelligence_api_redis_pool_timeouts_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" }, + { "refId": "C", "legendFormat": "stale removed", "expr": "sum(rate(intelligence_api_redis_pool_stale_connections_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval]))" } + ], + "fieldConfig": { "defaults": { "unit": "ops", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + + { "id": 104, "type": "row", "title": "Runtime", "collapsed": false, "gridPos": { "x": 0, "y": 56, "w": 24, "h": 1 }, "panels": [] }, + { + "id": 50, "type": "timeseries", "title": "GeoIP database loaded", + "description": "1 while an instance has a GeoIP database loaded. 0 for the first seconds after a start on an empty volume is normal: the database downloads in the background.", + "gridPos": { "x": 0, "y": 57, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{instance}}", "expr": "intelligence_api_geoip_database_loaded{app=~\"$app\",instance=~\"$instance\"}" } + ], + "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": 1, "decimals": 0, "custom": { "lineInterpolation": "stepAfter", "fillOpacity": 10 } }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + { + "id": 51, "type": "timeseries", "title": "Goroutines", + "gridPos": { "x": 6, "y": 57, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{instance}}", "expr": "go_goroutines{app=~\"$app\",instance=~\"$instance\"}" } + ], + "fieldConfig": { "defaults": { "unit": "short", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + { + "id": 52, "type": "timeseries", "title": "Memory", + "gridPos": { "x": 12, "y": 57, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "resident {{instance}}", "expr": "process_resident_memory_bytes{app=~\"$app\",instance=~\"$instance\"}" }, + { "refId": "B", "legendFormat": "heap in use {{instance}}", "expr": "go_memstats_heap_inuse_bytes{app=~\"$app\",instance=~\"$instance\"}" } + ], + "fieldConfig": { "defaults": { "unit": "bytes", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + }, + { + "id": 53, "type": "timeseries", "title": "CPU", + "description": "CPU seconds used per second: 1 is one full core.", + "gridPos": { "x": 18, "y": 57, "w": 6, "h": 8 }, + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "targets": [ + { "refId": "A", "legendFormat": "{{instance}}", "expr": "rate(process_cpu_seconds_total{app=~\"$app\",instance=~\"$instance\"}[$__rate_interval])" } + ], + "fieldConfig": { "defaults": { "unit": "short", "min": 0 }, "overrides": [] }, + "options": { "legend": { "displayMode": "list", "placement": "bottom" }, "tooltip": { "mode": "multi" } } + } + ] +} diff --git a/deploy/grafana/provisioning/dashboards/dashboards.yml b/deploy/grafana/provisioning/dashboards/dashboards.yml new file mode 100644 index 0000000..4976381 --- /dev/null +++ b/deploy/grafana/provisioning/dashboards/dashboards.yml @@ -0,0 +1,11 @@ +# Local development only: loads the committed dashboard from the path +# docker-compose.yaml mounts it at. Edits made in the UI are not written back; +# export the JSON and replace deploy/grafana/intelligence-api.json to keep one. +apiVersion: 1 + +providers: + - name: intelligence-api + type: file + allowUiUpdates: true + options: + path: /var/lib/grafana/dashboards diff --git a/deploy/grafana/provisioning/datasources/prometheus.yml b/deploy/grafana/provisioning/datasources/prometheus.yml new file mode 100644 index 0000000..3d26f90 --- /dev/null +++ b/deploy/grafana/provisioning/datasources/prometheus.yml @@ -0,0 +1,9 @@ +# Local development only: points Grafana at the Prometheus container. +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true diff --git a/deploy/prometheus/prometheus.yml b/deploy/prometheus/prometheus.yml new file mode 100644 index 0000000..3a68aaa --- /dev/null +++ b/deploy/prometheus/prometheus.yml @@ -0,0 +1,18 @@ +# Local development only: the Prometheus half of the pair in docker-compose.yaml. +# Production is scraped by Fly's managed Prometheus, configured by the [metrics] +# section of fly.toml; see docs/operations.md. + +global: + # Fly scrapes every 15s. Matching it keeps rate() windows on the dashboard + # behaving the same here as they will in production. + scrape_interval: 15s + +scrape_configs: + - job_name: intelligence-api + static_configs: + - targets: ["api:9091"] + labels: + # Fly stamps every series with the app's name, and the dashboard + # filters on it. Setting one here lets the same dashboard file work + # against both. + app: intelligence-api diff --git a/docker-compose.yaml b/docker-compose.yaml index dd2c941..abd140e 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -8,8 +8,13 @@ services: VERSION: ${VERSION:-dev} ports: - "8080:8080" + # The metrics listener, for curl and the smoke test. Loopback only: the + # endpoint is unauthenticated, and Prometheus reaches it over the compose + # network without this. + - "127.0.0.1:9091:9091" environment: POSTGRES_SSLMODE: disable + METRICS_ENABLED: "true" volumes: - /etc/security-accelerated/intelligence-api.yaml:/etc/security-accelerated/intelligence-api.yaml:ro # The app downloads the GeoIP database on first boot (no key needed) and @@ -47,7 +52,35 @@ services: interval: 2s timeout: 3s retries: 10 + # Prometheus and Grafana, so the dashboard in deploy/grafana can be checked + # against real series before a release. Development only: production is + # scraped by Fly (see docs/operations.md). Both are published on loopback. + prometheus: + image: prom/prometheus:v3.16.0 + ports: + - "127.0.0.1:9090:9090" + volumes: + - ./deploy/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro + - prometheus-data:/prometheus + depends_on: + - api + grafana: + image: grafana/grafana:13.3.1 + ports: + - "127.0.0.1:3000:3000" + environment: + # No login on a loopback-only dev instance. Editor is enough to change a + # panel and export the result; nothing here is ever reachable off-host. + GF_AUTH_ANONYMOUS_ENABLED: "true" + GF_AUTH_ANONYMOUS_ORG_ROLE: Editor + GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/intelligence-api.json + volumes: + - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro + - ./deploy/grafana/intelligence-api.json:/var/lib/grafana/dashboards/intelligence-api.json:ro + depends_on: + - prometheus volumes: redis-data: postgres-data: + prometheus-data: diff --git a/docs/api-reference.md b/docs/api-reference.md index 76b2d37..5b5c4e6 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -754,7 +754,7 @@ Returns application configuration, service status, and live runtime metrics. The | `database` | PostgreSQL host, port, dbname, sslmode, connection pool settings | | `cache` | Redis addr/db, cache TTL, auth cache TTL, plus live: key count, memory used, ping latency (ms), auth cache size | | `lookups` | Lookup timeout, subdomain/CT log limits, rate limits and max domains per plan | -| `services` | Notifications (enabled + alert config), billing (enabled, Stripe key mode `live`/`test`, config), threat intel (provider names), fingerprint (source + rule count), usage (buffer config) | +| `services` | Notifications (enabled + alert config), billing (enabled, Stripe key mode `live`/`test`, config), threat intel (provider names), fingerprint (source + rule count), usage (buffer config), GeoIP (database file), metrics (whether the Prometheus listener is on, and its address) | | `runtime` | Goroutines, heap alloc/inuse (MB), sys memory (MB), GC count | | `postgres_pool` | Connection pool: open, in use, idle, wait count, wait duration | diff --git a/docs/decisions.md b/docs/decisions.md index 8e4ad47..7e450ba 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -327,6 +327,61 @@ select. ## Operations +### 2026-11-16 — `/metrics` is on its own listener, with no authentication + +**Decision.** Prometheus metrics are served at `GET /metrics` on a second listener +(`metrics.listen_addr`, `:9091`), started only when `metrics.enabled` is set. The public +router has no such route and answers 404. The endpoint checks no credential; the port +having no public address is the whole access control. + +**Why.** Whoever scrapes has to be able to reach it, and the scraper we use (Fly's, see +the next entry) takes a port and a path and cannot send a header. Behind the admin key, +every scrape would also run the protected chain: an auth lookup, then a row in +`request_log` every 15 seconds per machine, which is the table this work exists to stop +leaning on. And the admin key would have to be copied into a +scraper's configuration, where a key that can rewrite fingerprint rules and read every +customer's usage has no business being. + +**Instead of.** `GET /v1/admin/metrics` behind `RequireAdmin()`, which looks safer because +it has a lock on it. The lock is the problem: it puts the most powerful credential in the +least protected place, and it makes reading the metrics depend on authentication, so on +Postgres, working: they would go dark in the outage they are there to explain. + +**Cost.** Nothing in the code stops someone publishing the port. A `[[services]]` block +for 9091 in `fly.toml`, or a Compose mapping without `127.0.0.1:`, exposes route names, +plan names, traffic volumes and the running version to anyone. On Fly the port is also +reachable by other apps in the same organization over the private network. + +**Revisit if.** The organization gains an app that should not see these numbers, or the +metrics have to be scraped from outside the platform. Then put a bearer token on this +listener; do not move it onto the API port. + +### 2026-11-16 — Production metrics use Fly's managed Prometheus and Grafana + +**Decision.** `fly.toml` carries a `[metrics]` section and Fly scrapes each machine. The +dashboard in `deploy/grafana/` is imported into Fly's hosted Grafana at fly-metrics.net. +We run no Prometheus, no Grafana and no agent in production. The pair in +`docker-compose.yaml` is for checking the dashboard locally and nothing else. + +**Why.** It is the only option that adds nothing to operate. The scrape happens inside +Fly's network, so the endpoint stays off the internet, and there is no token to store or +rotate. It also fits machines that stop when idle: a stopped machine is simply not +scraped, where a push-based setup needs an agent running beside the app on each one. + +**Instead of.** Grafana Cloud's free tier, which needs either a public, authenticated +scrape endpoint or a collector process in every machine pushing with a stored credential, +on machines sized at 1 GB. And instead of self-hosting both on Fly, which is another app +with a volume, upgrades and its own uptime to watch, for one service's dashboard. + +**Cost.** Retention, limits and price are Fly's to change; it is free today and they say +they will give notice. There is no alerting from these metrics yet: alerts still come from +the Pushover alerter and the uptime workflow. The dashboard is imported by hand, so the +hosted copy can fall behind the file. + +**Revisit if.** Fly starts charging enough to matter, alerting on these metrics is wanted +(Grafana Cloud's alerting is the likely reason to move), or the service leaves Fly. The +metrics endpoint itself is plain Prometheus and does not change in any of those cases. + ### 2026-08-27 — Deploys trigger on a release tag, not a release branch **Decision.** `fly-deploy.yml` fires on any `v*` tag. `make release V=` pushes the tag and diff --git a/docs/operations.md b/docs/operations.md index 5dcc523..564e1fc 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -103,6 +103,8 @@ Config values: | `billing.cleanup_interval` | `10m` | Sweep cadence for expired provisioned keys | | `usage.buffer_size` | `256` | Channel buffer size for async usage logging | | `usage.flush_interval` | `5s` | How often buffered usage entries are flushed to PostgreSQL | +| `metrics.enabled` | `false` | Serve Prometheus metrics at `GET /metrics` on a separate listener. See [Metrics](#metrics) | +| `metrics.listen_addr` | `:9091` | Address of the metrics listener. The endpoint is unauthenticated, so this port must never be published | ### Database Setup @@ -145,6 +147,14 @@ docker compose up --build This starts the API, Redis, and PostgreSQL. The API is available at `http://localhost:8080`. The migration and seed files run automatically on first start. +It also starts Prometheus and Grafana, so the dashboard can be checked against real series before a release. All three ports are published on loopback only: + +| URL | What | +|-----|------| +| `http://localhost:9091/metrics` | The API's metrics listener (Compose sets `METRICS_ENABLED=true`) | +| `http://localhost:9090` | Prometheus, scraping the API every 15s | +| `http://localhost:3000` | Grafana, no login, opening on the Intelligence API dashboard | + ### Run from GitHub Container Registry On any machine with Docker — no code checkout needed: @@ -221,7 +231,7 @@ make run # builds and starts the server | `make release V=v0.1.0` | Tag and push a release | | `make deploy` | Deploy to Fly.io with git version baked in | | `make docker-reset` | **Dev only.** Tear down containers, wipe volumes, rebuild, and wait for healthy startup. The wait is bounded by `RESET_TIMEOUT` (default 300s), after which it names the dependency that never came up | -| `./scripts/smoke-test.sh [host]` | Hit every endpoint with every key/plan combo and check expected status codes. Defaults to `http://localhost:8080` | +| `./scripts/smoke-test.sh [host]` | Hit every endpoint with every key/plan combo and check expected status codes. Defaults to `http://localhost:8080`. Also checks that `/metrics` is a 404 on the public port, and reads the metrics listener where it is reachable: `localhost:9091` for a local host, `SMOKE_METRICS_URL` otherwise, skipped when neither applies | | `./scripts/status.sh` | Check Fly.io deployment health, Postgres, and Redis status | | `./scripts/info.sh [host]` | Pretty-print the `/v1/info` admin endpoint. Defaults to production | @@ -289,6 +299,7 @@ hostname is added to Fly and DNS before `site.home_host` is released, not after. | `STRIPE_PORTAL_LINK` | Stripe Customer Portal URL for subscription management | | `SITE_HOME_HOST` | Hostname whose `/` is the company home page; set in `fly.toml` `[env]`, not a secret | | `SITE_API_HOST` | Hostname the home page's docs links point at; set in `fly.toml` `[env]`, not a secret | +| `METRICS_ENABLED` | Serve Prometheus metrics on the metrics listener; set to `true` in `fly.toml` `[env]`, not a secret. The `[metrics]` section of the same file tells Fly where to scrape | | `USAGE_BUFFER_SIZE` | Channel buffer size for async usage logging (default: 256) | | `USAGE_FLUSH_INTERVAL` | Flush interval for usage batch inserts (default: 5s) | | `THREAT_INTEL_VIRUSTOTAL_API_KEY` | VirusTotal API key (enables VirusTotal threat intel provider) | @@ -418,6 +429,60 @@ Alerts include a cooldown period (default: 15 minutes) to prevent alert fatigue. Additionally, a GitHub Actions workflow monitors uptime by pinging the API every 5 minutes and sending a Pushover alert on failure. +## Metrics + +With `metrics.enabled` set, the API serves Prometheus metrics at `GET /metrics` on a second listener (`metrics.listen_addr`, default `:9091`). That listener serves nothing else, and the API's own port does not serve `/metrics` at all: it answers 404 there. + +**The endpoint has no authentication. The port being unreachable from outside is the access control**, so it is never added to a Fly service, a load balancer, or a public Compose port mapping. The reasoning is in [decisions.md](decisions.md). + +Metrics are recorded whether or not the endpoint is enabled; the key only decides whether anything serves them. `GET /v1/info` reports the state under `services.metrics`. + +### What is exported + +| Metric | Labels | What it measures | +|--------|--------|------------------| +| `intelligence_api_http_request_duration_seconds` | `route`, `method`, `status`, `plan` | Latency histogram of every HTTP request. Its `_count` series is the request count | +| `intelligence_api_lookup_duration_seconds` | `scope`, `lookup`, `outcome` | Duration histogram of live lookup fetches. Its `_count` series is the outcome count | +| `intelligence_api_cache_requests_total` | `scope`, `lookup`, `result` | Lookup cache reads, `result` being `hit` or `miss` | +| `intelligence_api_singleflight_coalesced_total` | `scope`, `lookup` | Lookup calls that waited on another caller's in-flight fetch instead of making their own | +| `intelligence_api_rate_limit_rejections_total` | `plan` | Requests the rate limiter answered with 429 | +| `intelligence_api_redis_pool_*` | | The Redis connection pool: `connections`, `idle_connections`, and the counters `hits_total`, `misses_total`, `timeouts_total`, `stale_connections_total` | +| `go_sql_*` | `db_name="postgres"` | The PostgreSQL connection pool, from `database/sql` | +| `intelligence_api_geoip_database_loaded` | | 1 while a GeoIP database is loaded | +| `intelligence_api_geoip_database_build_timestamp_seconds` | | When the loaded GeoIP database was built. Its age is `time()` minus this. Absent while nothing is loaded | +| `intelligence_api_build_info` | `version`, `goversion` | Always 1; the labels carry the running version | +| `go_*`, `process_*` | | Go runtime and process metrics from the client library | + +How to read the labels: + +- **`route`** is the route template (`/v1/domain/lookup`), never the path as sent. A request that matched no route is `unmatched`, and a method outside the standard set is `other`, so a scanner cannot create a series per request. +- **`plan`** is the caller's plan, or `none` for a request with no valid API key (docs pages, health checks, failed authentication). +- **`scope`** is `domain` or `ip`. It keeps the two `threat_intel` lookups apart. +- **`lookup`** is a name from the lookup catalog. `fingerprint` and `risk` never appear: both are computed from other lookups and are never fetched. A `fingerprint` request shows up as `headers`. +- **`outcome`** is `ok`, `error` or `timeout`. A fan-out lookup (`reputation`, `threat_intel`) is an `error` when any provider failed, even if it returned a partial result, which is the same line the cache draws when it declines to store one. +- **Cache hits and coalesced callers are not in the lookup histogram.** It times work that went upstream, so a warm cache does not make a slow provider look fast. +- **`geo` has no cache series.** It is never cached, so it is neither a hit nor a miss. + +One limitation to keep in mind when reading a graph: Prometheus cannot see a series' first value as an increase. The lookup, cache and rate-limit series are created at zero on startup so their first event is visible. The HTTP series cannot be, because the combinations of route, status and plan are not enumerable, so the first request of each combination after a start does not show in a `rate()`. A lone 500 right after a restart is in the raw counter and not on the graph. + +### Dashboard + +`deploy/grafana/intelligence-api.json` is the Grafana dashboard for these metrics: traffic and latency, per-lookup fetch rate, duration and failure ratio, cache hit ratio, rate-limit rejections, both connection pools, GeoIP database age, and the runtime. It selects its data source and the app through variables at the top, so the same file works locally and in production. + +To change it, edit it in the local Grafana, export the dashboard as JSON, and replace the file. + +### Production: Fly's managed Prometheus and Grafana + +Production uses the metrics stack Fly runs for every organization. Nothing is deployed for it and no credential is involved: + +- The `[metrics]` section of `fly.toml` (`port = 9091`, `path = '/metrics'`) tells Fly to scrape each machine every 15 seconds. `METRICS_ENABLED = 'true'` in `[env]` is what makes the app listen there; the two are changed together. +- The port is not in `[http_service]`, so it has no public address. It is reachable from other apps in the same Fly organization over the private network. +- Fly adds `app`, `region`, `host` and `instance` labels to every series. +- Grafana is at [fly-metrics.net](https://fly-metrics.net), signed in with the Fly account. Import the dashboard once with Dashboards → New → Import, upload `deploy/grafana/intelligence-api.json`, and choose the Prometheus data source. After a change to the file, import it again and overwrite. +- For ad-hoc queries, the Prometheus API is at `https://api.fly.io/prometheus//`, authenticated with a Fly access token. + +A stopped machine is not scraped, and its counters start again from zero when it next starts; `rate()` handles the reset. The managed stack is free at the time of writing, drops a scrape larger than 16 MB, and may drop metrics with very high cardinality, which is one more reason the label values above are bounded. + ## Billing Self-service billing via Stripe. Customers click a Stripe Payment Link on the docs page, pay through Stripe Checkout, and receive an API key on a one-time success page. diff --git a/fly.toml b/fly.toml index a7bdc22..2bed5ef 100644 --- a/fly.toml +++ b/fly.toml @@ -18,6 +18,17 @@ primary_region = 'iad' # before this ships, or the home page has no address at all. SITE_HOME_HOST = 'securityaccelerated.com' SITE_API_HOST = 'api.securityaccelerated.com' + # Makes the app serve Prometheus metrics on :9091, which the [metrics] + # section below tells Fly to scrape. The two go together: without this + # nothing listens there, and without that nothing reads it. + METRICS_ENABLED = 'true' + +# Fly's managed Prometheus scrapes this port on every machine. It is +# deliberately not in [http_service]: the endpoint is unauthenticated and must +# have no public address. See docs/operations.md, "Metrics". +[metrics] + port = 9091 + path = '/metrics' [http_service] internal_port = 8080 diff --git a/go.mod b/go.mod index bd43ec8..5557038 100644 --- a/go.mod +++ b/go.mod @@ -12,6 +12,7 @@ require ( github.com/likexian/whois-parser v1.24.21 github.com/miekg/dns v1.1.73 github.com/oschwald/geoip2-golang v1.13.0 + github.com/prometheus/client_golang v1.25.0 github.com/redis/go-redis/v9 v9.23.0 github.com/spf13/viper v1.21.0 github.com/stripe/stripe-go/v84 v84.4.1 @@ -23,6 +24,7 @@ require ( ) require ( + github.com/beorn7/perks v1.0.1 // indirect github.com/bytedance/gopkg v0.1.4 // indirect github.com/bytedance/sonic v1.15.4 // indirect github.com/bytedance/sonic/loader v0.5.2 // indirect @@ -47,9 +49,13 @@ require ( github.com/mattn/go-isatty v0.0.24 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/oschwald/maxminddb-golang v1.13.1 // indirect github.com/pelletier/go-toml/v2 v2.4.3 // indirect github.com/pkg/errors v0.9.1 // indirect + github.com/prometheus/client_model v0.6.3 // indirect + github.com/prometheus/common v0.72.0 // indirect + github.com/prometheus/procfs v0.22.0 // indirect github.com/quic-go/qpack v0.6.0 // indirect github.com/quic-go/quic-go v0.64.0 // indirect github.com/sagikazarmark/locafero v0.12.0 // indirect diff --git a/go.sum b/go.sum index 4e299bf..d80f8ca 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7Oputl github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU= github.com/alicebob/miniredis/v2 v2.36.1 h1:Dvc5oAnNOr7BIfPn7tF269U8DvRW1dBG2D5n0WrfYMI= github.com/alicebob/miniredis/v2 v2.36.1/go.mod h1:TcL7YfarKPGDAthEtl5NBeHZfeUQj6OXMm/+iu5cLMM= +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs= github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c= github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA= @@ -59,12 +61,16 @@ github.com/jackc/puddle/v2 v2.2.3/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFr github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/kisielk/sqlstruct v0.0.0-20201105191214-5f3e10d3ab46/go.mod h1:yyMNCyc/Ib3bDTKd379tNMpB/7/H5TjM2Y9QJ5THLbE= +github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ= +github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI= github.com/klauspost/cpuid/v2 v2.4.1 h1:I9KxadAkvo+QjtcEE2f2tyfEvmIla/T0m+14PrZNl0U= github.com/klauspost/cpuid/v2 v2.4.1/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/leodido/go-urn v1.5.0 h1:pLqT2kq1zpHW/1D18QMjMpdtX7cekxqtJJjg5ANyWw0= github.com/leodido/go-urn v1.5.0/go.mod h1:9BORnCDhdPBJNDEX+w1bJisa8yOKYi116VeO96s4ifE= github.com/likexian/gokit v0.25.16 h1:wwBeUIN/OdoPp6t00xTnZE8Di/+s969Bl5N2Kw6bzP8= @@ -82,6 +88,8 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/oschwald/geoip2-golang v1.13.0 h1:Q44/Ldc703pasJeP5V9+aFSZFmBN7DKHbNsSFzQATJI= github.com/oschwald/geoip2-golang v1.13.0/go.mod h1:P9zG+54KPEFOliZ29i7SeYZ/GM6tfEL+rgSn03hYuUo= github.com/oschwald/maxminddb-golang v1.13.1 h1:G3wwjdN9JmIK2o/ermkHM+98oX5fS+k5MbwsmL4MRQE= @@ -91,6 +99,14 @@ github.com/pelletier/go-toml/v2 v2.4.3/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_golang v1.25.0 h1:cDtW1TcFsMi1RJc8khwCKB4VhzwV04aKqbYtNOXuiK0= +github.com/prometheus/client_golang v1.25.0/go.mod h1:j2X/4/kaQHLPnmyEU8zjmkndo1T317VpCIQEml8LTKk= +github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo= +github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM= +github.com/prometheus/common v0.72.0 h1:tAYsE+sPJxIncDAobm4H5aQjmox9ZxEIIqPbiffa8G4= +github.com/prometheus/common v0.72.0/go.mod h1:77NWqAQ2tXT7BIK40qjJdw5Acrsrg1TlHAnsQi3i6mk= +github.com/prometheus/procfs v0.22.0 h1:6q9+/JL9IKAPbCmBrv9n5O5Ty3NKnciV5X7YGw0oics= +github.com/prometheus/procfs v0.22.0/go.mod h1:CvmFr/GVhIjIvWJZW3tgkODBQMRIf0EyWMQLHCHab58= github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0= github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk= github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8= @@ -149,6 +165,8 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= go.uber.org/zap v1.28.0 h1:IZzaP1Fv73/T/pBMLk4VutPl36uNC+OSUh3JLG3FIjo= go.uber.org/zap v1.28.0/go.mod h1:rDLpOi171uODNm/mxFcuYWxDsqWSAVkFdX4XojSKg/Q= +go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ= +go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ= go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw= go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg= golang.org/x/arch v0.32.0 h1:NjGf8/xGf5JgAJ4dw4NtME5R1L7xatagqAg95T3LgII= diff --git a/integration_test.go b/integration_test.go index 516c5b1..2a09074 100644 --- a/integration_test.go +++ b/integration_test.go @@ -213,7 +213,7 @@ func setupRouterWithUsage(t *testing.T) (*gin.Engine, *miniredis.Miniredis, *rec {Name: "TestServer", Category: "web-server", Headers: map[string]string{"Server": "test"}}, }) - svc := service.NewDomainService(rc, defaultTestTTL, testCacheTTLs, 30*time.Second, 500, 100, fpEngine, ¬ifications.NoopRecorder{}, nil) + svc := service.NewDomainService(rc, defaultTestTTL, testCacheTTLs, 30*time.Second, 500, 100, fpEngine, ¬ifications.NoopRecorder{}, nil, nil) // Replace real lookups with stubs so we don't hit the network. svc.SetLookups( func(_ context.Context, _ string) (*domain.WhoisInfo, error) { @@ -311,7 +311,7 @@ func setupRouterWithUsage(t *testing.T) (*gin.Engine, *miniredis.Miniredis, *rec }) // IP service with stubs - ipSvc := service.NewIPService(rc, defaultTestTTL, testCacheTTLs, 30*time.Second, nil, ¬ifications.NoopRecorder{}, nil) + ipSvc := service.NewIPService(rc, defaultTestTTL, testCacheTTLs, 30*time.Second, nil, ¬ifications.NoopRecorder{}, nil, nil) ipSvc.SetLookups( func(_ net.IP) (*domain.GeoIPInfo, error) { return &domain.GeoIPInfo{Country: "United States", CountryCode: "US", City: "Mountain View", Latitude: 37.386, Longitude: -122.084}, nil diff --git a/internal/app/app.go b/internal/app/app.go index 014b4bb..7447fdb 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -15,6 +15,7 @@ import ( "github.com/security-accelerated/intelligence-api/internal/domain" "github.com/security-accelerated/intelligence-api/internal/handler" "github.com/security-accelerated/intelligence-api/internal/httpapi" + "github.com/security-accelerated/intelligence-api/internal/metrics" "github.com/security-accelerated/intelligence-api/internal/middleware" "github.com/security-accelerated/intelligence-api/internal/notifications" "github.com/security-accelerated/intelligence-api/internal/service" @@ -32,9 +33,17 @@ func Run(logger *zap.Logger, version string) error { status := config.NewStatusSnapshot(version, lookupNames) + // Built whether or not the endpoint is enabled: recording is a few atomic + // operations, and metrics.enabled only decides whether anything serves it. + m, err := metrics.New(version) + if err != nil { + return fmt.Errorf("build metrics: %w", err) + } + rlCfg := middleware.RateLimitConfig{ - Free: cfg.GetInt("rate_limit.free"), - Pro: cfg.GetInt("rate_limit.pro"), + Free: cfg.GetInt("rate_limit.free"), + Pro: cfg.GetInt("rate_limit.pro"), + Metrics: m, } logger.Info("initializing application", @@ -56,6 +65,9 @@ func Run(logger *zap.Logger, version string) error { } }() logger.Info("connected to postgres", zap.Any("config", status.Database)) + if err := m.RegisterDB(db); err != nil { + return err + } // Auth lastUsedBatcher := auth.NewLastUsedBatcher( @@ -84,6 +96,9 @@ func Run(logger *zap.Logger, version string) error { return fmt.Errorf("connect to redis: %w", err) } logger.Info("connected to redis", zap.Any("config", status.Cache)) + if err := m.RegisterRedisPool(redisCache.Client()); err != nil { + return err + } // DNS resolver for intelligence lookups. Scoped to the domain package via // domain.Resolver(), so this no longer has to run after Postgres and Redis @@ -189,7 +204,7 @@ func Run(logger *zap.Logger, version string) error { domain.CTLogsHTTPTimeout = v } - domainService := service.NewDomainService(redisCache, defaultCacheTTL, cacheTTLs, cfg.GetDuration("domain.lookup_timeout"), cfg.GetInt("subdomain.max_results"), cfg.GetInt("ct_logs.max_entries"), fpService.Engine(), alerter, tiProviders) + domainService := service.NewDomainService(redisCache, defaultCacheTTL, cacheTTLs, cfg.GetDuration("domain.lookup_timeout"), cfg.GetInt("subdomain.max_results"), cfg.GetInt("ct_logs.max_entries"), fpService.Engine(), alerter, tiProviders, m) // Billing billingRepo := billing.NewRepository(db) @@ -257,6 +272,9 @@ func Run(logger *zap.Logger, version string) error { logger.Error("geoip reader close failed", zap.Error(err)) } }() + if err := m.RegisterGeoIP(geoReader); err != nil { + return err + } // IP threat intel providers (adapters around the domain providers) var ipTIProviders []domain.IPThreatIntelProvider @@ -272,7 +290,7 @@ func Run(logger *zap.Logger, version string) error { } // IP intelligence service - ipService := service.NewIPService(redisCache, defaultCacheTTL, cacheTTLs, cfg.GetDuration("ip.lookup_timeout"), geoReader, alerter, ipTIProviders) + ipService := service.NewIPService(redisCache, defaultCacheTTL, cacheTTLs, cfg.GetDuration("ip.lookup_timeout"), geoReader, alerter, ipTIProviders, m) // HTTP h := handler.New(handler.Deps{ @@ -289,7 +307,7 @@ func Run(logger *zap.Logger, version string) error { DB: db, Status: status, }) - router := httpapi.NewRouter(h, cachedRepo, redisCache.Client(), rlCfg, usageLogger, cfg.GetString("application.mode"), int64(cfg.GetInt("server.body_limit"))) + router := httpapi.NewRouter(h, cachedRepo, redisCache.Client(), rlCfg, usageLogger, cfg.GetString("application.mode"), int64(cfg.GetInt("server.body_limit")), m) serverCfg := httpapi.ServerConfig{ Port: status.System.Port, ReadTimeout: cfg.GetDuration("server.read_timeout"), @@ -299,6 +317,14 @@ func Run(logger *zap.Logger, version string) error { } server := httpapi.NewServer(router, serverCfg) + // Prometheus endpoint, on its own listener. Started before the API so a + // port it cannot bind stops the process before it takes any traffic. + stopMetrics, err := startMetricsServer(ctx, cfg, m, serverCfg, logger, status) + if err != nil { + return err + } + defer stopMetrics() + // Shutdown signal sigCh := make(chan os.Signal, 1) signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) diff --git a/internal/app/metrics.go b/internal/app/metrics.go new file mode 100644 index 0000000..a44825b --- /dev/null +++ b/internal/app/metrics.go @@ -0,0 +1,50 @@ +package app + +import ( + "context" + "fmt" + + "github.com/security-accelerated/intelligence-api/internal/config" + "github.com/security-accelerated/intelligence-api/internal/httpapi" + "github.com/security-accelerated/intelligence-api/internal/metrics" + "github.com/spf13/viper" + "go.uber.org/zap" +) + +// startMetricsServer starts the Prometheus listener when metrics.enabled is +// set and returns the function that stops it. With metrics disabled it starts +// nothing and returns a no-op, so the caller defers the result either way. +// +// The listener shares the API server's timeouts: it is the same process +// answering the same kind of short request, and a second set of keys would be +// three more values nobody has a reason to set differently. +func startMetricsServer(ctx context.Context, cfg *viper.Viper, m *metrics.Metrics, serverCfg httpapi.ServerConfig, logger *zap.Logger, status *config.StatusSnapshot) (stop func(), err error) { + if !cfg.GetBool("metrics.enabled") { + logger.Info("metrics endpoint disabled (metrics.enabled is false)") + return func() {}, nil + } + + addr := cfg.GetString("metrics.listen_addr") + shutdown, err := httpapi.StartMetricsServer(ctx, httpapi.MetricsServerConfig{ + Addr: addr, + ReadTimeout: serverCfg.ReadTimeout, + WriteTimeout: serverCfg.WriteTimeout, + IdleTimeout: serverCfg.IdleTimeout, + }, m.Handler(), logger) + if err != nil { + return nil, fmt.Errorf("start metrics server: %w", err) + } + + status.Services.Metrics = config.MetricsStatus{Enabled: true, ListenAddr: addr} + logger.Info("metrics endpoint enabled", zap.Any("config", status.Services.Metrics)) + + return func() { + // Run's own context is already canceled by the time this runs, so the + // drain gets a fresh one bounded by the same shutdown timeout. + stopCtx, cancel := context.WithTimeout(context.Background(), serverCfg.ShutdownTimeout) + defer cancel() + if err := shutdown(stopCtx); err != nil { + logger.Error("metrics server shutdown failed", zap.Error(err)) + } + }, nil +} diff --git a/internal/config/config.go b/internal/config/config.go index f24b69f..458a345 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -149,6 +149,10 @@ func setDefaults(v *viper.Viper) { v.SetDefault("ip.lookup_timeout", "30s") v.SetDefault("ip.max_ips.free", 1) v.SetDefault("ip.max_ips.pro", 20) + // The Prometheus endpoint is off unless asked for: enabling it opens a + // second listener, which should never appear on an upgrade unannounced. + v.SetDefault("metrics.enabled", false) + v.SetDefault("metrics.listen_addr", ":9091") v.SetDefault("dns.servers", []string{}) v.SetDefault("dns.timeout", "30s") } diff --git a/internal/config/config_test.go b/internal/config/config_test.go index ddaee85..1605594 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -46,6 +46,9 @@ func TestConfig_Defaults(t *testing.T) { {"geoip.max_age", "720h"}, {"cache.ttl.asn", "24h"}, {"postgres.sslmode", "require"}, + // Off by default: turning it on opens a second, unauthenticated listener. + {"metrics.enabled", false}, + {"metrics.listen_addr", ":9091"}, } for _, tt := range tests { diff --git a/internal/config/sample-config.yml b/internal/config/sample-config.yml index b098e93..27012ea 100644 --- a/internal/config/sample-config.yml +++ b/internal/config/sample-config.yml @@ -132,6 +132,13 @@ geoip: # DB-IP publishes Lite releases monthly. 0 disables the check. max_age: "720h" +# Prometheus metrics, served at GET /metrics on a listener of their own. The +# endpoint is unauthenticated: the port being unreachable from outside is what +# protects it, so never publish it. Off by default. +metrics: + enabled: false + listen_addr: ":9091" + threat_intel: virustotal_api_key: "" alienvault_api_key: "" diff --git a/internal/config/status.go b/internal/config/status.go index 89f657e..6c1e344 100644 --- a/internal/config/status.go +++ b/internal/config/status.go @@ -95,6 +95,7 @@ type ServicesStatus struct { Fingerprint FingerprintStatus `json:"fingerprint"` Usage UsageStatus `json:"usage"` GeoIP GeoIPStatus `json:"geoip"` + Metrics MetricsStatus `json:"metrics"` } type NotificationsStatus struct { @@ -138,6 +139,13 @@ type GeoIPStatus struct { DBPath string `json:"db_path"` } +// MetricsStatus reports whether the Prometheus listener is serving. ListenAddr +// is empty when it is not. +type MetricsStatus struct { + Enabled bool `json:"enabled"` + ListenAddr string `json:"listen_addr,omitempty"` +} + // NewStatusSnapshot builds a StatusSnapshot from the current config values. // Service-level fields (Services.*) are left at zero values — the caller // populates them as each service is initialized. lookupNames enumerates the diff --git a/internal/domain/geoip.go b/internal/domain/geoip.go index 63161f9..0439077 100644 --- a/internal/domain/geoip.go +++ b/internal/domain/geoip.go @@ -7,6 +7,7 @@ import ( "errors" "fmt" "io" + "math" "net" "net/http" "os" @@ -67,6 +68,23 @@ func (r *GeoIPReader) Loaded() bool { return r.cityDB != nil } +// BuildTime reports when the loaded database was built, read from the +// database's own metadata, and false when no database is loaded. It is the age +// of the data being served, which the file's modification time is not: that +// records when the file was downloaded. +func (r *GeoIPReader) BuildTime() (time.Time, bool) { + r.mu.RLock() + defer r.mu.RUnlock() + if r.cityDB == nil { + return time.Time{}, false + } + epoch := r.cityDB.Metadata().BuildEpoch + if epoch > math.MaxInt64 { + return time.Time{}, false + } + return time.Unix(int64(epoch), 0), true +} + // LookupGeo returns geographic information for the given IP. DB-IP Lite carries // no postal code, timezone, accuracy radius, or subdivision code, so GeoIPInfo // has no fields for them. diff --git a/internal/domain/geoip_test.go b/internal/domain/geoip_test.go index 7ac4874..6083210 100644 --- a/internal/domain/geoip_test.go +++ b/internal/domain/geoip_test.go @@ -50,6 +50,14 @@ func stubDBIP(t *testing.T, release string, body []byte) { t.Cleanup(func() { trustedTransport = origTransport }) } +// The GeoIP metrics collector relies on this to report "not loaded" instead of +// a build time of zero. +func TestGeoIPReader_BuildTime_NothingLoaded(t *testing.T) { + if built, ok := NewEmptyGeoIPReader().BuildTime(); ok { + t.Errorf("BuildTime on an empty reader = %v, true; want false", built) + } +} + func TestDBIPReleases(t *testing.T) { tests := []struct { now time.Time diff --git a/internal/handler/changelog.yaml b/internal/handler/changelog.yaml index eb3a9c2..80c2e35 100644 --- a/internal/handler/changelog.yaml +++ b/internal/handler/changelog.yaml @@ -10,6 +10,18 @@ # terms (#91). An entry a later release reversed keeps its text and gains a # "superseded in vX" note, rather than being hidden or rewritten. +- version: v1.2.0 + date: 2026-11-16 + changes: + - text: "Prometheus metrics (#133). With `metrics.enabled` (`METRICS_ENABLED`) the app serves `GET /metrics` on its own listener, `metrics.listen_addr` (default `:9091`). The endpoint has no authentication and the API port answers 404 for it: the separate, unpublished port is the access control, recorded in docs/decisions.md. Off by default; `fly.toml` turns it on and gains a `[metrics]` section so Fly's managed Prometheus scrapes it" + audience: internal + - text: "Exported: an HTTP latency histogram by route template, method, status and plan; a per-lookup fetch duration histogram by scope, lookup and outcome (ok, error, timeout); cache hits and misses and singleflight-coalesced calls per lookup; rate-limit rejections per plan; the Postgres and Redis connection pools; GeoIP database build time; a build-info gauge with the version; Go runtime and process metrics. Each histogram's `_count` is the matching counter. `GET /v1/info` reports `services.metrics`" + audience: internal + - text: "`deploy/grafana/intelligence-api.json` is the Grafana dashboard for these metrics. `docker compose up` now also starts Prometheus and Grafana on loopback (9090 and 3000) to check it locally. Production uses Fly's hosted Grafana at fly-metrics.net, where the dashboard is imported by hand. The smoke test asserts `/metrics` is a 404 on the public port and reads the metrics listener where it can reach it (`SMOKE_METRICS_URL`, or localhost:9091 for a local host)" + audience: internal + - text: "New dependency: github.com/prometheus/client_golang 1.25.0" + audience: internal + - version: v1.1.3 date: 2026-11-12 changes: diff --git a/internal/httpapi/metrics_server.go b/internal/httpapi/metrics_server.go new file mode 100644 index 0000000..15252a7 --- /dev/null +++ b/internal/httpapi/metrics_server.go @@ -0,0 +1,60 @@ +package httpapi + +import ( + "context" + "errors" + "fmt" + "net" + "net/http" + "time" + + "go.uber.org/zap" +) + +// MetricsServerConfig configures the listener that serves /metrics. +type MetricsServerConfig struct { + // Addr is the listen address, e.g. ":9091". + Addr string + ReadTimeout time.Duration + WriteTimeout time.Duration + IdleTimeout time.Duration +} + +// StartMetricsServer serves handler at GET /metrics on a listener of its own, +// apart from the API's port, and returns once that listener is bound. +// +// The separate listener is the access control: nothing routes the public port +// here, so the endpoint needs no credential a scraper would have to hold. See +// docs/decisions.md. +// +// A bind failure is returned, so a deployment that asks for metrics and cannot +// serve them fails at startup instead of running without them. A failure after +// that is logged and leaves the API serving. The caller owns the server's +// lifetime and must call the returned stop function, which waits for an +// in-flight scrape until ctx ends. +func StartMetricsServer(ctx context.Context, cfg MetricsServerConfig, handler http.Handler, logger *zap.Logger) (stop func(context.Context) error, err error) { + var lc net.ListenConfig + ln, err := lc.Listen(ctx, "tcp", cfg.Addr) + if err != nil { + return nil, fmt.Errorf("listen on %s: %w", cfg.Addr, err) + } + + mux := http.NewServeMux() + mux.Handle("GET /metrics", handler) + server := &http.Server{ + Handler: mux, + ReadHeaderTimeout: cfg.ReadTimeout, + ReadTimeout: cfg.ReadTimeout, + WriteTimeout: cfg.WriteTimeout, + IdleTimeout: cfg.IdleTimeout, + } + + go func() { + logger.Info("metrics server started", zap.String("addr", ln.Addr().String())) + if err := server.Serve(ln); !errors.Is(err, http.ErrServerClosed) { + logger.Error("metrics server stopped", zap.Error(err)) + } + }() + + return server.Shutdown, nil +} diff --git a/internal/httpapi/metrics_server_test.go b/internal/httpapi/metrics_server_test.go new file mode 100644 index 0000000..3536687 --- /dev/null +++ b/internal/httpapi/metrics_server_test.go @@ -0,0 +1,162 @@ +package httpapi + +import ( + "context" + "io" + "net" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/redis/go-redis/v9" + "github.com/security-accelerated/intelligence-api/internal/auth" + "github.com/security-accelerated/intelligence-api/internal/handler" + "github.com/security-accelerated/intelligence-api/internal/metrics" + "github.com/security-accelerated/intelligence-api/internal/metrics/metricstest" + "github.com/security-accelerated/intelligence-api/internal/middleware" + "github.com/security-accelerated/intelligence-api/internal/usage" + "go.uber.org/zap" +) + +func newTestMetrics(t *testing.T) *metrics.Metrics { + t.Helper() + m, err := metrics.New("v1.2.3") + if err != nil { + t.Fatalf("metrics.New: %v", err) + } + return m +} + +func testMetricsServerConfig(addr string) MetricsServerConfig { + return MetricsServerConfig{ + Addr: addr, + ReadTimeout: 5 * time.Second, + WriteTimeout: 5 * time.Second, + IdleTimeout: 5 * time.Second, + } +} + +// freeAddr returns a loopback address nothing is listening on. +func freeAddr(t *testing.T) string { + t.Helper() + var lc net.ListenConfig + ln, err := lc.Listen(t.Context(), "tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + addr := ln.Addr().String() + if err := ln.Close(); err != nil { + t.Fatalf("close: %v", err) + } + return addr +} + +func get(t *testing.T, url string) (status int, body string) { + t.Helper() + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, url, http.NoBody) + if err != nil { + t.Fatalf("new request: %v", err) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("GET %s: %v", url, err) + } + defer resp.Body.Close() + b, err := io.ReadAll(resp.Body) + if err != nil { + t.Fatalf("read body: %v", err) + } + return resp.StatusCode, string(b) +} + +func TestStartMetricsServer_ServesOnlyMetrics(t *testing.T) { + addr := freeAddr(t) + stop, err := StartMetricsServer(t.Context(), testMetricsServerConfig(addr), newTestMetrics(t).Handler(), zap.NewNop()) + if err != nil { + t.Fatalf("StartMetricsServer: %v", err) + } + // Registered before the first request, so an assertion that fails early + // does not leave the listener and its goroutine behind for the rest of the + // test binary. Shutting down twice is harmless; the explicit stop below is + // the one the test asserts on. + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = stop(ctx) + }) + + status, body := get(t, "http://"+addr+"/metrics") + if status != http.StatusOK { + t.Fatalf("GET /metrics = %d, want 200", status) + } + if !strings.Contains(body, `intelligence_api_build_info{goversion="`) || !strings.Contains(body, `version="v1.2.3"`) { + t.Errorf("GET /metrics did not return the build info series:\n%.400s", body) + } + + // The listener carries the one path. Nothing of the API is reachable on it. + for _, path := range []string{"/", "/v1/health", "/v1/info", "/metrics/x"} { + if status, _ := get(t, "http://"+addr+path); status != http.StatusNotFound { + t.Errorf("GET %s on the metrics listener = %d, want 404", path, status) + } + } + + stopCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stop(stopCtx); err != nil { + t.Fatalf("stop: %v", err) + } + req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, "http://"+addr+"/metrics", http.NoBody) + if err != nil { + t.Fatalf("new request: %v", err) + } + if resp, err := http.DefaultClient.Do(req); err == nil { + resp.Body.Close() + t.Error("the listener still answers after stop") + } +} + +// A port that cannot be bound has to fail startup, not leave the process +// running with metrics silently absent. +func TestStartMetricsServer_BindFailureIsReturned(t *testing.T) { + var lc net.ListenConfig + taken, err := lc.Listen(t.Context(), "tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + defer taken.Close() + + stop, err := StartMetricsServer(t.Context(), testMetricsServerConfig(taken.Addr().String()), http.NotFoundHandler(), zap.NewNop()) + if err == nil { + _ = stop(context.Background()) + t.Fatal("StartMetricsServer on a taken port returned nil, want an error") + } +} + +// The public router must not serve /metrics: the endpoint is unauthenticated, +// and its own listener is the only thing standing between it and the internet. +func TestNewRouter_DoesNotServeMetricsAndRecordsTheAttempt(t *testing.T) { + gin.SetMode(gin.TestMode) + m := newTestMetrics(t) + // The Redis client is never dialed: nothing here reaches the rate limiter. + client := redis.NewClient(&redis.Options{Addr: "127.0.0.1:0"}) + t.Cleanup(func() { _ = client.Close() }) + + router := NewRouter(handler.New(handler.Deps{}), auth.NewCachedRepository(nil, time.Minute), client, + middleware.RateLimitConfig{}, usage.NoopRecorder{}, "test", 1<<20, m) + + w := httptest.NewRecorder() + router.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/metrics", http.NoBody)) + if w.Code != http.StatusNotFound { + t.Fatalf("GET /metrics on the public router = %d, want 404", w.Code) + } + if strings.Contains(w.Body.String(), "intelligence_api_") { + t.Error("the public router's 404 body contains metric names") + } + + // The request itself went through the metrics middleware like any other. + metricstest.Want(t, m, + `intelligence_api_http_request_duration_seconds_count{method="GET",plan="none",route="unmatched",status="404"}`, 1) +} diff --git a/internal/httpapi/router.go b/internal/httpapi/router.go index 6f88069..f8af7b0 100644 --- a/internal/httpapi/router.go +++ b/internal/httpapi/router.go @@ -8,13 +8,16 @@ import ( "github.com/security-accelerated/intelligence-api/internal/auth" "github.com/security-accelerated/intelligence-api/internal/handler" "github.com/security-accelerated/intelligence-api/internal/logging" + "github.com/security-accelerated/intelligence-api/internal/metrics" "github.com/security-accelerated/intelligence-api/internal/middleware" "github.com/security-accelerated/intelligence-api/internal/usage" "go.uber.org/zap" ) // NewRouter builds the gin engine with all middleware and routes registered. -func NewRouter(h *handler.Handler, cachedRepo *auth.CachedRepository, redisClient *redis.Client, rlCfg middleware.RateLimitConfig, recorder usage.Recorder, mode string, bodyLimit int64) *gin.Engine { +// m may be nil, which records no request metrics. /metrics itself is not a +// route here: it is served on its own listener by StartMetricsServer. +func NewRouter(h *handler.Handler, cachedRepo *auth.CachedRepository, redisClient *redis.Client, rlCfg middleware.RateLimitConfig, recorder usage.Recorder, mode string, bodyLimit int64, m *metrics.Metrics) *gin.Engine { if mode == "production" { gin.SetMode(gin.ReleaseMode) } @@ -26,6 +29,9 @@ func NewRouter(h *handler.Handler, cachedRepo *auth.CachedRepository, redisClien if mode == "production" { router.TrustedPlatform = "Fly-Client-IP" } + // Metrics goes ahead of Recovery so a request that panics is still + // recorded, as the 500 Recovery turns it into. + router.Use(middleware.Metrics(m)) router.Use(gin.Recovery()) router.Use(middleware.SecurityHeaders()) router.Use(middleware.BodyLimit(bodyLimit)) diff --git a/internal/metrics/collectors.go b/internal/metrics/collectors.go new file mode 100644 index 0000000..dc35c70 --- /dev/null +++ b/internal/metrics/collectors.go @@ -0,0 +1,123 @@ +package metrics + +import ( + "database/sql" + "fmt" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" + "github.com/redis/go-redis/v9" +) + +// Dependency collectors. Each reads its source when Prometheus scrapes, so +// nothing here runs on the request path and there is no background sampler to +// own or stop. + +// RegisterDB exports the Postgres connection pool (database/sql's DBStats) as +// the client library's standard go_sql_* series, labeled db_name="postgres". +func (m *Metrics) RegisterDB(db *sql.DB) error { + if err := m.registry.Register(collectors.NewDBStatsCollector(db, "postgres")); err != nil { + return fmt.Errorf("register postgres pool collector: %w", err) + } + return nil +} + +// RedisPool is the part of the Redis client the pool collector reads. +type RedisPool interface { + PoolStats() *redis.PoolStats +} + +// RegisterRedisPool exports the Redis client's connection pool. The cache and +// the rate limiter share one client, so this one pool is all of the service's +// Redis traffic. +func (m *Metrics) RegisterRedisPool(pool RedisPool) error { + desc := func(name, help string) *prometheus.Desc { + return prometheus.NewDesc(prometheus.BuildFQName(namespace, "redis_pool", name), help, nil, nil) + } + c := &redisPoolCollector{ + pool: pool, + totalConns: desc("connections", "Connections currently in the Redis pool, in use or idle."), + idleConns: desc("idle_connections", "Idle connections currently in the Redis pool."), + hits: desc("hits_total", "Times a free connection was found in the Redis pool."), + misses: desc("misses_total", "Times no free connection was found in the Redis pool and a new one was dialed."), + timeouts: desc("timeouts_total", "Times waiting for a Redis pool connection timed out."), + stale: desc("stale_connections_total", "Stale connections removed from the Redis pool."), + } + if err := m.registry.Register(c); err != nil { + return fmt.Errorf("register redis pool collector: %w", err) + } + return nil +} + +// redisPoolCollector takes one PoolStats snapshot per scrape, so the series in +// a scrape are consistent with each other. +type redisPoolCollector struct { + pool RedisPool + + totalConns, idleConns *prometheus.Desc + hits, misses, timeouts, stale *prometheus.Desc +} + +func (c *redisPoolCollector) Describe(ch chan<- *prometheus.Desc) { + prometheus.DescribeByCollect(c, ch) +} + +func (c *redisPoolCollector) Collect(ch chan<- prometheus.Metric) { + s := c.pool.PoolStats() + ch <- prometheus.MustNewConstMetric(c.totalConns, prometheus.GaugeValue, float64(s.TotalConns)) + ch <- prometheus.MustNewConstMetric(c.idleConns, prometheus.GaugeValue, float64(s.IdleConns)) + ch <- prometheus.MustNewConstMetric(c.hits, prometheus.CounterValue, float64(s.Hits)) + ch <- prometheus.MustNewConstMetric(c.misses, prometheus.CounterValue, float64(s.Misses)) + ch <- prometheus.MustNewConstMetric(c.timeouts, prometheus.CounterValue, float64(s.Timeouts)) + ch <- prometheus.MustNewConstMetric(c.stale, prometheus.CounterValue, float64(s.StaleConns)) +} + +// GeoIPDatabase is the part of the GeoIP reader the collector reads. +// BuildTime reports when the loaded database was built, and false when no +// database is loaded. +type GeoIPDatabase interface { + BuildTime() (time.Time, bool) +} + +// RegisterGeoIP exports whether a GeoIP database is loaded and, when one is, +// the time it was built. The build time is exported as a timestamp rather than +// an age so the age is computed where it is read (time() - the gauge) and +// stays correct between scrapes. It is absent, not zero, while nothing is +// loaded: a zero would read as a database built in 1970. +func (m *Metrics) RegisterGeoIP(db GeoIPDatabase) error { + c := &geoIPCollector{ + db: db, + loaded: prometheus.NewDesc(prometheus.BuildFQName(namespace, "geoip", "database_loaded"), + "1 when a GeoIP database is loaded and the geo lookup can answer, 0 otherwise.", nil, nil), + built: prometheus.NewDesc(prometheus.BuildFQName(namespace, "geoip", "database_build_timestamp_seconds"), + "Unix time the loaded GeoIP database was built. Absent while no database is loaded.", nil, nil), + } + if err := m.registry.Register(c); err != nil { + return fmt.Errorf("register geoip collector: %w", err) + } + return nil +} + +type geoIPCollector struct { + db GeoIPDatabase + loaded, built *prometheus.Desc +} + +// Describe lists both series explicitly: Collect emits the build time only +// while a database is loaded, so describing by collecting would register a +// different set depending on when the process started. +func (c *geoIPCollector) Describe(ch chan<- *prometheus.Desc) { + ch <- c.loaded + ch <- c.built +} + +func (c *geoIPCollector) Collect(ch chan<- prometheus.Metric) { + built, ok := c.db.BuildTime() + if !ok { + ch <- prometheus.MustNewConstMetric(c.loaded, prometheus.GaugeValue, 0) + return + } + ch <- prometheus.MustNewConstMetric(c.loaded, prometheus.GaugeValue, 1) + ch <- prometheus.MustNewConstMetric(c.built, prometheus.GaugeValue, float64(built.Unix())) +} diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go new file mode 100644 index 0000000..244aedd --- /dev/null +++ b/internal/metrics/metrics.go @@ -0,0 +1,216 @@ +// Package metrics holds the service's Prometheus instrumentation: the +// collectors, the methods the rest of the code calls to record into them, and +// the handler that renders them in the exposition format. +package metrics + +import ( + "fmt" + "net/http" + "runtime" + "strconv" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +// namespace prefixes every metric this package defines. The dashboard in +// deploy/grafana queries these names, so renaming one is a change to both. +const namespace = "intelligence_api" + +// Lookup outcomes, the values of the lookup histogram's "outcome" label. +const ( + OutcomeOK = "ok" + OutcomeError = "error" + OutcomeTimeout = "timeout" +) + +// Label values for requests that have no better one. A request that matched no +// route is recorded under one fixed route, and a method outside the standard +// set under one fixed method: both come straight from the caller, so recording +// them verbatim would let anyone mint an unbounded number of series. +const ( + RouteUnmatched = "unmatched" + MethodOther = "other" + PlanNone = "none" +) + +// Bucket bounds in seconds. They are part of the metric's schema rather than +// tuning: changing them breaks the continuity of every stored series, so they +// are fixed here and not config keys. Both run past the timeouts that bound +// what they measure (server.write_timeout at 45s, the lookup timeouts at 30s). +var ( + httpBuckets = []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 45} + lookupBuckets = []float64{0.005, 0.025, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 15, 30} +) + +// Metrics records what the service does and renders it for Prometheus. +// +// It owns a registry of its own instead of using the client library's global +// one, so two instances (every test builds its own) never collide on a metric +// name. Recording is always on and costs a few atomic operations; the +// metrics.enabled config key decides only whether a listener serves the +// result. +// +// Every recording method is safe on a nil *Metrics and does nothing there, so +// a caller holding no Metrics needs neither a check nor a stand-in. +type Metrics struct { + registry *prometheus.Registry + + httpDuration *prometheus.HistogramVec + lookupDuration *prometheus.HistogramVec + cacheRequests *prometheus.CounterVec + coalesced *prometheus.CounterVec + rateLimitRejections *prometheus.CounterVec +} + +// New builds a Metrics with the request, lookup, cache and rate-limit +// collectors registered, along with the Go runtime and process collectors and +// a build-info gauge carrying version. The dependency collectors are added +// separately (RegisterDB, RegisterRedisPool, RegisterGeoIP) as each dependency +// comes up. +func New(version string) (*Metrics, error) { + m := &Metrics{ + registry: prometheus.NewRegistry(), + // One histogram per event rather than a histogram and a counter: its + // _count series is the request counter, so the two cannot disagree. + httpDuration: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Namespace: namespace, + Name: "http_request_duration_seconds", + Help: "HTTP request latency by route template, method, status code and plan. The _count series is the request count.", + Buckets: httpBuckets, + }, []string{"route", "method", "status", "plan"}), + lookupDuration: prometheus.NewHistogramVec(prometheus.HistogramOpts{ + Namespace: namespace, + Name: "lookup_duration_seconds", + Help: "Duration of live lookup fetches by scope (domain or ip), lookup and outcome (ok, error, timeout). Cache hits and coalesced callers are not observed. The _count series is the outcome count.", + Buckets: lookupBuckets, + }, []string{"scope", "lookup", "outcome"}), + cacheRequests: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: namespace, + Name: "cache_requests_total", + Help: "Lookup cache reads by scope, lookup and result (hit or miss). Lookups that are never cached are not counted.", + }, []string{"scope", "lookup", "result"}), + coalesced: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: namespace, + Name: "singleflight_coalesced_total", + Help: "Lookup calls served by another caller's in-flight fetch instead of their own, by scope and lookup.", + }, []string{"scope", "lookup"}), + rateLimitRejections: prometheus.NewCounterVec(prometheus.CounterOpts{ + Namespace: namespace, + Name: "rate_limit_rejections_total", + Help: "Requests rejected with 429 by the rate limiter, by plan.", + }, []string{"plan"}), + } + + buildInfo := prometheus.NewGauge(prometheus.GaugeOpts{ + Namespace: namespace, + Name: "build_info", + Help: "Always 1; the labels carry the running version and the Go version it was built with.", + ConstLabels: prometheus.Labels{"version": version, "goversion": runtime.Version()}, + }) + buildInfo.Set(1) + + for _, c := range []prometheus.Collector{ + m.httpDuration, + m.lookupDuration, + m.cacheRequests, + m.coalesced, + m.rateLimitRejections, + buildInfo, + collectors.NewGoCollector(), + collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}), + } { + if err := m.registry.Register(c); err != nil { + return nil, fmt.Errorf("register collector: %w", err) + } + } + return m, nil +} + +// Handler serves the registry in the Prometheus exposition format. +func (m *Metrics) Handler() http.Handler { + return promhttp.HandlerFor(m.registry, promhttp.HandlerOpts{}) +} + +// InitLookup creates one lookup's series at zero. Prometheus cannot see the +// step from "no such series" to its first value, so a series that first appears +// when its first event happens hides that event from rate(): without this, the +// first timeout of a lookup after a start would never reach a graph. cached +// says whether the lookup reads the cache at all; one that does not gets no +// hit or miss series. +func (m *Metrics) InitLookup(scope, lookup string, cached bool) { + if m == nil { + return + } + for _, outcome := range []string{OutcomeOK, OutcomeError, OutcomeTimeout} { + m.lookupDuration.WithLabelValues(scope, lookup, outcome) + } + m.coalesced.WithLabelValues(scope, lookup) + if cached { + m.cacheRequests.WithLabelValues(scope, lookup, "hit") + m.cacheRequests.WithLabelValues(scope, lookup, "miss") + } +} + +// InitRateLimitPlan creates a plan's rejection counter at zero, for the same +// reason as InitLookup. +func (m *Metrics) InitRateLimitPlan(plan string) { + if m == nil { + return + } + m.rateLimitRejections.WithLabelValues(plan) +} + +// ObserveHTTP records one finished HTTP request. Its series cannot be created +// ahead of time the way the lookup series are: the combinations of route, +// status and plan are not enumerable, so the first request of each is the one +// rate() does not see. route is the route template +// (never the raw path), and the caller is responsible for bounding route, +// method and plan; see RouteUnmatched, MethodOther and PlanNone. +func (m *Metrics) ObserveHTTP(route, method, plan string, status int, d time.Duration) { + if m == nil { + return + } + m.httpDuration.WithLabelValues(route, method, strconv.Itoa(status), plan).Observe(d.Seconds()) +} + +// ObserveLookup records one live fetch of a lookup: how long it took and how +// it ended (OutcomeOK, OutcomeError or OutcomeTimeout). scope is "domain" or +// "ip", which keeps the two threat_intel lookups apart. +func (m *Metrics) ObserveLookup(scope, lookup, outcome string, d time.Duration) { + if m == nil { + return + } + m.lookupDuration.WithLabelValues(scope, lookup, outcome).Observe(d.Seconds()) +} + +// CacheResult records one cache read for a lookup as a hit or a miss. +func (m *Metrics) CacheResult(scope, lookup string, hit bool) { + if m == nil { + return + } + result := "miss" + if hit { + result = "hit" + } + m.cacheRequests.WithLabelValues(scope, lookup, result).Inc() +} + +// Coalesced records a lookup call that waited on another caller's fetch +// through singleflight rather than running its own. +func (m *Metrics) Coalesced(scope, lookup string) { + if m == nil { + return + } + m.coalesced.WithLabelValues(scope, lookup).Inc() +} + +// RateLimitRejected records a request the rate limiter turned away. +func (m *Metrics) RateLimitRejected(plan string) { + if m == nil { + return + } + m.rateLimitRejections.WithLabelValues(plan).Inc() +} diff --git a/internal/metrics/metrics_test.go b/internal/metrics/metrics_test.go new file mode 100644 index 0000000..4cdb3eb --- /dev/null +++ b/internal/metrics/metrics_test.go @@ -0,0 +1,143 @@ +package metrics_test + +import ( + "runtime" + "strings" + "testing" + "time" + + "github.com/alicebob/miniredis/v2" + "github.com/redis/go-redis/v9" + "github.com/security-accelerated/intelligence-api/internal/metrics" + "github.com/security-accelerated/intelligence-api/internal/metrics/metricstest" +) + +func newMetrics(t *testing.T) *metrics.Metrics { + t.Helper() + m, err := metrics.New("v9.9.9") + if err != nil { + t.Fatalf("New: %v", err) + } + return m +} + +func TestNew_ExposesBuildInfoAndRuntime(t *testing.T) { + m := newMetrics(t) + + metricstest.Want(t, m, `intelligence_api_build_info{goversion="`+runtime.Version()+`",version="v9.9.9"}`, 1) + + body := metricstest.Scrape(t, m) + for _, name := range []string{"go_goroutines", "process_start_time_seconds"} { + if !strings.Contains(body, "\n"+name+" ") { + t.Errorf("scrape is missing %s", name) + } + } +} + +// Two instances in one process is every test binary that builds more than one. +// It works only because each owns its registry. +func TestNew_InstancesDoNotCollide(t *testing.T) { + a, b := newMetrics(t), newMetrics(t) + a.RateLimitRejected("free") + + metricstest.Want(t, a, `intelligence_api_rate_limit_rejections_total{plan="free"}`, 1) + metricstest.WantAbsent(t, b, `intelligence_api_rate_limit_rejections_total{plan="free"}`) +} + +func TestRecording(t *testing.T) { + m := newMetrics(t) + + m.ObserveHTTP("/v1/domain/lookup", "POST", "pro", 200, 120*time.Millisecond) + m.ObserveLookup("domain", "dns", metrics.OutcomeOK, 40*time.Millisecond) + m.ObserveLookup("domain", "dns", metrics.OutcomeTimeout, 30*time.Second) + m.CacheResult("ip", "asn", true) + m.CacheResult("ip", "asn", false) + m.CacheResult("ip", "asn", false) + m.Coalesced("domain", "whois") + m.RateLimitRejected("free") + + for series, want := range map[string]float64{ + `intelligence_api_http_request_duration_seconds_count{method="POST",plan="pro",route="/v1/domain/lookup",status="200"}`: 1, + // 120ms lands in the 0.25s bucket and not the 0.1s one. + `intelligence_api_http_request_duration_seconds_bucket{method="POST",plan="pro",route="/v1/domain/lookup",status="200",le="0.1"}`: 0, + `intelligence_api_http_request_duration_seconds_bucket{method="POST",plan="pro",route="/v1/domain/lookup",status="200",le="0.25"}`: 1, + `intelligence_api_lookup_duration_seconds_count{lookup="dns",outcome="ok",scope="domain"}`: 1, + `intelligence_api_lookup_duration_seconds_count{lookup="dns",outcome="timeout",scope="domain"}`: 1, + // A lookup that ran to the 30s timeout must still land in a finite bucket. + `intelligence_api_lookup_duration_seconds_bucket{lookup="dns",outcome="timeout",scope="domain",le="30"}`: 1, + `intelligence_api_cache_requests_total{lookup="asn",result="hit",scope="ip"}`: 1, + `intelligence_api_cache_requests_total{lookup="asn",result="miss",scope="ip"}`: 2, + `intelligence_api_singleflight_coalesced_total{lookup="whois",scope="domain"}`: 1, + `intelligence_api_rate_limit_rejections_total{plan="free"}`: 1, + } { + metricstest.Want(t, m, series, want) + } +} + +// Callers hold a possibly-nil *Metrics and call it without a check. The test +// passes by not panicking. +func TestNilMetricsRecordsNothing(_ *testing.T) { + var m *metrics.Metrics + m.InitLookup("domain", "dns", true) + m.InitRateLimitPlan("free") + m.ObserveHTTP("/", "GET", "none", 200, time.Millisecond) + m.ObserveLookup("domain", "dns", metrics.OutcomeOK, time.Millisecond) + m.CacheResult("domain", "dns", true) + m.Coalesced("domain", "dns") + m.RateLimitRejected("free") +} + +type fakeGeoIP struct { + built time.Time + loaded bool +} + +func (f *fakeGeoIP) BuildTime() (time.Time, bool) { return f.built, f.loaded } + +func TestRegisterGeoIP(t *testing.T) { + m := newMetrics(t) + geo := &fakeGeoIP{} + if err := m.RegisterGeoIP(geo); err != nil { + t.Fatalf("RegisterGeoIP: %v", err) + } + + // Nothing loaded: the build time is absent rather than zero, which a + // dashboard would render as a database 56 years old. + metricstest.Want(t, m, "intelligence_api_geoip_database_loaded", 0) + metricstest.WantAbsent(t, m, "intelligence_api_geoip_database_build_timestamp_seconds") + + // The collector reads at scrape time, so a database that lands later + // (the background download) shows up without re-registering anything. + geo.built, geo.loaded = time.Unix(1_790_000_000, 0), true + metricstest.Want(t, m, "intelligence_api_geoip_database_loaded", 1) + metricstest.Want(t, m, "intelligence_api_geoip_database_build_timestamp_seconds", 1_790_000_000) +} + +func TestRegisterRedisPool(t *testing.T) { + m := newMetrics(t) + mr := miniredis.RunT(t) + client := redis.NewClient(&redis.Options{Addr: mr.Addr()}) + t.Cleanup(func() { _ = client.Close() }) + if err := m.RegisterRedisPool(client); err != nil { + t.Fatalf("RegisterRedisPool: %v", err) + } + + if err := client.Ping(t.Context()).Err(); err != nil { + t.Fatalf("ping: %v", err) + } + + // One command on a fresh client dials one connection and returns it. + metricstest.Want(t, m, "intelligence_api_redis_pool_connections", 1) + metricstest.Want(t, m, "intelligence_api_redis_pool_idle_connections", 1) + metricstest.Want(t, m, "intelligence_api_redis_pool_misses_total", 1) +} + +func TestRegisterTwiceIsAnError(t *testing.T) { + m := newMetrics(t) + if err := m.RegisterGeoIP(&fakeGeoIP{}); err != nil { + t.Fatalf("first RegisterGeoIP: %v", err) + } + if err := m.RegisterGeoIP(&fakeGeoIP{}); err == nil { + t.Error("second RegisterGeoIP returned nil, want an error") + } +} diff --git a/internal/metrics/metricstest/metricstest.go b/internal/metrics/metricstest/metricstest.go new file mode 100644 index 0000000..7f6a319 --- /dev/null +++ b/internal/metrics/metricstest/metricstest.go @@ -0,0 +1,64 @@ +// Package metricstest reads a metrics.Metrics the way Prometheus does, so tests +// in any package can assert on what a scrape would actually return. +package metricstest + +import ( + "net/http" + "net/http/httptest" + "strconv" + "strings" + "testing" + + "github.com/security-accelerated/intelligence-api/internal/metrics" +) + +// Scrape returns the exposition text a scrape of m produces. +func Scrape(t testing.TB, m *metrics.Metrics) string { + t.Helper() + w := httptest.NewRecorder() + m.Handler().ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/metrics", http.NoBody)) + if w.Code != http.StatusOK { + t.Fatalf("scrape status = %d, want 200", w.Code) + } + return w.Body.String() +} + +// Value returns the sample for one series and whether the scrape contains it. +// series is the name and label set exactly as exposed, labels in alphabetical +// order: `intelligence_api_rate_limit_rejections_total{plan="free"}`. +func Value(t testing.TB, m *metrics.Metrics, series string) (float64, bool) { + t.Helper() + for line := range strings.SplitSeq(Scrape(t, m), "\n") { + rest, ok := strings.CutPrefix(line, series+" ") + if !ok { + continue + } + v, err := strconv.ParseFloat(rest, 64) + if err != nil { + t.Fatalf("series %s has unparseable value %q: %v", series, rest, err) + } + return v, true + } + return 0, false +} + +// Want fails the test unless series is present with exactly the value want. +func Want(t testing.TB, m *metrics.Metrics, series string, want float64) { + t.Helper() + got, ok := Value(t, m, series) + if !ok { + t.Errorf("series %s is absent, want %v", series, want) + return + } + if got != want { + t.Errorf("series %s = %v, want %v", series, got, want) + } +} + +// WantAbsent fails the test if the scrape contains series. +func WantAbsent(t testing.TB, m *metrics.Metrics, series string) { + t.Helper() + if got, ok := Value(t, m, series); ok { + t.Errorf("series %s = %v, want it absent", series, got) + } +} diff --git a/internal/middleware/metrics.go b/internal/middleware/metrics.go new file mode 100644 index 0000000..0402020 --- /dev/null +++ b/internal/middleware/metrics.go @@ -0,0 +1,51 @@ +package middleware + +import ( + "net/http" + "time" + + "github.com/gin-gonic/gin" + "github.com/security-accelerated/intelligence-api/internal/metrics" +) + +// Metrics returns middleware that records every request's latency by route +// template, method, status and plan. +// +// It must be the first middleware in the chain, ahead of gin.Recovery. The +// recording happens after c.Next() returns, and a panic unwinds past every +// middleware between it and Recovery without running that code: placed after +// Recovery, a panicking request would never be counted, and the 500s a +// dashboard most needs to see would be the ones missing from it. +func Metrics(m *metrics.Metrics) gin.HandlerFunc { + return func(c *gin.Context) { + start := time.Now() + c.Next() + + // The route template, never the raw path: /v1/nope-1, /v1/nope-2, ... + // would otherwise each become a series of their own. + route := c.FullPath() + if route == "" { + route = metrics.RouteUnmatched + } + + plan := metrics.PlanNone + if v, ok := c.Get("api_key_plan"); ok { + if s, ok := v.(string); ok && s != "" { + plan = s + } + } + + m.ObserveHTTP(route, metricMethod(c.Request.Method), plan, c.Writer.Status(), time.Since(start)) + } +} + +// metricMethod bounds the method label. net/http accepts any token as a +// method, so an unrecognized one is recorded as "other" rather than verbatim. +func metricMethod(method string) string { + switch method { + case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut, http.MethodPatch, + http.MethodDelete, http.MethodOptions: + return method + } + return metrics.MethodOther +} diff --git a/internal/middleware/metrics_test.go b/internal/middleware/metrics_test.go new file mode 100644 index 0000000..59d8154 --- /dev/null +++ b/internal/middleware/metrics_test.go @@ -0,0 +1,111 @@ +package middleware + +import ( + "net/http" + "net/http/httptest" + "testing" + + "github.com/gin-gonic/gin" + "github.com/security-accelerated/intelligence-api/internal/metrics" + "github.com/security-accelerated/intelligence-api/internal/metrics/metricstest" +) + +func newTestMetrics(t *testing.T) *metrics.Metrics { + t.Helper() + m, err := metrics.New("test") + if err != nil { + t.Fatalf("metrics.New: %v", err) + } + return m +} + +// metricsRouter mirrors the production order: Metrics first, then Recovery. +func metricsRouter(m *metrics.Metrics) *gin.Engine { + gin.SetMode(gin.TestMode) + r := gin.New() + r.Use(Metrics(m)) + r.Use(gin.Recovery()) + r.GET("/v1/things/:id", func(c *gin.Context) { c.Status(http.StatusOK) }) + r.GET("/v1/authed", func(c *gin.Context) { + c.Set("api_key_plan", "pro") + c.Status(http.StatusOK) + }) + r.GET("/v1/boom", func(*gin.Context) { panic("boom") }) + return r +} + +func serve(r *gin.Engine, method, path string) int { + w := httptest.NewRecorder() + r.ServeHTTP(w, httptest.NewRequest(method, path, http.NoBody)) + return w.Code +} + +const httpCount = "intelligence_api_http_request_duration_seconds_count" + +func TestMetrics_LabelsByRouteTemplateMethodStatusAndPlan(t *testing.T) { + m := newTestMetrics(t) + r := metricsRouter(m) + + serve(r, http.MethodGet, "/v1/things/1") + serve(r, http.MethodGet, "/v1/things/2") + serve(r, http.MethodGet, "/v1/authed") + + // Two different paths, one route: the label is the template. + metricstest.Want(t, m, httpCount+`{method="GET",plan="none",route="/v1/things/:id",status="200"}`, 2) + metricstest.Want(t, m, httpCount+`{method="GET",plan="pro",route="/v1/authed",status="200"}`, 1) +} + +// Paths and methods are chosen by the caller. Neither may become a label +// value as sent, or a scanner walking /a, /b, /c mints a series per request. +func TestMetrics_CallerChosenValuesAreBounded(t *testing.T) { + m := newTestMetrics(t) + r := metricsRouter(m) + + serve(r, http.MethodGet, "/nope/1") + serve(r, http.MethodGet, "/nope/2") + serve(r, "PROPFIND", "/nope/3") + serve(r, "MADEUP", "/nope/4") + + metricstest.Want(t, m, httpCount+`{method="GET",plan="none",route="unmatched",status="404"}`, 2) + metricstest.Want(t, m, httpCount+`{method="other",plan="none",route="unmatched",status="404"}`, 2) +} + +// The reason Metrics sits ahead of Recovery in the chain. +func TestMetrics_PanicIsRecordedAs500(t *testing.T) { + m := newTestMetrics(t) + r := metricsRouter(m) + + if code := serve(r, http.MethodGet, "/v1/boom"); code != http.StatusInternalServerError { + t.Fatalf("status = %d, want 500", code) + } + + metricstest.Want(t, m, httpCount+`{method="GET",plan="none",route="/v1/boom",status="500"}`, 1) +} + +func TestRateLimit_CountsRejectionsByPlan(t *testing.T) { + m := newTestMetrics(t) + r, _, _ := setupRateLimit(t, RateLimitConfig{Free: 2, Pro: 10, Metrics: m}) + + const rejections = `intelligence_api_rate_limit_rejections_total{plan="free"}` + + // Allowed requests are not rejections. The series already exists at zero, + // so the first rejection is an increase rate() can see. + doRequest(r) + doRequest(r) + metricstest.Want(t, m, rejections, 0) + metricstest.Want(t, m, `intelligence_api_rate_limit_rejections_total{plan="pro"}`, 0) + + for range 3 { + if w := doRequest(r); w.Code != http.StatusTooManyRequests { + t.Fatalf("status = %d, want 429", w.Code) + } + } + metricstest.Want(t, m, rejections, 3) +} + +func TestMetrics_NilRecordsNothing(t *testing.T) { + r := metricsRouter(nil) + if code := serve(r, http.MethodGet, "/v1/things/1"); code != http.StatusOK { + t.Fatalf("status = %d, want 200", code) + } +} diff --git a/internal/middleware/rate_limit.go b/internal/middleware/rate_limit.go index 5a7787a..310d18b 100644 --- a/internal/middleware/rate_limit.go +++ b/internal/middleware/rate_limit.go @@ -10,6 +10,7 @@ import ( "github.com/gin-gonic/gin" "github.com/redis/go-redis/v9" "github.com/security-accelerated/intelligence-api/internal/logging" + "github.com/security-accelerated/intelligence-api/internal/metrics" "go.uber.org/zap" ) @@ -17,6 +18,8 @@ import ( type RateLimitConfig struct { Free int Pro int + // Metrics counts rejections per plan. Optional: nil records nothing. + Metrics *metrics.Metrics } // RateLimit returns middleware that enforces per-API-key, plan-aware rate @@ -35,6 +38,11 @@ func RateLimit(client *redis.Client, cfg RateLimitConfig) gin.HandlerFunc { // RateLimitWithLogger is like RateLimit but accepts a logger for testing. func RateLimitWithLogger(client *redis.Client, cfg RateLimitConfig, logger *zap.Logger) gin.HandlerFunc { + // The two plans this limiter has limits for start at zero rejections, so a + // plan's first rejection is a visible change rather than a new series. + cfg.Metrics.InitRateLimitPlan("free") + cfg.Metrics.InitRateLimitPlan("pro") + return func(c *gin.Context) { keyID, exists := c.Get("api_key_id") if !exists { @@ -108,6 +116,7 @@ func RateLimitWithLogger(client *redis.Client, cfg RateLimitConfig, logger *zap. zap.Int64("count", count), zap.Int("limit", limit), ) + cfg.Metrics.RateLimitRejected(planStr) c.AbortWithStatusJSON(http.StatusTooManyRequests, gin.H{"error": "rate limit exceeded"}) return } diff --git a/internal/service/domain.go b/internal/service/domain.go index 1f66745..1d82667 100644 --- a/internal/service/domain.go +++ b/internal/service/domain.go @@ -9,6 +9,7 @@ import ( "github.com/security-accelerated/intelligence-api/internal/cache" "github.com/security-accelerated/intelligence-api/internal/domain" "github.com/security-accelerated/intelligence-api/internal/logging" + "github.com/security-accelerated/intelligence-api/internal/metrics" "github.com/security-accelerated/intelligence-api/internal/notifications" "github.com/security-accelerated/intelligence-api/internal/safenet" "golang.org/x/sync/singleflight" @@ -55,6 +56,7 @@ type DomainService struct { ctLogsMaxEntries int fingerprintEngine *domain.FingerprintEngine alerter notifications.ErrorRecorder + metrics *metrics.Metrics lookupWhois func(ctx context.Context, d string) (*domain.WhoisInfo, error) lookupDNS func(ctx context.Context, d string) (*domain.DNSInfo, error) @@ -74,8 +76,9 @@ type DomainService struct { // defaultCacheTTL covers any lookup without an entry in cacheTTLs; see ttlFor. // tiProviders is the set of threat intelligence backends built from config at // startup. Non-positive numeric arguments fall back to sane defaults, so a -// zero-valued config cannot produce an unusable service. -func NewDomainService(c cache.Cache, defaultCacheTTL time.Duration, cacheTTLs map[string]time.Duration, lookupTimeout time.Duration, subdomainMaxResults, ctLogsMaxEntries int, fpEngine *domain.FingerprintEngine, alerter notifications.ErrorRecorder, tiProviders []domain.ThreatIntelProvider) *DomainService { +// zero-valued config cannot produce an unusable service. m may be nil, which +// records no metrics. +func NewDomainService(c cache.Cache, defaultCacheTTL time.Duration, cacheTTLs map[string]time.Duration, lookupTimeout time.Duration, subdomainMaxResults, ctLogsMaxEntries int, fpEngine *domain.FingerprintEngine, alerter notifications.ErrorRecorder, tiProviders []domain.ThreatIntelProvider, m *metrics.Metrics) *DomainService { if subdomainMaxResults <= 0 { subdomainMaxResults = 500 } @@ -88,7 +91,7 @@ func NewDomainService(c cache.Cache, defaultCacheTTL time.Duration, cacheTTLs ma if alerter == nil { alerter = ¬ifications.NoopRecorder{} } - return &DomainService{ + svc := &DomainService{ cache: c, defaultCacheTTL: defaultCacheTTL, cacheTTLs: cacheTTLs, @@ -97,6 +100,7 @@ func NewDomainService(c cache.Cache, defaultCacheTTL time.Duration, cacheTTLs ma ctLogsMaxEntries: ctLogsMaxEntries, fingerprintEngine: fpEngine, alerter: alerter, + metrics: m, lookupWhois: domain.LookupWhois, lookupDNS: domain.LookupDNS, lookupSSL: domain.LookupSSL, @@ -111,6 +115,22 @@ func NewDomainService(c cache.Cache, defaultCacheTTL time.Duration, cacheTTLs ma }, lookupDNSSecurity: domain.LookupDNSSecurity, } + svc.initMetrics() + return svc +} + +// initMetrics creates the metric series for every lookup this service fetches, +// so each exists at zero from startup (see metrics.InitLookup for why). +// fingerprint and risk are left out: both are computed from other lookups' +// data and never fetched, so no series of theirs would ever move. A test +// asserts this set matches what a request for every lookup actually records. +func (s *DomainService) initMetrics() { + for _, name := range domain.AllLookupNames { + if name == domain.LookupFingerprintName || name == domain.LookupRiskName { + continue + } + s.metrics.InitLookup(scopeDomain, name, true) + } } // SetLookups replaces the base lookup functions. Test-only: production wiring @@ -291,7 +311,7 @@ func (s *DomainService) Lookup(ctx context.Context, domainName string, opts doma errs: &resp.Errors, logger: logging.Logger(), target: domainName, - targetField: "domain", + targetField: scopeDomain, keyFor: func(name string) string { return cache.LookupKey(domainName, name) }, } @@ -300,6 +320,7 @@ func (s *DomainService) Lookup(ctx context.Context, domainName string, opts doma sf: &s.sf, lookupTimeout: s.lookupTimeout, ttlFor: s.ttlFor, + metrics: s.metrics, }, st, s.buildRunners(st, data, opts)) s.assemble(resp, data, opts, hits) diff --git a/internal/service/domain_test.go b/internal/service/domain_test.go index 668b55c..7198e87 100644 --- a/internal/service/domain_test.go +++ b/internal/service/domain_test.go @@ -1370,7 +1370,7 @@ func TestLookup_Risk_PartiallyCachedInputsCountAsMiss(t *testing.T) { // --- Setters and constructor --- func TestNewDomainService_Defaults(t *testing.T) { - svc := NewDomainService(newMockCache(), time.Minute, nil, 0, 0, 0, nil, nil, nil) + svc := NewDomainService(newMockCache(), time.Minute, nil, 0, 0, 0, nil, nil, nil, nil) if svc.subdomainMaxResults != 500 { t.Errorf("subdomainMaxResults = %d, want 500", svc.subdomainMaxResults) } diff --git a/internal/service/ip.go b/internal/service/ip.go index 2bbba74..ce50935 100644 --- a/internal/service/ip.go +++ b/internal/service/ip.go @@ -8,6 +8,7 @@ import ( "github.com/security-accelerated/intelligence-api/internal/cache" "github.com/security-accelerated/intelligence-api/internal/domain" "github.com/security-accelerated/intelligence-api/internal/logging" + "github.com/security-accelerated/intelligence-api/internal/metrics" "github.com/security-accelerated/intelligence-api/internal/notifications" "golang.org/x/sync/singleflight" ) @@ -19,6 +20,7 @@ type IPService struct { sf singleflight.Group lookupTimeout time.Duration alerter notifications.ErrorRecorder + metrics *metrics.Metrics lookupGeo func(ip net.IP) (*domain.GeoIPInfo, error) lookupASN func(ctx context.Context, ip net.IP) (*domain.IPASNInfo, error) @@ -31,7 +33,8 @@ type IPService struct { // // defaultCacheTTL covers any lookup without an entry in cacheTTLs. geoReader // may be an empty reader — geo then reports ErrDBNotLoaded until the database -// download lands. A nil geoReader leaves geo unwired; asn never needs it. +// download lands. A nil geoReader leaves geo unwired; asn never needs it. m may +// be nil, which records no metrics. func NewIPService( c cache.Cache, defaultCacheTTL time.Duration, @@ -40,6 +43,7 @@ func NewIPService( geoReader *domain.GeoIPReader, alerter notifications.ErrorRecorder, tiProviders []domain.IPThreatIntelProvider, + m *metrics.Metrics, ) *IPService { if lookupTimeout <= 0 { lookupTimeout = 30 * time.Second @@ -54,6 +58,7 @@ func NewIPService( cacheTTLs: cacheTTLs, lookupTimeout: lookupTimeout, alerter: alerter, + metrics: m, lookupASN: domain.LookupIPASN, lookupReverseDNS: domain.LookupReverseDNS, lookupAbuseContact: domain.LookupAbuseContact, @@ -65,10 +70,20 @@ func NewIPService( if geoReader != nil { svc.lookupGeo = geoReader.LookupGeo } + svc.initMetrics() return svc } +// initMetrics creates the metric series for every IP lookup, so each exists at +// zero from startup (see metrics.InitLookup for why). geo gets no cache series: +// it is the one lookup that never reads the cache. +func (s *IPService) initMetrics() { + for _, name := range domain.AllIPLookupNames { + s.metrics.InitLookup(scopeIP, name, name != domain.LookupGeoName) + } +} + // SetLookups replaces all lookup functions. Test-only: production wiring // happens in NewIPService. func (s *IPService) SetLookups( @@ -167,7 +182,7 @@ func (s *IPService) Lookup(ctx context.Context, ip string, opts domain.IPLookupO errs: &resp.Errors, logger: logging.Logger(), target: ip, - targetField: "ip", + targetField: scopeIP, keyFor: func(name string) string { return cache.IPLookupKey(ip, name) }, } @@ -176,6 +191,7 @@ func (s *IPService) Lookup(ctx context.Context, ip string, opts domain.IPLookupO sf: &s.sf, lookupTimeout: s.lookupTimeout, ttlFor: s.ttlFor, + metrics: s.metrics, }, st, s.buildRunners(st, data, opts, net.ParseIP(ip))) s.assemble(resp, data, opts, hits) diff --git a/internal/service/ip_test.go b/internal/service/ip_test.go index e5427f5..533f9cc 100644 --- a/internal/service/ip_test.go +++ b/internal/service/ip_test.go @@ -490,7 +490,7 @@ func TestIPLookup_GeoDBNotLoadedDoesNotAlert(t *testing.T) { // --- Constructor --- func TestNewIPService_Defaults(t *testing.T) { - svc := NewIPService(newMockCache(), time.Minute, nil, 0, nil, nil, nil) + svc := NewIPService(newMockCache(), time.Minute, nil, 0, nil, nil, nil, nil) if svc.lookupTimeout != 30*time.Second { t.Errorf("lookupTimeout = %v, want 30s", svc.lookupTimeout) } diff --git a/internal/service/metrics_test.go b/internal/service/metrics_test.go new file mode 100644 index 0000000..dca0319 --- /dev/null +++ b/internal/service/metrics_test.go @@ -0,0 +1,270 @@ +package service + +import ( + "context" + "errors" + "fmt" + "net" + "strings" + "sync" + "testing" + "testing/synctest" + + "github.com/security-accelerated/intelligence-api/internal/domain" + "github.com/security-accelerated/intelligence-api/internal/metrics" + "github.com/security-accelerated/intelligence-api/internal/metrics/metricstest" +) + +func newTestMetrics(t *testing.T) *metrics.Metrics { + t.Helper() + m, err := metrics.New("test") + if err != nil { + t.Fatalf("metrics.New: %v", err) + } + return m +} + +// lookupSeries returns every lookup, cache and coalescing series in a scrape, +// keyed by name and labels, with its value. +func lookupSeries(t *testing.T, m *metrics.Metrics) map[string]string { + t.Helper() + series := make(map[string]string) + for line := range strings.SplitSeq(metricstest.Scrape(t, m), "\n") { + for _, prefix := range []string{ + "intelligence_api_lookup_duration_seconds_count{", + "intelligence_api_cache_requests_total{", + "intelligence_api_singleflight_coalesced_total{", + } { + if strings.HasPrefix(line, prefix) { + name, value, _ := strings.Cut(line, "} ") + series[name+"}"] = value + } + } + } + return series +} + +// assertInitMatchesUse runs after a service has fetched every one of its +// lookups once and then read each back from the cache. It fails in both +// directions: a series the requests created that initMetrics had not (a lookup +// added without being initialized, so its first event after a start would be +// invisible to rate()), and an initialized series that the requests never +// moved (a lookup initialized although nothing fetches it). +func assertInitMatchesUse(t *testing.T, before, after map[string]string) { + t.Helper() + if len(before) == 0 { + t.Fatal("initMetrics created no series") + } + for name := range after { + if _, ok := before[name]; !ok { + t.Errorf("series %s appeared on first use; initMetrics should have created it", name) + } + } + for name := range before { + moved := strings.Contains(name, `outcome="ok"`) || strings.Contains(name, `result="`) + if moved && after[name] != "1" { + t.Errorf("series %s = %s after one fetch and one cache hit, want 1; is it initialized for a lookup nothing fetches?", name, after[name]) + } + } +} + +func TestDomainService_InitMetricsMatchesWhatLookupsRecord(t *testing.T) { + m := newTestMetrics(t) + svc := newTestService(newMockCache()) + svc.metrics = m + svc.initMetrics() + before := lookupSeries(t, m) + + everything := opts(domain.AllLookupNames...) + svc.Lookup(context.Background(), "example.com", everything) + svc.Lookup(context.Background(), "example.com", everything) + + assertInitMatchesUse(t, before, lookupSeries(t, m)) +} + +func TestIPService_InitMetricsMatchesWhatLookupsRecord(t *testing.T) { + m := newTestMetrics(t) + svc := newTestIPService(newMockCache()) + svc.metrics = m + svc.initMetrics() + before := lookupSeries(t, m) + + everything := domain.IPLookupOptions{Lookups: domain.AllIPLookupNames} + svc.Lookup(context.Background(), "8.8.8.8", everything) + // geo is uncached, so a second request would fetch it again while the + // others are served from the cache. Unwiring it keeps every lookup at + // exactly one fetch, which is what the assertion counts. + svc.lookupGeo = nil + svc.Lookup(context.Background(), "8.8.8.8", everything) + + assertInitMatchesUse(t, before, lookupSeries(t, m)) +} + +func TestLookupMetrics_CacheHitsAndMisses(t *testing.T) { + m := newTestMetrics(t) + svc := newTestService(newMockCache()) + svc.metrics = m + + // First request fetches both; the second finds both in the cache. + svc.Lookup(context.Background(), "example.com", opts(domain.LookupDNSName, domain.LookupSSLName)) + svc.Lookup(context.Background(), "example.com", opts(domain.LookupDNSName, domain.LookupSSLName)) + + for _, lookup := range []string{"dns", "ssl"} { + metricstest.Want(t, m, `intelligence_api_cache_requests_total{lookup="`+lookup+`",result="miss",scope="domain"}`, 1) + metricstest.Want(t, m, `intelligence_api_cache_requests_total{lookup="`+lookup+`",result="hit",scope="domain"}`, 1) + // Only the miss ran a fetch. A cache hit must not look like a fast lookup. + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="`+lookup+`",outcome="ok",scope="domain"}`, 1) + } +} + +// fingerprint is computed from headers and has no fetch or cache entry of its +// own, so it shows up under the headers lookup and nowhere else. +func TestLookupMetrics_FingerprintIsRecordedAsHeaders(t *testing.T) { + m := newTestMetrics(t) + svc := newTestService(newMockCache()) + svc.metrics = m + + svc.Lookup(context.Background(), "example.com", opts(domain.LookupFingerprintName)) + + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="headers",outcome="ok",scope="domain"}`, 1) + metricstest.WantAbsent(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="fingerprint",outcome="ok",scope="domain"}`) +} + +func TestLookupMetrics_UncachedLookupIsNotACacheMiss(t *testing.T) { + m := newTestMetrics(t) + svc := newTestIPService(newMockCache()) + svc.metrics = m + + svc.Lookup(context.Background(), "8.8.8.8", domain.IPLookupOptions{Lookups: []string{domain.LookupGeoName, domain.LookupASNName}}) + + // geo never consults the cache, so it is fetched and timed but counts as + // neither a hit nor a miss. asn is cached like any other lookup. + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="geo",outcome="ok",scope="ip"}`, 1) + metricstest.WantAbsent(t, m, `intelligence_api_cache_requests_total{lookup="geo",result="miss",scope="ip"}`) + metricstest.Want(t, m, `intelligence_api_cache_requests_total{lookup="asn",result="miss",scope="ip"}`, 1) +} + +func TestLookupMetrics_Outcomes(t *testing.T) { + m := newTestMetrics(t) + svc := newTestService(newMockCache()) + svc.metrics = m + + svc.lookupDNS = failDNS + // What a slow upstream actually returns: a network timeout wrapped a few + // layers down, with the lookup's own deadline nowhere near. + svc.lookupSSL = func(context.Context, string) (*domain.SSLInfo, error) { + return nil, fmt.Errorf("dial: %w", &net.DNSError{Err: "i/o timeout", IsTimeout: true}) + } + // A fan-out lookup's partial result: data, plus one provider's error. + svc.lookupReputation = func(context.Context, string) (*domain.ReputationInfo, []domain.LookupError) { + return &domain.ReputationInfo{}, []domain.LookupError{{Lookup: "reputation", Message: "provider unavailable"}} + } + + svc.Lookup(context.Background(), "example.com", + opts(domain.LookupWhoisName, domain.LookupDNSName, domain.LookupSSLName, domain.LookupReputationName)) + + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="whois",outcome="ok",scope="domain"}`, 1) + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="dns",outcome="error",scope="domain"}`, 1) + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="ssl",outcome="timeout",scope="domain"}`, 1) + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="reputation",outcome="error",scope="domain"}`, 1) +} + +func TestLookupOutcome(t *testing.T) { + expired, cancel := context.WithTimeout(context.Background(), 0) + defer cancel() + <-expired.Done() + + canceled, cancelNow := context.WithCancel(context.Background()) + cancelNow() + + failed := []domain.LookupError{{Lookup: "dns", Message: "failed"}} + + tests := []struct { + name string + ctx context.Context + res fetchResult + want string + }{ + {"no errors", context.Background(), fetchResult{payload: []byte(`{}`)}, metrics.OutcomeOK}, + {"no errors and no data", context.Background(), fetchResult{}, metrics.OutcomeOK}, + {"error", context.Background(), fetchResult{errs: failed}, metrics.OutcomeError}, + {"runner saw a timeout", context.Background(), fetchResult{errs: failed, timedOut: true}, metrics.OutcomeTimeout}, + // The fan-out lookups cannot report what kind of error a provider + // returned, so an expired deadline is what marks theirs as timeouts. + {"deadline passed", expired, fetchResult{errs: failed}, metrics.OutcomeTimeout}, + {"partial result after the deadline", expired, fetchResult{payload: []byte(`{}`), errs: failed}, metrics.OutcomeTimeout}, + // A lookup that finished cleanly just as the deadline passed succeeded. + {"success at the deadline", expired, fetchResult{payload: []byte(`{}`)}, metrics.OutcomeOK}, + {"canceled is not a timeout", canceled, fetchResult{errs: failed}, metrics.OutcomeError}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := lookupOutcome(tt.ctx, tt.res); got != tt.want { + t.Errorf("lookupOutcome = %q, want %q", got, tt.want) + } + }) + } +} + +func TestIsTimeout(t *testing.T) { + tests := []struct { + name string + err error + want bool + }{ + {"deadline exceeded", context.DeadlineExceeded, true}, + {"wrapped deadline exceeded", fmt.Errorf("fetch: %w", context.DeadlineExceeded), true}, + {"dns timeout", &net.DNSError{Err: "i/o timeout", IsTimeout: true}, true}, + {"wrapped net timeout", fmt.Errorf("get: %w", &net.OpError{Op: "read", Err: context.DeadlineExceeded}), true}, + {"dns not found", &net.DNSError{Err: "no such host", IsNotFound: true}, false}, + {"canceled", context.Canceled, false}, + {"plain error", errors.New("boom"), false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := isTimeout(tt.err); got != tt.want { + t.Errorf("isTimeout(%v) = %v, want %v", tt.err, got, tt.want) + } + }) + } +} + +// Eight callers ask for the same lookup at once: one fetches, seven wait on it. +// The fetch is observed once and the seven are counted as coalesced. +// +// Run in a synctest bubble so "every caller is waiting" is something the test +// can ask for instead of something it sleeps and hopes for. synctest.Wait +// returns only once every goroutine is durably blocked: the one that started +// the flight on the release channel, the other seven inside singleflight. +// Releasing the fetch any earlier lets a straggler arrive after the flight has +// finished and start a second one, which fails the counts below with correct +// instrumentation. +func TestLookupMetrics_CoalescedCallers(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + m := newTestMetrics(t) + svc := newTestService(newMockCache()) + svc.metrics = m + + release := make(chan struct{}) + svc.lookupDNS = func(context.Context, string) (*domain.DNSInfo, error) { + <-release + return okDNS(context.TODO(), "") + } + + const callers = 8 + var wg sync.WaitGroup + for range callers { + wg.Go(func() { + svc.Lookup(context.Background(), "example.com", opts(domain.LookupDNSName)) + }) + } + synctest.Wait() + close(release) + wg.Wait() + + metricstest.Want(t, m, `intelligence_api_lookup_duration_seconds_count{lookup="dns",outcome="ok",scope="domain"}`, 1) + metricstest.Want(t, m, `intelligence_api_singleflight_coalesced_total{lookup="dns",scope="domain"}`, callers-1) + // Every caller read the cache and missed; coalescing happens after that. + metricstest.Want(t, m, `intelligence_api_cache_requests_total{lookup="dns",result="miss",scope="domain"}`, callers) + }) +} diff --git a/internal/service/runner.go b/internal/service/runner.go index 727a67a..2d460df 100644 --- a/internal/service/runner.go +++ b/internal/service/runner.go @@ -3,11 +3,14 @@ package service import ( "context" "encoding/json" + "errors" + "net" "sync" "time" "github.com/security-accelerated/intelligence-api/internal/cache" "github.com/security-accelerated/intelligence-api/internal/domain" + "github.com/security-accelerated/intelligence-api/internal/metrics" "github.com/security-accelerated/intelligence-api/internal/notifications" "go.uber.org/zap" "golang.org/x/sync/singleflight" @@ -19,6 +22,13 @@ import ( // exist, how they are fetched and how the response is assembled stays in the // owning service. +// The two kinds of target. Each is a runState's targetField, which names the +// target in log output and is the "scope" label on every lookup metric. +const ( + scopeDomain = "domain" + scopeIP = "ip" +) + // lookupRunner is one cacheable lookup for one target: how to decode a payload // into the caller's own storage, and how to fetch that payload live. // @@ -46,6 +56,38 @@ type lookupRunner struct { type fetchResult struct { payload []byte errs []domain.LookupError + // timedOut marks a failure that was a timeout, for the outcome metric. The + // errs above carry only sanitized text, so the runner classifies while it + // still holds the error. + timedOut bool +} + +// isTimeout reports whether a lookup failed by running out of time: the +// lookup's own deadline, or a network timeout further down (a DNS query, an +// HTTP client timeout), which is where most of them come from. +func isTimeout(err error) bool { + if errors.Is(err, context.DeadlineExceeded) { + return true + } + var netErr net.Error + return errors.As(err, &netErr) && netErr.Timeout() +} + +// lookupOutcome classifies a finished fetch for the lookup metric. Any +// reported error makes it a failure, including a fan-out lookup's partial +// result, which is the same line the cache draws when it refuses to store one. +// A failure counts as a timeout when the runner saw a timeout error or the +// fetch's own deadline has passed; the fan-out lookups report sanitized +// per-provider errors only, so the deadline is all there is to go on for them. +func lookupOutcome(ctx context.Context, res fetchResult) string { + switch { + case len(res.errs) == 0: + return metrics.OutcomeOK + case res.timedOut || errors.Is(ctx.Err(), context.DeadlineExceeded): + return metrics.OutcomeTimeout + default: + return metrics.OutcomeError + } } // runState is the mutable per-request state shared by concurrently running @@ -122,10 +164,13 @@ func simpleRunner[T any](alerter notifications.ErrorRecorder, st *runState, name if err != nil { st.logger.Warn("lookup failed", zap.String("lookup", name), st.logTarget(), zap.Error(err)) recordIfUnexpected(alerter, name, err) - return fetchResult{errs: []domain.LookupError{{ - Lookup: name, - Message: sanitizeErr(err), - }}} + return fetchResult{ + errs: []domain.LookupError{{ + Lookup: name, + Message: sanitizeErr(err), + }}, + timedOut: isTimeout(err), + } } return fetchResult{payload: marshalForCache(st, name, result)} }, @@ -156,6 +201,8 @@ type runnerDeps struct { sf *singleflight.Group lookupTimeout time.Duration ttlFor func(lookupName string) time.Duration + // metrics may be nil, which records nothing. + metrics *metrics.Metrics } // runLookups resolves every runner: one MGET for the cacheable ones, then a @@ -190,7 +237,13 @@ func runLookups(ctx context.Context, d runnerDeps, st *runState, runners []looku // target — only one runs and every caller gets its payload. Keying // per lookup rather than per response means two requests asking for // overlapping-but-different lookup sets still share the overlap. + // + // singleflight runs the function on the goroutine of the caller + // that started the flight, so ran is only ever written here and + // tells this caller whether it fetched or waited on someone else. + ran := false v, _, _ := d.sf.Do(key, func() (any, error) { + ran = true // Shared work is detached from the triggering caller's context // and bounded by the service's own timeout instead. One client // disconnecting — or holding a tight deadline — must not cancel @@ -200,7 +253,11 @@ func runLookups(ctx context.Context, d runnerDeps, st *runState, runners []looku fetchCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), d.lookupTimeout) defer cancel() + // Timed around the fetch alone, so the cache write below does + // not count toward the lookup's duration. + start := time.Now() res := r.fetch(fetchCtx) + d.metrics.ObserveLookup(st.targetField, r.name, lookupOutcome(fetchCtx, res), time.Since(start)) // Only the goroutine that actually ran the fetch writes it // back; coalesced callers would rewrite the identical value. // @@ -220,6 +277,9 @@ func runLookups(ctx context.Context, d runnerDeps, st *runState, runners []looku return res, nil }) res := v.(fetchResult) + if !ran { + d.metrics.Coalesced(st.targetField, r.name) + } st.mu.Lock() defer st.mu.Unlock() @@ -265,6 +325,10 @@ func readCachedLookups(ctx context.Context, d runnerDeps, st *runState, runners cached = nil } + // Hits and misses are counted for the cacheable runners only. An uncached + // lookup never asked the cache anything, and counting it as a miss would + // drag the hit ratio down for a reason nobody can act on. A failed MGET and + // an unusable entry both count as misses, because both end in a live fetch. for i, r := range cacheable { if i < len(cached) && cached[i] != nil { if err := r.decode(cached[i]); err != nil { @@ -273,9 +337,11 @@ func readCachedLookups(ctx context.Context, d runnerDeps, st *runState, runners zap.String("lookup", r.name), st.logTarget(), zap.Error(err)) } else { hits[r.name] = true + d.metrics.CacheResult(st.targetField, r.name, true) continue } } + d.metrics.CacheResult(st.targetField, r.name, false) misses = append(misses, r) } return misses diff --git a/scripts/smoke-test.sh b/scripts/smoke-test.sh index 5a31ec7..91cbdcf 100755 --- a/scripts/smoke-test.sh +++ b/scripts/smoke-test.sh @@ -599,6 +599,66 @@ echo "=== GeoIP Admin ===" post 403 "refresh (pro, forbidden)" /v1/admin/geoip/refresh "$PRO" '{}' post 401 "refresh (no key)" /v1/admin/geoip/refresh "" '{}' +echo "=== Metrics ===" +# The Prometheus endpoint is unauthenticated and protected only by living on a +# listener of its own. The check that holds against every target, production +# included, is that the public port does not serve it. +check 404 "GET /metrics is not served on the public port" "$HOST/metrics" + +# The listener itself can only be checked where its port is reachable. The +# local stack publishes it on loopback; production publishes it nowhere, so +# against production this is skipped unless SMOKE_METRICS_URL points at a +# tunnel to it. A skip here is the normal case, so it is not a warning. +# +# Last in the script on purpose: the series asserted below exist because of the +# requests made above. +METRICS_URL="${SMOKE_METRICS_URL:-}" +if [ -z "$METRICS_URL" ] && [[ "$HOST" == *"localhost"* || "$HOST" == *"127.0.0.1"* ]]; then + METRICS_URL="http://localhost:9091/metrics" +fi + +# check_metric — asserts the captured scrape has a line matching a regex. +# $1 description +# $2 extended regex matched against each line of $metrics_body +check_metric() { + local desc="$1" pattern="$2" + if grep -Eq "$pattern" <<< "$metrics_body"; then + printf " \033[32m✓\033[0m %s\n" "$desc" + pass=$((pass + 1)) + else + printf " \033[31m✗\033[0m %s (no series matching %s)\n" "$desc" "$pattern" + fail=$((fail + 1)) + errors+=("$desc: no series matching $pattern") + fi +} + +if [ -z "$METRICS_URL" ]; then + printf " - metrics listener not checked (not reachable from here; set SMOKE_METRICS_URL to check it)\n" +else + check 200 "GET /metrics on the metrics listener" "$METRICS_URL" + metrics_body=$(curl -s -m 10 "$METRICS_URL") || true + check_metric "build info carries the version" '^intelligence_api_build_info\{.*version="[^"]+"' + check_metric "requests are recorded by route template and plan" \ + '^intelligence_api_http_request_duration_seconds_count\{.*plan="admin".*route="/v1/ip/lookup"' + check_metric "unknown paths are recorded as one unmatched route" \ + '^intelligence_api_http_request_duration_seconds_count\{.*route="unmatched"' + # These three series exist at zero from startup, so matching the name alone + # would pass on a server that had recorded nothing. Each pattern requires a + # value that does not start with 0, which for a count means at least one. + # + # geo is never cached, so the runs above always fetched it at least once. + check_metric "live lookups are timed" \ + '^intelligence_api_lookup_duration_seconds_count\{lookup="geo",outcome="ok",scope="ip"\} [1-9]' + check_metric "cache reads are counted" '^intelligence_api_cache_requests_total\{.*\} [1-9]' + if [ -n "$RATELIMIT_KEY" ]; then + check_metric "rate-limit rejections are counted" \ + '^intelligence_api_rate_limit_rejections_total\{plan="free"\} [1-9]' + fi + check_metric "postgres pool is exported" '^go_sql_open_connections\{db_name="postgres"\}' + check_metric "redis pool is exported" '^intelligence_api_redis_pool_connections ' + check_metric "geoip database state is exported" '^intelligence_api_geoip_database_loaded 1$' +fi + # Summary echo "" echo "================================"