Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 30 additions & 10 deletions .claude/rules/http-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,43 @@ paths:

## Middleware chain — order matters

1. `gin.Recovery()` — panics → 500 (handler goroutines only; lookups recover themselves)
2. `SecurityHeaders()` — X-Content-Type-Options, X-Frame-Options, `Cache-Control:
1. `Metrics()` — records latency by route template, method, status and plan after
`c.Next()` returns. **First, ahead of Recovery**: a panic unwinds past every middleware
between it and Recovery, so anywhere later a panicking request would not be counted
2. `gin.Recovery()` — panics → 500 (handler goroutines only; lookups recover themselves)
3. `SecurityHeaders()` — X-Content-Type-Options, X-Frame-Options, `Cache-Control:
no-store`, HSTS, Referrer-Policy, and a strict `default-src 'none'` CSP for the JSON
API. HTML pages override the CSP with `htmlPageCSP` in the handler package.
3. `BodyLimit(1MB)`
4. `ClientRequestID()` — validates/echoes `X-Client-Request-ID`, sets a context key
5. `RequestLogger()` — structured logging; runs after `c.Next()` returns
6. *(protected)* `APIKeyAuth()` — sets `customer_id`, `api_key_id`, `api_key_plan`
7. *(protected)* `RateLimit()` — per-key sliding window in Redis, **fail-open**, admin
4. `BodyLimit(1MB)`
5. `ClientRequestID()` — validates/echoes `X-Client-Request-ID`, sets a context key
6. `RequestLogger()` — structured logging; runs after `c.Next()` returns
7. *(protected)* `APIKeyAuth()` — sets `customer_id`, `api_key_id`, `api_key_plan`
8. *(protected)* `RateLimit()` — per-key sliding window in Redis, **fail-open**, admin
bypass
8. *(protected + admin)* `UsageLog()` — records to `usage.Recorder` after the response
9. *(admin)* `RequireAdmin()` — `plan == "admin"`
9. *(protected + admin)* `UsageLog()` — records to `usage.Recorder` after the response
10. *(admin)* `RequireAdmin()` — `plan == "admin"`

Adding middleware means choosing a position in this list, and the position is part of the
change. Auth must precede rate limiting (the limit is per key); usage logging must follow
auth (it needs the key) and must be after `c.Next()` so it sees the outcome.

## Metrics — `internal/metrics`

One `*metrics.Metrics`, built in `app.go` and passed to the router, the rate limiter
(`RateLimitConfig.Metrics`) and both services. It is a concrete type, not an interface:
there is one implementation, and every recording method is a no-op on a nil pointer, so
tests pass nil or build their own (each owns its registry; `metricstest` reads one the
way a scrape does). Recording is always on. `metrics.enabled` only decides whether
`httpapi.StartMetricsServer` serves it, on **its own listener** (`metrics.listen_addr`).
`/metrics` is never a route on the API router: the endpoint is unauthenticated and the
separate port is its access control (`docs/decisions.md`).

**Label values must be bounded.** Anything a caller chooses (path, method) goes through a
fixed fallback (`RouteUnmatched`, `MethodOther`) before it becomes a label. A new label
whose values come from a request is a way to exhaust the scraper. Metric names and labels
are also what `deploy/grafana/intelligence-api.json` queries: renaming one means editing
the dashboard and `docs/operations.md` in the same change.

## Handler construction

`handler.New` takes a single `Deps` struct. **Every field is optional**; a nil service
Expand Down Expand Up @@ -71,7 +91,7 @@ domain.AllLookupNames)` and populated progressively in `app.go`. It feeds both t
structured startup log and `GET /v1/info` — **adding a field to a struct surfaces it in
both places**. Live data (runtime stats, DB pool, Redis ping, auth cache size) is computed
at request time in the info handler and merged in. Response groups: `system`, `database`,
`cache`, `lookups`, `services`, `runtime`, `postgres_pool`. `./scripts/info.sh`
`cache`, `lookups`, `services` (including `services.metrics`), `runtime`, `postgres_pool`. `./scripts/info.sh`
pretty-prints it (needs `SMOKE_ADMIN_KEY` from `.env`).

## Health
Expand Down
19 changes: 18 additions & 1 deletion .claude/rules/service-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,29 @@ coalescing anywhere; `service/whole_response.go` is gone and must not come back.
reads each key individually so `CACHE_TTL_WHOIS` and friends work through Viper's
`AutomaticEnv`. Pulling the subtree as a map silently drops env overrides.
- Expiry is absolute. There is no stale serving and no negative cache.
- `geo` and `asn` are `uncached`. `fingerprint` has no key of its own — it is computed
- `geo` is `uncached`. `fingerprint` has no key of its own — it is computed
from `headers` at assembly, and its `cache_hit` mirrors the headers entry.

## Metrics

`runLookups` records everything, so a lookup wired through a runner is instrumented with
no code of its own: fetch duration and outcome (`ok` / `error` / `timeout`) around the
fetch inside singleflight, a hit or miss per cacheable runner in the MGET pre-flight, and
a coalesced call for every caller that waited on someone else's fetch. Cache hits and
coalesced callers are deliberately absent from the duration histogram, which times
upstream work. The label is the **runner** name, so `fingerprint` is recorded as `headers`
and `risk` not at all.

Each service's `initMetrics()` creates its lookups' series at zero at construction,
because `rate()` cannot see a series' first value. It derives the set from
`domain.AllLookupNames` / `AllIPLookupNames`, minus the lookups nothing fetches; a test
per service fails if that set and what a request actually records ever differ.

## Adding a lookup

Runner in `buildRunners()`, branch in `assemble()`, function field + setter on the service.
A lookup with no runner of its own (computed from another's data, like `fingerprint`) must
also be excluded in `initMetrics()`; the test there says so if it is not.
The full multi-file checklist is the `adding-features` skill; use it rather than
reconstructing the list.

Expand Down
12 changes: 10 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,13 @@ internal/database/ shared PostgreSQL connection
internal/billing/ Stripe webhooks, key provisioning, noop fallback
internal/usage/ buffered usage logging, admin query
internal/notifications/ Pushover alerting, throttle and spike detection
internal/metrics/ Prometheus collectors and recording methods; metricstest reads them in tests
internal/safenet/ SSRF prevention at dial time
internal/logging/ Zap (ECS in prod, pretty in dev)
internal/config/ Viper, YAML + env overrides, StatusSnapshot
migrations/ PostgreSQL schema
deploy/ Grafana dashboard (deploy/grafana/intelligence-api.json), plus the
Prometheus and Grafana config the local Compose stack mounts
scripts/ docker-reset.sh, smoke-test.sh, status.sh, info.sh — dev only
changelog.yaml symlink to internal/handler/changelog.yaml; edit from the root
```
Expand Down Expand Up @@ -110,11 +113,16 @@ open the file.
`sanitizeErr()`.
- Every outbound HTTP client goes through `safenet` — a new one that doesn't is a
vulnerability, not a style issue.
- `/metrics` has no authentication; its own listener (`:9091`) is the access control.
Never route it on the API port, add it to a Fly service, or publish it from Compose
beyond loopback. Keep it out of the public docs, like `/v1/hello`.

## Infrastructure

Docker Compose locally (API + Redis + Postgres). Production on Fly.io (iad, auto-stop) with
Neon Postgres and Upstash Redis. One app answers for two hostnames:
Docker Compose locally (API + Redis + Postgres, plus Prometheus and Grafana for checking the
dashboard). Production on Fly.io (iad, auto-stop) with Neon Postgres and Upstash Redis;
Fly's managed Prometheus scrapes the metrics listener per `[metrics]` in `fly.toml`, and
the dashboard is imported into fly-metrics.net. One app answers for two hostnames:
`securityaccelerated.com` (`/` is the company home page) and `api.securityaccelerated.com`
(`/` redirects to `/docs`), told apart by `site.home_host`. A Fly volume `maxmind_data` (name predates DB-IP) at
`/data/geoip` persists the GeoIP database; `docker-entrypoint.sh` chowns it as root then
Expand Down
3 changes: 3 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,8 @@ RUN chmod +x /usr/local/bin/docker-entrypoint.sh
# root-owned and there is no uid/gid option on mounts, so this is the only way a
# non-root process can write to its own volume.
EXPOSE 8080
# The Prometheus listener, when metrics.enabled is set. Documentation only, as
# EXPOSE always is: never publish this port, the endpoint has no authentication.
EXPOSE 9091
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["intelligence-api"]
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ client ──> Gin router ──> middleware chain ──> handler ──> servi
| `internal/safenet` | SSRF prevention at dial time: every outbound HTTP client in the server goes through it |
| `internal/auth`, `internal/billing`, `internal/usage` | Hashed API keys with rotation, Stripe webhooks and key provisioning, buffered usage logging |
| `internal/notifications` | Pushover alerting with throttle and spike detection |
| `internal/metrics` | Prometheus instrumentation, served on a listener separate from the API's; the Grafana dashboard for it is in `deploy/grafana` |
| `internal/cache`, `internal/database`, `internal/config`, `internal/logging` | Redis, PostgreSQL, Viper config with env overrides, Zap in ECS format |
| `cmd/api`, `cmd/sactl` | The server, and the operator CLI that reuses the server's own packages |

Expand Down
Loading
Loading