Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/rules/auth-billing.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ docs and success pages. Config: `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`,
the aggregation SQL behind it. `UsageLogger` buffers on a channel and batch-inserts to
`request_log` every 5s or 50 entries; a full channel drops with a warning. `NoopRecorder`
discards entries when tracking isn't wanted (tests). `request_log` rows carry per-request
metadata including `domain_count`, `ip_count`, and a per-feature flag per lookup. The `UsageLog()` middleware runs after `c.Next()` and reads the
metadata including `domain_count`, `ip_count`, `host_count`, and a per-feature flag per lookup. The `UsageLog()` middleware runs after `c.Next()` and reads the
`lookups_requested`, `cache_hits`, `cache_misses` context keys the handlers set.

Two rules a drift test enforces:
Expand Down
2 changes: 1 addition & 1 deletion .claude/rules/http-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ auth (it needs the key) and must be after `c.Next()` so it sees the outcome.
## Metrics — `internal/metrics`

One `*metrics.Metrics`, built in `app.go` and passed to the router, the rate limiter
(`RateLimitConfig.Metrics`) and both services. It is a concrete type, not an interface:
(`RateLimitConfig.Metrics`) and the three lookup services. It is a concrete type, not an interface:
there is one implementation, and every recording method is a no-op on a nil pointer, so
tests pass nil or build their own (each owns its registry; `metricstest` reads one the
way a scrape does). Recording is always on. `metrics.enabled` only decides whether
Expand Down
34 changes: 34 additions & 0 deletions .claude/rules/lookups.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ paths:
- "internal/safenet/**"
- "internal/handler/domain.go"
- "internal/handler/ip.go"
- "internal/handler/encryption.go"
- "internal/handler/fingerprint.go"
- "internal/handler/geoip_admin.go"
---
Expand Down Expand Up @@ -39,6 +40,39 @@ non-globally-routable addresses, and gates `threat_intel` to pro/admin. `ipLooku
is a separate table — `whois` is valid for a domain and a 400 for an IP. Keys live under
`ip:`, so `threat_intel` on the two endpoints cannot collide.

## Encryption lookups use it too, and the probe has rules of its own

`/v1/encryption/lookup` takes `hosts` and the same `lookups` array. `encryptionLookupMinPlan`
in `internal/handler/encryption.go` is a third table (all free: the lookups spend no
upstream quota). Hosts go through `normalizeDomain`/`groupDomains` unchanged, so the port a
caller writes is dropped. Keys live under `encryption:`. Types are in
`internal/domain/encryption_model.go`; the catalog is `AllEncryptionLookupNames`.

The probe is `internal/domain/encryption_probe.go`. Each rule below is a way to return a
wrong answer that would then be cached for hours (`docs/decisions.md`, "The encryption
probe"):

- **A handshake has three outcomes: accepted, refused, unknown.** Refused (`rejectedError`)
means the TCP connection opened and the handshake then failed. A connect failure or a
timeout is unknown, and unknown fails the lookup. Never record it as a refusal.
- **A lookup where nothing was accepted returns `ErrNoHandshake`**, not a result saying the
host accepts nothing.
- **All handshakes in one lookup go to one address**, picked once in `newSession` after
resolving through `domain.Resolver()`.
- **Port 443 only, hostnames only.** A caller-supplied port makes this a port prober.
- **`InsecureSkipVerify` during the handshake, on purpose**, as in `LookupSSL`. The
`certificate` lookup judges the chain *afterwards* (`trustVerdict`, `hostnameVerdict`) and
reports verdicts. The x509 error is dropped there and must never be returned as the
lookup's error: `IsExpectedError` treats x509 failures as ours and they page.
- **The key exchange probe uses default (ECDHE-only) cipher suites and checks the negotiated
group**; the other two offer every suite so legacy servers still answer, and every group
(`everyGroup()`), because the standard library's default offer leaves some out and a
server requiring one of those would read as refusing TLS.
- Every connection goes through `safenet.SafeDialer`.

Tests aim a `session` at TLS servers on loopback with an empty trust store; nothing touches
the network.

## Resolver — never `net.DefaultResolver`

`domain.Resolver()` returns the resolver lookups use: the external servers from
Expand Down
12 changes: 7 additions & 5 deletions .claude/rules/service-layer.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ The *why* behind each of these is in `docs/decisions.md` under "Lookups and cach

## Shared runner core — `internal/service/runner.go`

Both `DomainService` and `IPService` run on the same machinery: `lookupRunner`,
`DomainService`, `IPService` and `EncryptionService` run on the same machinery: `lookupRunner`,
`fetchResult`, `runState`, the runner constructors, panic recovery, the MGET pre-flight,
and the singleflight + SET fan-out. `runState` is target-agnostic — a target string, a
field name for logs, and a `keyFor` closure — so the domain service supplies
`lookup:<domain>:<name>` keys and the IP service supplies `ip:<ip>:<name>`. Each service
`lookup:<domain>:<name>` keys, the IP service `ip:<ip>:<name>`, and the encryption service
`encryption:<host>:<name>`. Each service
keeps only what is specific to it: which lookups exist, how they are fetched, how the
response is assembled.

Expand Down Expand Up @@ -55,9 +56,10 @@ coalescing anywhere; `service/whole_response.go` is gone and must not come back.

## Cache rules

- Keys are flat: `lookup:<domain>:<name>` / `ip:<ip>:<name>` (e.g.
`lookup:example.net:ssl`), built by `cache.LookupKey()` with the `cache.LookupKeyPrefix`
constant. The builder **does not normalize** — that is the handler's job.
- Keys are flat: `lookup:<domain>:<name>` / `ip:<ip>:<name>` / `encryption:<host>:<name>`
(e.g. `lookup:example.net:ssl`), built by `cache.LookupKey()`, `IPLookupKey()` and
`EncryptionLookupKey()` with their prefix constants. The builders **do not normalize** —
that is the handler's job.
- Redis stores the **bare payload only**. Response wrappers embed the payload type and add
`cache_hit`, set at assembly time; only the embedded type is ever marshaled.
- TTLs come from `cache.default_ttl` + `cache.ttl.<name>` via `config.CacheTTLs()`, which
Expand Down
12 changes: 11 additions & 1 deletion .claude/skills/adding-features/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,5 +31,15 @@ surfaces late — usually as a drift test failing in CI or a stale docs surface.
5. Register the route in `RegisterProtectedRoutes()` with `RequireAdmin()` if admin-only
6. Wire the service in `internal/app/app.go`
7. Add a field to `handler.Deps` and set it in `app.go` — existing test call sites need no change, since they name only the dependencies they use
8. Add context keys to `middleware/request_logger.go` if needed
8. Add context keys to `middleware/request_logger.go` if needed. A per-request count such as `host_count` also needs a `UsageEntry` field, a block in `middleware/usage.go`, a `request_log` column, the insert in `usage/logger.go`, the `SUM` and `UsageSummary` field in `usage/query.go`, and the same field in `cmd/sactl/usage.go`, `openapi.yaml` and `docs_account.html`
9. Update `docs/api-reference.md` (endpoint table and a section for the endpoint), api.http, smoke test, integration tests, changelog.yaml

An endpoint with its own lookup catalog (`AllEncryptionLookupNames` beside `AllLookupNames` and `AllIPLookupNames`) also has to be added everywhere the catalogs are unioned. Nothing fails if one is missed, which is why they are listed:

- `internal/app/app.go` `lookupNames` — left out, the catalog's `cache.ttl.*` keys are silently ignored and every lookup gets `cache.default_ttl`
- `internal/usage/logger_test.go` `TestLookupColumns_MatchesLookupCatalogs` — left out along with the columns, the drift test passes and protects nothing
- `internal/handler/openapi_contract_test.go` and `openapi_test.go` — the request-schema table and the expected-paths list
- `internal/handler/docs_request_body_test.go` and the hard-coded page lists in `docs_test.go`
- `cmd/sactl/lookup.go` and `interactive.go` — the `lookup` command and menu

It also needs its own cache key prefix and builder in `internal/cache/cache.go`, its own `<name>LookupMinPlan` table, and, for a docs page, a `docsPages` row plus the topbar nav link in all six templates (`TestDocs_MobileNavReachesEveryPage` fails until every one has it).
6 changes: 4 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@ Domain and IP intelligence REST API in Go with Gin. Concurrent WHOIS, DNS, SSL,
headers, reputation, email security (SPF/DKIM/DMARC/MTA-STS/TLS-RPT/BIMI),
subdomains, certificate transparency, BGP/ASN, technology fingerprinting, threat
intelligence, and DNS security (DNSSEC/CAA) for domains, plus an aggregate risk score. IP
lookups: GeoIP, ASN, reverse DNS, RDAP abuse contact, threat intelligence. Results are
cached per lookup in Redis with singleflight coalescing; failures are never cached. Auth is
lookups: GeoIP, ASN, reverse DNS, RDAP abuse contact, threat intelligence. Encryption
lookups for hosts, from TLS handshakes the API makes itself to port 443: accepted TLS
versions, key exchange groups including post-quantum ones, and the certificate chain.
Results are cached per lookup in Redis with singleflight coalescing; failures are never cached. Auth is
API-key based with PostgreSQL; customers self-rotate keys and query their own usage.
OpenAPI 3.1 at `/docs/openapi.yaml`.

Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ A domain and IP intelligence REST API in Go. One request fans out to concurrent
(WHOIS, DNS, TLS, HTTP headers, email security, certificate transparency, BGP/ASN,
technology fingerprinting, reputation, threat intelligence, DNSSEC/CAA, GeoIP, reverse
DNS, abuse contacts) and returns one JSON document with a per-lookup cache flag and an
optional aggregate risk score. It is a paid, metered service with self-service API keys,
optional aggregate risk score. A third endpoint reports a host's encryption posture from
handshakes it makes itself: accepted TLS versions, post-quantum key exchange, and the
certificate chain. It is a paid, metered service with self-service API keys,
Stripe billing, and per-plan rate limits, running in production at
**<https://api.securityaccelerated.com>**.

Expand Down
91 changes: 91 additions & 0 deletions api.http
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ GET {{host}}/docs/domain
### IP Intelligence docs (no auth required)
GET {{host}}/docs/ip

### Encryption Posture docs (no auth required)
GET {{host}}/docs/encryption

### OpenAPI specification (no auth required)
GET {{host}}/docs/openapi.yaml

Expand Down Expand Up @@ -405,6 +408,94 @@ X-API-Key: {{adminKey}}
"lookups": ["geo", "asn", "reverse_dns", "abuse_contact"]
}

### Encryption lookup - all three lookups (admin key)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{adminKey}}

{
"hosts": ["example.com"],
"lookups": ["protocols", "key_exchange", "certificate"]
}

### Encryption lookup - every lookup is available on the free plan (free key)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{freeKey}}

{
"hosts": ["github.com"],
"lookups": ["key_exchange"]
}

### Encryption lookup - URL and port are stripped; the probe always goes to 443 (pro key)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{proKey}}

{
"hosts": ["https://Example.com:8443/path", "example.com", "cloudflare.com"],
"lookups": ["certificate"]
}

### Encryption lookup - expired certificate is a result, not an error (admin key)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{adminKey}}

{
"hosts": ["expired.badssl.com"],
"lookups": ["certificate"]
}

### Encryption lookup - should return 400 (lookups missing)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{adminKey}}

{
"hosts": ["example.com"]
}

### Encryption lookup - should return 400 (ssl is a domain-only lookup)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{adminKey}}

{
"hosts": ["example.com"],
"lookups": ["ssl"]
}

### Encryption lookup - should return 400 (raw IP)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{adminKey}}

{
"hosts": ["93.184.216.34"],
"lookups": ["certificate"]
}

### Encryption lookup - should return 400 (free plan allows one host)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json
X-API-Key: {{freeKey}}

{
"hosts": ["example.com", "example.org"],
"lookups": ["certificate"]
}

### Encryption lookup - should return 401 (no key)
POST {{host}}/v1/encryption/lookup
Content-Type: application/json

{
"hosts": ["example.com"],
"lookups": ["certificate"]
}

### My usage (any plan)
GET {{host}}/v1/usage
X-API-Key: {{lookupKey}}
Expand Down
4 changes: 4 additions & 0 deletions cmd/sactl/interactive.go
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,10 @@ func (s *shell) lookupMenu(ctx context.Context) error {
return s.lookupAction(ctx, "IP", "ips", "/v1/ip/lookup",
domain.AllIPLookupNames, defaultIPLookups)
}},
{"3", "Encryption posture", func(ctx context.Context) error {
return s.lookupAction(ctx, "host", "hosts", "/v1/encryption/lookup",
domain.AllEncryptionLookupNames, domain.AllEncryptionLookupNames)
}},
}, "b) Back")
}

Expand Down
18 changes: 11 additions & 7 deletions cmd/sactl/lookup.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,11 @@ import (
func lookupGroup() group {
return group{
name: "lookup",
summary: "run domain and IP intelligence lookups",
summary: "run domain, IP and encryption lookups",
commands: []command{
lookupCommandFor("domain", "domains", "/v1/domain/lookup", domain.AllLookupNames, defaultDomainLookups),
lookupCommandFor("ip", "ips", "/v1/ip/lookup", domain.AllIPLookupNames, defaultIPLookups),
lookupCommandFor("domain", "domain", "domains", "/v1/domain/lookup", domain.AllLookupNames, defaultDomainLookups),
lookupCommandFor("ip", "ip", "ips", "/v1/ip/lookup", domain.AllIPLookupNames, defaultIPLookups),
lookupCommandFor("encryption", "host", "hosts", "/v1/encryption/lookup", domain.AllEncryptionLookupNames, domain.AllEncryptionLookupNames),
},
}
}
Expand All @@ -45,11 +46,14 @@ type lookupFlags struct {
lookups string
}

func lookupCommandFor(name, field, path string, all, defaults []string) command {
// lookupCommandFor builds one `sactl lookup <name>` command. target is what the
// command takes ("domain", "ip", "host"), which is not always the command's
// name: the encryption endpoint takes hosts. field is the request body key.
func lookupCommandFor(name, target, field, path string, all, defaults []string) command {
return command{
name: name,
summary: "look up one or more " + name + "s",
usage: strings.ToUpper(name) + " [" + strings.ToUpper(name) + "...] [--lookups a,b,c]",
summary: "look up one or more " + target + "s",
usage: strings.ToUpper(target) + " [" + strings.ToUpper(target) + "...] [--lookups a,b,c]",
flags: func(fs *flag.FlagSet) any {
f := &lookupFlags{api: registerAPIFlags(fs)}
fs.StringVar(&f.lookups, "lookups", "", fmt.Sprintf(
Expand All @@ -60,7 +64,7 @@ func lookupCommandFor(name, field, path string, all, defaults []string) command
run: func(ctx context.Context, cfg any, args []string) error {
f := cfg.(*lookupFlags)
if len(args) == 0 {
return fmt.Errorf("give at least one %s, e.g. sactl lookup %s example.com", name, name)
return fmt.Errorf("give at least one %s, e.g. sactl lookup %s example.com", target, name)
}
names, err := resolveLookupNames(f.lookups, all, defaults)
if err != nil {
Expand Down
2 changes: 2 additions & 0 deletions cmd/sactl/usage.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ type usageSummary struct {
AvgLatencyMs float64 `json:"avg_latency_ms"`
TotalDomains int `json:"total_domains"`
TotalIPs int `json:"total_ips"`
TotalHosts int `json:"total_hosts"`
StatusCodes map[string]int `json:"status_codes"`
FeatureUsage map[string]int `json:"feature_usage"`
RequestsByPath map[string]int `json:"requests_by_path"`
Expand Down Expand Up @@ -113,6 +114,7 @@ func renderUsage(s usageSummary, f *usageFlags) {
fmt.Printf("\n requests %d\n", s.TotalRequests)
fmt.Printf(" domains %d\n", s.TotalDomains)
fmt.Printf(" IPs %d\n", s.TotalIPs)
fmt.Printf(" hosts %d\n", s.TotalHosts)
fmt.Printf(" latency %.1f ms average\n", s.AvgLatencyMs)
// cache_hit_rate already arrives as a percentage (see usage/query.go), not
// a fraction — scaling it again here produced "7922.3%".
Expand Down
12 changes: 6 additions & 6 deletions deploy/grafana/intelligence-api.json
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,11 @@
},
{
"id": 4, "type": "stat", "title": "Lookup p95",
"description": "95th percentile latency of POST /v1/domain/lookup and POST /v1/ip/lookup, as the caller sees it.",
"description": "95th percentile latency of POST /v1/domain/lookup, POST /v1/ip/lookup and POST /v1/encryption/lookup, as the caller sees it.",
"gridPos": { "x": 12, "y": 0, "w": 4, "h": 4 },
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"targets": [
{ "refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" }
{ "refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip|encryption)/lookup\"}[$__rate_interval])))" }
],
"fieldConfig": { "defaults": { "unit": "s", "decimals": 2 }, "overrides": [] },
"options": { "colorMode": "none", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"] } }
Expand Down Expand Up @@ -150,13 +150,13 @@
},
{
"id": 12, "type": "timeseries", "title": "Lookup endpoint latency",
"description": "Request latency of the two lookup endpoints, end to end. The per-lookup fetch times are in the Lookups row.",
"description": "Request latency of the three lookup endpoints, end to end. The per-lookup fetch times are in the Lookups row.",
"gridPos": { "x": 0, "y": 13, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"targets": [
{ "refId": "A", "legendFormat": "p50 {{route}}", "expr": "histogram_quantile(0.50, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" },
{ "refId": "B", "legendFormat": "p95 {{route}}", "expr": "histogram_quantile(0.95, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" },
{ "refId": "C", "legendFormat": "p99 {{route}}", "expr": "histogram_quantile(0.99, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip)/lookup\"}[$__rate_interval])))" }
{ "refId": "A", "legendFormat": "p50 {{route}}", "expr": "histogram_quantile(0.50, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip|encryption)/lookup\"}[$__rate_interval])))" },
{ "refId": "B", "legendFormat": "p95 {{route}}", "expr": "histogram_quantile(0.95, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip|encryption)/lookup\"}[$__rate_interval])))" },
{ "refId": "C", "legendFormat": "p99 {{route}}", "expr": "histogram_quantile(0.99, sum by (le, route) (rate(intelligence_api_http_request_duration_seconds_bucket{app=~\"$app\",instance=~\"$instance\",route=~\"/v1/(domain|ip|encryption)/lookup\"}[$__rate_interval])))" }
],
"fieldConfig": { "defaults": { "unit": "s", "min": 0 }, "overrides": [] },
"options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["lastNotNull", "max"] }, "tooltip": { "mode": "multi", "sort": "desc" } }
Expand Down
Loading
Loading