Skip to content

Latest commit

 

History

History
626 lines (450 loc) · 32.9 KB

decisions.md

File metadata and controls

626 lines (450 loc) · 32.9 KB

Decisions — Intelligence API

Things in this codebase that look removable and are not, with the reason attached.

The fleet's global instructions say don't preserve complexity because it already exists. That rule is right, and it is dangerous on its own: some complexity is load-bearing, and code cannot say so about itself. domain.Resolver() looks like a pointless wrapper around net.DefaultResolver right up until you learn the global one broke every Postgres connection the process later opened. This file is the counterweight — read it before simplifying.

The bar for an entry: would a competent engineer, seeing only the code, reasonably try to undo this? If yes, it belongs here. If the code explains itself, it does not. This is not a design document and it is not a changelog.

Lookups and caching

Failed lookups are never cached

Decision. A lookup that errors is retried on the next request. There is no negative cache and no redis.negative_cache_ttl config key.

Why. Per-lookup TTLs removed the problem that key was compensating for. One flaky provider can no longer shorten every other lookup's cache lifetime, because each lookup now expires on its own schedule.

Instead of. Caching failures briefly to protect upstreams. That trades a real cost — serving a stale failure to someone who would have gotten a good answer — for a saving that per-lookup TTLs already deliver.

geo is never cached

Decision. Its runner sets uncached, so it is excluded from the MGET pre-flight, never written back, and always reports cache_hit: false. asn was uncached too until it moved to Team Cymru DNS; as a network lookup it is now cached for 24h like bgp.

Why. It reads a local mmap. A Redis round trip costs more than recomputing it. The old whole-document cache only stored it because it shared a key with RDAP.

Instead of. Caching them for consistency with every other lookup. Consistency is not the goal; latency is, and here the cache is the slow path. This also deleted the "skip caching when the GeoIP database is missing" special case — with geo uncached there is nothing to skip.

fingerprint has no cache key of its own

Decision. It is deterministic computation over headers data. Requesting headers, fingerprint, or both creates exactly one headers runner, and fingerprint.cache_hit mirrors the headers entry it was computed from.

Why. Caching headers already caches it. A separate entry would be a second copy of derived data that can go stale independently.

Instead of. Giving it a key like every other lookup. That would also mean admin fingerprint-rule updates sat behind a TTL instead of taking effect immediately.

Redis stores the bare payload, never the response wrapper

Decision. Response types (WhoisResult, DNSResult, …) embed the payload type and add cache_hit, which is set at assembly time. Only the embedded type is ever marshaled.

Why. cache_hit is a fact about this request. Persisting it would mean a cached entry asserting it was a cache hit before it was ever read back.

Instead of. Storing the wrapper for convenience. The type embedding is what makes this structurally impossible rather than merely avoided.

Cache keys are flat and never re-normalized

Decision. Three namespaces, one shape: lookup:<domain>:<name> from cache.LookupKey(), ip:<ip>:<name> from cache.IPLookupKey(), and encryption:<host>:<name> from cache.EncryptionLookupKey(), with the prefixes as constants beside them. No builder normalizes; normalization is the handler's job. The separate ip: prefix is what keeps threat_intel on the two endpoints from colliding. The encryption: prefix looks redundant today, since no encryption lookup shares a name with a domain one, but a host is a domain name: the first shared name would otherwise read the other endpoint's payload.

Why. A missed normalization step surfaces as a distinct key — visibly wrong — rather than being silently papered over at the cache layer.

Instead of. Defensive normalization in the key builder, or a composite key covering the whole requested lookup set. The composite version meant two callers asking for overlapping but different lookups shared nothing.

Cache problems never fail a request

Decision. MGET error → treat everything as a miss. Corrupt entry → treat as a miss, and the live fetch overwrites it. Failed SET → serve the result anyway. None of these ever reach resp.Errors.

Why. The cache is an optimization. A request that could be served should not fail because the optimization is unavailable. resp.Errors is reserved for real lookup failures, so putting cache noise there would train callers to ignore it.

Instead of. Surfacing cache errors to the caller, which looks more honest and makes Redis a hard dependency of every request. It also mixes two kinds of error in one field, and the field callers actually act on is the one that then becomes noisy.

Revisit if. Cache failures ever become invisible for long enough to matter — they are logged, and that is the only signal.

Concurrency and failure

Lookups use a scoped resolver, not net.DefaultResolver

Decision. domain.Resolver() returns an atomic.Pointer[net.Resolver] set by InitResolver. Lookups that resolve user-supplied names use it. Infrastructure keeps using the system resolver.

Why. Overriding net.DefaultResolver sent every resolution in the process to public DNS, including the service's own connections to Postgres and Redis. Anything addressed by a platform-internal name — a Compose service, a Kubernetes service, a Fly .internal address — stopped resolving. Calling InitResolver after those connected only protected the initial connections: database/sql recycles on conn_max_lifetime, and the replacements failed with "no such host" (issue #62).

Instead of. Setting net.DefaultResolver once at startup, which is shorter and looks equivalent. It is not — the blast radius is the entire process. resolver_test.go asserts InitResolver installs no Dial hook on the global resolver, so this cannot regress quietly.

Revisit if. The service stops depending on any name only the platform's resolver knows.

Panic recovery happens inside the fetch, not at the goroutine boundary

Decision. Every lookup fetch runs behind recoverFetch, which degrades a panic to a sanitized internal error on the response.

Why. A panic escaping into singleflight is re-raised in every coalesced caller. Recovering at the goroutine boundary would let one bad lookup take down unrelated in-flight requests.

Instead of. Relying on gin.Recovery(), which only covers the handler's own goroutine, or recovering where the goroutine is spawned, which is too late.

The shared singleflight fetch runs under context.WithoutCancel

Decision. The coalesced fetch detaches from the caller's context and reapplies the service's own lookupTimeout.

Why. One caller disconnecting, or holding a tight deadline, must not cancel or truncate work other callers are waiting on.

Instead of. Passing the first caller's context through. That makes the shared work hostage to whichever request happened to arrive first.

Coalesced callers each decode their own copy

Decision. Every caller unmarshals the shared payload into its own response value.

Why. Nobody shares a mutable pointer, so one caller mutating its result cannot corrupt another's.

Instead of. Handing out the same decoded value and defending with a deep copy. This replaced exactly that — the old copyViaJSON guard — by making the sharing impossible rather than compensating for it.

Screenshots were removed before v1.0

Decision. The screenshot lookup is gone; asking for it is a 400. The Chromium provider, the capture semaphore, the browser SSRF proxy, chromedp, and Chromium in the image went with it. request_log.screenshot stays for its history. #110 tracks the return.

Why. In-process headless Chromium never reliably started inside its budget on a shared-CPU Fly machine, and failed captures return 200 with the reason in errors, so they looked like success. Three rounds of fixes (startup timeout, /dev/shm, concurrency 4→1, 1 GB) did not get it there. The likely remaining cause — the 45s capture timeout nested under the 30s lookup timeout — was never confirmed in production. Shipping a paid lookup that silently fails was worse than not shipping it.

Instead of. Keeping it behind a warning, or fixing the timeout nesting under launch pressure. The code is one git revert away, and #110 records what was learned: a hosted provider is the preferred return path, since local Chromium also costs ~500 MB per capture and runs with --no-sandbox.

Dependency injection: interfaces only where a second implementation is real

Decision. Lookups with one implementation and no state are function fields, swapped in tests via setters. Collaborators that could plausibly have a second implementation get an interface — cache.Cache, notifications.Notifier, usage.Recorder, billing.Service, domain.ThreatIntelProvider.

Why. The test is whether an alternative is real, not whether one is imaginable. Screenshots earned an interface for the same reason — a hosted provider was a live option — until the lookup was removed; see #110.

Instead of. Wrapping everything in interfaces for symmetry. LookupDNS behind an interface buys nothing and costs a layer of indirection at every call site.

The encryption probe

internal/domain/encryption_probe.go and the three lookups built on it (#136). The common thread: these results are cached for hours, so a wrong answer is worse than no answer.

2026-11-16 — A handshake that times out fails the lookup; it is never a refusal

Decision. A handshake has three outcomes: accepted, refused (rejectedError: the TCP connection opened and the handshake then failed), and unknown (the connection never opened, or time ran out). Unknown fails the whole lookup.

Why. A failed lookup is retried on the next request. A wrong "TLS 1.3 not accepted" sits in the cache for twelve hours, and one lost packet is enough to produce it.

Instead of. Counting silence as refusal. Some middleboxes drop offers they dislike instead of refusing them, so those hosts fail the lookup every time. That cost was weighed and accepted when the endpoint shipped.

2026-11-16 — A lookup where nothing was accepted is an error, not an empty result

Decision. When every offer is refused the lookup returns ErrNoHandshake. IsExpectedError knows it, so it does not page.

Why. "This host accepts no TLS version" cannot be told apart from "this host is blocking us", and only the second is common.

Instead of. Returning a payload with every accepted false, which reads as a finding and would be cached as one.

2026-11-16 — Every handshake in a lookup goes to one address, and only to port 443

Decision. newSession resolves once through domain.Resolver(), picks one public address (IPv4 first), and every handshake in that lookup dials it. The payload reports it as ip. The port is a constant.

Why. A host with several addresses can be configured differently on each; mixing them gives a result true of no single server. IPv4 first because a missing IPv6 route on our side would be reported as the target's failure. A caller-supplied port turns the endpoint into a port prober, which is a product decision and not a parameter.

Instead of. Dialing the hostname each time and letting the dialer pick, as LookupSSL does for its single connection.

2026-11-16 — The handshake skips verification; the certificate lookup judges the chain afterwards

Decision. Every probe sets InsecureSkipVerify. certificate then runs x509.Verify itself and reports trust_verdict and hostname_verdict. The verification error is dropped, never returned. Trust is judged at a moment inside the leaf's validity window (now, or the nearest end of the window), using only the certificates the server sent.

Why. Reporting expired, self-signed and mismatched certificates is the feature; verifying in the handshake would turn the answer into an error. The error is dropped because IsExpectedError deliberately treats x509 failures as ours (they can otherwise only come from trusted upstreams), so one escaping would page on every untrusted host a customer checks. The clamped time keeps trust and expiry independent: otherwise every expired certificate is also untrusted, and "renewal was missed" is indistinguishable from "no public CA issued this".

Instead of. Verifying at probed_at, or fetching missing intermediates the way some browsers do. A chain that only verifies after a repair fails in strict clients, which is what the caller wants to know.

The trust store is checked before it is used. checkTrustStore fails the lookup with ErrTrustStoreUnavailable when the system roots are missing or empty. The empty case looks unnecessary and is the one that happens: on Linux an image with no CA bundle loads an empty pool with no error, and every chain then verifies as "unknown authority". Without the check every host would be untrusted, cached for hours, with nothing wrong in the logs. That error is ours, so it is deliberately not in IsExpectedError and it alerts.

2026-11-16 — The key exchange probe offers default cipher suites; the others offer every suite

Decision. protocols and certificate offer every suite the standard library has, insecure ones included, and every key exchange group it can speak. key_exchange leaves the default (ECDHE-only) suite list and checks that the negotiated group is the one offered.

Why. A legacy server with only RSA key exchange or CBC suites left still speaks TLS, and leaving those suites out would report it as refusing. The groups are listed explicitly for the same reason at the other end of the range: the default offer omits some groups (pure ML-KEM), so a server that requires one refused the handshake and was reported as accepting no TLS version and having no certificate. For the group probe the extra suites are a hazard: a TLS 1.2 server could finish the handshake over RSA key exchange without using the group at all, and the group would read as accepted.

Instead of. One shared tls.Config for all three probes.

2026-11-16 — All three encryption lookups are on the free plan

Decision. encryptionLookupMinPlan gates nothing.

Why. Pro gating in this API has meant "spends scarce upstream quota" (reputation, threat_intel). These spend none: measured at about 1 ms of CPU and 60–100 ms of wall time per handshake, with the per-plan host cap and rate limit bounding the rest. Gating would be a pricing decision, and none has been made.

Revisit if. Pricing for the encryption product is decided. It is a one-word change per row, and TestEncryptionLookup_EveryLookupIsFree is there to make it a deliberate one.

HTTP surface and API contract

The lookup endpoint returns {"results": [...]}, not a bare array

Decision. Every response is a JSON object.

Why. Consistent with {"error": "..."}, and response-level fields can be added later without breaking clients.

Instead of. Returning the array directly, which is tidier and permanently unextendable.

handler.New takes a Deps struct

Decision. One struct, every field optional, a nil service making its endpoints return 503 rather than panicking.

Why. Most fields are nilable and several share a type, so a misordered positional argument list would have compiled silently. Naming them also documents which surfaces a given Handler serves.

Instead of. New(nil, nil, nil, nil, nil, nil, nil, nil, rc, nil, nil, nil). Adding a dependency is now additive; every existing call site is untouched.

SSRF is blocked at the dialer, after DNS resolution

Decision. safenet.DialControl rejects private, loopback, link-local, multicast, unspecified, and reserved addresses at connect time. All HTTP clients in domain use it via the shared transports in domain/transport.go.

Why. Checking after resolution means there is no TOCTOU window between validating a hostname and connecting to it.

Instead of. Validating the URL or hostname before the request. A name that resolves to a private address passes that check and connects anyway.

2026-11-16 — A refused private address is logged, not alerted on

Decision. safenet.BlockedError is an expected error in IsExpectedError. The refusal is returned to the caller and logged at warn level with the host; it does not count toward the throttle or spike alerts.

Why. It was unexpected on purpose, as a possible SSRF probe worth a look. But an alert carries a lookup name and a count, never the host or the customer, so the alert could not be investigated; the log line can. Meanwhile the trigger is a hostname the caller chooses, and names that point at private addresses are common in public DNS (internal hosts published by mistake, localhost. records, split-DNS leftovers). A customer checking a list of someone's subdomains would raise alerts by accident, and anyone could raise them on purpose. The encryption endpoint made this sharper, because all three of its lookups resolve the host and so fail together.

Instead of. Keeping the page as an attack signal. If that signal is wanted, a counter of blocked dials in internal/metrics is the place for it: it can be graphed and alerted on at a threshold that means something, without tying it to the pager.

The 404 page never echoes the requested path

Decision. Handler.NotFound renders a static page. Paths under /v1 or /billing get the JSON error envelope instead, so a mistyped endpoint lands in a caller's existing error path.

Why. Reflecting caller-chosen text onto our own domain is a phishing primitive.

Instead of. The conventional "no such page: /foo", which is friendlier and turns the domain into a rendering surface for arbitrary attacker-supplied strings.

Docs pages

docsPages is the canonical page set

Decision. One table in docs.go. Routes are registered by looping it, and the tests derive their page lists from it.

Why. The set was previously spelled out in the router, twice in docs_test.go, twice more in docs_layout_test.go, and in every template's topbar nav. Adding /docs/account meant finding all six by hand.

Instead of. Listing pages where they are used. Tests that apply to only some pages select rather than re-list — docsPagesWithLanguageTabs() — so a page that gains a Go example gets it compiled without anyone remembering.

The company home page is served by the API and is not in docsPages

Decision. / renders home.html, the Security Accelerated company page, from the same binary as the docs. It stays out of the docsPages table. It uses the docs' design language (the shared stylesheet's tokens, typefaces and theme toggle) with a layout of its own. It says what the product is and carries no reference material: no lookup list, no request or response sample.

Why. The root used to be a 301 to /docs, which left the company with no page of its own. Serving it here means no second host, deploy or toolchain, and the price it shows comes from the same config keys as the Plans table on /docs. The split between shared language and own layout was arrived at by getting it wrong in both directions: the first version reused the docs page shell and looked like a copy of the docs overview; the second had its own typeface and palette and no longer looked like the same company. Through v1.1.1 the page also listed every lookup and showed a request and its response. That was a second copy of the docs, one link away from the first, and a list every new lookup had to remember to update; it was replaced with a short description of what the API is for.

Instead of. A separate static site, which would need its own hosting and would quote plan limits and a price that nothing ties to this service's config. And instead of a sixth docs page, which is what reusing the docs shell produced.

The hostname decides what / is, and the redirect is a 302

Decision. One app answers for two names. site.home_host names the company's domain: / is the home page there (and on www.), and a 302 to /docs on every other host. The home page links to the docs on site.api_host, and the docs' brand link goes to the home host. Both keys empty, the default, serves the home page at / on every host with relative links.

Why. v1.1.0 served the home page at the root of the API's hostname, which put the company page at api. and left the company's own domain on a registrar placeholder. The links are absolute because a relative one keeps the reader on the wrong name: the docs would be read at the company's domain, and the docs' brand link would go to a root that redirects straight back to the docs. They are built from config, never from the request's Host header, so a caller cannot choose where they point. The redirect is a 302 because which host gets the page is configuration, and a browser keeps a 301 after the configuration has changed: anyone holding the pre-v1.1.0 301 never saw the home page at the API's root.

Instead of. A second app or a static site for the company's domain, for the reasons in the entry above. And instead of redirecting everything except / on the home host over to the API host: the docs and the API still answer there, which is untidy but harmless, and the redirect would need its own exceptions for the stylesheet and scripts the home page loads.

Every table is wrapped in .table-scroll, and grid tracks use minmax(0, 1fr)

Decision. Both are pinned by tests in docs_layout_test.go.

Why. Both are invisible at a desktop viewport and made every page unreadable on a phone. width: 100% on a table is a floor, not a ceiling: when the columns' min-content widths exceed the container, the table grows past it and takes the document's scroll width with it. A bare 1fr track floors at the widest item's min-content — the unbreakable base URL — and .hero-card clips its overflow, so hero copy was cut off mid-sentence.

Instead of. Trusting width: 100% and 1fr to mean what they appear to mean. They do not, and nothing on a desktop screen tells you.

The copy button and page search are injected by docs.js, never written into templates

Decision. Pinned by docs_controls_test.go, which fails if either control's class names appear in a template or diverge between script and stylesheet.

Why. Markup for either one is left behind when the script does not run — a dead control on a page that is otherwise readable static HTML. A search index authored by hand drifts from tables that are edited constantly.

Instead of. Putting the button in the markup. It also must live in the .code-block wrapper rather than inside the <pre>: a <pre> is its own horizontal scroll container, so a control inside one slides away with the code exactly when a 40-line payload is hardest to select.

Operations

2026-11-16 — /metrics is on its own listener, with no authentication

Decision. Prometheus metrics are served at GET /metrics on a second listener (metrics.listen_addr, :9091), started only when metrics.enabled is set. The public router has no such route and answers 404. The endpoint checks no credential; the port having no public address is the whole access control.

Why. Whoever scrapes has to be able to reach it, and the scraper we use (Fly's, see the next entry) takes a port and a path and cannot send a header. Behind the admin key, every scrape would also run the protected chain: an auth lookup, then a row in request_log every 15 seconds per machine, which is the table this work exists to stop leaning on. And the admin key would have to be copied into a scraper's configuration, where a key that can rewrite fingerprint rules and read every customer's usage has no business being.

Instead of. GET /v1/admin/metrics behind RequireAdmin(), which looks safer because it has a lock on it. The lock is the problem: it puts the most powerful credential in the least protected place, and it makes reading the metrics depend on authentication, so on Postgres, working: they would go dark in the outage they are there to explain.

Cost. Nothing in the code stops someone publishing the port. A [[services]] block for 9091 in fly.toml, or a Compose mapping without 127.0.0.1:, exposes route names, plan names, traffic volumes and the running version to anyone. On Fly the port is also reachable by other apps in the same organization over the private network.

Revisit if. The organization gains an app that should not see these numbers, or the metrics have to be scraped from outside the platform. Then put a bearer token on this listener; do not move it onto the API port.

2026-11-16 — Production metrics use Fly's managed Prometheus and Grafana

Decision. fly.toml carries a [metrics] section and Fly scrapes each machine. The dashboard in deploy/grafana/ is imported into Fly's hosted Grafana at fly-metrics.net. We run no Prometheus, no Grafana and no agent in production. The pair in docker-compose.yaml is for checking the dashboard locally and nothing else.

Why. It is the only option that adds nothing to operate. The scrape happens inside Fly's network, so the endpoint stays off the internet, and there is no token to store or rotate. It also fits machines that stop when idle: a stopped machine is simply not scraped, where a push-based setup needs an agent running beside the app on each one.

Instead of. Grafana Cloud's free tier, which needs either a public, authenticated scrape endpoint or a collector process in every machine pushing with a stored credential, on machines sized at 1 GB. And instead of self-hosting both on Fly, which is another app with a volume, upgrades and its own uptime to watch, for one service's dashboard.

Cost. Retention, limits and price are Fly's to change; it is free today and they say they will give notice. There is no alerting from these metrics yet: alerts still come from the Pushover alerter and the uptime workflow. The dashboard is imported by hand, so the hosted copy can fall behind the file.

Revisit if. Fly starts charging enough to matter, alerting on these metrics is wanted (Grafana Cloud's alerting is the likely reason to move), or the service leaves Fly. The metrics endpoint itself is plain Prometheus and does not change in any of those cases.

2026-08-27 — Deploys trigger on a release tag, not a release branch

Decision. fly-deploy.yml fires on any v* tag. make release V= pushes the tag and that is the deploy. The version baked into the binary is the tag name. Rollback is workflow_dispatch with an earlier tag. There is no release branch.

Why. The previous flow — PR from main to release, deploy on push to release, version from git describe --tags --abbrev=0 — mislabelled a deploy whenever that PR was squash- or rebase-merged: the tagged commit dropped out of release's history and describe walked back to the previous tag, so production ran new code under the old version. Guarding it took a repo setting, a rule in two files, and a verification step. The tag-triggered form has nothing to guard: the checkout is the tag.

Instead of. Keeping the release branch and enforcing merge commits, which we did first (#90). It worked, but it protected a mechanism whose only remaining value — a human click before production — make release already provides. The release PR also re-reviewed code that had already been reviewed into main; CodeRabbit had nothing to say on #94 and never would have.

Revisit if. A second person needs to approve deploys. Then add required reviewers to the production environment rather than bringing the branch back.

feature_usage counts requested lookups, not served ones

Decision. The usage middleware records what the caller asked for, regardless of whether it came from cache. cache_hits/cache_misses are separate counters on request_log.

Why. A cache hit must not make a feature look less used. Those are different questions and were previously answered by the same number.

Instead of. Counting upstream work, which measures cost rather than demand.

usage.LookupColumns is asserted to match domain.AllLookupNames

Decision. A test fails when they diverge. It compares against the union of every catalog (domain, IP, encryption), so a new catalog has to be added to the test as well as to LookupColumns.

Why. A new lookup that nobody added a column for fails loudly in CI rather than silently in production, where the symptom is a feature that appears unused forever.

Instead of. A checklist entry in the "adding a lookup" docs. Checklists are followed until they are not, and this failure is undetectable by anyone reading the diff.

sactl enforces --allow-writes at each mutation site

Decision. requireWrites is called at every point that changes deployed state, rather than threading a field through commands.

Why. A path that forgot to check a field would be silently unguarded. There is one function to consult instead.

Instead of. A dryRun bool on the command struct — the standard approach, and one where the failure mode is invisible.

sactl parses .env rather than sourcing it

Decision. The CLI reads and parses the file itself.

Why. Connection strings contain &, which a shell treats as a background operator, truncating the value at the first query parameter. scripts/status.sh works around the same problem by grepping individual keys out of the file.

Instead of. source .env or set -a; . ./.env, which is what everyone reaches for first. It does not error — it silently hands you a connection string cut off at the first &, and the failure appears later as an authentication error against the wrong database.

The smoke test and docker reset are deliberately not in sactl

Decision. scripts/smoke-test.sh and scripts/docker-reset.sh are the single source of truth for both.

Why. They existed in sactl briefly and were removed. Two implementations of the same procedure drift, and the one you are not looking at is the one that is wrong.

Instead of. Folding them in for a single operator entry point, which is a real convenience and the reason it was tried once already. If the CLI should run them, it should shell out to the scripts rather than reimplement them.

GeoIP comes from DB-IP Lite and Team Cymru, not MaxMind

Decision. geo uses the DB-IP Lite City database (CC BY 4.0), downloaded monthly; asn uses Team Cymru's IP-to-ASN DNS service. /docs/ip carries the DB-IP attribution.

Why. The MaxMind GeoLite EULA grants an internal-use license and requires MaxMind's written consent before the data is disclosed to third parties. Returning it from a paid API is disclosure. CC BY 4.0 explicitly allows commercial use and redistribution with a link back. Cymru publishes no restrictions and was already behind the domain bgp lookup, so asn gained no dependency — and gained the network prefix, which the MaxMind reader never exposed.

Cost. DB-IP Lite is less accurate than GeoLite and has no postal code, timezone, accuracy radius, or subdivision code; those fields left the response rather than being documented as always-absent. asn is now a DNS round trip on a cache miss instead of an mmap read.

Instead of. A commercial GeoIP2 license, which keeps accuracy at a recurring cost that is hard to justify before launch. Revisit if customers need finer geolocation.

The Fly volume is chowned by the entrypoint, running as root then dropping

Decision. docker-entrypoint.sh chowns the mount as root, then execs the app as appuser via su-exec. The container sets no USER.

Why. Fly volumes mount root-owned and Fly offers no uid/gid option. Without this step the GeoIP download fails with permission denied on every boot — silently.

Instead of. Setting USER appuser in the Dockerfile, which is the correct default everywhere else and produces a service that looks healthy while never populating its database.

A missing GeoIP database does not degrade /v1/health

Decision. Postgres or Redis down returns 503. Missing GeoIP does not.

Why. It self-bootstraps at startup and is absent for the first few seconds of a cold start. Degrading on it would page someone for normal startup, every time.

Instead of. Treating every dependency uniformly, which reads as more rigorous and produces an alert that fires on every deploy until people mute it. A missing GeoIP database is also classified in isExpectedError for the same reason.