feat(transport): bounded retry for active services, over an API-neutral layer

[thodson-usgs]

Retitled. This started as a boundary extraction and was described as one,
but 3822f04f also switched WQP, NLDI, StreamStats, and Water Use onto a
retrying request path. That is a user-visible resilience change, not a
refactor, so it now leads the description. Reviewers should evaluate the
retry behavior on its own merits; the layering change is the second half.

What changes for users

Four services now retry transient failures where they previously failed on the
first attempt: wqp, nldi, streamstats, and wateruse. A request that
hits a 429 or a gateway 5xx is re-sent up to API_USGS_RETRIES times (default 4)
with exponential backoff and full jitter, honoring a server Retry-After.

This costs latency and quota on failing requests, so the retry is bounded on two
independent axes and narrowed to failures a later attempt could actually
survive:

• API_USGS_STALL_TIMEOUT (new; default 60 s, 0 disables) bounds how long
  a call may go without receiving any data. API_USGS_RETRIES counts
  attempts, not seconds, so on its own four retries of a request that times out
  after a minute is four silent minutes. Progress restarts the budget — a page
  received, or a queued sub-request acquiring its concurrency slot — and an
  attempt already in flight is never interrupted. The first retry is never
  withheld, so one slow attempt cannot disable retry by itself. A dead
  connection costs about two read timeouts (~2 min) instead of five attempts
  (~5 min).
• Which statuses are re-sent is per-adapter. WQP answers an over-large query
  with a 500 and StreamStats answers out-of-network coordinates with one, so
  those one-shot adapters re-send only for 429/502/503/504. The Water Data OGC
  API is a query interface where a 500 is an upstream hiccup, so the chunker
  keeps re-sending for every 5xx, as it always has.
• Failures already settled are not retried — an unsupported scheme, a
  malformed request, or a hostname the resolver rejects outright fails on the
  first attempt. A temporary resolver failure (EAI_AGAIN: a resolver still
  coming up, a VPN reconnect, a laptop waking) stays retryable.
• Backoff always includes jitter, including on a server-named Retry-After,
  so sub-requests handed one hint do not wake in lockstep, and a Retry-After
  of 0 cannot become a zero-delay re-send. An HTTP-date that has already
  passed is treated as no hint rather than as "retry now", since the likelier
  cause is client/server clock skew.

Water Data (waterdata) already retried via the chunker and keeps doing so;
what is new for it is the stall budget, the deterministic-failure
classification, and the jitter above.

Deprecated nwis is untouched — it still uses utils.query, which retains
its exact signature and performs no retry.

A bad setting now raises ConfigurationError (both a DataRetrievalError
and a ValueError), so a typo in API_USGS_RETRIES, API_USGS_STALL_TIMEOUT,
or API_USGS_CONCURRENT no longer escapes a request path as a bare
ValueError.

Measured against the live API

A 4-state, 30-year get_daily over 800 sites at parallel_chunks(1) runs 91.8 s
wall-clock and returns 581,070 rows. Its longest gap between progress events is
12.1 s — the budget measures silence, not duration, so long successful queries
are not at risk. A 2,000-site / 35-year variant: 74.0 s total, 11.2 s worst gap.

The layering change

dataretrieval.transport is a new internal, API-neutral execution layer owning
guarded HTTP client lifecycle and timeout defaults, host-scoped authentication,
cursor pagination, bounded retry, response aggregation, progress, and
sync-over-async dispatch. dataretrieval.ogc keeps its protocol concerns:
dialects, CQL2, request construction, feature shaping, URL-byte chunk planning,
resumable ChunkedCall state, and interruption types.

This is what makes the retry change tractable: before it, generic execution
behavior lived under OGC even where non-OGC services used it, so Water Use
depended on private protocol modules and retry policy was uneven across
services. There is now one retry policy to reason about instead of several.

Internally, transport.liveness is a stdlib-only leaf recording when data last
arrived, so the page loop that observes progress and the retry loop that acts on
it both depend on it rather than on each other.

Not a public framework: dataretrieval.transport is internal and carries no API
promise.

Compatibility

Public imports, service signatures, return shapes, metadata, deprecations,
exception types, OGC chunking/resume behavior, and the exact four-symbol OGC
facade are unchanged. utils.query retains its exact signature and still
performs no retry.

Compatibility aliases are kept where a consumer exists: the private utils
transport names (_get, _default_headers, HTTPX_DEFAULTS, …) and the
ogc.engine wrappers are live re-exports, so existing monkeypatch sites keep
working. Two modules were removed rather than aliased, since nothing in the
tree imported them: dataretrieval.ogc.progress and
dataretrieval.ogc.combining (now dataretrieval.transport.progress /
dataretrieval.transport.combining). dataretrieval.ogc.retry keeps only its
OGC interruption classifiers; the retry tunables it used to re-export were
copies by value that patching could not reach, and now live solely in
dataretrieval.transport.retry.

What changes deliberately is the retry behavior described above.

Commits

• 3822f04f — extract API-neutral execution policy (also the retry opt-in)
• 966bb618 — simplify shared policy use
• 01268fd3 — bound retry by elapsed silence and failure kind
• c1121265 — don't spend the stall budget on queue time
• 2ce45372 — never let the budget withhold the first retry
• 5c666d20 — credit the queue wait instead of resetting the clock
• 99fdbe5d — give the retry loop the gate and the whole stop rule
• 5dfa267b — scope the retry-status rule and stop over-classifying DNS
• 8ed5889a — repair three Retry-After invariants review found broken

Validation

Re-run on the current tip (8ed5889a): full suite 666 passed with 57 expected
warnings; strict mypy clean over 41 source files; Ruff lint and format clean;
all configured pre-commit hooks pass.

From the post-rebase validation of 966bb618 (not re-run for the later
commits): coverage 97%, Sphinx HTML build with the same four pre-existing
warnings, isolated wheel build/install/import outside the checkout, and the
GitHub Actions matrix across lint, typing, artifact, docs, Ubuntu, and Windows.

Architecture fitness functions in tests/architecture_test.py enforce transport
dependency direction, an acyclic transport graph, and Water Use isolation from
OGC. ADR 0006 records the decision.

Test-coverage caveat

tests/conftest.py::_pin_chunker_env pins API_USGS_RETRIES=0 for the whole
suite, so the newly default-on retry paths are exercised only where a test opts
back in explicitly. The green suite therefore verifies less of this behavior
change than its size suggests; the live-API measurements above and the targeted
retry tests are the real evidence.

Known follow-ups (recorded, not fixed here)

• .retryable on the exception taxonomy stays the broader "might succeed" hint
  and does not agree with the stricter auto-re-send sets; unifying them changes
  a documented public field and belongs in its own change.
• ogc.retry._classify_chunk_error still maps some deterministic failures to a
  resumable ServiceInterrupted (pre-existing).
• Three next-cursor validators exist with three failure modes (wateruse,
  ogc/engine, waterdata/ratings); a shared resolver belongs in transport.
• The stall default equals the 60 s read timeout, so a request burning a full
  read timeout gets one retry rather than several. Deliberate.

Scope

Collection-family splitting, additional adapter restructuring, and legacy NWIS
retirement remain separate follow-up work.