Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
**08/02/2026:** Phase 1 OGC boundary stabilization: the `dataretrieval.ogc` package now exposes a deliberate, small facade (`OgcDialect`, `prepare_request_args`, `get_ogc_data`, `fetch_ogc_request`) that service adapters (NGWMN, Water Data's generic wrapper) import. Internal request-construction helpers moved to a new `ogc.requests` module; the dialect type and endpoint constants live in the leaf `ogc.policy` module. `ogc.shaping` no longer depends on `ogc.engine` at all, and the complete runtime OGC import graph is now acyclic. `_default_headers` now accepts a target URL and adds `X-Api-Key` only for `api.waterdata.usgs.gov`; shared sync and async clients also strip the key before following any cross-host redirect, including external rating-asset downloads. `waterdata.utils` no longer bulk-re-exports private OGC symbols, and consumers import implementation helpers from their canonical modules. No public API, return-value, or deprecation changes.

**08/02/2026:** Fixed source-distribution and wheel package discovery so the `dataretrieval.ogc` and `dataretrieval.waterdata` subpackages are included in installed artifacts. CI now builds and installs the wheel outside the source checkout before importing the core service modules. Added an architecture baseline, initial decision records, and executable dependency-direction guardrails for the existing modular-monolith boundaries.

**06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected.
Expand Down
13 changes: 8 additions & 5 deletions dataretrieval/ngwmn.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
The NGWMN exposes its data through a dedicated OGC API
(``https://api.waterdata.usgs.gov/ngwmn/ogcapi``) with five collections:
``sites``, ``waterLevelObs``, ``lithologyObs``, ``constructionObs``, and
``providers``. Each getter below delegates to the shared OGC engine
(:func:`~dataretrieval.ogc.engine.get_ogc_data`) with
``providers``. Each getter below delegates to the shared OGC facade
(:func:`~dataretrieval.ogc.get_ogc_data`) with
``base_url=NGWMN_OGC_API_URL``, so multi-value chunking, pagination,
retry/resume, and result shaping all behave exactly as they do for the main
Water Data getters.
Expand All @@ -24,9 +24,12 @@
import pandas as pd

from dataretrieval.codes.states import apply_state
from dataretrieval.ogc.engine import BASE_URL, OgcDialect, _get_args, get_ogc_data
from dataretrieval.ogc import OgcDialect, get_ogc_data, prepare_request_args
from dataretrieval.utils import BaseMetadata

# The Water Data API base URL, defined locally to avoid importing policy internals.
BASE_URL = "https://api.waterdata.usgs.gov"

# The National Ground-Water Monitoring Network exposes its own OGC API at a
# separate, unversioned base.
NGWMN_OGC_API_URL = f"{BASE_URL}/ngwmn/ogcapi"
Expand Down Expand Up @@ -72,15 +75,15 @@


def _get(service: str, local_vars: dict[str, Any]) -> tuple[pd.DataFrame, BaseMetadata]:
"""Marshal a getter's arguments and dispatch to the shared OGC engine.
"""Marshal a getter's arguments and dispatch to the shared OGC facade.

Every NGWMN getter ends with this same call; centralizing it keeps the
NGWMN base URL, output id, and dialect wired up in exactly one place.
"""
queryable = _STATE_QUERYABLE.get(service)
if queryable is not None:
apply_state(local_vars, to=queryable["to"], into=queryable["into"])
args = _get_args(local_vars)
args = prepare_request_args(local_vars)
return get_ogc_data(
args,
service,
Expand Down
27 changes: 26 additions & 1 deletion dataretrieval/ogc/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,26 @@
"""Generic OGC API engine shared by the Water Data and NGWMN getters."""
"""Generic OGC API engine shared by the Water Data and NGWMN getters.

The public facade exposes only the minimal service-adapter seam:

- :class:`OgcDialect` — per-API request/response quirks.
- :func:`prepare_request_args` — normalize caller kwargs for the engine.
- :func:`get_ogc_data` — full orchestrated OGC fetch (chunking + pagination).
- :func:`fetch_ogc_request` — execute a pre-built request with pagination.

Service adapters (NGWMN, Water Data's generic wrapper) import from this
facade rather than reaching into engine internals. The engine module remains
available for lower-level orchestration needs (e.g. ``_paginate``,
``_run_sync``) that sibling modules like ``wateruse`` use under the accepted
temporary variance.
"""

from dataretrieval.ogc.engine import fetch_ogc_request, get_ogc_data
from dataretrieval.ogc.policy import OgcDialect
from dataretrieval.ogc.requests import prepare_request_args

__all__ = [
"OgcDialect",
"fetch_ogc_request",
"get_ogc_data",
"prepare_request_args",
]
6 changes: 3 additions & 3 deletions dataretrieval/ogc/chunking.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@
import pandas as pd
from anyio.from_thread import start_blocking_portal

from dataretrieval.utils import HTTPX_DEFAULTS, Ambient, _require_positive_int
from dataretrieval.utils import HTTPX_ASYNC_DEFAULTS, Ambient, _require_positive_int

from . import progress as _progress
from .combining import (
Expand Down Expand Up @@ -600,7 +600,7 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]:
The semaphore, not the pool, is deliberately the throttle. If the
pool throttled instead, the excess sub-requests would queue
*inside* httpx waiting for a connection, and that wait counts
against the pool-acquire timeout (60 s, from ``HTTPX_DEFAULTS``).
against the pool-acquire timeout (60 s, from ``HTTPX_ASYNC_DEFAULTS``).
A batch of slow pages that keeps every connection busy past that
window would then trip ``httpx.PoolTimeout`` on the queued tail —
a purely client-side failure that consumes the retry budget and
Expand Down Expand Up @@ -650,7 +650,7 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]:
self.plan.total if max_concurrent is None else max_concurrent
)

async with httpx.AsyncClient(limits=limits, **HTTPX_DEFAULTS) as client:
async with httpx.AsyncClient(limits=limits, **HTTPX_ASYNC_DEFAULTS) as client:
with _chunked_client(client):
reporter = _progress.current()
if reporter is not None:
Expand Down
Loading