Skip to content

test: make the suite offline by default, trim fixtures and snapshots - #359

Merged
thodson-usgs merged 1 commit into
DOI-USGS:mainfrom
thodson-usgs:test/minimize-live-tests
Aug 7, 2026
Merged

test: make the suite offline by default, trim fixtures and snapshots#359
thodson-usgs merged 1 commit into
DOI-USGS:mainfrom
thodson-usgs:test/minimize-live-tests

Conversation

@thodson-usgs

Copy link
Copy Markdown
Collaborator

Summary

The test suite depended on USGS uptime. 46 tests queried the real services on every push, so an unrelated commit's CI could go red because a gage didn't report or a service returned a transient 503. Those tests also asserted on live data volumes — len(df) > 0, df.shape[1] == 97 — which is a claim about today's water, not about this package.

  • Mock the request/response wiring using real captured response shapes trimmed to two features. Property names, nesting, and value types are verbatim (including the numeric-looking strings the API really sends); only the row count is reduced. Assertions move to what we own: the outgoing request and the shape of the returned frame.
  • Keep 12 genuine contract tests behind a live marker, deselected by default via addopts = "-m 'not live'" in pyproject.toml and run nightly by a new live-api.yml workflow. A mock can't detect upstream drift, because the mock is the thing that would need updating.
  • Shrink the fixtures: tests/data/ goes 53 MB → 1.1 MB. The two 22 MB files existed only to satisfy assert df.size == 5087443-style magic numbers.
  • Replace two brittle snapshots: the character-exact str(inspect.signature(f)) snapshot in contracts/public_api_test.py and the frozen name sets in architecture_test.py. Both failed on renames, reorders, and annotation reflows — changes that break no caller — while the property-based checks that replace them still catch a new required argument, a stale re-export, or a family growing a second public home.

Why this is worth the diff

Writing the mocks surfaced seven wrong assumptions the live tests could not catch, because a test that only asserts "some rows came back" passes regardless of whether the request was built correctly. The clearest case: test_get_peaks_water_year_filter asserted only that returned rows fell inside the requested years — which an empty frame satisfies, so it could never fail. Others included the wrong skipGeometry casing, a CQL2 body envelope that doesn't exist, and a multi-site fan-out that actually comma-joins into one request.

Results

before after
default suite 46 tests hitting live APIs 761 passed, 0 network calls, 5.6 s
tests/data/ 53 MB 1.1 MB
upstream drift invisible (mocks would be updated) 12 live tests, nightly

Network isolation was verified empirically with a socket-blocking pytest plugin: the only test that opens a socket is test_fan_out_outlives_pool_timeout_on_real_transport, which binds a localhost server on purpose (a MockTransport bypasses the connection pool it's testing).

Test plan

  • pytest tests/ — 761 passed, 12 deselected, 5.6 s
  • Same run under a socket-blocking plugin — one offender, the intentional localhost test
  • pytest tests/ -m live — 12 passed against the real APIs
  • ruff check / ruff format --check / mypy clean; all pre-commit hooks pass
  • Mutation-checked the replacement snapshots: adding a required argument to get_queryables and making a family re-export another family's getter both fail as intended
  • Confirm the nightly live-api.yml schedule fires once merged to the default branch

🤖 Generated with Claude Code

The suite depended on USGS uptime. 46 tests queried the real services on
every push, so an unrelated commit's CI could go red because a gage did not
report or a service returned a transient 503. Those tests also asserted on
live data volumes (``len(df) > 0``, ``df.shape[1] == 97``), which is a claim
about today's water, not about this package.

Mock the request/response wiring instead, using real captured response shapes
trimmed to two features -- property names, nesting, and value types verbatim,
only the row count reduced. The assertions move to what we own: the outgoing
request and the shape of the returned frame. Doing this surfaced seven wrong
assumptions the live tests could not catch, including a water-year filter test
that an empty frame satisfied and so could never fail.

Keep the 12 tests whose subject genuinely is the upstream API behind a ``live``
marker, deselected by default and run nightly. A mock cannot detect upstream
drift, because the mock is the thing that would need updating.

Also shrink the test fixtures (53 MB -> 1.1 MB) and replace two brittle
snapshots. The 22 MB fixtures existed to satisfy ``df.size == 5087443``
assertions; the signature and export snapshots failed on renames and reflows,
which break no caller, while the property-based checks that replace them still
catch a new required argument or a family growing a second public home.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thodson-usgs
thodson-usgs marked this pull request as ready for review August 7, 2026 20:20
@thodson-usgs
thodson-usgs merged commit 316939b into DOI-USGS:main Aug 7, 2026
11 checks passed
@thodson-usgs
thodson-usgs deleted the test/minimize-live-tests branch August 7, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant