MAAP credential broker: retry, a clear error, and no global warnings filter

[SmithB]

Why

On 2026-10-01, 556 ATL14/15 jobs were submitted to MAAP DPS at once. 61 of them failed with

AttributeError: 'NoneType' object has no attribute 'get_s3_filesystem'

What happened in each: the one call to MAAP's credential broker for NSIDC failed (the fit step sat for a median of 136 s at ~0 CPU first, which matches a connect timeout to the MAAP API), get_s3fs fell back to earthaccess, and earthaccess has no login on a worker. The warning that carried the broker's actual error was never printed, because ps_scale_for_lat.py calls warnings.filterwarnings("ignore") at import, which silences every warning in the process.

What changes

• io_utils._s3fs_from_maap: the broker call is tried 5 times, 10 s apart (MAAP_BROKER_ATTEMPTS, MAAP_BROKER_PAUSE_S) before giving up.
• io_utils.get_s3fs: if the broker gave up and the earthaccess fallback also fails, raise a RuntimeError that names both the broker's last error and the earthaccess error. Where earthaccess does have a login (the ADE), the fallback still serves as before. Off MAAP nothing changes.
• ps_scale_for_lat: the module-level filter is removed. The function's own warnings (divide by zero at the pole, all-NaN input) are silenced where they arise; its values are unchanged.

Worth knowing before merging

• Removing the global filter means warnings from anywhere in a process that imports pointCollection are visible again. Logs will be noisier, and anything run with warnings-as-errors may now fail on warnings that were hidden.
• The retry adds up to 40 s of pauses when the broker is down, plus however long each failed call takes (about 130 s each for a connect timeout, since maap-py sets no timeout).

Tests

tests/test_maap_broker.py (12 tests, no network; stand-ins for maap-py and earthaccess). They fail on main and pass here. Full suite: 319 passed, 3 skipped.

🤖 Generated with Claude Code