mosaic: read tiles in place from S3, in a process pool; fix add_to_band

[SmithB]

For mosaicking ATL1415 tiles that live on S3 (MAAP DPS jobs read them in place; nothing is copied to /home).

Changes

• io_utils.glob_remote(pattern): the remote glob.glob, returning sorted URIs.
• make_mosaic.py: --directory may be a URI (listed with glob_remote, tiles read in place); --output must then be a local absolute path. New --block_size (default for remote tiles: DEFAULT_REMOTE_BLOCK_SIZE) and -j/--workers.
• grid.mosaic.from_list(block_size=None, workers=1):
  • block_size now reaches from_h5/from_nc for remote files. Without it, s3fs's 50 MiB default reads most of each tile for a few fields: 4 tiles pulled ~1 GB.
  • workers > 1 reads files in a process pool using forkserver. Forking after s3fs has started fails ("This class is not fork-safe"). Threads give no speed-up, because h5py's lock serializes the reads (measured).
  • Tiles are still added in list order, at most 2 × workers ahead, so output is identical to the serial read.
  • workers=1 with no block_size is the original code path.
• Fix, add_to_band: item[:,:,in_band] on an in-memory mosaic returns a plain grid.data (via grid.data.__copy__), which then failed on update_spacing. It now converts back with mosaic().from_grid(), as the neighbouring grid.data branch already did. This affected from_list(by_band=True) with any 3-D in-memory mosaic input.

Measured

557 ATL1415 GL-north tiles on S3, run on the MAAP ADE:

run	serial	parallel	output
40 km average	357 s	44.5 s (8 workers)	bit-identical
z0, weighted	802 s	195 s (4 workers)	bit-identical

Each worker costs about 0.35 GiB resident. The ADE container is capped at 7.3 GiB, and 8 workers on z0 exceeded it.

Tests

tests/test_mosaic_list.py:

• every read option against the original path, and remote reads through fsspec's memory filesystem
• block_size reaching open_remote
• make_mosaic.py on a remote directory, and its output check
• the add_to_band regression, which fails on main

Suite: 306 passed, 3 skipped.

🤖 Generated with Claude Code