Tile dft_stac_fetch to bound download over sparse-floodplain bounding boxes (#36)#39
Open
NewGraphEnvironment wants to merge 7 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Phase 1 of #36: the offline, network-free core of bounding dft_stac_fetch()'s download to the AOI footprint. tile_size_check() validates a tile_size and snaps it to a multiple of res (so tile pixel grids align for a seam-free merge); tile_grid() splits the AOI bbox into a res-aligned grid anchored at the lower- left and keeps only tiles intersecting the AOI polygon — a thin corridor fetches near its footprint, not its full bbox. Boundary tiles are left un-trimmed (the overhang is masked away; trimming would break res-alignment). Tests written first (validation, snapping, intersecting-subset, alignment, single-tile, degenerate-AOI); 40 pass / 1 skip; lint clean; code-check clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Phase 2 of #36: stac_cache_key() gains an optional tile_size, appended to the hash only when non-NULL. An untiled fetch (tile_size = NULL) reproduces the exact pre-tiling 9-element hash, so existing io-lulc caches stay valid on upgrade; a tiled fetch keys distinctly, so its terra .tif mosaic is never served as an untiled gdalcubes .nc (or vice versa). A frozen golden-hash regression guards the legacy-preservation invariant. tile_size arrives already snapped by the caller (single normalization site — no gate/key desync). 45 pass / 1 skip; lint + code-check clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
#36) Phase 3 of #36: pull the cube_view + raster_cube + write_ncdf block out of the per-year loop into fetch_extent_to(). The untiled path now routes through it, writing straight to the existing <yr>_<key>.nc cache — identical filename, format, and I/O to before (faithful code-motion; cube_view built identically). Sharing this primitive is what will guarantee each tile fetches identically to the corresponding slice of the untiled cube when the tiled branch lands in Phase 4. 45 pass / 1 skip; lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Phase 4 of #36: tiled fetch end-to-end. When tile_size is set, dft_stac_fetch() splits the AOI bbox into a res-aligned grid, streams only tiles that intersect the AOI polygon (via the shared fetch_extent_to()), mosaics them with terra::merge(terra::sprc(...)) (new mosaic_tiles() helper), and caches a terra GeoTIFF — so a thin corridor fetches near its footprint instead of its full bounding box. Extension routes on is.null(tile_size) (.nc untiled / .tif tiled), matching the distinct cache key. tile_size is normalized once (snap to res multiple) so the path gate and cache key agree; GDAL /vsicurl config (mirroring dft_stac_cube) is set with an on.exit restore, scoped to the tiled path, since tiling multiplies per-item COG opens. Default tile_size = NULL is unchanged. Offline merge oracle (byte-for-byte reassembly, single-tile, .tif round-trip) and an opt-in DRIFT_TEST_NETWORK e2e (tiled == untiled over the in-AOI overlap, extension routing). 352 pass / 5 skip; document + lint + code-check clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Phase 5 of #36. Document the download-tiling workaround in the gdalcubes gotchas note (the filter_geom-independent way to bound the read, cross-referencing the cube-path residual #38); add the NEWS 0.6.0 entry for tile_size; bump DESCRIPTION 0.5.0 -> 0.6.0. devtools::check() clean: 0 errors / 0 warnings / 0 notes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tile_sizetodft_stac_fetch()that bounds the STAC download to the AOI footprint. By default one gdalcubes cube is streamed over the whole AOI bounding box, so a thin, diagonal floodplain corridor downloads ~10× more pixels than the polygon needs (measured 10.1% of bbox cells inside a real io-lulc floodplain). Whentile_size(CRS units) is set, the bbox is split into ares-aligned grid, only tiles intersecting the AOI polygon are streamed, and the results are mosaicked withterra::merge().filter_geom-independent path — the polygon clip that would do this in the cube pipeline segfaults on the pinned gdalcubes build (dft_stac_cube: restore AOI-polygon clip (filter_geom segfault workaround) #32). Tiled fetches cache a terra GeoTIFF (.tif) rather than a gdalcubes NetCDF (.nc) and key distinctly, so existing untiled caches are untouched;tile_size = NULL(default) is byte-for-byte the previous behavior.tile_grid()+tile_size_check()(offline), a conditional cache-key append with a frozen golden-hash regression, extraction of the sharedfetch_extent_to()primitive, the tiled branch (extension split + GDAL/vsicurlconfig scoped to the tiled path), and release docs.Related Issues
dft_stac_cube()path stays open as Bound dft_stac_cube() streaming to the AOI (fetch-time read still bbox-bound after #32) #38Test plan
devtools::test()— 352 pass / 5 skip / 0 fail (network + bfast tests opt-in)devtools::check()— 0 errors / 0 warnings / 0 notes; vignettes rebuildlintrclean; roxygen regenerated.tifround-trip); cache-key golden regression preserves the legacy untiled keyDRIFT_TEST_NETWORK=true): tiled == untiled over the in-AOI overlap;.nc/.tifextension routing — runs only with network + Planetary Computer accessNotes
dft_stac_cube()); the cache key keys the two formats apart.resand anchored at the bbox lower-left so per-tile grids are co-lattice — otherwise the merge seams. Boundary tiles are left un-trimmed (overhang masked away).tile_sizeis normalized once so the path gate and the cache key derive from the same snapped scalar (no gate/key desync).🤖 Generated with Claude Code