Skip to content

Single-pass patch vectorization (fixes OOM on large-extent rasters)#29

Merged
NewGraphEnvironment merged 5 commits into
mainfrom
27-dft-transition-vectors-ooms-on-large-ext
Jul 7, 2026
Merged

Single-pass patch vectorization (fixes OOM on large-extent rasters)#29
NewGraphEnvironment merged 5 commits into
mainfrom
27-dft-transition-vectors-ooms-on-large-ext

Conversation

@NewGraphEnvironment

Copy link
Copy Markdown
Owner

Summary

  • Fix OOM in dft_transition_vectors() on large-extent rasters: the per-class loop allocated full-grid vectors per class and two full-grid logicals per patch (ncell × n_patches scaling — ~18+ GB extrapolated on the 102.6M-cell UFRA floodplain). Replaced with a single terra::patches(x, directions = 8, values = TRUE) pass plus a sparse patch→label map that touches only the ~2% non-NA cells.
  • Output is identical to the old implementation — pinned by a regression test whose reference numbers (185 patches / 123.11 ha / 57 at patch_area_min = 1000) were captured from the old code before the rewrite. Only patch_id numbering / row order changes (now raster scan order).
  • terra floored at >= 1.8-10: older versions had an edge-wraparound bug in patches(values = TRUE) that silently merged patches touching opposite raster edges.
  • Benchmark at 24M cells: 1.9 s for a 4,799-patch raster vs 122 s for the old code on a milder 1,232-patch raster. Reporter can drop the column-tiled stop-gap from the per-area pipeline driver.
  • Follow-up filed as dft_rast_transition builds 6+ full-grid vectors (incl. two character) — same OOM class as #27 #28: dft_rast_transition() has the same disease (6+ full-grid vectors incl. two character vectors) and needs its own refactor.

Fixes #27

Related Issues

Test plan

  • devtools::test(): 211 pass, 0 fail — all 10 pre-existing tests unchanged (behavior contract), plus new brute-force decomposition test (engineered topology: diagonal merges, class boundaries, crossed diagonals, code-0 class, holes), all-NA edge case, and fixture regression guard
  • lintr clean on touched files (one pre-existing vignette lint, untouched)
  • 24M-cell synthetic benchmark against the final implementation: 1.9 s
  • /code-check clean on each commit

Notes

Approach was selected empirically during planning — the issue's tiling and sparse-window options benchmarked worse (sparse-window degenerates on mainstem-shaped data where every class's bounding window spans the grid) and as.polygons+explode was disqualified (GDAL polygonize is 4-connected, silently splits diagonal joins). Full analysis in planning/archive/2026-07-issue-27-transition-vectors-oom/.

🤖 Generated with Claude Code

NewGraphEnvironment and others added 5 commits July 7, 2026 09:27
dft_transition_vectors() looped over transition classes, allocating
full-grid vectors per class and two full-grid logicals per patch —
ncell x n_patches scaling that OOM-killed a 102.6M-cell, 56-class
floodplain (1.87% non-NA; trim() useless when data follows the mainstem
across the whole bbox).

Replace the loop with terra::patches(x, directions = 8, values = TRUE)
— one C++ pass computing 8-connected components of same-valued cells —
then as.polygons() and a sparse pid-to-label map that touches only the
non-NA cells. Output verified identical on fixtures (185 patches,
123.11 ha, 57 at patch_area_min = 1000; pinned as a regression test
captured from the old implementation). Benchmark at 24M cells:
1.9 s vs 122 s. patch_id numbering is now scan-order.

terra floor >= 1.8-10: earlier versions falsely connected patches
across the left/right raster edges with values = TRUE.

Fixes #27

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Relates to #27

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@NewGraphEnvironment NewGraphEnvironment merged commit 228c84e into main Jul 7, 2026
1 check passed
@NewGraphEnvironment NewGraphEnvironment deleted the 27-dft-transition-vectors-ooms-on-large-ext branch July 7, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dft_transition_vectors OOMs on large-extent rasters (processes full grid per class)

1 participant