Skip to content

Continuous index-trajectory change detection (Sentinel-2 + BFAST) (#30)#33

Merged
NewGraphEnvironment merged 16 commits into
mainfrom
30-continuous-index-trajectory-change-detec
Jul 9, 2026
Merged

Continuous index-trajectory change detection (Sentinel-2 + BFAST) (#30)#33
NewGraphEnvironment merged 16 commits into
mainfrom
30-continuous-index-trajectory-change-detec

Conversation

@NewGraphEnvironment

@NewGraphEnvironment NewGraphEnvironment commented Jul 9, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a continuous change-detection pipeline alongside the categorical one (v0.3.0; R CMD check clean, 0/0/0):

  • dft_stac_cube() — fetch a cloud-masked monthly spectral-index stack from Sentinel-2 (new role-based "sentinel-2-l2a" source in dft_stac_config()). Growing-season months filter, snow masking, eo:cloud_cover scene pre-filter + pagination, and a baseline-conditional offset split at 2022-01-25 so multi-year series carry no false step. Returns a terra SpatRaster stack.
  • dft_index_expr() / dft_index_table() — table-driven spectral-index registry (NDVI, kNDVI, NDMI) written over band roles, with reflectance scale/offset folded into the expression.
  • dft_rast_break() — per-pixel bfast::bfastmonitor() reduction (forked mclapply) to a two-band break-date / break-magnitude raster (negative = index drop = scour / veg loss).
  • Vignette trajectory-break-detection.Rmd on the Neexdzii Kwa reach, loading a 132 KB committed artifact (data-raw pattern; no network/bfast at build).

Empirical findings that reshaped the plan

Validated end-to-end on real Sentinel-2; three things could only be settled by running code:

  1. gdalcubes runs the reduce_time R-callback in spawned workers where closures fail — ultimately mooted by the terra pivot.
  2. gdalcubes::filter_geom() segfaults / yields an all-NA cube on the pinned build → dropped; cube spans the AOI bbox, callers clip the result (AOI-polygon clip deferred to dft_stac_cube: restore AOI-polygon clip (filter_geom segfault workaround) #32).
  3. The Planetary Computer +1000 DN offset only applies from 2022-01-25; a uniform offset produced a false whole-AOI break at the boundary (99% of pixels). Fixed with the baseline-conditional split — per-year kNDVI now aligns across 2022 and breaks fall to 25% of pixels (real change). Independently confirmed by a peer validation session.

Related Issues

Test plan

  • devtools::test() — 286 pass (network/bfast-gated tests skip)
  • R CMD check — 0 errors / 0 warnings / 0 notes
  • lintr::lint_package() clean
  • Vignette renders from the committed artifact (no network/bfast at build)
  • Offset split validated on real 2018-2023 Sentinel-2 (kNDVI aligns across 2022; breaks 25% not 99%)

Notes

10 feature commits + PWF baseline; per-commit fresh-eyes code-checks. Archived planning: planning/archive/2026-07-issue-30-index-trajectory/.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DKqnW6Fni3tc44Lo5D5fJz

NewGraphEnvironment and others added 16 commits July 8, 2026 16:51
@NewGraphEnvironment @claude
Initialize PWF baseline for #30
6da759c 
Continuous index-trajectory change detection (Sentinel-2 + BFAST).
Four new exports across a config restructure, index registry, cube fetch mode,
and a bfast reducer, plus a network-decoupled vignette. One branch, 0.3.0 release.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Add role-based STAC config schema + Sentinel-2 cube source (#30)
aa55171 
Restructure dft_stac_config() so cube-capable sources carry a role->asset
map (red/nir/swir16/mask), mask values, and reflectance scale/offset, marked
with cube = TRUE. Categorical sources (io-lulc, esa-worldcover) keep their
flat 4-field shape verbatim (backward compat, pinned by expect_named); the
cube marker is absence = FALSE. Add sentinel-2-l2a (Planetary Computer band
names B04/B08/B11/SCL). Guard dft_stac_fetch() to reject cube sources with a
pointer to dft_stac_cube().

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Add index registry + dft_stac_cube() index-cube fetch (#30)
96076df 
Phase 2 (index registry): inst/indices/indices.csv holds spectral-index
formulas over band roles in gdalcubes tinyexpr C syntax (pow(), not ^).
dft_index_table() reads it (mirrors dft_class_table); dft_index_expr()
resolves roles to per-source assets, folds the reflectance scale/offset into
each band token (a non-zero offset does not cancel in a ratio index), and
applies the expression to a cube.

Phase 3 (cube fetch): dft_stac_cube() is the trajectory sibling of
dft_stac_fetch() -- STAC query with intersects (unioned AOI) + eo:cloud_cover
scene pre-filter + items_fetch pagination, an explicit AOI-bounded cube_view,
SCL cloud masking, the index applied, and the masked index cube materialized
to <source>/cube_<key>.nc keyed by every cube-affecting parameter. Returns the
lazy cube for dft_rast_break().

Fresh-eyes code-check on each: the cube's intersects used only the first AOI
feature (silent NoData holes for multi-feature AOIs) -- fixed by unioning.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Add dft_rast_break() per-pixel trajectory breakpoint reducer (#30)
18e9995 
Reduces an index cube over time with bfast::bfastmonitor(), returning a
two-band SpatRaster of break date (decimal year) and magnitude (negative =
index drop = scour / veg loss).

Empirical gate (findings.md): the gdalcubes reduce_time R callback runs in a
spawned worker at EVERY parallel setting, and a closure over enclosing locals
fails there ("object 'band' not found"). The reducer is therefore built
self-contained -- build_break_reducer() embeds the testable .dft_break_pixel()
helper as a literal object (env detached to baseenv) and inlines every scalar
parameter via substitute(), with load_pkgs="bfast" -- proven correct at
parallel 1 and 2. ts start / seasonal frequency are derived from the cube time
axis in the main process; a caller frequency that disagrees with the cadence
errors. Degenerate pixels (all-NA / < min_obs) short-circuit before any bfast
call, so those paths test with no bfast dependency.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Fix dft_stac_cube() all-NA cube from filter_geom (#30)
c6953a0 
The real-Sentinel-2 end-to-end run produced an entirely-NA kNDVI cube (and
intermittently segfaulted the gdalcubes compute worker). Isolated to
gdalcubes::filter_geom(): stepping through the pipeline, the raw bands, SCL
masking, and the kNDVI index all yield valid data (kNDVI in 0..1) WITHOUT
filter_geom, and the proven sibling dft_stac_fetch() works on the same AOI.

Drop the in-pipeline polygon clip. The cube now spans the AOI bounding box
(cloud-masked); callers clip the reduced raster with terra::mask(), matching
how dft_stac_fetch() masks after reading. Verified end-to-end: 48-month cube
has valid kNDVI throughout and dft_rast_break() reduces it (no segfault) to a
plausible break raster (dates in the 2022-2023 monitoring window).

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Add growing-season tuning: months filter, snow mask, order, /vsicurl …
61eaa5d 
…config (#30)

For high-latitude floodplains the all-months fetch was slow (~30 min for 100 ha
-- the Sentinel-2 streaming, not the compute) and injected winter snow/low-sun
noise that has no vegetation signal. Focus the analysis on the growing season:

- dft_stac_cube() gains `months` (calendar-month filter, default NULL/all).
  Fetching only e.g. June-September streams ~3x fewer scenes AND sharpens the
  signal; empty months become NA so the monthly series stays regular at
  frequency 12 for bfast.
- Snow/ice (SCL 11) added to the default Sentinel-2 mask.
- A GDAL /vsicurl config block (DISABLE_READDIR_ON_OPEN etc.), restored on exit,
  trims per-COG open latency.
- dft_rast_break() gains `order` (bfastmonitor harmonic order); lower it for a
  growing-season-only series that samples a narrow part of the annual cycle.

Both cache keys extended with the new params. Growing-season E2E (months=6:9,
2018-2023): 12 min vs 30, 24 clean summer observations per pixel, breaks dated
to summer months.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Split Sentinel-2 at the baseline offset boundary; reduce via terra (#30)
34ecbd4 
The Planetary Computer +1000 DN reflectance offset only applies from processing
baseline 04.00 (2022-01-25 on); earlier scenes have no offset. A uniform
offset was therefore wrong for pre-2022 scenes and produced a false
whole-AOI index step at the boundary (validated: 99% of pixels showed a
negative "break" dated to exactly the boundary -- an artifact, not vegetation).

Fix (baseline-conditional split):
- dft_stac_config() sentinel-2-l2a gains offset_boundary = "2022-01-25" and
  offset_before = 0.
- dft_stac_cube() splits items at the boundary, corrects each side with its own
  offset, and coalesces onto the shared monthly grid. It now returns a terra
  SpatRaster stack (materialized GeoTIFF, time per layer) instead of a gdalcubes
  cube -- gdalcubes cannot read a terra-written NetCDF, so the coalesce lives in
  terra.
- dft_rast_break() reduces the stack per pixel via parallel::mclapply (forked
  workers inherit the namespace and closures, so the whole gdalcubes reduce_time
  worker-serialization machinery -- build_break_reducer, load_pkgs -- is gone;
  .dft_break_pixel is unchanged). ts start / frequency derive from the layer
  times.

Validated end-to-end on 2018-2023 growing-season Sentinel-2: split fired
75 pre / 85 post, per-year kNDVI now aligns across 2022 (2021=0.52, 2022=0.53),
and breaks fell from 99% to 25% of usable pixels -- real change, not the
boundary artifact. Addresses the offset half of #32.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Note egress cost (free) on cube sources in dft_stac_config (#30)
42aae75 
Flag that the Sentinel-2 (and Landsat template) cube sources read free
Planetary Computer / Azure Open Data with no egress charge, prompting anyone
adding a new source to check egress terms before shipping a requester-pays one.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Add trajectory-break vignette + committed artifact (#30)
aa9b6b9 
vignettes/trajectory-break-detection.Rmd demonstrates the continuous
index-trajectory pipeline on the same Neexdzii Kwa reach as the land-cover
vignette. Following drift's data-raw pattern, data-raw/vignette_data_break.R
does the network + bfast work (growing-season Sentinel-2 kNDVI 2018-2023 with
the baseline-conditional offset split, reduced to break date + magnitude) and
saves a small committed artifact (inst/testdata/neexdzii_break.rds, 132 KB) so
every vignette chunk runs live under pkgdown with no network or bfast.

The vignette shows the break-magnitude map (negative = scour / veg loss), the
break-year map, and scour-vs-stable pixel kNDVI trajectories with the detected
break marked. Renders clean.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Clean up R CMD check NOTEs (#30)
21e0de4 
- .Rbuildignore: exclude planning/ and .claude/ from the built package (they
  belong in git for the SRED trail but are not package sources).
- .gitignore: data-raw/.break_cache/ (local STAC cube cache).
- Declare the rlang .data pronoun as a global (pre-existing note in
  dft_rast_classify's apply_remap), alongside eo:cloud_cover.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Release v0.3.0
c145899 
Continuous index-trajectory change detection (Sentinel-2 + BFAST) for
floodplain reaches (#30). R CMD check: 0 errors / 0 warnings / 0 notes.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Archive planning files for issue #30
b3748ed 
Continuous index-trajectory change detection shipped as v0.3.0. See the archive
README for the outcome and the three empirical findings that reshaped the plan
(worker-closure failure, filter_geom segfault, Sentinel-2 offset boundary).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Reopen PWF for trajectory vignette redo (#30, continued)
c918491 
Continuation of the archived #30 cycle (v0.3.0 / PR #33). Refines the trajectory
vignette example to 2017-2023 (apples-to-apples with the LULC vignette) and to
surface the known 2022-2023 logging cut, which the shipped 2018-2023 /
monitor-from-2022 example did not (region-wide 2023 dip dominated). PWF headers
carry the prior journey forward and reference the archive.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Rework trajectory vignette: 2017-2023, LULC-comparable (#30)
5437dcf 
Redo of the trajectory vignette example so it is apples-to-apples with the
land-cover vignette (same Neexdzii Kwa reach, same 2017-2023 window) and
actually surfaces the logging the shipped 2018-start / monitor-from-2022
example buried under a region-wide 2023 dip.

- data-raw/vignette_data_break.R: fetch 2017-2023 growing-season kNDVI (offset
  split), monitor from 2022 with 2017-2021 history (longer baseline -> 13k
  breaks vs 25k), and precompute a LULC ground-truth grouping (Trees ->
  Rangeland/Bare vs intact forest vs background) + agreement stats into a
  140 KB artifact.
- vignette reframed as the complement to LULC: LULC says what and roughly when
  (annual snapshots), the trajectory says exactly when (monthly). Shows the
  break map with LULC tree-loss outlined, an agreement table (tree-loss stands
  break 2x as often, 2.7x stronger, dated summer 2023), and the grouped
  trajectory -- with the basin-wide 2023 dip honestly acknowledged.

Vignette re-builds clean under R CMD check; 286 tests pass. Lands on PR #33.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Add dft_rast_trend() + reframe vignette as a QA check on land cover (#30
b5dcbf3 
)

Reconciling why the bfast trajectory disagreed with LULC's dramatic
Trees->Rangeland *inside* the floodplain revealed the tool's real value for the
project goal (floodplain veg change to justify leave-standing / restoration):
the LULC "tree-loss" pixels were borderline (kNDVI baseline 0.43 vs 0.50 intact,
median drop 0.037, only 2% a real crash). On this deciduous-riparian reach the
categorical product overstates forest loss (label flips, not canopy removal),
and the continuous trajectory's job is to QA that and to catch gradual
degradation/recovery the annual labels cannot.

- dft_rast_trend(): new export. Per-pixel robust Theil-Sen slope (index/yr) +
  Mann-Kendall significance over the full stack, via forked mclapply like
  dft_rast_break. Resistant to a single anomalous season. Guards for <2 layers
  / min_obs<2. Code-check verified the statistics.
- Vignette rebuilt as "Trajectories as a Check on Land-Cover Change": a
  degradation/recovery trend map (floodplain is stable-to-greening), a QA table
  showing the mapped tree-loss carries a median trend of ~0 and only ~6%
  significant decline (a caution on the area numbers), the grouped trajectory,
  and an honest "when to use which" (strong on evergreen/abrupt/gradual, weak at
  deciduous-riparian-vs-grass). Fixed an inverted map palette in self-review.

302 tests pass; R CMD check clean (bar a spurious future-timestamp NOTE); lint
clean. Lands on PR #33.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment @claude
Reframe trajectory vignette (patch-level, complementary) + interactiv…
ac3ada0 
…e map (#30)

Rework the vignette from patch-level ground truth: real cutblocks in the
floodplain coincide with red patches in the break/trend maps, and some red
falls outside the LULC Trees->Rangeland polygons -- the continuous trajectory
and the categorical map catch different real change, so they are complementary,
not one correcting the other.

- data-raw: derive the actual LULC Trees->Rangeland transition polygons
  (dft_rast_transition + dft_transition_vectors, filtered to the loss
  transition, not stable Trees->Trees) for the map + overlap check; drop the
  aggregate QA table and averaged group trajectory (whole-floodplain averaging
  hides the patchy signal).
- vignette: neutral methods-first framing; break-magnitude map with the
  transition polygons outlined (red in/out/absent = confirmed / LULC-missed /
  unsupported); trend map relabelled per-year 2017-2023; and an interactive
  leaflet mirroring the land-cover vignette (Light/Esri/Google basemaps, AOI
  centering) with toggleable break, trend, and beige Trees->Rangeland layers to
  inspect the cutblocks on satellite imagery.

R CMD check clean (bar the spurious future-timestamp NOTE); vignette builds;
302 tests pass; lint + code-check clean. Lands on PR #33.

Relates to NewGraphEnvironment/sred#16

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NewGraphEnvironment NewGraphEnvironment merged commit f126422 into main Jul 9, 2026
1 check passed
@NewGraphEnvironment NewGraphEnvironment deleted the 30-continuous-index-trajectory-change-detec branch July 9, 2026 16:30
NewGraphEnvironment added a commit that referenced this pull request Jul 9, 2026
@NewGraphEnvironment @claude
Archive planning files for #30 vignette QA/interactive-map cycle
6ba10bb 
Continuation cycle that reworked the trajectory vignette (QA framing +
dft_rast_trend + interactive map) and shipped v0.3.0 (PR #33). See the archive
README for the outcome and the categorical-vs-continuous reconciliation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Continuous index-trajectory change detection (Sentinel-2 + BFAST) for floodplain reaches

1 participant

@NewGraphEnvironment