Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Experimental package — breaking all the time and loving the learning curve. St
**Prefix:** `lnk_`
**Branch:** `main` (v0.44.0 as of 2026-07-03)

## Status (2026-07-04) — v0.44.1 shipped (#226 vignette accessible_km)

Extended the PARS vignette with an **Accessible habitat (km)** section proving `accessible_km` bcfp-equivalence (link 6,822.5 vs bcfp 6,822.9 km BT, **−0.01%**; table from cached `inst/vignette-data/pars_accessible.rds`). **Gotcha that bit hard:** "regenerate the vignette artifacts" was NOT docs-only — the two persist configs drift in segmentation because only WSGs re-modelled post-#223 are dense. `fresh` (bcfp config) had PARS at 97,538 segs but `fresh_default` (default/grayling) was still pre-#223 (48,558); the gpkg's single `streams` layer joins `fresh` geometry to `fresh_default` `mapping_code_gr` on `id_segment`, so a naive regen attaches grayling tokens to mismatched geometry → corrupt GR map. Fix: re-model the lagging config (`data-raw/wsg_run_one.R` + `merge=TRUE` recompute via `wsg_recompute_one.R` for cross-WSG `;DAM`) so both share segmentation; `wsg_vignette_data.R` now carries a **segmentation-parity guard** that refuses a mixed build. Any cross-config artifact joined on `id_segment` must verify both sides share segmentation first. mapping_code parity refreshed 99.04%→98.91% (denser post-#223). Open follow-ups unchanged: **#224**, **#225**, **#227**.

## Status (2026-07-03) — v0.44.0 shipped (#221 + #223 accessible_km)

Fixed the BT/ST `accessible_km` over-credit: streams now break at **every** gradient frontier (`lnk_pipeline_prepare.R` unions the raw per-model positions into `gradient_barriers_minimal`, not the `frs_barriers_minimal` reduction) — matching bcfp. Added the `accessible_km` roll-up column + `lnk_rollup_wsg()` (#221). Proven across 11 WSGs × 8 species: `accessible_km` 44/44 within 0.05%, habitat holds (parked BULK SK = fresh#190). Validator/proof: `data-raw/parity_crosssection.R` + `research/parity_accessible_habitat_2026_07_03.md`. **Gotcha that bit hard:** `fresh.streams_vw_bcfp` spawning/rearing/access_<sp> are coded 0/1/2/3 → parity uses `IN (1,2)` (a `= 1` under-counts; see memory `bcfp-vw-column-coding`). Segment count now 2–3.5× (bcfp-matching; intersects #205). Open follow-ups: **#225** (rename `gradient_barriers_minimal` → `gradient_barriers_break`), **#226** (vignette accessible_km demo), **#227** (`wsg_outlet` builder + single-WSG guard; relates to #222), **#224** (bcfp `dam_dnstr_ind` reservoir-inflow quirk — reference-side, not ours).
Expand Down
4 changes: 2 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Package: link
Title: Stream Network Habitat Interpretation (Experimental)
Version: 0.44.0
Date: 2026-07-03
Version: 0.44.1
Date: 2026-07-04
Authors@R: c(
person("Allan", "Irvine", , "airvine@newgraphenvironment.com",
role = c("aut", "cre"),
Expand Down
4 changes: 4 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# link 0.44.1

Extend the PARS vignette to demonstrate `accessible_km` bcfishpass-equivalence ([#226](https://github.com/NewGraphEnvironment/link/issues/226)). `vignettes/pars-habitat-connectivity.Rmd` gains an **Accessible habitat (km)** section: a link-vs-bcfishpass roll-up table (bull-trout accessible / spawning / rearing, from the new cached `pars_accessible.rds`) plus computed prose — link models 6,822.5 km of accessible bull-trout habitat against bcfishpass's 6,822.9 km, a −0.01% difference (the [#223](https://github.com/NewGraphEnvironment/link/issues/223) access-segmentation fix; spawning/rearing agree within the 5% habitat band). `data-raw/wsg_vignette_data.R` grows the accessible roll-up (mirroring `data-raw/parity_crosssection.R`'s `lnk_rollup_wsg()` + `fresh.streams_vw_bcfp` `IN (1,2)` computation) and a **segmentation-parity guard** that refuses to build when `fresh` and `fresh_default` disagree by more than 1%. All cached artifacts were regenerated from the post-#223 state: this required re-modelling the `default` (Arctic grayling) config for `PARS` into `fresh_default`, which had been left on the pre-#223 segmentation (48,558 segments vs `fresh`'s 97,538) — a naive regeneration would have joined the grayling `mapping_code` onto mismatched geometry and corrupted the map. The re-model (`data-raw/wsg_run_one.R` + a `merge = TRUE` recompute for cross-WSG `;DAM`) brings both configs onto the same segmentation; the previously hardcoded grayling map-caption counts are now computed inline from the loaded layer so they track the artifact. Bull-trout `mapping_code` parity refreshes to 98.91% (from 99.04%, on the denser post-#223 segmentation). Verified with a database-free knit.

# link 0.44.0

Fix a BT/ST `accessible_km` over-credit and add the per-WSG `accessible_km` roll-up ([#223](https://github.com/NewGraphEnvironment/link/issues/223), [#221](https://github.com/NewGraphEnvironment/link/issues/221)). The stream break source `gradient_barriers_minimal` was fed the `fresh::frs_barriers_minimal()` downstream-most reduction as a *segmentation* source, so a single stream segment straddled the accessibility frontier and its whole reach — including the blocked reach above the barrier — was credited accessible (`accessible_km` +40.4% PCEA, +23.6% FINA). The fix (`R/lnk_pipeline_prepare.R`) unions the **raw** per-model gradient + falls positions into `gradient_barriers_minimal` so streams break at every frontier — mirroring the orphan path already in the code and matching bcfishpass, which segments at every gradient barrier; `fresh::frs_barriers_minimal()` is now unused in link. `lnk_compare_rollup()` gains an `accessible_km` column and a reusable `lnk_rollup_wsg()` (tunnel-free vs `fresh.streams_vw_bcfp`, `access_<sp> IN (1,2)`). Proven with `data-raw/parity_crosssection.R` across 11 WSGs × 8 species (Peace / Fraser / Skeena / Columbia): `accessible_km` converges 44/44 within 0.05%, spawning/rearing hold — the only two over-tolerance being the documented parked BULK SK ([fresh#190](https://github.com/NewGraphEnvironment/fresh/issues/190) dual-rearing-lake topology). The fix is habitat-neutral (byte-identical spawn/rear pre/post); segment count grows 2–3.5× (inherent to bcfp-matching, intersects #205). Follow-ups: #224 (bcfp `dam_dnstr_ind` reservoir-inflow propagation), #225 (rename `gradient_barriers_minimal` → `gradient_barriers_break`), #226 (PARS vignette `accessible_km` demo), #227 (`public.wsg_outlet` builder + single-WSG guard).
Expand Down
50 changes: 50 additions & 0 deletions data-raw/wsg_vignette_data.R
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,62 @@ if (!persisted("fresh") || !persisted("fresh_default")) {
}
message("[wsg_vignette_data] authoritative ", aoi, " state present in fresh + fresh_default")

# --- segmentation-parity guard (link#226) ------------------------------------
# The gpkg's streams layer joins `fresh` geometry + mapping_code_bt to
# `fresh_default` mapping_code_gr on id_segment, so BOTH configs must sit on the
# SAME post-#223 segmentation. If one lags (e.g. fresh_default not re-modelled
# after the #223 access-segmentation fix), the GR token lands on the wrong
# geometry — refuse to build rather than ship a corrupt grayling map.
seg_count <- function(schema) DBI::dbGetQuery(conn, sprintf(
"SELECT count(*)::int n FROM %s.streams WHERE watershed_group_code = %s",
schema, DBI::dbQuoteLiteral(conn, aoi)))$n
.n_fresh <- seg_count("fresh")
.n_fdef <- seg_count("fresh_default")
if (abs(.n_fresh - .n_fdef) > 0.01 * max(.n_fresh, .n_fdef)) {
stop(sprintf(
paste0("%s segmentation mismatch: fresh %d vs fresh_default %d segments (>1%%). ",
"Re-model the lagging config post-#223 before generating vignette data ",
"(link#226) — the gpkg joins fresh geometry to fresh_default ",
"mapping_code_gr on id_segment."),
aoi, .n_fresh, .n_fdef), call. = FALSE)
}
message(sprintf("[wsg_vignette_data] segmentation parity OK: fresh %d / fresh_default %d segments",
.n_fresh, .n_fdef))

# --- tunnel-free parity (BT is the only bcfp-config species in the Peace) -----
parity <- lnk_compare_mapping_code(conn, aoi = aoi, cfg = cfg_bcfp)
saveRDS(parity, file.path(out_dir, paste0(stub, "_parity.rds")))
message("[wsg_vignette_data] parity rows: ", nrow(parity),
" (median match_pct = ", stats::median(parity$match_pct, na.rm = TRUE), ")")

# --- accessible / spawning / rearing km parity (BT, tunnel-free; link#226) ----
# link side = the #221 roll-up; bcfp side = the fresh.streams_vw_bcfp reference
# with the IN (1,2) predicate (bcfp codes access/spawn/rear 0/1/2/3, so a bare
# `= 1` under-counts). Same computation as the #223 proof
# (data-raw/parity_crosssection.R). accessible_km is the #223 target and matches
# exactly; spawning/rearing agree within habitat-methodology tolerance.
metrics_km <- c("accessible_km", "spawning_km", "rearing_km")
acc_link <- lnk_rollup_wsg(conn, aoi = aoi, species = "BT", schema = "fresh")
acc_bcfp <- DBI::dbGetQuery(conn, sprintf(
"SELECT
round((coalesce(sum(length_metre) FILTER (WHERE access_bt IN (1,2)),0)/1000)::numeric,2) AS accessible_km,
round((coalesce(sum(length_metre) FILTER (WHERE spawning_bt IN (1,2)),0)/1000)::numeric,2) AS spawning_km,
round((coalesce(sum(length_metre) FILTER (WHERE rearing_bt IN (1,2)),0)/1000)::numeric,2) AS rearing_km
FROM fresh.streams_vw_bcfp WHERE watershed_group_code = %s",
DBI::dbQuoteLiteral(conn, aoi)))
.link_v <- as.numeric(unlist(acc_link[1, metrics_km]))
.bcfp_v <- as.numeric(unlist(acc_bcfp[1, metrics_km]))
accessible <- data.frame(
metric = c("accessible", "spawning", "rearing"),
link_km = .link_v,
bcfp_km = .bcfp_v,
diff_pct = ifelse(.bcfp_v == 0, NA_real_, round(100 * (.link_v - .bcfp_v) / .bcfp_v, 2)),
stringsAsFactors = FALSE)
saveRDS(accessible, file.path(out_dir, paste0(stub, "_accessible.rds")))
message("[wsg_vignette_data] accessible km (link|bcfp|diff%): ",
paste(sprintf("%s %.1f|%.1f|%+.2f", accessible$metric, accessible$link_km,
accessible$bcfp_km, accessible$diff_pct), collapse = " "))

# --- spatial layers -----------------------------------------------------------
aoi_lit <- DBI::dbQuoteLiteral(conn, aoi)

Expand Down
Binary file modified inst/vignette-data/pars.gpkg
Binary file not shown.
Binary file added inst/vignette-data/pars_accessible.rds
Binary file not shown.
Binary file modified inst/vignette-data/pars_parity.rds
Binary file not shown.
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
## Outcome

Extended `vignettes/pars-habitat-connectivity.Rmd` to demonstrate `accessible_km` bcfishpass-equivalence
(#226): a new **Accessible habitat (km)** section with a link-vs-bcfp roll-up table (from cached
`pars_accessible.rds`) + computed prose — link 6,822.5 km vs bcfp 6,822.9 km accessible bull-trout habitat,
**−0.01%** (the #223 access-segmentation fix); spawning/rearing within the 5% habitat band. `data-raw/wsg_vignette_data.R`
gained the accessible roll-up cache (mirroring `parity_crosssection.R`'s `lnk_rollup_wsg()` + `streams_vw_bcfp`
`IN (1,2)`) and a segmentation-parity guard.

**The load-bearing discovery:** "regenerate the artifacts from the merged state" was *not* docs-only. The Plan
agent + live DB probes found `fresh_default` (the Arctic-grayling schema) was still on **pre-#223 segmentation**
(48,558 PARS segs vs `fresh`'s 97,538). The gpkg's single `streams` layer joins `fresh` geometry +
`mapping_code_bt` to `fresh_default` `mapping_code_gr` on `id_segment`, so a naive regeneration would have
attached grayling tokens to mismatched geometry and shipped a corrupted GR map. Fixed by re-modelling the
`default` config for PARS into `fresh_default` (`wsg_run_one.R`, 5.1 min) + a `merge=TRUE` recompute
(`wsg_recompute_one.R`, 1.6 min) for cross-WSG `;DAM`, bringing both configs onto the same segmentation. The
join is then faithful (exact GR count 19,232; 99.93% aggregate length; residual per-segment wiggle is local
habitat-break redistribution, sub-pixel at basin scale — single-layer gpkg design retained). Stale hardcoded
map-caption counts (19,233/31,932/1,764) are now computed inline (→ 19,232/38,622/257). mapping_code parity
refreshed 99.04% → 98.91% (denser post-#223). Verified with a DB-free knit; `/code-check` fixed a latent
`diff_pct` div-by-zero on the generic-reuse path.

**Reusable lesson:** the two persist configs (`fresh` / `fresh_default`) can drift in segmentation because only
WSGs re-modelled post-#223 are dense. Any cross-config artifact joined on `id_segment` must verify both sides
share segmentation first — the new guard in `wsg_vignette_data.R` enforces this.

Closed by: branch `226-vignette-accessible-km` (commits `fb0de90..7cb748d`), shipped v0.44.1. Closes #226.
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Findings — Extend PARS vignette to demonstrate accessible_km bcfp-equivalence (#226)

## Issue context

#221 added the per-WSG `accessible_km` roll-up and #223 fixed the access segmentation so `accessible_km`
converges to the bcfp reference **exactly** (proof: `research/parity_accessible_habitat_2026_07_03.md` —
44/44 within 0.05% across 8 species × 11 WSGs).

Extend `vignettes/pars-habitat-connectivity.Rmd` (currently demonstrates `mapping_code` parity at 99.04% BT)
to also demonstrate **accessible_km bcfp-equivalence**, and regenerate the cached artifacts via
`data-raw/wsg_vignette_data.R` from the merged state. Kept as its own PR (user-facing docs + a regenerated
binary `.gpkg` + pkgdown CI build). Relates to #221, #223.

## Live-verified DB state (2026-07-04, local `:5432` fwapg, read-only)

**PARS·BT roll-up — link vs bcfp (the numbers the vignette will assert):**

| metric | link km | bcfp km | diff |
|---|---|---|---|
| accessible | 6822.47 | 6822.88 | **−0.01%** (exact; the #223 target) |
| spawning | 1683.38 | 1667.92 | +0.93% (within habitat tol) |
| rearing | 2575.06 | 2588.91 | −0.53% (within habitat tol) |

- Link side reproduces `lnk_rollup_wsg(conn, aoi="PARS", species="BT", schema="fresh")`.
- Bcfp side = `fresh.streams_vw_bcfp` PARS, `access_bt/spawning_bt/rearing_bt IN (1,2)`, `sum(length_metre)/1000`.
- accessible −0.01% matches the proof doc's PARS·BT row exactly → persisted `fresh` is post-#223.

**The blocker (verified — drove the full-regen scope choice):**

| schema | PARS segs | avg length | state |
|---|---|---|---|
| `fresh` (BT) | 97,538 | 142.2 m | post-#223 |
| `fresh_default` (GR) | 48,558 | 285.6 m | **pre-#223** |

The gpkg `streams` query (`wsg_vignette_data.R:103-118`) pulls geometry + `mapping_code_bt` from `fresh` but
LEFT-JOINs `mapping_code_gr` from `fresh_default` on `(id_segment, watershed_group_code)`. Two different
segmentations → the GR token attaches to the wrong geometry. Regenerating the gpkg *now* corrupts the GR map.
Plan-agent measured: `gr_only` 1,764 → 12,749, streams layer 33,696 → 51,371 — pure join artifact.

**Why a local re-model suffices (no cyphers):** PARS downstream closure WSGs are all persisted in
`fresh_default` — PCEA (33,763), FINA (26,094), PARA (19,079), LBTN (28,304), LPCE (11,483), UPCE (25,024),
NECR (27,989). So the `;DAM` recompute can run locally. Pre-#223 both configs segmented PARS identically
(48,558 = 48,558) → expect both → ~97,538 post-#223, restoring a valid `id_segment` join.

`fresh` is a **patchwork**: only the proof WSGs (FINA/PARS/PCEA + the other 7) are post-#223; the rest still
match `fresh_default` (pre-#223). Only PARS's own segmentation matters for this gpkg.

## Existing structure (from Explore agents)

- Vignette output `bookdown::html_vignette2`; loads `inst/vignette-data/pars.gpkg` (11.5 MB) +
`pars_parity.rds` (272 B, 1-row: wsg/species/total_segs/match_pct/n_diffs/top_pattern/top_pattern_count).
- Parity section `## Reproducing bcfishpass (parity)` (~L212): `parity-table` kable + `parity-pct` computed
prose (99.04% inline, auto-updates). No `\@ref` cross-refs anywhere.
- Two-species: BT (bcfp parity) + GR (link extension, "nothing to compare"). Parity portion is BT-only.
- **Blocker 2:** map-gr `fig.cap` (`Rmd:388`) + map-detail prose (`Rmd:411`) hardcode 19,233 / 31,932 / 1,764
— won't auto-update; must inline-compute post-regen.
- `data-raw/wsg_vignette_data.R`: `LNK_LOAD=loadall Rscript ...`, reads persisted `fresh` + `fresh_default`
on `:5432`; computes ONLY mapping_code + geometry (no rollup). `bookdown`/`sf`/`gq` in Suggests;
`ggplot2` NOT declared (new content stays kable+prose). pkgdown CI builds on push to main, cache-only.

## API to reuse

- `R/lnk_rollup_wsg.R` — `lnk_rollup_wsg(conn, aoi, species, schema="fresh")` → data.frame
`wsg/species/accessible_km/spawning_km/rearing_km`. accessible = `access_<sp> IN (1,2)` from
`streams_access` LEFT-joined to `streams` on full PK; length from `streams`.
- `data-raw/parity_crosssection.R:55-69` — the exact bcfp `streams_vw_bcfp` `IN (1,2)` rollup SQL to mirror.
- `data-raw/wsg_run_one.R` / `wsg_recompute_one.R` — Phase 1 re-model + cheap `merge=TRUE` recompute.

## Reference
- bcfp reference: local `fresh.streams_vw_bcfp` snapshot, `smnorris/bcfishpass@v0.7.15-41-g2917790` (tunnel-free).
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Progress — Extend PARS vignette to demonstrate accessible_km bcfp-equivalence (#226)

## Session 2026-07-04

- Cleared leftover memory-migration + run-logs onto `main` (commits `3846a3d`, `16b41aa`; pushed). Left
`.claude/settings.local.json` uncommitted — it carries a live bcfp-tunnel password (flagged to user).
- Plan-mode exploration: 3 Explore agents (vignette / data-gen+build / API+numbers) + 1 Plan-agent review.
- Live-verified the PARS·BT roll-up numbers and the `fresh_default` pre-#223 blocker (see findings.md).
- User chose **full faithful regeneration** (re-model the stale grayling schema, not accessible-only).
- Created branch `226-vignette-accessible-km` off main; scaffolded PWF baseline with approved phases.
- **Phase 1 done:** re-modelled PARS default → `fresh_default` (`wsg_run_one.R`, 5.1 min) + `merge=TRUE`
recompute (`wsg_recompute_one.R`, 1.6 min). Gate: `fresh_default` PARS = 97,537 segs / 142.2 m (matches
`fresh` 97,538). Join nuance surfaced + reassessed: 94.49% exact-length but **99.93% aggregate-length +
exact GR count** (19,232 = native); residual is local habitat-break redistribution, sub-pixel at basin
scale → single-layer gpkg design retained (user informed).
- **Phase 2 done:** `data-raw/wsg_vignette_data.R` — added accessible/spawning/rearing km cache
(`pars_accessible.rds`, mirrors `parity_crosssection.R` link+bcfp `IN (1,2)`) + segmentation-parity guard
(refuses mixed-segmentation gpkg).
- **Phase 3 done:** regenerated all artifacts. accessible **6822.47 | 6822.88 | −0.01%** (exact),
spawning +0.93%, rearing −0.53%; mapping_code parity 98.91%; gpkg 11.9 MB (no balloon), all 8 layers,
tunnel context included.
- **Phase 4 done:** vignette gains `### Accessible habitat (km)` (bridging table + computed prose:
6,822.5 vs 6,822.9 km, −0.01%), loads `pars_accessible.rds`, and the stale map captions
(19,233/31,932/1,764) are now inline-computed (`n_gr`/`n_bt`/`n_gronly` → 19,232/38,622/257).
- **Phase 5 verify:** cache-only knit passed (exit 0) — accessible numbers + computed captions render,
stale values gone; the only `lnk_db_conn()` sits in the `eval=FALSE` demo chunk. `/code-check` round 1:
1 low latent finding (diff_pct div-by-zero on the generic-reuse path) fixed with an NA guard; PARS
artifact unchanged.
- Next: NEWS + patch bump, archive PWF, open PR (Closes #226; ref sred#24).
Loading