[ewlarson]

Download-only datasets can prepare COG or vector PMTiles derivatives for both OGM Viewer and content thumbnails, plus compressed Parquet derivatives for searchable table previews. IIIF image services and Presentation 2/3 manifests can now produce hosted static Image API 2 level 0 bundles, so published image previews do not depend on provider CORS or continued availability. Separate coverage manifests supply labeled location geometry and static maps. The feature defaults off, with an explicit per-record publication allowlist. Original downloads, harvested references, curated thumbnails, and working previews retain priority.

This branch starts at feature/ogm-viewer-resource-page (ab8458ed) and targets develop; it includes that branch’s unmerged integration changes.

Tabular derivatives and dataset-local search

CSV, TSV, and Parquet downloads now prepare compressed Parquet derivatives and expose schema, row count, and asset size in the shared manifest. The resource page uses a lazy-loaded, same-origin DuckDB WASM worker for literal search within the opened dataset, typed sorting, and 50-row pagination. Collection-wide search is unchanged. CSV identifiers remain text; existing Parquet numeric types are preserved, with exact string display for large integers and decimals.

The worker uses the existing bounded conversion and publication path. Browser queries have cancellation, a 30-second watchdog, and explicit failure/retry states. The S3-backed asset route now handles range-aware HEAD requests required by strict DuckDB range reads. Original downloads remain unchanged; no geography or illustrative thumbnails are fabricated from a table.

Validation at 04ee1518: 82 backend tests, 57 frontend unit/component tests, and 6 browser acceptance checks across development and production builds passed. Browser checks verify partial reads, leading-zero IDs, exact large integers, literal search, completed sorting, pagination, unavailable files, and cancellation. Backend lint/format, frontend lint (12 existing warnings), and the production application build passed. New CI results remain separate from these local checks.

Supplemental real-source validation: a public USGS daily earthquake CSV produced 202 rows, 41,386 source bytes → 24,069 Parquet bytes in 0.877 seconds. This is not a BTAA pilot record or collection-wide compression estimate. Table storage is additional measured Parquet bytes × retained versions; the existing cost estimate has no measured tabular census yet. Full-text scans can read most of a file despite Range support.

Table formats, API, limits, workflow, and validation

Production versus preview

Actual browser screenshots captured October 31, 2026, using the same 12 resource IDs in gallery view, sorted by Title (A–Z). Both captures use the same 1,280-pixel page width and include every result.

Production — current thumbnails	Preview — generated content thumbnails

Open production screenshot at full size · Open preview screenshot at full size

Production returns 11 records; the local preview returns 12. The additional preview result is Purdue Campus: West Lafayette, Indiana, 1895 (008b35d8-d227-43aa-86d3-f3236019af87), which is absent from these production search results. The first ten results align by title; vegetation is result 11 in production and 12 in the preview. These screenshots show the selected pilot records, not a production deployment or collection-wide success claim. The production Location facet is expanded and the preview Location facet is collapsed.

Hosted IIIF and image optimization (November 1 extension)

• Generate complete bounded snapshots of embedded Presentation 2/3 canvases, hosted info.json, full-image JPEGs, overview sizes and 512-pixel image pyramids. Preserve source dimensions, structure, labels, rights and attribution. Unsupported compositions/authentication/external painting pages receive explicit outcomes.
• Runtime viewer references select hosted IIIF without changing harvested references or original downloads. Nested asset delivery uses the same publication/withdrawal gates, CORS and immutable local/S3-compatible storage.
• Optimized progressive JPEG quality 82 with 4:4:4 chroma, reuse of smaller full-image source JPEGs, duplicate-object upload elimination, and bounded-error 64/128/256-color PNG quantization. Numeric raster values and class identifiers are unaffected.
• Real Princeton map: 29,470,206 -> 19,736,552 IIIF bytes (33.03% smaller); thumbnail 398,117 -> 189,282 bytes (52.46% smaller). The viewer loaded five tiles initially and 41 after zoom, with provider requests blocked and no page errors. A second source correctly exceeded the pixel limit.
• Local validation: 72 pipeline tests passed, followed by 28 targeted IIIF/service/index tests including the additional recipe-identity test (73 distinct pipeline tests now covered). Nine frontend adapter tests passed, frontend lint/build passed (existing lint warnings), and backend lint/format passed. New GitHub checks are pending; the earlier full-CI result above is historical.
• Storage estimates now count IIIF canvases, tile objects and retained versions separately. A single measured map is not extrapolated as a collection average. Detailed budget assumptions remain in the separate manager brief.
• Direct CDN serving still requires the documented bundle-path export/resolver and cache withdrawal integration. Current hosted delivery works through the API; this is not a production deployment.

Implementation, limits and validation evidence

Implementation

• Bounded isolated raster/vector conversion, source/service/index discovery, immutable local/S3 storage, access-checked HTTP Range delivery, and database publication with last-valid retention.
• Existing COG/PMTiles sources use bounded partial inspection. Named layers and shared rendering recipes feed the thumbnail renderer and runtime viewer adapter.
• WMS version/CRS ordering, ArcGIS export fallback, deterministic index-member selection, one-hop child records, and supported IIIF painting images.
• Shared display coverage with dateline/disconnected-region handling, explicit representative framing, independent thumbnail outcomes, retry limits, and delivery/status metrics.
• Vector derivatives retain exact source feature identifiers separately from generated tile IDs. Decoding all 651 tiles from eight refreshed records verified 171,098 feature instances, unchanged geometry/original attributes, and preserved source IDs.
• The frontend pins published OGM Viewer v1.7.0-btaa.5. This includes a tested correction for custom GeoTIFF projection offsets recorded in feet; source pixels and georeferencing remain unchanged.
• Local/API documentation, a fixed 40-record audit fixture, and self-contained HTML/PDF comparison tooling. Conversion provenance records the inspected curation recipes revision.

Validation

• 61 isolated pipeline tests pass, including real conversions in the pinned Linux worker (GDAL 3.11.0 / Tippecanoe 2.79.0). The optional PostgreSQL publication/withdrawal test passes separately against a local transactional schema. All 1,765 viewer tests pass; application lint/build and browser contracts pass.
• All 40 pilot records have processing, thumbnail, browser, and visual-review outcomes: 21 preparation-ready, 18 thumbnail-ready, 19 browser-passed. Two live services remain under review for dark/blank rendering. These selected-pilot outcomes are not a collection success-rate estimate.
• Real conversions from georeferenced imagery TIFF, elevation TIFF, classified TIFF, Shapefile ZIP, and GeoPackage pass both thumbnail and interactive acceptance. Browser evidence covers partial reads, pan/zoom, placement, shared styling, and feature inspection where supported.
• Supplemental standalone elevation acceptance uses the previously download-only Sheboygan 2004 county DEM: 178.7 MB source ZIP, original 7,591 × 7,797 grid retained, 30.5 seconds preparation. Correct placement and pan/zoom passed; a clean browser run read 3.4 MB through eight HTTP 206 responses. Vertical units remain unspecified because the source band does not encode them. This is separate from the fixed 40 records.
• Supplemental native-source checks cover a Tacoma catalog PMTiles archive and public Nelson COG fixture, also outside the pilot counts.
• The comparison contains 160 embedded images with zero broken browser images. All 40 PDF pages and the two-page standalone-elevation supplement were rendered and visually reviewed. Failures and retained curated assets are explicit.

Rollout and limitations

The collection maintainer confirms that BTAA Geoportal inclusion establishes reuse eligibility. Public/published/not-suppressed checks and the technical/visual rollout allowlist remain enforced. Seventeen eligible reviewed records (16 fixed-pilot records plus the standalone elevation supplement) were freshly prepared and published in an isolated local API. All 17 pass API/image checks and feature-off rollback with original attributes preserved; every prepared file layer passes HTTP Range/CORS checks. All regenerated thumbnails match reviewed images byte for byte. Seven vector derivatives retain identical decoded geometry, properties and feature IDs. One absent local record and one retired local record remain excluded. The original screenshot-only USGS DEM index still lacks retrievable metadata and is not claimed as validated. Numeric raster pixel inspection is unsupported. Review-only service thumbnails remain withheld; over-limit records retain their existing presentation. No production rollout or collection-wide acceptance claim is made.

Before the IIIF extension, all checks passed on commit 4b478c19: backend (2,292 passed, 124 skipped, one expected-failure test unexpectedly passed), frontend (1,398 passed across 132 files, plus production build), pinned conversion worker (61 passed; optional database test skipped), viewer browser contracts, Ruff, CodeQL, CLI, QGIS, and MkDocs. All automated review threads are resolved.

The additional local rollout report contains 35 verified embedded images. The revised elevation PDF’s two pages were rendered and visually checked. Expansion priorities follow the measured pilot outcomes: supported downloads, supported index members, then focused service batches; all remain subject to technical and visual review.

Code-quality review (a18ebe2)

Fixed typed Parquet sorting (display casts must not change numeric order), stale image selection when a table takes precedence, public-response leakage/bloat from internal IIIF registries, missing source invalidation for service-layer/title changes, and subprocess error-log/descendant cleanup. Public preview descriptors now use an explicit allowlist; the full delivery manifest remains internal. Existing pilot manifests need re-preparation because the record signature now includes layer and title.

Validation: 87 backend preview tests, one transactional PostgreSQL test, 10 targeted frontend unit tests, and six development/production browser table tests passed. Backend lint/format and frontend lint passed (existing frontend warnings); modified frontend files pass formatting. All CI checks passed on the preceding head; new-head CI must also pass.

Remaining broad-enablement gates:

• Response caching: enabling previews currently bypasses resource/search/home response caches globally. Implement cached base representations with a fresh, batch-loaded publication overlay while preserving withdrawal behavior.
• Retention: manifest history and reference-aware garbage collection are not implemented. Failed/obsolete uploads can accumulate; estimates assuming two retained versions remain conditional on that work.
• Failure isolation: some converted-data paths render a thumbnail before publishing layers, so a renderer exception can suppress an otherwise valid interactive derivative. Split those publication outcomes before claiming full acceptance.

Further maintainability work: reuse a DuckDB session across table queries instead of recreating it for every sort/page/filter; review the inherited MapLibre foundation separately from derivative lifecycle/adapters. Process limits and controlled retrieval are valuable but do not constitute a strong sandbox for native parsers. Keep this PR draft and adapters gated pending the remaining acceptance work.

Client-facing API asset discovery (0c21368)

Resource detail, search-result, and homepage resource objects now expose a versioned meta.assets catalog independently of meta.ui. Published items include URLs, roles, format/media type, rendering recipes, layer identifiers, bounds, available sizes, and Parquet column types/row counts. The catalog includes labeled resource coverage and provenance; representative samples remain explicit. Generated thumbnail and coverage-map links are discoverable without exposing private tile registries or review images. Original downloads and curated thumbnail precedence remain unchanged; access/approval/feature gates still apply. WMS preparation now persists the selected source_layer explicitly.

Client documentation: preview asset API contract. Validation: 98 preview-suite tests passed, followed by all 12 client-contract tests including the additional WMS preparation regression; backend lint/format and diff checks passed.