[ZFDB] FDB-721 Gribjump extractor by tbkr · Pull Request #335 · ecmwf/fdb

tbkr · 2026-08-18T09:21:58Z

Description

Implementation of GribJump backend with a CustomStoreBuilder.

Contributor Declaration

By opening this pull request, I affirm the following:

All authors agree to the Contributor License Agreement.
The code follows the project's coding standards.
I have performed self-review and added comments where needed.
I have added or updated tests to verify that my changes are effective and functional.
I have run all existing tests and confirmed they pass.

…nment, and GRIB field validation Refactoring and new features across the chunked data view (CDV), Z3FDB Zarr store, and GRIB extraction pipeline. Chunked data view: - Refactor extractor signature: move request out of extractor, remove extractor from view part, rename shape to extension in ViewPart - Add arbitrary chunk sizes in Axis - Add fill value support for virtual views; default fill value is NaN - Fix concurrency bug in data setup (session/function scope) - Add sanity checks and orientation checks for BoundingBox; remove dead code causing size_t underflow - Fix bounds check for extensionAxisIndex (moved before first write access) - Fix missing optional creation for extensionAxisIndex - Remove old Buffer class; move internal mappings logic to own subfolder - Fix CDVBuilder nullptr issue on second call (builder was moving content) Z3FDB / Zarr store: - Align internal functions with Zarr API - Use MarsSelections as dict instead of strings for MARS requests - Refactor prefix function to use consolidated paths from constructor - Fix prefix computation - Add ruff.toml for Python linting GRIB / extraction: - Check returned parameters against view definition to detect FDB-derived parameters (e.g. u/v) that may surprise users - Explicitly reject unsupported field derivation Tests: - Add tests for individual chunking (C++ and Python) - Add test for missing data during chunk query - Add zarr store interface tests using Zarr's own test harness - Restructure and rename zarr API conformity and permutation tests Docs: - Add technical docs for chunking and individual chunking - Add docs to chunked_data_view classes - Reformulate user-facing docs for Z3FDB - Add copyright headers to exception files - Shorten FDB name in docs for consistency

Restructured the bindings to contain the logic for the fdb5lib loading. This was formerly done in the pychunked data view leading to issues with the import of the bindings from python repl without loading fdb5lib via findlibs first. Also added a mechanism to have pyi stubs for the bindings layer (autocompletion as the *.so aren't picked up for security reasons).

Adjusted paths and pickup of test cases of pytest.

Newer versions need python 3.12

Copilot

Pull request overview

Implements GribJump-backed partial GRIB extraction and hierarchical Zarr store construction, alongside extractor API, packaging, documentation, and test restructuring.

Changes:

Adds GribJump value selection and implicit-dimension chunking.
Introduces CustomStoreBuilder and dimension metadata.
Refactors extractor ownership, bindings, packaging, tests, and documentation.

Reviewed changes

Copilot reviewed 110 out of 111 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`src/chunked_data_view/extractors/gribjump/GribJumpExtractor.cc`	Implements GribJump extraction.
`src/chunked_data_view/extractors/gribjump/GribJumpExtractor.h`	Declares GribJump extractor.
`src/chunked_data_view/extractors/gribjump/GribJumpExtractorDefinition.cc`	Builds configured extractors.
`src/chunked_data_view/extractors/gribjump/GribJumpExtractorDefinition.h`	Defines extractor factory.
`src/chunked_data_view/extractors/grib/GribExtractor.cc`	Refactors GRIB retrieval.
`src/chunked_data_view/extractors/grib/GribExtractor.h`	Updates GRIB extractor API.
`src/chunked_data_view/extractors/grib/GribExtractorDefinition.cc`	Adds GRIB factory.
`src/chunked_data_view/extractors/grib/GribExtractorDefinition.h`	Declares GRIB factory.
`src/chunked_data_view/ChunkedDataViewBuilder.cc`	Builds per-part extractors.
`src/chunked_data_view/ChunkedDataViewImpl.cc`	Supports implicit-axis chunks.
`src/chunked_data_view/ChunkedDataViewImpl.h`	Updates extractor ownership.
`src/chunked_data_view/ViewPart.cc`	Removes stored layout.
`src/chunked_data_view/ViewPart.h`	Revises part metadata.
`src/chunked_data_view/RequestManipulation.cc`	Adds sample-request reduction.
`src/chunked_data_view/RequestManipulation.h`	Exposes request helper.
`src/chunked_data_view/Axis.cc`	Improves lookup errors.
`src/chunked_data_view/Axis.h`	Clarifies growing chunks.
`src/chunked_data_view/Fdb.cc`	Updates FDB integration.
`src/chunked_data_view/ListIterator.cc`	Returns field locations.
`src/chunked_data_view/LibChunkedDataView.cc`	Updates licensing.
`src/chunked_data_view/mapping/AxisMapper.cc`	Updates licensing.
`src/chunked_data_view/mapping/AxisMapper.h`	Updates licensing.
`src/chunked_data_view/mapping/IndexMapper.cc`	Updates licensing.
`src/chunked_data_view/mapping/IndexMapper.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/AxisDefinition.h`	Adds dimension names.
`src/chunked_data_view/include/chunked_data_view/ChunkedDataView.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/ChunkedDataViewBuilder.h`	Adds extractor definitions.
`src/chunked_data_view/include/chunked_data_view/DataLayout.h`	Adds per-chunk size.
`src/chunked_data_view/include/chunked_data_view/Extractor.h`	Defines extractor configurations.
`src/chunked_data_view/include/chunked_data_view/Fdb.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/LibChunkedDataView.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/ListIterator.h`	Adds location-backed entries.
`src/chunked_data_view/include/chunked_data_view/Types.h`	Adds chunk bounding-box alias.
`src/chunked_data_view/include/chunked_data_view/exception/AxisMapperException.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/exception/BoundingBoxException.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/exception/GribExtractorException.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/exception/GribJumpExtractorException.h`	Adds GribJump exception.
`src/chunked_data_view/include/chunked_data_view/exception/RequestManipulationException.h`	Updates licensing.
`src/chunked_data_view/include/chunked_data_view/exception/UnknownExtractorException.h`	Updates licensing.
`src/chunked_data_view/exception/AxisMapperException.cc`	Updates licensing.
`src/chunked_data_view/exception/BoundingBoxException.cc`	Updates licensing.
`src/chunked_data_view/exception/GribExtractorException.cc`	Updates licensing.
`src/chunked_data_view/exception/GribJumpExtractorException.cc`	Implements new exception.
`src/chunked_data_view/exception/RequestManipulationException.cc`	Updates licensing.
`src/chunked_data_view/exception/UnknownExtractorException.cc`	Updates licensing.
`src/chunked_data_view/CMakeLists.txt`	Builds and links GribJump.
`src/chunked_data_view_bindings/bindings.cc`	Exposes new Python API.
`src/chunked_data_view_bindings/CMakeLists.txt`	Links binding dependencies.
`src/chunked_data_view_bindings/__init__.py`	Initializes packaged bindings.
`src/pychunked_data_view/chunked_data_view.py`	Adds extractor configuration wrappers.
`src/pychunked_data_view/__init__.py`	Exports value requests.
`src/pychunked_data_view/exceptions.py`	Updates licensing.
`src/z3fdb/custom_store_builder.py`	Adds hierarchical store builder.
`src/z3fdb/simple_store_builder.py`	Accepts configurable extractors.
`src/z3fdb/__init__.py`	Exports custom builder.
`src/z3fdb/_internal/__init__.py`	Updates licensing.
`src/z3fdb/z3fdb_error.py`	Updates licensing.
`src/CMakeLists.txt`	Reworks Python staging and stubs.
`tests/chunked_data_view/test_axis.cc`	Updates licensing.
`tests/chunked_data_view/test_bounding_box.cc`	Updates licensing.
`tests/chunked_data_view/test_index_mapper.cc`	Updates licensing.
`tests/chunked_data_view/test_mock_helpers.h`	Adapts extractor mocks.
`tests/chunked_data_view/test_request_manipulation_bounding_box.cc`	Updates licensing.
`tests/chunked_data_view/test_view_individual_chunking.cc`	Uses extractor definitions.
`tests/conftest.py`	Adds divergence/vorticity fixtures.
`tests/pychunked_data_view/CMakeLists.txt`	Revises Python test registration.
`tests/pychunked_data_view/fdb/test_chunked_data_view_fdb.py`	Updates GRIB builder tests.
`tests/pychunked_data_view/gribjump/test_chunked_data_view_gribjump.py`	Tests value requests.
`tests/pychunked_data_view/test_chunked_data_view_errors.py`	Tests Python errors.
`tests/pychunked_data_view/test_mars_selection.py`	Updates licensing.
`tests/z3fdb/CMakeLists.txt`	Reorganizes Z3FDB tests.
`tests/z3fdb/integration/test_gribjump_extractor.py`	Tests GribJump integration.
`tests/z3fdb/integration/test_mixed_extractors.py`	Tests mixed backends.
`tests/z3fdb/permutation_tests/test_axis_definition_ordering.py`	Tests axis ordering.
`tests/z3fdb/permutation_tests/test_axis_definition_permutations.py`	Tests axis permutations.
`tests/z3fdb/permutation_tests/test_scrambled_request_ordering.py`	Tests request ordering.
`tests/z3fdb/permutation_tests/test_store_missing_values.py`	Updates missing-value tests.
`tests/z3fdb/test_store_v3.py`	Removes superseded tests.
`tests/z3fdb/test_store_v3_pattern.py`	Removes superseded tests.
`tests/z3fdb/test_store_v3_random_axis.py`	Removes superseded tests.
`tests/z3fdb/unit/test_store_v3.py`	Adds store and chunk-list tests.
`tests/z3fdb/unit/test_store_v3_errors.py`	Updates extractor API usage.
`tests/z3fdb/unit/test_u_v_vo_d_retrieval.py`	Tests parameter derivation behavior.
`cmake/z3fdb_setup.py.in`	Packages generated stubs.
`cmake/z3fdb_setup.cfg.in`	Updates licensing.
`requirements.txt`	Pins Zarr and adds stubgen.
`pytest.toml`	Configures import mode.
`ruff.toml`	Adds Python lint settings.
`.github/workflows/z3fdb.yml`	Installs build requirements.
`docs/conf.py`	Updates documentation branding.
`docs/fdb/index.rst`	Simplifies FDB title.
`docs/index.rst`	Adds indices page.
`docs/indices_and_tables.rst`	Adds index links.
`docs/pyfdb/conftest.py`	Updates licensing.
`docs/z3fdb/api.rst`	Documents extractor API.
`docs/z3fdb/architecture.rst`	Updates licensing.
`docs/z3fdb/conf.py`	Updates licensing.
`docs/z3fdb/dimension_mapping.rst`	Updates chunking syntax.
`docs/z3fdb/getting_started.rst`	Uses new extractor API.
`docs/z3fdb/index.rst`	Reorganizes documentation.
`docs/z3fdb/installation.rst`	Adds installation guide.
`docs/z3fdb/technical_insights.rst`	Removes monolithic guide.
`docs/z3fdb/technical_insights/buffer_layout.rst`	Documents buffer layout.
`docs/z3fdb/technical_insights/chunk_access.rst`	Documents chunk access.
`docs/z3fdb/technical_insights/dev_setup.rst`	Adds development guide.
`docs/z3fdb/technical_insights/extractor.rst`	Documents extractor internals.

Suppressed comments (2)

src/chunked_data_view/extractors/grib/GribExtractor.cc:122

FieldLocation::dataHandle() returns an owning raw pointer (the previous iterator wrapped it in unique_ptr). Keeping it in auto leaks one handle for every GRIB field read, so repeated chunk access steadily consumes resources. Restore RAII ownership here.
src/chunked_data_view/extractors/gribjump/GribJumpExtractor.cc:202
Every field in every chunk prints its storage fragment directly to stdout. Large reads will emit thousands of lines, interfere with callers' output, and add avoidable I/O overhead. Remove this debugging output (and the adjacent unused variable).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+std::unique_ptr<Extractor> GribJumpExtractorDefinition::buildExtractor(const metkit::mars::MarsRequest& request) const {
+    if (gribjumpConfig) {
+        ::setenv("GRIBJUMP_CONFIG_FILE", gribjumpConfig->c_str(), /*overwrite=*/1);
+    }


…s support Introduce the GribJump-backed extractor alongside a suite of related refactors that improve the extractor ownership model, Python API surface, and zarr metadata output. Add `GribJumpExtractor` / `ExtractorType.GribJump` as a second extractor implementation. It uses the GribJump library to retrieve a configurable sub-range of grid-point values without a full GRIB decode, making it efficient for spatial sub-region access. Field-level chunking of the implicit grid-point axis is controlled by `field_chunking` (default `WholeAxisChunking`). Mixing `GribExtractor` and `GribJumpExtractor` parts is valid only when `GribJumpExtractor` uses `WholeAxisChunking`; all other `field_chunking` variants raise at build time. - Replace `std::shared_ptr<Extractor>` with `std::unique_ptr<Extractor>`. Each `ViewPart` owns its extractor exclusively; sharing between parts is no longer supported. - Introduce the `ExtractorDefinition` factory pattern: `addPart` records a lightweight config object; `build()` calls `buildExtractor(request)` once per part, deferring FDB/GribJump initialisation to build time. - `ChunkedDataViewBuilder` is now non-copyable (copy constructor and assignment operator explicitly deleted) because it stores `unique_ptr<ExtractorDefinition>`. - `ListIteratorWrapperImpl::next()` now returns `ListElement` directly. Add `CustomStoreBuilder` alongside `SimpleStoreBuilder`. It accepts an arbitrary path-based hierarchy of `ChunkedDataViewBuilder` instances and materialises them into a nested `FdbZarrGroup` / `FdbZarrArray` tree, enabling multi-array zarr stores from a single builder. - Add `std::optional<std::string> name` to the `AxisDefinition` C++ struct and expose it via pybind11 `def_readwrite`. - Track dimension names in the Python layer only: `ChunkedDataViewBuilder` derives `dim_names` from `axis.name` (explicit) or `"_".join(axis.keys)` (auto) on the first `add_part` call; the implicit grid-point axis is always named `"values"`. - `FdbSource` accepts `dim_names` and forwards them to `DotZarrArrayJson.dimension_names`, so zarr v3 array metadata carries labelled dimensions consumable by xarray. - Rename `Chunking.FixedSizeChunk.chunkShape` → `chunk_shape`. - Remove unused store flags. - `extend_on_axis` now supports extension on any axis index, not just the first. - Restructure API tests: move `user_tests/` and `zarr_interface_conformity/` under a new `interface/` directory; update `CMakeLists.txt` accordingly. - Add `test_metadata_mapping.py`: verifies explicit and auto-derived axis names appear in zarr array metadata. - Add `test_xarray_access.py`: exercises xarray integration via `dimension_names` (single-part, fixed-size chunking, multi-part SFC+PL, named-dim reduction). - Update extractor docs: ownership model (`unique_ptr`, non-copyable), `ExtractorDefinition` factory, `GribJumpExtractor` value ranges and configuration.

Introduced a mechanism for detecting whether gribjump is available and build accordingly. If the dependency is missing the gribjump tests are skipped.

In case there are mixed extractors make sure that they fit in terms of the value dimension.

- Thread safety for the fdb/gribjump calls - Asserts for writeInto and the uri fragment conversion

Addresses the architecture findings from the FDB-721 review, plus two tests that were passing without checking what they claimed to check.

tbkr requested a balanced review from Copilot August 18, 2026 09:22

Copilot started reviewing on behalf of tbkr August 18, 2026 09:22 View session

tbkr added 11 commits August 18, 2026 11:24

fix: Review remarks

bf3ff5c

docs(fdb-678): Added docs for bindings.cc

862b9f8

tests(fdb-678): Negative tests for pychunked_data_view

eb1c8dd

tests(fdb-678): Introduce skip on 429 for online tests

896c60f

docs(fdb-678): Aligned docs and introduced tech insights

ea75810

docs(fdb-678): Local testing improvement

1069a4d

Adjusted paths and pickup of test cases of pytest.

chore(fdb-678): License header adjustments

a33861a

chore(fdb-678): Pin to zarr 3.1.6

2ddd246

Newer versions need python 3.12

chore(fdb-678): Minor adjustment to test cases

c6acd3b

tbkr force-pushed the feature/zfdb/fdb-721-gribjump-extractor branch from b24127c to 93a9748 Compare August 18, 2026 09:25

Copilot AI reviewed Aug 18, 2026

View reviewed changes

tbkr force-pushed the feature/zfdb/fdb-721-gribjump-extractor branch 15 times, most recently from 50495e0 to e70310c Compare August 20, 2026 09:21

caraghbiner self-requested a review August 20, 2026 10:13

tbkr force-pushed the feature/zfdb/fdb-721-gribjump-extractor branch 3 times, most recently from 5e97a84 to 6e956e2 Compare August 21, 2026 08:20

tbkr force-pushed the feature/zfdb/fdb-721-gribjump-extractor branch from 6e956e2 to 0abce29 Compare August 21, 2026 09:41

tbkr added 9 commits August 21, 2026 13:35

review(fdb-721): Extractor Definition refinement

5f899bb

review(fdb-721): Decoupled build from optional GJ dependency

c393892

Introduced a mechanism for detecting whether gribjump is available and build accordingly. If the dependency is missing the gribjump tests are skipped.

review(fdb-721): Fix datahandle leak

6354f5f

review(fdb-721): Added test for the layout validation

cef7ffc

In case there are mixed extractors make sure that they fit in terms of the value dimension.

review(fdb-721): Correctness changes

d38aa8a

- Thread safety for the fdb/gribjump calls - Asserts for writeInto and the uri fragment conversion

review(fdb-721): Change testing to reflect latest additions

bc2eaf5

review(fdb-721): Fixing seam issues of Python/C++

ef33a12

review(fdb-721): Store builder cleanup and test corrections

2609f36

Addresses the architecture findings from the FDB-721 review, plus two tests that were passing without checking what they claimed to check.

review(fdb-721): Adjust docs

c1c761b

tbkr force-pushed the feature/zfdb/fdb-721-gribjump-extractor branch 5 times, most recently from 49fa850 to 585208d Compare August 24, 2026 15:16

feat(fdb-721): Use uvx in the runner for stub creation

5b19b29

tbkr force-pushed the feature/zfdb/fdb-721-gribjump-extractor branch from 585208d to 5b19b29 Compare August 24, 2026 15:51

Navigation Menu

Uh oh!

[ZFDB] FDB-721 Gribjump extractor - #335

[ZFDB] FDB-721 Gribjump extractor#335
tbkr wants to merge 22 commits into
developfrom
feature/zfdb/fdb-721-gribjump-extractor

tbkr commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

tbkr commented Aug 18, 2026

Description

Contributor Declaration

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants