Multi shard consolidation - #862
Open
blasscoc wants to merge 4 commits into
Open
Conversation
Additive feature for consolidating many SEG-Y shards (e.g. a shot survey split across files) into one MDIO store, cloud-to-cloud, without changing the default single-file segy_to_mdio behavior. - allocate_mdio_grid / append_segy_shard: two-phase region-write workflow that builds an empty global grid, then writes each shard into its region in place. - plan_consolidation (+ ConsolidationPlan): deterministic planner computing per shard chunk ownership, per-chunk fill vs merge (read-modify-write) mode, the shard conflict graph, and concurrency waves. Orchestration is left to callers. - Optional read-modify-write in trace_worker / blocked_io.to_zarr via a merge_chunks set, so shards sharing a boundary chunk don't clobber each other. Defaults to the original fast pure-write path when not provided. - Export the new API from mdio, mdio.converters.
A shared template's "required" set should describe the data category, not one contributor's acquisition rig. `gun` (StreamerShotGathers3D) and the new elevation coords (ShotReceiverLineGathers3D) are now optional coordinates: populated when the source carries them, silently omitted otherwise. - base: add _optional_coord_names + optional_coordinate_names property - validation: subtract optional coords from required-field check; add prune_absent_optional_coordinates to drop them from the resolved schema when the segy_spec doesn't carry them - pipeline + consolidate: prune optional coords after schema resolution - shot-receiver-line: add optional source/receiver elevation for true 3D land/areal geometry Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce an optional shard grid so many chunks pack into one storage object (far fewer objects / S3 GETs) while keeping the small chunk as the partial-read unit. Adds shard_grid to VariableMetadata and shard_shape to ResolvedSchema, a full_shard_shape property on the template base, resolver pass-through, and shard resolution/validation (whole multiple of the chunk per dimension) in the dataset factory. The xarray builder emits the sharding_indexed codec (v3 only) and aligns Dask lazy blocks to shard boundaries so to_zarr's safe-chunks guard passes; blocked_io writes whole shards to avoid read-modify-write races. Headers are intentionally left unsharded: their structured (void) dtype trips Zarr's sharding partial-encode path, and they are tiny relative to the sample cube. Sharding applies to the data variable. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a little unhinged, but I want to post here in case there was an interest.
The multi-SEGY consolidation could also be done using Xarray directly and the "concat" feature.
Otherwise the "gun" needs to be optional.
Zarr V3 sharding on ingestion is also a win.