feat(signals): extract RaggedStore from SpyDEDiffractionVectors - #129
Merged
Conversation
…n store One offsets array + schema columns over a rectangular scan grid, with the outer CSR levels derived as zero-copy strided views of the leaf (the rectangular-grid stride identity). Streaming fill (append_batch in any chunk order, stable-sort finalize with an adopt-in-place already-sorted fast path), count_map/map with the count-law pinned by test, and a versioned save/load with append-only column padding. _AxisLite moves in and is re-exported from diffraction_vectors for its existing importers. Base only — the vectors class adopts it in the next commits.
…ing only __post_init__ aliases the base state over the dataclass fields (packed backing = flat_buffer, levels = the stored nav_offsets list verbatim, leaf offsets = nav_offsets[-1]); every method body is untouched, so any breakage from the class-identity change alone (pickling, repr/eq, getattr gates) bisects to this commit. columns_schema is asserted against COLUMN_NAMES and the COL_* constants at import — those are the packed ABI of the GPU and embed consumers. The legacy 4-D-only offsets field keeps its semantics and shadows the base's public alias; base internals never read it.
_flat_pos/_slice_flat/flatten are now the inherited base implementations (reading the identical memory through the __post_init__ wiring); slice_at keeps its pinned name and (N, 6) packed return as a thin delegate to the base's _prefix_row_range (the stored nav_offsets list — hand-built and degenerate ones included — stays this instance's source of truth); count_map_series reshapes the base counts(); _build_nav_offsets delegates to the generic builders, its outer levels now the zero-copy strided views of the leaf that they were always element-equal to. The physics/ABI surface (at/at_t/at_nav, count_map, VVI, GPU, render, to_dense, legacy save/load) is untouched. Pinned 29-file vectors subset green (477 passed).
…seam setattr + provenance stamp (a container without its own provenance record inherits the tree's _commit_provenance), routed through by the single call site (find_vectors_action._finalize). resolve_vectors, the requires_vectors toolbar gate and the Save hook are untouched — they read the attribute the helper sets. The docstring records the later-PR design (requires_container: yaml key with requires_vectors as alias; commit.* container= kwarg) without implementing any of it.
streaming() + one append_batch + finalize() — so the whole existing vectors suite transitively exercises the seam chunked computes will stream through. finalize()'s already-sorted fast path adopts the caller's buffer verbatim (object identity preserved, nothing copied or reordered), so the orchestrate / carrier / legacy-npz paths are byte-identical; an unsorted buffer — a silent contract violation that used to yield broken CSR slices — now gets stable-sorted. Pinned subset + ragged-store tests green (477 passed).
CSSFrancis
marked this pull request as ready for review
August 9, 2026 04:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of the unification: the ragged per-nav-position container extracted from
SpyDEDiffractionVectorsasspyde/signals/ragged_store.py— pure refactor of main code, no quarry involved. The 29 vector-pinning test files are untouched and green at every step (5 commits, each independently green); full suite 2664 passed / 25 skipped / 1 xfailed.The base (~670 lines incl. contracts): CSR
offsets[n_positions+1]+ named flat columns (may be zero-copy strided views of a packed block — the packed layout is then the subclass's ABI and the base never copies/reorders),full_nav_shape+nav_axes(nav-space ONLY),streaming()/append_batch()(mutex, any chunk order)/finalize()(stable-sort with an already-sorted O(n) fast path),at()(dict of views),slice_at()(O(1) prefix),count_map()(O(positions)),map(column, reducer)(NaN fill for empty positions), versioned save/load with older-file prefix-column padding. Structure-frozen after finalize; values writable in place.The subclass keeps its keyword constructor and every field verbatim (three test files construct it directly);
__post_init__wires the base over the SAME memory —offsetsalias, levels list, packed backing. Physics, the packed(N,6)at()ABI, GPU buffer,to_densecache, display methods, and the legacy save/load all stay byte-for-byte.columns_schemais asserted equal toCOLUMN_NAMES/COL_*at import (the tensor-indexing ABI guard).The one seam:
lifecycle.attach_container(tree, store, name=...)—_finalizeroutes through it;resolve_vectors, the toolbar gate and the Save hook are unchanged readers. The laterrequires_container:/commit design is recorded in its docstring only.from_arraysnow runs throughstreaming()+append_batch+finalize()internally (single-sorted-batch fast path adopts the buffer by identity), so the entire existing vectors suite transitively exercises the streaming seam.Two behavior notes (previously-broken input only): the generic offsets builder clamps negative indices on every nav column (old code clamped only time; illegal input now clamps instead of raising out of bincount), and unsorted
from_arraysinput is now stable-sorted (previously produced silently broken CSR slices).Perf sanity (200×64 grid, 30 rows/pos, before-vs-after in one process):
at()−2 to −11%,count_map()within noise,from_arrays+3–20% run-to-run noise from the O(n) sortedness check on a one-shot constructor.Next consumers per the landing order: the particles container (quarry port) rides
streaming/append_batch/count_map/mapdirectly; OrientationMap adoption is a later judgment call and nothing here is designed for it; FitStore stays on HyperSpy's own store by decision.