Repository navigation
feat(io): a segment-keyed IO protocol for planner-driven scans - #10568
Draft
joseph-isaacs wants to merge 1 commit into
Draft
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The protocol half of #10527: replaces the byte-range planning IO protocol on
develop(vortex_io::request) withvortex_io::segments, keyed by segment source and source request. It contains only the protocol types and the planning-trait change. #10527, stacked on this PR, adds the implementation (FileSegmentIo).Nothing drives this protocol on
developyet. No v1 code changes: the only users ofvortex_io::requestondevelopwere the planning traits.Changes
vortex-io:segments.IoKey { source: SegmentSourceId, request: SourceRequest }, whereSourceRequest ::= Size | Segment(id), with anIoIntent:Fetch,PrefetchorAnnounce. One service serves many segment sources, and a source can answer more than segment reads.SourceResult:Size(u64)orBytes(BufferHandle), with checked accessorssize()andinto_bytes().SegmentIoSession::submitregisters a whole batch before any read starts, and returns oneIoHandleper request. The session holds fetches and prefetches until they finish; an announcement is withdrawn by dropping its handles.SegmentIoSession::poll_readyreports every fetch's owner once it finishes (including fetches finished at submit) and drives the session's prefetches.FetchTrackerimplements this for sessions.SegmentIoService::session()opens one session per scope.RegisteredSourceis a source's registration with a service.request(IoSource,IoTarget,IoResult,IoConsumer,IoSlot,ReadAtIoSource).vortex-scan: planning traits.PlannerandMorseldrop theIoConsumersupertrait, returnNeedsIO(Vec<IoRequest>), and receivedeliver(key, result)once per successful fetch.Tests: handles, results checked against what was asked, and
FetchTrackerreporting owners and failing with nothing in flight.Checks run:
cargo clippy -p vortex-io -p vortex-scan -p vortex-layout -p vortex-file --all-targets --all-features -- -D warningscargo +nightly-2026-09-10 fmt -p vortex-io -p vortex-scan --checkRUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps --document-private-items -p vortex-io -p vortex-scan --all-featurescargo test --lib:vortex-io(118),vortex-scan(29)API Changes
vortex_io::requestis removed in favour ofvortex_io::segments.PlannerandMorseldrop theIoConsumersupertrait and gaindeliver(IoKey, SourceResult);NeedsIOcarriesVec<IoRequest>.🤖 Generated with Claude Code
https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9
Generated by Claude Code