Skip to content

feat(io): a segment-keyed IO protocol for planner-driven scans - #10568

Draft
joseph-isaacs wants to merge 1 commit into
developfrom
ji/segment-io-protocol
Draft

joseph-isaacs wants to merge 1 commit into
developfrom
ji/segment-io-protocol

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

The protocol half of #10527: replaces the byte-range planning IO protocol on develop (vortex_io::request) with vortex_io::segments, keyed by segment source and source request. It contains only the protocol types and the planning-trait change. #10527, stacked on this PR, adds the implementation (FileSegmentIo).

Nothing drives this protocol on develop yet. No v1 code changes: the only users of vortex_io::request on develop were the planning traits.

Changes

  • vortex-io: segments.
    • A request names IoKey { source: SegmentSourceId, request: SourceRequest }, where SourceRequest ::= Size | Segment(id), with an IoIntent: Fetch, Prefetch or Announce. One service serves many segment sources, and a source can answer more than segment reads.
    • Reads resolve to a SourceResult: Size(u64) or Bytes(BufferHandle), with checked accessors size() and into_bytes().
    • SegmentIoSession::submit registers a whole batch before any read starts, and returns one IoHandle per request. The session holds fetches and prefetches until they finish; an announcement is withdrawn by dropping its handles.
    • SegmentIoSession::poll_ready reports every fetch's owner once it finishes (including fetches finished at submit) and drives the session's prefetches. FetchTracker implements this for sessions.
    • SegmentIoService::session() opens one session per scope. RegisteredSource is a source's registration with a service.
    • Removed: request (IoSource, IoTarget, IoResult, IoConsumer, IoSlot, ReadAtIoSource).
  • vortex-scan: planning traits. Planner and Morsel drop the IoConsumer supertrait, return NeedsIO(Vec<IoRequest>), and receive deliver(key, result) once per successful fetch.

Tests: handles, results checked against what was asked, and FetchTracker reporting owners and failing with nothing in flight.

Checks run:

  • cargo clippy -p vortex-io -p vortex-scan -p vortex-layout -p vortex-file --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-09-10 fmt -p vortex-io -p vortex-scan --check
  • RUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps --document-private-items -p vortex-io -p vortex-scan --all-features
  • cargo test --lib: vortex-io (118), vortex-scan (29)

API Changes

  • vortex_io::request is removed in favour of vortex_io::segments.
  • Planner and Morsel drop the IoConsumer supertrait and gain deliver(IoKey, SourceResult); NeedsIO carries Vec<IoRequest>.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9


Generated by Claude Code

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants