Skip to content

[Rust] Add Arrow mux API - #866

Merged
danilonajkov-db merged 6 commits into
mainfrom
danilo-najkov_data/effort/mux-streams-arrow-api
Oct 14, 2026
Merged

[Rust] Add Arrow mux API#866
danilonajkov-db merged 6 commits into
mainfrom
danilo-najkov_data/effort/mux-streams-arrow-api

Conversation

@danilonajkov-db

@danilonajkov-db danilonajkov-db commented Sep 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What changes are proposed in this pull request?

Add Rust-only multiplexed Arrow Flight ingestion through .arrow(schema).multiplexed(n).build_arrow(), returning MultiplexedArrowStream. Whole RecordBatches route round-robin, MessageId identifies the lane and local batch offset, and the mux-wide max_inflight_batches budget is divided evenly among lanes. No global ordering is promised.

Private Arrow reservation and admission hooks connect to the shared core from #865, inheriting its poisoning and cancellation handling. Close preserves an already-recorded mux failure; otherwise it returns the first observed lane-close error and logs additional errors. Arrow retains its recovery and close supervisor. All lanes share the stats exporter; events retain lane-local offsets. Arrow acknowledgment callbacks remain unsupported.

For standalone Arrow streams, ingest_batch calls blocked on max_inflight_batches now fail as soon as close() starts instead of waiting for close to finish. Other closed-admission checks retain the prompt rejection behavior already present in 2.10. Includes focused tests for both cases, builder validation, routing/IPC/message-ID/telemetry tests, API docs, a runnable example, and changelog entries. No FFI or wrapper API changes.

Stack base: #865. Follow-up: #867 adds broader lifecycle and failure tests.

How is this tested?

  • Arrow integration suite at this layer: 104 passed.
  • SDK unit tests with all features: 388 passed, 1 ignored.
  • Full Rust make test and make lint matrices passed on the combined implementation, plus the standalone Avro mux regression.
  • Rust formatting and diff whitespace checks passed.

This was referenced Sep 22, 2026
Draft
Merged
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from fbcdcbe to 846642f Compare September 23, 2026 11:08
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch 2 times, most recently from cd29799 to fe13be4 Compare September 23, 2026 12:52
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch 2 times, most recently from fe46935 to e208838 Compare September 23, 2026 14:54
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from fe13be4 to 9328853 Compare September 23, 2026 14:54
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from e208838 to 71e31cb Compare September 23, 2026 15:18
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from 9328853 to fe13be4 Compare September 23, 2026 15:18
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 71e31cb to 5e2aa6d Compare September 25, 2026 10:49
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from ebeb366 to 85a48bb Compare September 25, 2026 13:32
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch 2 times, most recently from 7876529 to 28e40c3 Compare September 25, 2026 15:58
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch 2 times, most recently from f55b303 to 8534e17 Compare September 28, 2026 16:59
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 28e40c3 to 21a7300 Compare September 28, 2026 16:59
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-core branch from 8534e17 to 6ee8720 Compare September 30, 2026 14:58
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 21a7300 to 117c993 Compare September 30, 2026 14:58
Base automatically changed from danilo-najkov_data/effort/mux-streams-core to main October 1, 2026 12:46
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch 2 times, most recently from 97aa0ee to 48c4c0d Compare October 5, 2026 15:31
@danilonajkov-db
danilonajkov-db marked this pull request as ready for review October 5, 2026 15:35
elenagaljak-db added a commit that referenced this pull request Oct 6, 2026
@elenagaljak-db Isaac
[Rust] Move ephemeral builder entry points
f003332 
Move StreamBuilder::build into stream_builder/ephemeral.rs, and move
multiplexed() and multiplexed_ack_callback() into multiplexed.rs, so
mod.rs keeps the builder state and the setters that the persistent
builder also uses. No public API or behavior change. The Arrow Flight
setters and build_arrow() stay in mod.rs until the Arrow mux work in
#866 lands. Addresses a review nit on #763.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
@elenagaljak-db elenagaljak-db mentioned this pull request Oct 8, 2026
Merged
@danilonajkov-db
[Rust] Simplify Arrow mux lane hooks
4012e04 
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
[Rust] Fix async-trait lint on Rust 1.99
eea6f48 
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
[Rust] Tighten Arrow mux ingestion and docs
85c275b 
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 48c4c0d to 85c275b Compare October 13, 2026 14:01
@danilonajkov-db
danilonajkov-db force-pushed the danilo-najkov_data/effort/mux-streams-arrow-api branch from 48c4c0d to 85c275b Compare October 13, 2026 14:01
teodordelibasic-db
teodordelibasic-db reviewed Oct 14, 2026
View reviewed changes
Comment thread rust/sdk/src/headers_provider.rs Outdated
Comment thread rust/sdk/src/lib.rs Outdated
Comment thread rust/sdk/src/multiplexed_stream/lane.rs Outdated
Comment thread rust/sdk/src/multiplexed_stream/lane.rs
Comment thread rust/sdk/src/builder/stream_builder/multiplexed.rs Outdated
.map(|(delay, headers_provider)| {
let table_properties = table_properties.clone();
let options = options.clone();
let stats_exporter = inner.stats_exporter.clone();

@teodordelibasic-db teodordelibasic-db Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since every lane reports into this one exporter with its own lane-local offsets, the same offset shows up once per lane and none of them match the MessageId that ingest_batch returned. Totals still add up, but per-batch telemetry such as send-to-ack latency or retransmits cannot be tied to a batch.

It's probably best to leave stats off Arrow mux streams for now. We could reject stats_exporter in MultiplexedStreamBuilder::validate with InvalidArgument, as validate_common already does for gRPC formats and drop the shared-exporter wording from the docs? We can add it later.

As a follow-up, the ideal is probably for mux telemetry to use the same IDs as the rest of the mux API: wrap each lane's exporter so BatchSent and BatchAcked report MessageId::raw() instead of the lane-local offset, the same way multiplexed_ack_callback adapts gRPC lane offsets, and note on the offset docs that mux events carry a raw MessageId.

WDYT?

@danilonajkov-db danilonajkov-db Oct 14, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it doesn't make much sense currently with the message id. I agree its best to reject it in validate now and have a proper solution in a followup

@danilonajkov-db
[Rust] Clarify mux lane shutdown
1ab3d6f 
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
[Rust] Reject telemetry on Arrow mux
6ee1b83 
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
@danilonajkov-db
[Rust] Remove Arrow mux telemetry test
1de53a6 
Signed-off-by: danilo-najkov-db <danilo.najkov@databricks.com>
teodordelibasic-db
teodordelibasic-db approved these changes Oct 14, 2026
View reviewed changes

@teodordelibasic-db teodordelibasic-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@danilonajkov-db
danilonajkov-db added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit 8efcd30 Oct 14, 2026
51 checks passed
@danilonajkov-db
danilonajkov-db deleted the danilo-najkov_data/effort/mux-streams-arrow-api branch October 14, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teodordelibasic-db teodordelibasic-db teodordelibasic-db approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@danilonajkov-db @teodordelibasic-db