Skip to content

[Rust] Expose persistent stream API - #763

Merged
elenagaljak-db merged 1 commit into
mainfrom
stack/eos-public-api
Oct 5, 2026
Merged

[Rust] Expose persistent stream API#763
elenagaljak-db merged 1 commit into
mainfrom
stack/eos-public-api

Conversation

@elenagaljak-db

@elenagaljak-db elenagaljak-db commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What changes are proposed in this pull request?

This PR exposes the feature-gated Rust API for creating and resuming persistent streams on top of #762.

It adds:

  • ZerobusSdk::persistent_stream_builder().
  • PersistentStreamBuilder with the existing table, authentication, format, recovery, timeout, backpressure, and callback configuration.
  • build() for a new durable stream and resume(stream_id) for reconnecting.
  • PersistentStream, exposing stream_id, the resume watermark, ingestion, batch ingestion, flush, offset waiting, close, and unacknowledged-record retrieval.
  • Shared builder preparation so ephemeral and persistent streams resolve identical gRPC configuration.

JSON and protobuf are supported; Arrow Flight and a public retirement API are intentionally deferred. Documentation follows the pipelined loop-then-flush() pattern. The new public feature is recorded in rust/NEXT_CHANGELOG.md.

Stack

  1. [Rust] Add persistent stream proto #761 — protobuf contract
  2. [Rust] Add persistent gRPC transport #762 — gRPC transport and recovery engine
  3. [Rust] Expose persistent stream API #763 — public Rust API
  4. [Rust] Test persistent streams #764 — stateful mock and integration tests
  5. [Rust] Document persistent streams #765 — documentation and runnable example

How is this tested?

  • cargo check -p databricks-zerobus-ingest-sdk --features eos
  • cargo test -p databricks-zerobus-ingest-sdk --features eos --lib — 156 tests passed
  • cargo fmt --all -- --check
  • Public create/resume behavior is exercised in [Rust] Test persistent streams #764.

This was referenced Aug 20, 2026
Merged
Merged
Merged
@elenagaljak-db elenagaljak-db mentioned this pull request Aug 20, 2026
Open
@elenagaljak-db
elenagaljak-db force-pushed the stack/eos-grpc-core branch 2 times, most recently from 18a643f to abf5a92 Compare August 20, 2026 13:09
@elenagaljak-db
elenagaljak-db force-pushed the stack/eos-public-api branch 3 times, most recently from 0ecfe35 to c202abf Compare August 28, 2026 16:41
@elenagaljak-db
elenagaljak-db force-pushed the stack/eos-public-api branch 2 times, most recently from 8a1eda0 to c722ab2 Compare September 25, 2026 14:03
Base automatically changed from stack/eos-grpc-core to main September 25, 2026 15:34
@elenagaljak-db
elenagaljak-db force-pushed the stack/eos-public-api branch 2 times, most recently from f4ee1e6 to 17f6734 Compare October 2, 2026 14:16
@elenagaljak-db
[Rust] Expose persistent stream API
ef342eb 
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
teodordelibasic-db
teodordelibasic-db approved these changes Oct 3, 2026
View reviewed changes

@teodordelibasic-db teodordelibasic-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, couple of follow ups/nits.

Comment thread rust/sdk/src/builder/stream_builder/mod.rs
pub(crate) async fn prepare_grpc(
mut self,
) -> ZerobusResult<(
crate::databricks::zerobus::zerobus_client::ZerobusClient<tonic::transport::Channel>,

@teodordelibasic-db teodordelibasic-db Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be a follow up, but it would be probably be cleaner to have a structure/type alias containing all of this.

@elenagaljak-db elenagaljak-db Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will fix all of these in the next pr

Comment thread rust/sdk/src/builder/stream_builder/mod.rs
pub(crate) async fn prepare_grpc(
mut self,
) -> ZerobusResult<(
crate::databricks::zerobus::zerobus_client::ZerobusClient<tonic::transport::Channel>,

@teodordelibasic-db teodordelibasic-db Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also a nit, but I am personally not a fan of these fully qualified names, we can import them at the top of the file.

Comment thread rust/sdk/src/builder/stream_builder/mod.rs
/// let stream = builder.build().await?;
/// ```
#[must_use = "a StreamBuilder does nothing until `.build()` is called"]
pub struct StreamBuilder<'a> {

@teodordelibasic-db teodordelibasic-db Oct 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also a nit and a follow up, but we can extract the ephemeral specific constructs into a separate file and leave only common stuff in here.

@elenagaljak-db
elenagaljak-db added this pull request to the merge queue Oct 5, 2026
Merged via the queue into main with commit 385705d Oct 5, 2026
50 of 51 checks passed
@elenagaljak-db
elenagaljak-db deleted the stack/eos-public-api branch October 5, 2026 08:15
elenagaljak-db added a commit that referenced this pull request Oct 6, 2026
@elenagaljak-db Isaac
[Rust] Name shared gRPC stream parts
e131a70 
Replace the tuple that prepare_grpc returns with a crate-private
GrpcStreamParts struct, and import ZerobusClient and Channel instead of
using fully qualified paths. Addresses review nits on #763.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
elenagaljak-db added a commit that referenced this pull request Oct 6, 2026
@elenagaljak-db Isaac
[Rust] Move ephemeral builder entry points
f003332 
Move StreamBuilder::build into stream_builder/ephemeral.rs, and move
multiplexed() and multiplexed_ack_callback() into multiplexed.rs, so
mod.rs keeps the builder state and the setters that the persistent
builder also uses. No public API or behavior change. The Arrow Flight
setters and build_arrow() stay in mod.rs until the Arrow mux work in
#866 lands. Addresses a review nit on #763.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
elenagaljak-db added a commit that referenced this pull request Oct 16, 2026
@elenagaljak-db Isaac
[Rust] Name shared gRPC stream parts
590cd4d 
Replace the tuple that prepare_grpc returns with a crate-private
GrpcStreamParts struct, and import ZerobusClient and Channel instead of
using fully qualified paths. Addresses review nits on #763.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: elenagaljak-db <elena.galjak@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teodordelibasic-db teodordelibasic-db teodordelibasic-db approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elenagaljak-db @teodordelibasic-db