Skip to content

feat: add authenticated streamed Substrait reads - #6

Open
aunjgr wants to merge 6 commits into
tae-scan-supportfrom
feature/27586-stream-input
Open

feat: add authenticated streamed Substrait reads#6
aunjgr wants to merge 6 commits into
tae-scan-supportfrom
feature/27586-stream-input

Conversation

@aunjgr

@aunjgr aunjgr commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add the versioned matrixone.sirius.v1.StreamRead Substrait read extension and strict extract-enum validation
  • route mo_stream_scan through a dedicated GPU_MO_SCAN source that reuses the GPU-native TAE vector decoder
  • copy and coalesce MO-native input vectors into bounded 32 MiB staging batches (96 MiB hard bound) before H2D
  • expose synchronous Sirius data-batch result callbacks so the sidecar bypasses DuckDB DataChunk
  • add an adaptive GPU fixed-width MO result packer, with direct host-representation fallback for small, string, nullable, and coercion-heavy results
  • close per-query source state, task scheduling, EOF, cancellation, and repeated-query generation lifecycles

Scope

This is the Sirius portion of MatrixOne #27586. The existing authenticated direct TaeRead path remains supported. The dependent sidecar PR owns Flight framing and the MatrixOne PR owns local snapshot scans.

Validation

  • sirius_substrait_contract_unittest: 89 assertions
  • mo_sidecar_contract_unittest: 61 assertions
  • mo_sidecar_stream_unittest: 51 assertions
  • CUDA-enabled duckdb, mo_sidecar_loadable_extension, and native-result CUDA packer builds passed
  • all 22 TPC-H SF1 views matched native MatrixOne byte-for-byte through SIDECAR STREAM on one reused sidecar process

Depends on the matching protocol-v5 changes in sidecar PR sirius-db#14 and MatrixOne PR #27599.

@aunjgr
aunjgr requested review from XuPeng-SH and aptend August 28, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant