Repository navigation
[Refactor] TableProvider (RawSource renamed) with a ScanRequest, and one BindingDataSources for table slots and materialized-view slots #966
Description
Example
SELECT count(*) FROM requests WHERE status = 500 AND ts >= now() - INTERVAL '5 minutes', where Stage 3 keeps 1-minute panes as a materialized view.
The DAG is Scan(Source::Table { table_ref: "requests" }, predicate status = 500) under TimeRange(Range, 5m), and the pane node under it is materialized.
Today a deployment binds two kinds of data through two interfaces:
| Data | Interface today | What the deployment gets to see |
|---|---|---|
raw requests rows |
RawSource { schema(), boundedness(), scan(ctx) }, registered in DataSources by Source and wrapped in a Scan by bind_with_data_sources |
nothing about the scan: no time range, no predicates, no columns. It must return the whole table on every scan, and its schema must equal Scan.schema (DataSources::bind) |
| the pane view's stored state | an arbitrary Source (Box<dyn PhysicalOperator>) bound at the pane node in instantiate |
nothing about which panes this evaluation reads |
After this issue, both are a TableProvider:
requestsgetsscan(ScanRequest { time_range_ms: Some((T - 5m, T]), projection: Some(&[status, ts]), predicates: &[status = 500] }).- The pane view gets
scan(ScanRequest { time_range_ms: Some((T - 5m, T]), .. })and returns the five pane states.
Concrete maintenance problem
RawSourcegives a provider no way to skip rows or columns. That is the reason the raw-sample selectors exist (removed in the next PR of [Tracking] Physical DAG → deployment boundary: InstallablePlan, input slots, TableProvider, BindingDataSources #968).- Requiring the provider schema to equal the
Scan's rules out a real table with more columns than the query reads. - Materialized views have no interface of their own: the deployment hands the executor an operator. So the deployment's summary store cannot share code or contract with raw-data providers, and the executor cannot tell it which panes a run reads.
Behavior that must remain unchanged
- Every query returns the same answers.
- Compilation opens no readers.
- A connector error is never an empty successful scan.
- A provider that ignores every
ScanRequestfield stays correct.
Before and after structure
pub trait TableProvider { // RawSource, renamed
fn schema(&self) -> SchemaRef; // contains the slot's columns; may have more
fn boundedness(&self) -> Boundedness;
fn scan(&self, request: &ScanRequest, context: RunContext)
-> Result<OutputStream<'_, Batch>, Error>;
}
pub struct ScanRequest<'a> {
pub time_range_ms: Option<(i64, i64)>, // (start, end]
pub projection: Option<&'a [usize]>, // indices into the provider's schema
pub predicates: &'a [ScalarExpr], // over the provider's schema; a hint only
}
pub struct BindingDataSources {
pub tables: DataSources, // Source → provider
pub views: BTreeMap<NodeId, Arc<dyn TableProvider>>, // view slot → provider
}
impl CompiledPhysicalDAG {
pub fn instantiate<'a>(&self, data_sources: &'a BindingDataSources) -> Result<PhysicalDAG<'a, Batch, SchemaRef>, Error>;
}- Table slot. Every
Scanleaf compiles to a table slot holding itsSource, schema, compiled predicates and window, which is the offsets and widths of theTimeRange/TimeShifts above it. This is computed once, astable_time_operator(physical_planner/mod.rs:725) does today for SQL tables.instantiateturns the slot and its provider into aScan. At run time theScantakesTfromRunContext'sScope, builds theScanRequest, projects the provider's columns to the slot's schema, and re-applies the time range and the predicates. AScanread by several windows requests their union. - View slot.
instantiatewraps the view's provider in a schema-checked source. The request carries the view's time range for this run: the panes the query reads. - Pushdown is a hint. There is no
supports_filters_pushdownand noExactmode, so the executor always filters (decision Q4 of [Tracking] Physical DAG → deployment boundary: InstallablePlan, input slots, TableProvider, BindingDataSources #968). bindandbind_with_data_sourcesare replaced byinstantiate(&BindingDataSources).
Why the proposed structure is minimally complex
The trait is renamed and gains one argument. The two kinds of bound data share one trait. No pushdown negotiation is added, and the correctness of a scan never depends on the provider honoring the request.
Existing unit and end-to-end protection
crates/executor/tests/{raw_scan,current_timestamp,deployment_computation,precompute_candidates}.rs.crates/integration-tests/tests/{sql_to_physical,filtered_aggregates,sketch_rollup,operator_design_examples}.rs.crates/planner/tests/{sql_table_scans,sql_per_group_topk}.rs.- Add tests:
- a provider whose schema has an extra column;
- a provider that ignores the request and returns rows outside the range and the predicates (same answer);
- a provider that applies the request (same answer, fewer rows read);
- a view provider asked for exactly the panes of
(T - 5m, T].
Compatibility or migration impact
This is a public API change in asap_executor:
RawSourceis renamedTableProvider, and itsscantakes aScanRequest;bindandbind_with_data_sourcesare replaced byinstantiate(&BindingDataSources).
ASAPQuery-backend main pins an older Planner revision (f46cbf6c) and does not depend on asap_executor yet, so no backend code breaks. The backend's executor integration implements TableProvider from the start. A deployment that also links DataFusion names ours by path (asap_executor::sources::TableProvider).
Parent: #968 (PR 4 of 6; after #965). The next PR moves PromQL time-series windows onto this interface and deletes the raw-sample selectors.
Activity
[-][Refactor] asap_executor holds only the operator library, the compiled plan and the deployment interface; move raw-sample selectors and legacy types out[/-][+][Refactor] One general table provider for raw data (PromQL time series and SQL tables), replacing RawSource and the raw-sample selectors[/+][-][Refactor] One general table provider for raw data (PromQL time series and SQL tables), replacing RawSource and the raw-sample selectors[/-][+][Refactor] TableProvider (RawSource renamed) with a ScanRequest, and one Bindings for table slots and materialized-view slots[/+][-][Refactor] TableProvider (RawSource renamed) with a ScanRequest, and one Bindings for table slots and materialized-view slots[/-][+][Refactor] TableProvider (RawSource renamed) with a ScanRequest, and one BindingDataSources for table slots and materialized-view slots[/+]