Given the success of the Datasets + DuckDB integration, a similar integration might be worthwhile in this module.
The datasets API allows taking filters and columns subset, and provides an iterator of Arrow record batches. I think that could be wrapped in a TableProvider, though I'm unclear how predicate pushdown is implemented in Datafusion.
Given the success of the Datasets + DuckDB integration, a similar integration might be worthwhile in this module.
The datasets API allows taking filters and columns subset, and provides an iterator of Arrow record batches. I think that could be wrapped in a
TableProvider, though I'm unclear how predicate pushdown is implemented in Datafusion.