Search before asking
Motivation
paimon-rust can already search full-text global indexes (FullTextSearchBuilder,
full_text_search, hybrid search), but it cannot build or drop them:
CALL sys.create_global_index(..., index_type => 'full-text') is rejected. The procedure supports
only btree, bitmap, multivalue, fm, and the vindex types.
CALL sys.drop_global_index(..., index_type => 'full-text') is rejected as an unsupported type.
So a table written by Rust, pypaimon native, or C FFI needs a separate Java (Flink/Spark) job before
full-text search can use an index. Until then, every search either returns nothing (fast mode)
or builds a temporary in-memory index over the raw rows (full/detail mode).
Solution
Add a native full-text index build that mirrors Java's generic global-index build
(GenericIndexTopoBuilder driving NativeFullTextGlobalIndexWriter):
- Split the latest snapshot's row IDs into
global-index.row-count-per-shard shards. Skip ranges a
full-text index on the column already covers, so repeated builds are incremental.
- Feed each shard's text column to
paimon-ftindex-core with shard-relative row IDs. This is the
native core the Rust reader already uses. Write one full-text index file per non-empty shard
and commit all shards in one snapshot.
- Follow Java's semantics:
- only
full-text.* options reach the native writer, with the prefix removed;
index_meta is the flat JSON of those options;
- NULL rows count toward the row count but are not indexed.
- Expose it as
Table::new_full_text_index_build_builder() (behind the fulltext feature). Route
create_global_index(index_type => 'full-text') to it in DataFusion, and accept full-text in
drop_global_index.
Table requirements would match the existing global-index builders: row tracking, data evolution,
global index enabled, no primary keys, no deletion vectors. The column must be CHAR/VARCHAR.
Anything else?
Out of scope for the first PR, as possible follow-ups:
- allow tables with deletion vectors (Java's generic build does not reject them);
- refresh indexes on column updates (
global-index.column-update-action).
Willingness to contribute
Search before asking
Motivation
paimon-rust can already search
full-textglobal indexes (FullTextSearchBuilder,full_text_search, hybrid search), but it cannot build or drop them:CALL sys.create_global_index(..., index_type => 'full-text')is rejected. The procedure supportsonly
btree,bitmap,multivalue,fm, and the vindex types.CALL sys.drop_global_index(..., index_type => 'full-text')is rejected as an unsupported type.So a table written by Rust, pypaimon native, or C FFI needs a separate Java (Flink/Spark) job before
full-text search can use an index. Until then, every search either returns nothing (
fastmode)or builds a temporary in-memory index over the raw rows (
full/detailmode).Solution
Add a native full-text index build that mirrors Java's generic global-index build
(
GenericIndexTopoBuilderdrivingNativeFullTextGlobalIndexWriter):global-index.row-count-per-shardshards. Skip ranges afull-textindex on the column already covers, so repeated builds are incremental.paimon-ftindex-corewith shard-relative row IDs. This is thenative core the Rust reader already uses. Write one
full-textindex file per non-empty shardand commit all shards in one snapshot.
full-text.*options reach the native writer, with the prefix removed;index_metais the flat JSON of those options;Table::new_full_text_index_build_builder()(behind thefulltextfeature). Routecreate_global_index(index_type => 'full-text')to it in DataFusion, and acceptfull-textindrop_global_index.Table requirements would match the existing global-index builders: row tracking, data evolution,
global index enabled, no primary keys, no deletion vectors. The column must be CHAR/VARCHAR.
Anything else?
Out of scope for the first PR, as possible follow-ups:
global-index.column-update-action).Willingness to contribute