-
Notifications
You must be signed in to change notification settings - Fork 17
Expand file tree
/
Copy pathapi.html
More file actions
300 lines (269 loc) · 26.6 KB
/
Copy pathapi.html
File metadata and controls
300 lines (269 loc) · 26.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0.
-->
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Unified Paimon Vector Index APIs for Rust, C, C++, Java, and Python, including warm-up and metadata filter pushdown.">
<title>API and Language Bindings · Paimon Vector Index</title>
<link rel="stylesheet" href="styles.css">
<script src="docs.js" defer></script>
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<header class="site-header"><div class="header-inner">
<a class="brand" href="index.html" aria-label="Paimon Vector Index documentation home"><span class="brand-mark">VI</span><span>Paimon Vector Index</span></a>
<nav class="site-nav" data-site-nav aria-label="Documentation"><a href="index.html">Overview</a><a href="api.html" aria-current="page">API</a><a href="development.html">Development</a><a href="ivf-flat.html">IVF-FLAT</a><a href="ivf-pq.html">IVF-PQ</a><a href="ivf-rq.html">IVF-RQ</a><a href="ivf-sq.html">IVF-SQ</a><a href="diskann.html">DiskANN</a><a href="releases.html">Releases</a></nav>
<div class="header-actions"><button class="icon-button" type="button" data-theme-toggle aria-label="Switch color theme">◐</button><button class="nav-toggle" type="button" data-nav-toggle aria-expanded="false" aria-label="Open navigation">☰</button></div>
</div></header>
<main id="main"><div class="page-shell">
<section class="hero detail-hero">
<p class="eyebrow">One lifecycle · five language surfaces</p>
<h1>API and language bindings</h1>
<p class="hero-lead">The Rust core, C FFI, C++ RAII layer, Java/JNI, and Python ctypes package share one train → write → serialize → detect → search model. Build options select the index type; Readers discover it from the file magic.</p>
<div class="badge-row"><span class="badge strong">Unified lifecycle</span><span class="badge">Positional I/O</span><span class="badge">Single and batch search</span><span class="badge">Roaring64 prefilter</span></div>
</section>
<div class="doc-layout">
<aside class="toc" aria-label="On this page"><strong>On this page</strong><a href="#workspace">Modules</a><a href="#lifecycle">Lifecycle</a><a href="#params">Search parameters</a><a href="#reader-options">Reader options</a><a href="#warmup">Warm-up</a><a href="#rust">Rust</a><a href="#c">C FFI</a><a href="#cpp">C++</a><a href="#java">Java/JNI</a><a href="#python">Python</a><a href="#filter">Filter pushdown</a></aside>
<article class="article">
<section class="article-section" id="workspace">
<h2>Public integration layers</h2>
<div class="table-wrap"><table><thead><tr><th>Module</th><th>Role</th><th>Primary entry point</th></tr></thead><tbody>
<tr><td><code>core</code></td><td>Pure Rust indexes, file Readers/Writers, and benchmarks</td><td><code>paimon_vindex_core::index</code></td></tr>
<tr><td><code>ffi</code></td><td>C ABI over the Rust core; builds <code>libpaimon_vindex_ffi</code></td><td>Generated <code>include/paimon_vindex.h</code></td></tr>
<tr><td><code>include</code></td><td>C++ RAII wrapper</td><td><code>paimon_vindex.hpp</code></td></tr>
<tr><td><code>jni</code> + <code>java</code></td><td>JNI implementation and Java API</td><td><code>org.apache.paimon.index.vector</code></td></tr>
<tr><td><code>python</code></td><td>Pure Python package loading the C FFI through ctypes</td><td><code>paimon_vindex</code></td></tr>
</tbody></table></div>
<p>The top-level Cargo workspace contains <code>core</code>, <code>ffi</code>, and <code>jni</code>. The Python package loads the shared FFI library at runtime.</p>
</section>
<section class="article-section" id="lifecycle">
<h2>Shared lifecycle</h2>
<div class="flow" aria-label="Unified API lifecycle"><div class="flow-step"><small>01</small><strong>Create a Trainer<br>Parse and validate options</strong></div><div class="flow-step"><small>02</small><strong>Submit one or more<br>training batches</strong></div><div class="flow-step"><small>03</small><strong>Finish training and<br>create a one-shot Writer</strong></div><div class="flow-step"><small>04</small><strong>Add row IDs / vectors<br>and write the file</strong></div><div class="flow-step"><small>05</small><strong>Detect file magic<br>and execute searches</strong></div></div>
<ul><li>Vectors are contiguous <code>f32</code> values; length must equal <code>vector_count × dimension</code>.</li><li>Training data may arrive in batches. Every IVF trainer keeps a deterministic reservoir of at most <code>max(65,536, 64 × resolved nlist)</code> vectors. DiskANN starts from a 50,000-row cap and lowers it when necessary so the retained sample, optional cosine-normalized copy, codebook, and parallel PQ-training scratch fit <code>diskann.memory-budget-bytes</code>. Sampling is independent of batch boundaries.</li><li>The Python and Java one-shot <code>train</code> helpers infer <code>dimension</code> from the matrix and use its row count for automatic <code>nlist</code>. When the matrix is only a sample, pass the final corpus size as <code>expected-vector-count</code>. Streaming Trainer APIs require a concrete dimension before their first batch.</li><li>A Writer may receive production vectors in multiple batches. Row-ID count must equal vector count.</li><li>Readers expose metadata, single-query search, batch search, and Roaring64-filtered variants.</li><li>Files carry their type and resolved model sections. Callers do not pass index options again when opening a Reader.</li></ul>
</section>
<section class="article-section" id="params">
<h2>Shared search parameters</h2>
<div class="table-wrap"><table><thead><tr><th>Parameter</th><th>Applies to</th><th>Description</th></tr></thead><tbody>
<tr><td><code>top_k</code></td><td>All indexes</td><td>Number of nearest neighbors returned for each query.</td></tr>
<tr><td><code>Auto</code> width</td><td>All indexes</td><td>IVF starts at <code>max(8, ceil(nlist/16))</code>, adds enough average-list capacity for at least <code>4 × top_k</code> candidates, scales for filter selectivity, and progressively doubles when a filtered result is short. DiskANN uses a calibrated width when present, otherwise <code>max(100, 2 × top_k)</code>.</td></tr>
<tr><td><code>max_initial_filter_expansion_factor</code></td><td>Automatic IVF search in Rust, C, C++, and Java</td><td>Optional cap on inverse-selectivity expansion of the initial <code>nprobe</code>. Unset preserves the current unlimited behavior. A value of 1 keeps the unfiltered automatic width. Lower factors reduce initial search work but may reduce recall compared with uncapped automatic search. Progressive expansion occurs only when fewer than <code>top_k</code> valid results are returned.</td></tr>
<tr><td><code>nprobe</code></td><td>IVF families</td><td>Explicit expert override for the number of lists probed. A tagged IVF width is rejected by DiskANN rather than silently ignored.</td></tr>
<tr><td><code>l_search</code></td><td>DiskANN</td><td>Explicit expert override for graph search-list size; it is clamped to at least <code>top_k</code>. A tagged DiskANN width is rejected by IVF indexes.</td></tr>
<tr><td><code>ivfpq_batch_table_reuse</code></td><td>8-bit IVF-PQ batch search</td><td>Controls reuse of query distance tables across inverted lists: <code>off</code>, <code>on</code>, or <code>auto</code>. The default is <code>auto</code>.</td></tr>
<tr><td><code>ivfpq_batch_table_reuse_max_bytes</code></td><td>8-bit IVF-PQ batch search</td><td>Positive memory budget for reused query distance tables. The default is 512 MiB.</td></tr>
</tbody></table></div>
<p>Use the binding's <code>automatic</code> search-parameter constructor. For DiskANN, <code>calibrate_search_width</code> / <code>calibrateSearchWidth</code> evaluates widths 100, 200, and 400 on representative queries and remembers the smallest adjacent pair with at least 98% Top-K result overlap. This is a stability proxy, not a ground-truth recall guarantee; explicit widths always win.</p>
<div class="callout"><strong>IVF-RQ width is a build option</strong><code>rq.bits</code> accepts 1–8 and defaults to 4. It is persisted in the file and reported as <code>rq_bits</code> / <code>rqBits</code> metadata; search has no separate bit-width switch.</div>
</section>
<section class="article-section" id="reader-options">
<h2>Reader options for DiskANN</h2>
<p>Reader options are accepted by every binding and affect DiskANN only. Other index families continue to use their existing positional-I/O behavior.</p>
<div class="table-wrap"><table><thead><tr><th>Concept</th><th>Rust / Python</th><th>C / C++ / Java</th><th>Default</th></tr></thead><tbody>
<tr><td>Random-read latency hint</td><td>Input capability <code>estimated_random_read_latency_nanos</code></td><td>C/C++ field / Java <code>estimatedRandomReadLatencyNanos()</code></td><td>0: reuse the mandatory header read's elapsed time; positive values bypass measurement</td></tr>
<tr><td>Total Reader memory</td><td><code>memory_budget_bytes</code></td><td><code>memory_budget_bytes</code> / constructor argument</td><td>4 GiB; automatically partitioned among required resident state, a profile-sized adjacency prefix, a bounded cold-adjacency LRU, and a bounded raw-vector LRU</td></tr>
</tbody></table></div>
<p>The Reader has no public storage-tier switch. DiskANN selects an internal tier once during <code>open</code> from the latency hint or mandatory header-read timing; its latency, window, and beam policy then remain stable. <code>read_plan</code> / <code>readPlan</code> exposes that policy together with the current effective preload and shared-cache capacities. Those capacities are zero before resident initialization and may shrink when lazy row-ID lookup state consumes the same total budget. Cache partitioning remains an internal decision. The input callback receives all positional ranges for one round together and should execute them concurrently.</p>
<p>A storage adapter may additionally advertise <code>preferred_window_bytes</code> and a maximum range count named <code>max_ranges_per_pread</code> in Rust or <code>max_ranges_per_read</code> in C/C++/Python. Zero means unspecified. DiskANN rounds the requested window to complete 4 KiB logical pages, bounds it to 1 MiB, limits each <code>pread</code> batch to the advertised range count, and keeps physical alignment concerns inside the storage adapter. Build-time <code>deployment-profile</code> remains separate because it can select the persisted compact or interleaved layout. See <a href="diskann.html#reader-parameters">DiskANN reader tuning</a>.</p>
</section>
<section class="article-section" id="warmup">
<h2>Search warm-up</h2>
<p>After opening a Reader and before repeated searches, initialize resident state and, for DiskANN, optionally replay a small representative query set. Warm-up builds process-local caches without changing the file or search results.</p>
<div class="table-wrap"><table><thead><tr><th>Language</th><th>Resident initialization</th><th>Representative-query warm-up</th><th>DiskANN width calibration</th></tr></thead><tbody>
<tr><td>Rust</td><td><code>optimize_for_search</code></td><td><code>warmup_queries</code></td><td><code>calibrate_search_width</code></td></tr>
<tr><td>C</td><td><code>paimon_vindex_reader_optimize_for_search</code></td><td><code>paimon_vindex_reader_warmup_queries</code></td><td><code>paimon_vindex_reader_calibrate_search_width</code></td></tr>
<tr><td>C++</td><td><code>optimize_for_search</code></td><td><code>warmup_queries</code></td><td><code>calibrate_search_width</code></td></tr>
<tr><td>Java</td><td><code>optimizeForSearch</code></td><td><code>warmupQueries</code></td><td><code>calibrateSearchWidth</code></td></tr>
<tr><td>Python</td><td><code>optimize_for_search</code></td><td><code>warmup_queries</code></td><td><code>calibrate_search_width</code></td></tr>
</tbody></table></div>
<p><code>optimize_for_search</code> is optional for correctness: the first search performs the same initialization lazily. It builds IVF-PQ residual-L2 tables or DiskANN resident PQ/row-ID state and its automatically budgeted hot adjacency prefix. DiskANN <code>warmup_queries</code> then executes top-1 graph traversal and persisted-vector rerank for each supplied query, priming immutable adjacency and raw-vector LRUs. Other index families treat representative warm-up as resident initialization only.</p>
</section>
<section class="article-section" id="rust">
<h2>Rust</h2>
<div class="code-block"><span class="code-label">Rust · train, write, and search</span><pre><code>use std::fs::File;
use paimon_vindex_core::distance::MetricType;
use paimon_vindex_core::index::{
VectorIndexConfig, VectorIndexReader, VectorIndexTrainer,
VectorIndexWriter, VectorSearchParams,
};
use paimon_vindex_core::io::PosWriter;
let config = VectorIndexConfig::IvfSq {
dimension: 128,
nlist: 1024,
metric: MetricType::L2,
};
let training = VectorIndexTrainer::train(
config, &training_vectors, training_count)?;
let mut writer = VectorIndexWriter::new(training);
writer.add_vectors(&row_ids, &vectors, vector_count)?;
let mut file = File::create("vectors.pvindex")?;
let mut out = PosWriter::new(&mut file);
writer.write(&mut out)?;
let file = File::open("vectors.pvindex")?;
let mut reader = VectorIndexReader::open(file)?;
reader.optimize_for_search()?;
let params = VectorSearchParams::automatic(10);
let (ids, distances) = reader.search(&query, params)?;</code></pre></div>
<div class="code-block"><span class="code-label">Rust · optional filtered-IVF tuning</span><pre><code>// Example only: select the factor using workload-specific
// latency and Recall@K measurements.
let params = VectorSearchParams::automatic(10)
.with_max_initial_filter_expansion_factor(4);</code></pre></div>
<div class="code-block"><span class="code-label">Rust · other configurations</span><pre><code>VectorIndexConfig::IvfFlat {
dimension: 128, nlist: 1024, metric: MetricType::L2,
};
VectorIndexConfig::ivf_pq(
128, 1024, MetricType::L2, false,
)?;
VectorIndexConfig::IvfRq {
dimension: 128, nlist: 1024, bits: 4, metric: MetricType::L2,
};
VectorIndexConfig::IvfSq {
dimension: 128, nlist: 1024, metric: MetricType::L2,
};</code></pre></div>
<p>The IVF-PQ constructor uses the default relative PQ-code budget and resolves a concrete <code>m</code>. In every option-map API, <code>pq.m</code> is optional: <code>pq.code-ratio=0.0625</code> is the default, and an explicit <code>pq.m</code> takes precedence. Metadata and the on-disk header expose the resolved value.</p>
</section>
<section class="article-section" id="c">
<h2>C FFI</h2>
<p>The C ABI builds <code>libpaimon_vindex_ffi</code>; cbindgen produces the public header. Status-returning operations return <code>0</code> on success and <code>-1</code> on failure. Handle-producing functions such as <code>paimon_vindex_trainer_open()</code>, <code>paimon_vindex_trainer_finish()</code>, <code>paimon_vindex_writer_open()</code>, and <code>paimon_vindex_reader_open()</code> return a handle on success or <code>NULL</code> on failure. The corresponding free functions return <code>void</code> and accept <code>NULL</code>. After a failure, <code>paimon_vindex_last_error()</code> returns the thread-local diagnostic string, or <code>NULL</code> if no error has been recorded.</p>
<div class="code-block"><span class="code-label">C · complete lifecycle</span><pre><code>#include "paimon_vindex.h"
const char *keys[] = {"index.type", "dimension", "nlist", "metric"};
const char *values[] = {"ivf_flat", "128", "1024", "l2"};
PaimonVindexTrainerHandle *trainer =
paimon_vindex_trainer_open(keys, values, 4);
paimon_vindex_trainer_add_training_vectors(
trainer, training_vectors, training_count);
PaimonVindexTrainingHandle *training =
paimon_vindex_trainer_finish(trainer);
paimon_vindex_trainer_free(trainer);
PaimonVindexWriterHandle *writer = paimon_vindex_writer_open(training);
paimon_vindex_training_free(training);
paimon_vindex_writer_add_vectors(writer, row_ids, vectors, vector_count);
paimon_vindex_writer_write_index(writer, output_file);
paimon_vindex_writer_free(writer);
PaimonVindexReaderHandle *reader = paimon_vindex_reader_open(input_file);
PaimonVindexMetadata metadata;
paimon_vindex_reader_metadata(reader, &metadata);
paimon_vindex_reader_optimize_for_search(reader);
int64_t ids[10];
float distances[10];
PaimonVindexSearchParamsEx params =
paimon_vindex_search_params_ex_default();
params.top_k = 10;
params.max_initial_filter_expansion_factor = 4;
paimon_vindex_reader_warmup_queries(reader, representative_queries, 8, 0);
paimon_vindex_reader_search_ex(reader, query, &params, ids, distances, 10);
paimon_vindex_reader_free(reader);</code></pre></div>
<p><code>PaimonVindexSearchParamsEx</code> is append-only and passed by pointer. Prefer <code>paimon_vindex_search_params_ex_default()</code>, which fills the semantic defaults and sets <code>struct_size</code> to <code>PAIMON_VINDEX_SEARCH_PARAMS_EX_V1_SIZE</code>. The size is the exact end of the last initialized field, not <code>sizeof(PaimonVindexSearchParamsEx)</code>, because structure sizes may include tail padding. Newer libraries default fields outside an older caller's initialized prefix and older libraries ignore trailing fields from newer callers. The original by-value search functions remain available for ABI compatibility.</p>
<p><code>PaimonVindexOutputFile</code> and <code>PaimonVindexInputFile</code> are callback structures. The input callback receives every positional range in one I/O batch, allowing object-store implementations to issue reads concurrently. DiskANN batch search may also invoke the callback concurrently from separate query workers, so the callback and its context must be thread-safe. The input structure also carries the three optional read-capability hints described above. C callers own result buffers and pass their capacity explicitly. Metadata reports <code>pq_m</code>/<code>pq_bits</code> for PQ-backed indexes and <code>rq_bits</code> for IVF-RQ. IVF-SQ reports <code>pq_m=0</code> and <code>pq_bits=8</code> to expose its scalar-code width.</p>
</section>
<section class="article-section" id="cpp">
<h2>C++ RAII</h2>
<p><code>include/paimon_vindex.hpp</code> wraps the C ABI with explicit ownership and automatic cleanup.</p>
<div class="code-block"><span class="code-label">C++</span><pre><code>#include "paimon_vindex.hpp"
std::vector<std::pair<std::string, std::string>> options = {
{"index.type", "ivf_flat"}, {"dimension", "128"},
{"nlist", "1024"}, {"metric", "l2"},
};
paimon::vindex::Training training =
paimon::vindex::Trainer::train(
options, training_vectors.data(), training_count);
paimon::vindex::Writer writer(std::move(training));
writer.add_vectors(row_ids.data(), vectors.data(), vector_count);
writer.write_index(output_file);
paimon::vindex::Reader reader(input_file);
auto metadata = reader.metadata();
reader.optimize_for_search();
reader.warmup_queries(representative_queries.data(), 8);
auto params = paimon::vindex::SearchParams::automatic(10);
params.max_initial_filter_expansion_factor = 4;
params.ivfpq_batch_table_reuse = PAIMON_VINDEX_IVFPQ_BATCH_TABLE_REUSE_AUTO;
auto result = reader.search(query.data(), params);</code></pre></div>
</section>
<section class="article-section" id="java">
<h2>Java / JNI</h2>
<p>The package is <code>org.apache.paimon.index.vector</code>. String options map directly to Paimon table and index properties; Rust parses and validates them when a Trainer is created.</p>
<p>The Maven JAR contains the supported Linux x86-64, Linux aarch64, macOS arm64, and Windows x86-64 JNI libraries. The first native API call selects the current platform, extracts that library to a private temporary file, and loads it automatically. To use a separately built library instead, start the JVM with <code>-Dpaimon.vindex.native.path=/absolute/path/to/the/library</code>.</p>
<div class="code-block"><span class="code-label">Java · build and search</span><pre><code>Map<String, String> options = new HashMap<>();
options.put("index.type", "ivf_sq");
options.put("metric", "l2");
try (VectorIndexTraining training =
VectorIndexTrainer.train(options, trainingVectors, trainingCount);
VectorIndexWriter writer = new VectorIndexWriter(training)) {
writer.addVectors(rowIds, vectors, vectorCount);
writer.writeIndex(vectorIndexOutput);
}
try (VectorIndexReader reader = new VectorIndexReader(vectorIndexInput)) {
VectorIndexMetadata metadata = reader.metadata();
reader.optimizeForSearch();
VectorSearchResult result = reader.search(
query,
VectorSearchParams.automatic(10));
}</code></pre></div>
<div class="code-block"><span class="code-label">Java · optional filtered-IVF tuning</span><pre><code>// Example only: select the factor using workload-specific
// latency and Recall@K measurements.
VectorSearchParams params =
VectorSearchParams.automatic(10)
.withMaxInitialFilterExpansionFactor(4);</code></pre></div>
<h3>Batching a large training set</h3>
<div class="code-block"><span class="code-label">Java · Trainer-owned native staging</span><pre><code>try (VectorIndexTrainer trainer = VectorIndexTrainer.create(options)) {
for (float[] batch : trainingBatches) {
trainer.addTrainingVectors(batch, batch.length / dimension);
}
try (VectorIndexTraining training = trainer.finishTraining();
VectorIndexWriter writer = new VectorIndexWriter(training)) {
writer.addVectors(rowIds, vectors, vectorCount);
writer.writeIndex(vectorIndexOutput);
}
}</code></pre></div>
<p>Staging avoids one very large Java <code>float[]</code> and its array-length limit. Native reservoir sampling bounds retained training rows, but streaming creation still needs concrete <code>dimension</code> and either concrete <code>nlist</code> or <code>expected-vector-count</code>.</p>
</section>
<section class="article-section" id="python">
<h2>Python</h2>
<p>The C++, Java, and Python Readers serialize operations on one native handle across threads. Calling another method on that same handle from its input/output callback is rejected as reentrant—even when DiskANN invokes the callback from a native query worker—instead of deadlocking or re-entering a mutable native handle; use a separate handle if a callback needs vector-index work. The Python package loads <code>libpaimon_vindex_ffi</code> with ctypes. A single search returns one-dimensional NumPy arrays. <code>search_batch</code> accepts a two-dimensional query array and returns arrays shaped <code>(query_count, top_k)</code>.</p>
<div class="code-block"><span class="code-label">Python</span><pre><code>from paimon_vindex import (
SearchParams, VectorIndexReader,
VectorIndexTrainer, VectorIndexWriter,
)
class VectorIndexInput:
def __init__(self, data: bytes):
self.data = data
self.estimated_random_read_latency_nanos = 20_000_000
self.preferred_window_bytes = 64 * 1024
self.max_ranges_per_read = 32
def pread_many(self, ranges):
return [self.data[pos : pos + length] for pos, length in ranges]
options = {"index.type": "ivf_sq", "metric": "l2"}
training = VectorIndexTrainer.train(options, training_vectors)
writer = VectorIndexWriter(training)
writer.add_vectors(row_ids, vectors)
writer.write(output)
reader = VectorIndexReader(VectorIndexInput(index_bytes))
reader.optimize_for_search()
reader.warmup_queries(representative_queries)
ids, distances = reader.search(
query, SearchParams.automatic(top_k=10))</code></pre></div>
</section>
<section class="article-section" id="filter">
<h2>Metadata filter pushdown</h2>
<p>The query layer can evaluate metadata predicates through Paimon table or scalar indexes, serialize the allowed row IDs as a 64-bit Roaring bitmap, and pass that set into ANN search as a prefilter. Every binding uses the same wire format:</p>
<div class="table-wrap"><table><thead><tr><th>Language</th><th>Single / batch entry points</th></tr></thead><tbody>
<tr><td>Rust</td><td><code>search_with_roaring_filter</code> / <code>search_batch_with_roaring_filter</code></td></tr>
<tr><td>C</td><td><code>paimon_vindex_reader_search_with_roaring_filter</code> / <code>...search_batch_with_roaring_filter</code></td></tr>
<tr><td>Java</td><td><code>VectorIndexReader.search(..., byte[])</code> / <code>searchBatch(..., byte[])</code></td></tr>
<tr><td>Python</td><td><code>search(..., filter_bytes=...)</code> / <code>search_batch(..., filter_bytes=...)</code></td></tr>
</tbody></table></div>
<div class="callout warning"><strong>Row-ID constraint</strong><code>RoaringTreemap</code> uses the <code>u64</code> domain, so row IDs must be non-negative to match the filter. Filters are query payloads and are never written into index files.</div>
<p>DiskANN maps sparse filters through its persisted row-ID lookup after first use; dense filters scan resident row IDs once per batch. It normally scans PQ codes only for matching nodes, then exactly reranks a bounded candidate set. A quality-gated broad filter may use ordinary graph navigation and post-filter its candidates, with a complete matching-node PQ fallback before raw-vector reads. Benchmark filter-heavy traffic separately because the safe PQ path remains linear in the number of matching nodes.</p>
<nav class="pager" aria-label="Related documentation"><a href="index.html"><small>Choose an index</small>← Index overview</a><a href="development.html"><small>Next</small>Development and benchmarks →</a></nav>
</section>
</article>
</div>
</div></main>
<footer class="site-footer"><div class="footer-inner"><span>Apache Paimon Vector Index</span><span>API and language bindings</span></div></footer>
</body>
</html>