ivf: Retry only incomplete queries in automatic batch search - #72
Merged
Conversation
Automatic IVF batch search previously expanded nprobe for the entire batch whenever any query returned fewer than the required results. Track incomplete queries per round, retry only those queries, and scatter their results back to the original batch order. Explicit IVF nprobe searches remain a single unchanged batch call. Incremental reuse of previously probed lists is intentionally left to a follow-up change.
shyjsarah
force-pushed
the
codex/ivf-batch-partial-retry
branch
from
August 10, 2026 05:42
534857b to
8bab74a
Compare
shyjsarah
changed the title
shyjsarah
force-pushed
the
codex/ivf-batch-partial-retry
branch
from
August 10, 2026 05:45
8bab74a to
534857b
Compare
shyjsarah
changed the title
jerry-024
reviewed
Aug 10, 2026
jerry-024
reviewed
Aug 10, 2026
jerry-024
added a commit
to jerry-024/paimon-vector-index
that referenced
this pull request
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Automatic IVF batch search currently doubles
nprobefor the entire batch whenever any query returns fewer than the required number of results. A small number of difficult queries can therefore force already-complete queries to be searched repeatedly at every expansion round.Changes
nprobesearches as one unchanged full-batch call.This change intentionally does not reuse previously scanned IVF lists. Retried queries still search
0..nprobe; incremental new-list-only scanning is handled by the follow-up PR.Tests
cargo fmt --all -- --checkcargo test --workspacecargo clippy --workspace --all-targets -- -D warningsmvn -f java/pom.xml test