ivf: incrementally scan expanded batch probe ranges

[shyjsarah]

Summary

This is a follow-up to #72. After #72 removes completed queries from later automatic IVF batch rounds, incomplete queries still rerun the full larger nprobe and rescan old lists. This PR makes those retries incremental: a 16 -> 32 -> 64 expansion scans [16,32) and then [32,64) instead of rescanning [0,32) and [0,64).

Changes

• pass half-open probe ranges through automatic batch expansion for IVF-FLAT, IVF-SQ, IVF-PQ, and IVF-RQ
• seed each expansion round with the query's accumulated Top-K so prior heap results are retained
• scan only newly added lists in filtered and unfiltered batch paths
• preserve ivf: Retry only incomplete queries in automatic batch search #72's bounded retry buffers when packing active queries and their Top-K seeds
• preserve aggregate IVF-RQ search statistics across progressive rounds
• keep fixed-width/scalar searches, storage formats, public APIs, and the C ABI unchanged

Logging and JNI diagnostic bridging are intentionally excluded from this PR and handled separately by #75.

Testing

• cargo fmt --all -- --check
• cargo test -p paimon-vindex-core
  • 451 passed, 1 ignored
• cargo clippy -p paimon-vindex-core --all-targets -- -D warnings
• focused coverage for:
  • half-open probe ranges across multiple expansion rounds
  • carrying accumulated Top-K seeds between rounds
  • bounding packed query and seed buffers during partial retries
  • preserving a valid row ID of -1 via its finite distance
  • IVF-PQ range partitioning and seeded result equivalence with a one-shot search
  • aggregate IVF-RQ statistics across progressive rounds

Notes

• Rebased onto main after ivf: Retry only incomplete queries in automatic batch search #72 was merged.
• The PR now contains one functional commit only.
• Query preprocessing and coarse centroid selection are still recomputed for the smaller active-query set in each round; no cross-round public search-state API is introduced.

[shyjsarah]

Diagnostic follow-up:

• Confirmed the PR ivf: incrementally scan expanded batch probe ranges #73 IVF-PQ batch diagnostic path was executing in EMR Spark, including the filtered batch route across all 14 shards.
• Root cause of the missing messages: the EMR log collector did not capture writes made directly to the Rust/native stderr descriptor.
• Added a JNI bridge that captures the opt-in IVF-PQ diagnostic line on the search thread and mirrors the same line through Java System.err, while retaining the existing native stderr write.
• EMR smoke job jr-375b9ff8d65f53f3 succeeded: 14/14 filtered batch searches emitted complete diagnostics and there were no native errors.
• The smoke showed nq=100, k=3, fixed nprobe=16, and incomplete_queries=0 on all 14 shards. Payload read time dominated filter and PQ scan time.

Minimal validation after cleanup:

• cargo fmt --all -- --check
• focused diagnostic capture tests (3 passed)
• cargo check -p paimon-vindex-jni
• mvn -f java/pom.xml test

The final change is contained in paimon-vector-index; no Paimon source change is required.

[jerry-024]

New: 2; still open: 0; fixed candidates: 0.

[jerry-024]

+1