Skip to content

fix(index): correct cosine RabitQ query factor - #8393

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-5649-1
Open

fix(index): correct cosine RabitQ query factor#8393
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-5649-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • treat normalized cosine as L2 when calculating the residual-query RabitQ partition factor
  • preserve the Dot-specific offset
  • compare full cosine and L2 calculator outputs in a regression test

Root cause

Cosine vectors are normalized before IVF_RQ search, so the residual-query estimator operates on squared L2 distances. Its query-factor match still grouped Cosine with Dot and subtracted 1.0, shifting every approximate cosine distance.

Validation

  • cargo test -p lance-index test_residual_query_cosine_uses_l2_query_factor
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #5649

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Aug 7, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The residual-query estimator now uses the squared-L2 partition factor required by normalized cosine vectors while preserving the distinct dot-product offset. This is the narrow fix for the incorrect legacy score and bound semantics, and the focused regression covers the affected calculator path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(rust): wrong q_factor for Cosine in RabitQuantizationStorage::dist_calculator

0 participants