fix(index): search legacy truncated PQ indexes

[lance-gatefixer]

Summary

• bound PQ distance-table construction to the persisted sub-vector count
• preserve legacy writer semantics for non-divisible vector dimensions across L2 and dot-product search
• add regression coverage for legacy truncated PQ codebooks

Root cause

Older PQ writers accepted dimensions that were not divisible by num_sub_vectors and silently omitted the tail during training. Search split the full query dimension without applying the persisted sub-vector bound, so some configurations produced an extra chunk and panicked in get_sub_vector_centroids. Current writers already reject this configuration; this change keeps the legacy read path compatible.

Validation

• cargo fmt --all
• cargo test -p lance-index test_distance_with_legacy_truncated_dimension
• cargo test -p lance-index (986 passed, 2 ignored; 8 doctests passed)
• cargo clippy --all --tests --benches -- -D warnings

Fixes #2006

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The distance-table bound matches the legacy prefix-only contract, but the compatibility claim is not yet exercised by an index that the affected historical writer could emit. Please add released-history coverage through the legacy load/search path, plus numeric assertions for the changed L2 and dot-product semantics.

[lance-gatekeeper]

This regression does not exercise the legacy compatibility surface it claims to protect. Before non-divisible dimensions were rejected in #2234, PQ construction only supported 8-bit codes; 4-bit PQ was added later in #3144. Directly constructing ProductQuantizer also bypasses legacy deserialization, and len() == 1 cannot show that L2 and dot distances use the persisted prefix correctly. Please add a checked-in 8-bit index written by a released pre-#2234 version (including the required datagen.py version assertion) and search it end to end. Use nonzero data and assert the expected prefix-only distance for L2; retain synthetic numeric coverage for dot and the generic/prepared L2 branches as needed.

[lance-gatefixer]

Addressed in fa3d326. Added a checked-in Lance 0.10.15 8-bit fixture with a version-asserting datagen and an end-to-end search asserting the persisted-prefix L2 distance of 56. Synthetic coverage now numerically asserts prepared L2, generic L2, and dot-product prefix behavior.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The revision preserves the legacy prefix-only PQ search contract while keeping current writer validation unchanged. The added Lance 0.10.15 fixture and numeric prepared L2, generic L2, and dot-product coverage resolve the earlier compatibility-evidence gap.