[core] Validate ANN segment metric against the built-in metric - #10264
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
The metric guard in PkVectorAnnSegmentSearcher compared two values that both derive from the current table config, so it could never fail: changing the metric option after ANN segments were built let searches score old segments with the new metric and silently return wrong distances. Record the effective metric in the segment's vector index metadata when writing it, expose it through VectorGlobalIndexer.segmentMetric, and reject segments whose recorded metric differs from the current one. Legacy segments that record no metric are still accepted. Assisted-by: GLM-5.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
PkVectorAnnSegmentSearcheris meant to reject searching an ANN segment whose distance metric differs from the configured one, but the guard compares two values that both come from the current table configuration, so it is always true and never rejects anything. Once the configured metric changes, the already-built segments are scored with the new metric and the nearest-neighbor results are silently wrong.This records the metric each ANN segment was built with in
VectorIndexMetaand exposes it through a newVectorGlobalIndexer.segmentMetric(byte[]), a default method that returns null and is overridden byNativeVectorGlobalIndexer. The searcher now compares the segment's recorded metric against the configured metric and fails fast on a mismatch. Segments written before this change carry no metric and are treated as exempt, so existing indexes keep working, and otherVectorGlobalIndexerimplementations that record no metric return null and are skipped.This closes #10263.
Tests
PkVectorAnnSegmentSearcherMetricTestpins the guard: a matching metric passes, a differing metric fails fast, a legacy segment with no recorded metric is exempt, and alias or case differences (both sides go through the samenormalize) do not falsely reject.VectorIndexMetaTestpins that the metric round-trips through the segment metadata and that legacy metadata carrying no metric deserializes to null.API and Format
The persisted vector segment metadata gains an optional
metricfield. Metadata written before this change has no such field and deserializes to null, which is treated as exempt, so the change is backward compatible. Flagging for format review.Documentation
No.