[fix](clucene) Support block WAND in composite readers#401
Open
hoshinojyunn wants to merge 1 commit into
Open
Conversation
### What problem does this PR solve? Issue Number: close #62955 Related PR: None Problem Summary: Implement the block postings API in MultiTermDocs so composite readers globalize block docids, cross child-reader boundaries, and expose block statistics to WAND. Correct SegmentTermDocs handling of the skip-list sentinel returned for the first block, which otherwise lets a subsequent block read consume stale postings. Add MultiReader regression coverage for block reads and cross-reader skip targets. ### Release note None ### Check List (For Author) - Test: Unit Test (./bin/cl_test testReadRange) - Behavior changed: Yes (composite readers support block WAND APIs) - Does this need documentation: No
hoshinojyunn
force-pushed
the
fix/multi-reader-block-wand
branch
from
dd28de6 to
177dcf8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close apache/doris#62955
Problem Summary: MultiTermDocs did not implement the block postings APIs used by block WAND. Composite readers therefore could not provide block reads, block skipping, or block statistics across child readers. This change implements those APIs, translates child docIDs to global docIDs, handles reader boundaries, and corrects the first-block skip-list sentinel handling in SegmentTermDocs.
Why normalize the skip count?
MultiLevelSkipListReader::skipTo()returnsnumSkipped - skipInterval - 1. Its negative result is a sentinel meaning no postings have been skipped, not a valid SegmentTermDocs count.Example with two 2,048-document child readers and a skip interval of 128:
skipToBlock(2085)selects the second child reader and passes local target 37.0..127. Loading that entry makes numSkipped equal 128, soskipTo(37)returns 128 - 128 - 1 = -1.Release note
None
Check List (For Author)