feat(core): diversify retrieval results by document - #160
Merged
Conversation
Release highlights: - Keep runtime, build, and publishing dependencies current and vulnerability-free. Release details: - Upgrade compatible dependencies across Core, Chat, TTS, and the landing site. - Pin patched sharp and protobufjs transitive versions. - Align PDF cleanup and Chat runtime metadata with the updated packages. Verification: - pnpm validate - pnpm audit --audit-level moderate
Release highlights: - Return broader document coverage without reducing the requested result count. - Keep lexical fallback complete when the corpus exceeds one scan batch. Release details: - Add a configurable document cap with internal over-retrieval and ranked backfill. - Apply diversification to CLI, MCP, ask, evaluation, portable status, and explanations. - Scan fallback batches across the full eligible corpus and document MMR tradeoffs. Verification: - pnpm validate - Quality benchmark: 100/100 cases, reproducible, recall@10 1.000. - Manual CLI: 10/10 distinct documents by default versus 6/10 uncapped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Added
maxChunksPerDocumentacross configuration, TypeScript, CLI search and ask, MCP, evaluation, portable status, and score explanations.Changed
hybridTextScanLimit.sharpandprotobufjstransitive releases.How to Test
pnpm validate.rgr searchwith the default cap and--max-chunks-per-document 100; the default should increase distinct documents without reducing the requested result count.