Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom

[groksrc]

Context: fresh benchmark run (2026-06-10)

Full LoCoMo retrieval run via basic-memory-benchmarks (run 3e11241b9d56, 1,986 queries): Basic Memory main @ 0.21.6 (commit de53e0e) vs mem0ai 2.0.5, both at current latest with out-of-the-box defaults. Reproducible via the benchmarks repo with basic-memory-benchmarks#13 and basic-memory-benchmarks#14 applied.

Headline (LoCoMo categories 1–4):

	recall@5	recall@10	MRR	content-hit	mean latency	p95
bm-local	0.733	0.839	0.619	0.277	45ms	53ms
mem0-local	0.791	0.891	0.648	0.344	882ms	1,603ms

(Good news vs. the earlier benchmark issue basic-memory-benchmarks#2: content-hit went from 15.5% to ~30% overall since February.)

The finding

Head-to-head per query, BM uniquely missed 281 queries (recall@5) that mem0 got; mem0 uniquely missed 160. Decomposing BM's 281 misses:

• 135 (~48%) are ranking failures, not retrieval failures — the gold doc IS in BM's results, just at rank 6–10 (clustered at 6–8), or partially retrieved for multi-doc answers. recall@10 is already 0.843; the problem is converting it to recall@5.
• 146 are true top-10 misses (separate issue on entity boosting).

Proposal

Add an optional rerank stage over the top-N (e.g. 20) hybrid candidates before final ordering. BM answers in 45ms mean vs mem0's 882ms — a cross-encoder reranker (fastembed ships bge/jina reranker models, consistent with the existing fastembed dependency) costs roughly 50–150ms on this corpus size, leaving BM still ~5–10x faster while directly targeting the largest single bucket of quality loss. Converting even most of the 135 rank-6–10 misses flips the headline comparison.

Worth a diagnostic pass on hybrid fusion weights at the same time — near-misses clustered just below the cutoff suggest the semantic/keyword legs may fuse suboptimally for short queries.

Measurable commit-to-commit with the benchmarks repo's worktree workflow (--bm-local-path + deterministic run IDs).

🤖 Generated with Claude Code

[claude]

Claude finished @groksrc's task in 2m 36s —— View job

---

Issue Triage

• Analyze issue content and classify
• Explore relevant codebase (search/ranking code)
• Check for duplicate issues
• Apply appropriate labels
• Provide triage summary

---

Classification

Field	Assessment
Type	Enhancement — new optional rerank stage on top of existing hybrid search
Priority	High — directly addresses ~48% of retrieval quality loss (135/281 unique misses), backed by quantitative benchmark data
Component	Search pipeline (search_repository_base.py, search_service.py, fastembed_provider.py)
Complexity	Complex — requires new cross-encoder model integration, configuration surface, pipeline hook, and benchmark validation

Labels applied: enhancement

---

Related Issues

This issue overlaps with two existing open issues:

• Add reranking step to search pipeline (local cross-encoder) #618 ("Add reranking step to search pipeline (local cross-encoder)") — Most closely related. Add reranking step to search pipeline (local cross-encoder) #618 proposes a tiered approach: cloud reranker for subscribers, OpenAI-compatible API for local, and no-op default. This issue (Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950) is complementary but takes a different angle: use fastembed's built-in cross-encoder models (already a dep, zero new user-facing config) rather than external APIs/cloud. The benchmark data here provides the quantitative motivation that Add reranking step to search pipeline (local cross-encoder) #618 lacked.
• Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666 ("Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall") — Broader scope; reranking is one item in a larger list. Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950 is more actionable and focused.

Verdict: Not a duplicate — #950 is a focused, evidence-backed proposal for a specific implementation path (fastembed local cross-encoder, ~50–150ms overhead, zero new dependencies) that complements but doesn't duplicate the tiered approach in #618. The benchmark data makes this standalone.

---

Codebase Context

The current hybrid search pipeline in search_repository_base.py:

1. FTS (BM25/ts_rank) + vector (BGE small via fastembed) → scores in [0, 1]
2. Fusion: max(vec, fts) + 0.3 * min(vec, fts) (fixed formula, no reranking hooks)
3. Sort descending → paginate

No existing reranking code or extension hooks. The natural insertion point is after fusion in _search_hybrid() / SearchService.search(), before the final sort + paginate.

fastembed is already a dependency and ships cross-encoder/reranker models (BAAI/bge-reranker-v2-m3, jina-reranker). This means the implementation path the issue proposes — local reranking with no new install-time dependencies — is feasible.

---

Implementation Notes (for whoever picks this up)

1. Candidate expansion: Widen retrieval to top-20 (or configurable rerank_candidates, default 20) before reranking, then return top_k from the reranked set.
2. Provider: fastembed ships reranker models via fastembed.rerank.cross_encoder.TextCrossEncoder — same ONNX/CPU path as the embedder.
3. Config: New semantic_reranker_enabled / semantic_reranker_model config keys; default disabled to avoid latency regression for existing users.
4. Extension point: Add optional reranker hook in SearchRepositoryBase parallel to the existing embedding_provider.
5. Fusion weight tuning (mentioned in the issue): The FUSION_BONUS = 0.3 constant in search_repository_base.py is worth revisiting — short queries may benefit from higher semantic weight.
6. Benchmark gate: The issue correctly notes this is measurable commit-to-commit with the benchmarks repo. Target: recover most of the 135 rank-6–10 misses without regressing latency beyond ~200ms mean.