fix: apply fusion weighting uniformly and recalibrate the exact-keyword floor - #984
fix: apply fusion weighting uniformly and recalibrate the exact-keyword floor#984benjaml4 wants to merge 1 commit into
Conversation
…or (CortexReach#978) fuseResults() scored candidates differently by branch: candidates with a vector hit got clamp01(max(v*vectorWeight + b*bm25Weight, keywordFloor)), while BM25-only candidates got their normalised BM25 score directly — bypassing bm25Weight entirely. Since store.bm25Search sigmoid-normalises raw BM25 (floor 0.5 by construction), any keyword-only hit outranked semantic candidates capped at vectorWeight, regardless of configured weights. Score every candidate with the same weighted formula, and keep the exact-keyword floor on all branches — recalibrated to the sigmoid: the old 0.75 threshold corresponds to raw BM25 ~5.5, which nearly every FTS hit clears (the floor value b*0.92 >= 0.69 then outranks any vector hit capped at vectorWeight, quietly re-creating the bypass). 0.95 maps to raw ~14.7 — genuinely exceptional matches like identifiers and exact strings, preserving the floor's documented purpose of keeping those above minScore. Observed on a 7,253-row store: for queries with no relevant corpus content, unrelated keyword-only hits (normalised 0.92) were returned at floored confidence 0.85, above every semantic candidate. With this change they score b*bm25Weight (~0.28, below the default minScore) and are correctly suppressed, while on-topic queries are unaffected.
rwmjhb
left a comment
There was a problem hiding this comment.
Reviewed head 2043eb9. Applying configured fusion weights uniformly is the right direction, but the new 0.95 keyword-floor policy introduces two retrieval blockers.
test/retriever-rerank-regression.mjsfails because a strong lexical/semantic candidate (vector0.5006586, BM250.78) now fuses to about0.5845, falls belowminScore=0.6, and is removed before reranking.- BM25 graceful degradation is broken. With default
bm25Weight=0.3, every BM25-only score below0.95is weighted below0.285and cannot pass the defaultminScore=0.3; the existing graceful-degradation test reproduces this with a0.9BM25 hit when vector search fails.
Please preserve configured weighting when both backends participate while renormalizing or using the surviving backend's score when one backend fails, and replace the universal floor with a calibrated/configurable exact-match signal that keeps existing strong lexical matches. Add boundary tests around the floor and keep both existing regressions green.
Requesting changes.
rwmjhb
left a comment
There was a problem hiding this comment.
Fresh recheck of 2043eb9: there have been no author commits or replies since the August 4 changes-requested review, and the retrieval regressions remain reproducible.
node test/retriever-rerank-regression.mjsfails at the strong lexical-hit assertion: the expected result count is 1, but the new fusion/floor policy returns 0 before reranking.node --test test/retriever-graceful-degradation.test.mjsfails 2 of 8 tests, including the BM25-only fallback when vector search fails; the expected result count is 1, but weighting a surviving BM25 score by the default 0.3 removes it.
Please renormalize to the surviving backend when one search backend fails, and replace the universal 0.95 keyword floor with a calibrated/configurable exact-match rule that preserves existing strong lexical hits. Add boundary regressions before requesting re-review. The existing CHANGES_REQUESTED decision still applies.
Addresses the confirmed remainder of #978 (post-correction).
As the reporter's follow-up established, BM25 scores are sigmoid-normalised before fusion — but the structural asymmetry stands: the BM25-only branch used the normalised score directly, bypassing
bm25Weight, so keyword-only candidates (sigmoid floor 0.5) outranked semantic candidates capped atvectorWeightno matter how the weights were configured.Two changes:
clamp01(max(v·vectorWeight + b·bm25Weight, keywordFloor), 0.1).1/(1+e^(-raw/5))reaches 0.75 at raw ≈ 5.5, which almost every FTS hit clears, so the floor (b·0.92 ≥ 0.69) quietly re-created the bypass on both branches. 0.95 corresponds to raw ≈ 14.7 — genuinely exceptional matches (identifiers, exact strings), preserving the floor's documented purpose of keeping those aboveminScore.Observed effect on a 7,253-row store: queries with no relevant corpus content used to return unrelated keyword-only hits at floored confidence 0.85; they now score ~b·bm25Weight (≈0.28, below default
minScore) and are suppressed, while on-topic queries are unaffected (verified before/after on the same store).If you'd prefer the floor threshold configurable rather than a constant, happy to add a
retrieval.keywordFloorThresholdoption instead. An alternative worth considering longer-term is rank-based fusion (RRF) — which the comment above this code already names — as it sidesteps score-scale calibration entirely.tsc --noEmitclean.