From 86925756335417f130ae25461111e2f09e01afea Mon Sep 17 00:00:00 2001 From: Aleksandr Platonenkov Date: Wed, 5 Aug 2026 12:25:51 -0300 Subject: [PATCH 1/2] docs(embedding): record issue #35's relevance-floor investigation Measured the wallet corpus's Windows-only auto-lock wiring chunk (Platforms/Windows/App.xaml.cs's OnAppWindowChanged) directly against Ollama for both reported phrasings, bypassing the floor entirely: - "what fires when the auto-lock timeout elapses" -> cosine 0.4184, 0.13 below MinCosineSimilarity and squarely inside the measured noise band (0.3321-0.5402), not a borderline miss. - "how does the app lock on resume" -> cosine 0.5880 for the exact same chunk, comfortably above the floor. The method body never mentions "lock", "auto", or "timeout", and the file-path lexical signal RoslynChunker already embeds does not help here because the path itself carries no lock/timeout vocabulary either. This is a phrasing-sensitive content gap in one chunk's embedding, not a systemic threshold miscalibration -- lowering the floor far enough to admit 0.4184 would re-admit several of the genuinely off-topic queries the constant was measured against. No production behavior changes; the 0.55 default is left as-is. This records the measurement and reasoning on MinCosineSimilarity itself, matching how the constant's original derivation is already documented there. --- .../Embedding/EmbeddingOptions.cs | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/CodeIndex.Core/Embedding/EmbeddingOptions.cs b/src/CodeIndex.Core/Embedding/EmbeddingOptions.cs index 60d587d..4eafe8b 100644 --- a/src/CodeIndex.Core/Embedding/EmbeddingOptions.cs +++ b/src/CodeIndex.Core/Embedding/EmbeddingOptions.cs @@ -123,6 +123,28 @@ public sealed class EmbeddingOptions /// (rank-1 cosine for genuine vs. off-topic queries against a real index) before trusting this /// default on a different configuration. /// + /// + /// Issue #35 case study — a genuine miss is not always a threshold problem. The + /// wallet corpus's Windows-only lifecycle wiring — Platforms/Windows/App.xaml.cs's + /// OnAppWindowChanged method, which calls OnAppSleep/OnAppResume when the + /// window is minimized/restored — never surfaces for the query "what fires when the auto-lock + /// timeout elapses": measured directly against Ollama (bypassing this floor), that chunk's + /// cosine similarity for that query is 0.4184 — 0.13 below this floor, and squarely + /// inside the noise band above (0.3321–0.5402), not a marginal miss sitting at the boundary. + /// The method body never mentions "lock", "auto", or "timeout"; it only calls two + /// unrelated-looking methods on another type. already + /// prepends the file's own path as a lexical signal ("File: {path}\n" — see + /// BuildEmbedText), but that does not help here because the path itself + /// ("Platforms/Windows/App.xaml.cs") carries no lock/timeout vocabulary either. Tellingly, a + /// different, path-adjacent phrasing — "how does the app lock on resume" — scores 0.5880 + /// for the exact same chunk, comfortably above the floor: this is a phrasing-sensitive content + /// gap in one file's embedding, not a systemic threshold miscalibration. Lowering the floor far + /// enough to admit 0.4184 would also re-admit several of the genuinely off-topic queries this + /// constant was measured against (the noise band tops out at 0.5402), trading one known miss + /// for unmeasured false positives elsewhere — so the floor was left unchanged after this + /// investigation. See issue #35 for the full measurement and the alternatives considered (and + /// rejected, for lack of measurement) before reaching that conclusion. + /// /// public double MinCosineSimilarity { get; set; } = DefaultMinCosineSimilarity; From bebb680432c4869c565e3ae93edbcef390592db7 Mon Sep 17 00:00:00 2001 From: Aleksandr Platonenkov Date: Wed, 5 Aug 2026 12:28:22 -0300 Subject: [PATCH 2/2] docs: state the call-graph blind spot as a known limitation The #35 investigation is recorded on MinCosineSimilarity, where a maintainer deciding whether to move that constant will find it. The finding generalises past that constant, though, and the generalisation belongs where a user deciding which tool to reach for will see it. Similarity is computed against a chunk's own text, so code that takes part in a feature without naming it has nothing to match on. Grep reaches it in two hops -- find the symbol, then find its callers -- and embedding similarity has no equivalent second hop. That is a capability boundary rather than a tuning problem, and it is the concrete mechanism behind the coverage gap the benchmark measured (82% vs 88% of required facts), since platform-guarded variants and parallel code paths are precisely the shapes that participate without naming. Ends with the actionable form: for completeness questions, use Grep, or use code_search to find the symbol and Grep to find its callers. --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index af8ecb1..f3b2075 100644 --- a/README.md +++ b/README.md @@ -803,6 +803,23 @@ the wrapper regardless of case, spacing, or hidden Unicode tricks. - **Markdown can outrank the code it documents for doc-shaped queries.** See [Narrowing to code vs. documentation](#narrowing-to-code-vs-documentation) for what was measured and how to filter it out with `kind` when it matters. +- **Code that participates in a feature only by *calling into* it is close to invisible here.** + Similarity is computed against a chunk's own text, so a method that takes part in a feature + without naming it has nothing to match on. Measured case: `wallet`'s Windows lifecycle handler + `OnAppWindowChanged` calls `OnAppSleep`/`OnAppResume` on minimise/restore and is therefore part + of the auto-lock mechanism, but its body never says "lock", "auto" or "timeout", and neither + does its path. Against the query *"what fires when the auto-lock timeout elapses"* it scores + **0.4184** — well inside the band of genuinely irrelevant results, not marginally below the + floor. `Grep` finds it easily, because once you know the symbol `OnAppResume` a literal search + reaches every caller; embedding similarity has no equivalent of that second hop. + This is a capability boundary, not a tuning problem — lowering the relevance floor far enough + to admit 0.4184 would re-admit measured off-topic matches too (see `MinCosineSimilarity`'s + remarks and issue #35). It is also the concrete mechanism behind the coverage gap in + [Cost and accuracy](#cost-and-accuracy-what-four-measurements-actually-showed): platform-guarded + variants, alternative implementations and parallel code paths are exactly the shapes that + participate without naming. **If completeness is what you need — "find every caller", "is this + handled on all platforms" — use `Grep`, or use `code_search` to find the symbol and `Grep` to + find its callers.** - **Chunk ids do not survive a reindex — but reusing a stale one is now detected, not silently wrong.** Each id (e.g. `"myproject:3:4137"`) is an ordinal position in the specific index snapshot of *that project* a `code_search` call ran against, plus the generation number of that