Skip to content

code_search: relevant chunk falls below the relevance floor for some natural-language phrasings #35

Description

@Platonenkov

Found while fixing result crowding (#34). This is a separate defect, and it is the one that actually caused the benchmark miss — the diversity fix does not address it.

What was measured

Tracing the auto-lock feature on the wallet corpus, the Windows lifecycle file src\[Clients]\SB.MobileApp\Platforms\Windows\App.xaml.cs behaves differently depending on phrasing:

query result
what fires when the auto-lock timeout elapses absent at limit=40; path_filter restricted to that exact file returns 0 hits
how does the app lock on resume rank #10 at limit=15, absent at limit=5

The second case is crowding, and #34 addresses that class of failure. The first is not: with path_filter pinned to the file, there is nothing to crowd it out — the chunk simply scores below MinCosineSimilarity = 0.55 (EmbeddingOptions.cs) for that phrasing and is filtered before ranking ever happens.

Why it matters

Across two benchmark runs the semantic arm never opened this file, while the Grep arm found it every time. It is a platform-guarded variant of a feature the query is explicitly about — exactly the "peripheral file" class this tool is weakest on. A caller asking "is this handled on all platforms" gets a confident, incomplete answer.

Worth investigating

  • Is 0.55 too aggressive? It was chosen from a measurement where relevant chunks scored 0.61–0.91 and irrelevant ones 0.33–0.54 — a clean separation on that sample. This file is a counter-example, and one counter-example against a threshold picked from a single sample is worth taking seriously.
  • Does the floor even belong before ranking? Filtering by absolute similarity discards a chunk that might still be the best available answer. A floor applied to the returned set, or a warning when everything scores low, keeps the honest "nothing matched well" signal without silently hiding the best candidate.
  • Is the chunk itself poorly embedded? A platform partial class whose body is mostly window-management plumbing may genuinely not resemble "auto-lock timeout" text, in which case the fix is on the indexing side (including the file path as a lexical signal already exists — check whether it applies here).

Do not raise the threshold blindly to make this one file appear; that trades a known miss for unknown false positives. Measure against the golden-query suite either way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions