FTS retrieval quality: camelCase sub-word recall, column-weighted bm25, path filtering, dedupe

[LordXyTh]

Ran codegraph (1.5.0) on a ~13k-file multi-repo workspace (TS/Python/PHP/Liquid) and measured the FTS layer against the live DB. Five findings, each with a small, sourced fix. Happy to split into separate issues if you prefer.

1. camelCase identifiers are unreachable by sub-word query (biggest one)

nodes_fts uses the default unicode61 tokenizer. _ is a separator (Unicode Pc), so snake_case works — but there's no case-boundary splitting, so getShippingMethodIdFromCheckout indexes as one opaque token.

Measured on my index: 53.5% of symbols are camelCase/PascalCase, and 93.4% of them can't be found by any of their constituent words (e.g. query checkout reaches 6 of 76 camelCase symbols containing it; session 77/1,383; product 110/1,473).

Fix: a custom FTS5 tokenizer emitting the whole identifier as the primary token plus each sub-word with FTS5_TOKEN_COLOCATED at the same position — whole-identifier exact match and sub-word match from one index, prefix queries intact. Documented mechanism: https://www.sqlite.org/fts5.html#custom_tokenizers (§7.1, the synonym example transposes directly). Cheaper alternative: pre-split identifiers into a denormalized search column at insert time.

2. No column weighting in ranking

Default rank weights name, qualified_name, docstring, signature equally, so a docstring mention scores like a symbol-name hit. For code search the name should dominate — Zoekt weights symbol matches 14x over content words (scoreSymbol 7000 vs scoreWordMatch 500, https://github.com/sourcegraph/zoekt/blob/main/index/contentprovider.go).

One-line fix, persisted in the table: INSERT INTO nodes_fts(nodes_fts, rank) VALUES('rank', 'bm25(0.0, 10.0, 5.0, 1.0, 2.0)');

3. id is in the FTS index

Opaque node IDs are never a useful query target — pure index bloat and a source of accidental matches. Suggest dropping the column from the FTS table (or UNINDEXED).

4. No way to scope a query to a path/repo

file_path isn't in the FTS table and codegraph query has no --path/--repo flag. In a multi-repo workspace, "find X in repo Y" is the most common query shape, and today the only option is client-side filtering of global results. A --path <glob> flag (SQL WHERE file_path GLOB ... on the join) would cover it.

5. Duplicate results from scaffolded/copied code

Workspaces with scaffolded projects (cookiecutter, copied Shopify themes) produce many identical qualified_names; my index has 846 qualified_names spanning >1 repo, and generic names like Migration appear 1,600+ times. Top-N results fill up with copies of the same symbol. GROUP BY qualified_name (keep best rank, note the other locations) plus an optional per-path-prefix priority tiebreaker fixes the result list cheaply — Zoekt's per-repo rank tiebreaker is precedent.

---

Before/after sanity check: applying 2+5 manually against my DB (weighted bm25 subquery + GROUP BY qualified_name) turned generic queries like checkout from duplicate-heavy noise into exactly the current-code hits an agent wants. Great tool — these five are all at the SQL/tokenizer layer and would compound with the graph features that already work well.