Large repo indexing performance: parallel work, optional Rust BM25 tokenizer, faster cache validation

[mazulo]

Summary

I've been running a personal fork of semble for almost 2 months now against a large monorepo. Roughly 15k files and hundreds of thousands of chunks. Almost everything I changed was about performance. The Rust module exists for the same reason.

I follow CONTRIBUTING and wanted to open a discussion issue first before sending PRs.

Fork: https://github.com/mazulo/semble
Matched benchmarks: https://gist.github.com/mazulo/94c8243730bcee05f90e97e2c5059fe8

The fork is currently synced with MinishLab main and tagged as 0.5.2 on my side.

Problem

On large local checkouts, cold indexing is dominated by:

1. Tree-sitter parser load across many languages
2. Embedding hundreds of thousands of chunks
3. BM25 tokenization over every chunk, including identifier splitting for camelCase and snake_case
4. Cache validation that re-walks the tree on every warm start

For my agent workflows the important loops are:

• first semble search or MCP call after a clone or cache miss
• later searches against a warm cache while the repo did not change

Warm search latency was already fine for me. The painful part was time to first useful index, and warm cache validation on big trees.

Why this belongs in semble

These are optimizations inside the existing index, build, and cache pipeline. Not a separate search product.

• Same CLI, MCP, and Python API
• Same hybrid BM25 + dense ranking
• Optional native extension only for the hot tokenization loop, with a pure Python fallback
• No change to how agents call search / find_related

A wrapper cannot fix parser preload, embed batching, or cache mtime validation without forking the library anyway.

What changed in the fork

Four independent pieces. If there is interest I would rather send them as separate PRs.

1. Parallel tree-sitter parser preload before chunking
   Internal only. get_parser releases the GIL, so threads help.

2. Faster cache validation
   Store per-file and per-dir mtimes at save time. Validate with stat instead of a full walk. Legacy caches can fall back to the current walk.

3. Optional Rust BM25 tokenizer via semble-core
   No API change. tokens.tokenize tries semble_core and falls back to pure Python. This would be an optional install. I know CONTRIBUTING is careful about new dependencies, so I want your read on this before I open a PR.

4. Parallel chunk embedding with ThreadPoolExecutor when the chunk count is large
   Internal only. HF tokenizer + numpy pooling release the GIL. Skip below a small threshold.

There is also a small warmup on index load so the first search after a cache hit does not pay a one-shot mixed dtype GEMM cost. model2vec often returns float16 and the vector store runs in float32. Casting the query embedding to float32 and priming encode + a dummy semantic query during load_from_disk fixed that.

Fork reference: https://github.com/mazulo/semble

Relevant commits on the fork history:

• aef7959: parser preload + fast cache validation + large repo profiler
• c3b5091: semble-core Rust tokenizer
• 7e1adc5: parallel embedding

Minimal example

Behaviour stays the same:

semble init
semble search "authentication middleware" ./app/api/v1/auth

If the optional native tokenizer is accepted upstream:

pip install semble-core

No new CLI flags are required for the core perf path. Profiler used for measurement:

uv run python -m benchmarks.profile_large_repo /path/to/large/repo

Matched benchmarks vs current upstream

Same machine. Both runs forced to the same settings:

• _DESIRED_CHUNK_LENGTH_CHARS = 750
• SEMBLE_MODEL_NAME=minishlab/potion-code-16M-v2
• Same chunk count: 341,832 across 14,691 files
• Fork with semble-core installed

Metric	Upstream	Fork	Delta
Cold build	90.8s	83.4s	-8%
Warm cache hit	6.09s	3.80s	-38%
embed_chunks	29.3s	26.7s	-9%
tokenize BM25 prep	7.0s	2.6s	-64%
Search first cold	42.9ms	20.7ms	-52%
Search warm p50	6.8ms	6.3ms	-7%
Search warm p90	8.1ms	6.9ms	-15%
Search warm p99	8.3ms	7.2ms	-13%

Cold build phase detail:

Upstream: chunk 34.5s | embed 29.3s | tokenize 7.0s | bm25 10.1s | total 90.8s
Fork:     chunk 35.6s | embed 26.7s | tokenize 2.6s | bm25  8.9s | total 83.4s

Full raw output is in the gist linked above.

Earlier internal measurements while developing the patches, same settings inside the fork tree:

Patch	Effect
Parallel parser preload	Parser load ~33s → ~1s. One VSCode cold build ~97s → ~49s
Rust BM25 tokenizer	Tokenize ~13s → ~1.5s on that VSCode run
Parallel embedding	Embed ~28.7s → ~19.3s on ~105k chunks

Parallel parser preload does not move chunk_source much on my mostly Python tree. Multi-language repos may still benefit more.

Questions for maintainers

1. Are large repo indexing and warm cache improvements in scope for semble itself?
2. Would an optional semble-core wheel be acceptable if pure Python stays the default path? Or do you prefer staying pure Python only?
3. Prefer four tiny PRs for preload, cache mtimes, tokenize, and embed, plus a small warm-on-load PR? Or one larger perf PR after this discussion?
4. Anything here you would rather not take?

What I am not asking for yet

• Merging the fork as-is
• Shipping my packaging or release workflow differences
• Expanding scope beyond indexing and cache performance

Most importantly: thank you all for semble! It has been a daily driver on large codebases, which is why I spent time on this.

[Pringled]

Hey @mazulo,

Thanks for the investigation and for your interest in improving Semble! Unfortunately, after looking through the benchmarks and implementation, I don’t think we’ll be pursuing these changes. The overall gain for the most important part for us (-8% cold build time) isn’t large enough to justify the additional implementation and maintenance complexity of these changes. E.g. we don’t want to introduce a separate Rust package for BM25 tokenization, and we’re also not planning to adopt the parser preloading or early embedding changes at this point due to the additional complexity and threading considerations it would bring.

As for your question about large repo indexing: our current focus is on incremental/partial reindexing. That should significantly reduce the amount of work required after the initial index build by only reprocessing files that have changed, especially on larger repos. This is currently in PR (see #225) and will likely be part of our next release. Warm cache improvements are not our biggest priority at the moment because in the MCP path (which I think you are also using) this only has to happen once when starting the server, after which searches are very fast. It is something we can have another look at after all the incremental reindexing changes are released though.