Version
built from source, main @ 2278498 (v0.11.0-186)
Platform
Linux (x64)
Install channel
Built from source
Binary variant
standard
What happened, and what did you expect?
cbm_py_build_cross_registry passes Python defs to py_register_lsp_defs one at a time with idx_arena == NULL (internal/cbm/lsp/py_lsp.c:5130-5136). That skips the mid-build finalize (py_lsp.c:5004-5010), so every Method's receiver probe (py_lsp.c:5049) is a linear strcmp scan over all types registered so far (type_registry.c:830-834). On a Python-heavy tree the build runs single-threaded for over a minute while the other workers wait. Incremental reindexes repeat it whenever the closure exceeds 8 files (src/pipeline/pipeline_incremental.c:1325). This is a different path from #1527, which is per-file resolution (py_lsp.c:5095).
I have a fix that backs only the existence probe with a cbm_ht set of type QNs, as the Go (959f4de) and Rust (10b3b79) fixes did. It keeps the per-def order, so the graph is byte-identical and peak RSS is unchanged.
Reproduction
To reproduce, index generated modules of 10 classes with 8 methods each, where each class inherits the previous one and calls into an imported class. My real-world case is the .py/.pyi files of a Python 3.14 install (26,233 files, 40,923 classes, 222,253 methods), with site-packages renamed because discovery skips it and mpmath removed because it hits a separate bug.
| Linux x86_64, 32 logical CPUs, GCC 16 |
main |
fixed |
Python builder (lsp_cross_prepare.builders), Python 3.14 tree, 3 runs |
67-75 s |
0.20-0.25 s |
| Same builder, generated 2k / 8k / 32k classes |
0.11 / 1.5 / 22 s |
8 / 28 / 135 ms |
| Whole index wall time, Python 3.14 tree |
175-182 s |
113 s |
Logs
Diagnostics trajectory (memory / performance / leak issues)
Not a memory problem: one CPU-bound, single-threaded phase of a cold index run. The fix leaves peak RSS unchanged.
Project scale (if relevant)
No response
Confirmations
Version
built from source, main @ 2278498 (v0.11.0-186)
Platform
Linux (x64)
Install channel
Built from source
Binary variant
standard
What happened, and what did you expect?
cbm_py_build_cross_registrypasses Python defs topy_register_lsp_defsone at a time withidx_arena == NULL(internal/cbm/lsp/py_lsp.c:5130-5136). That skips the mid-build finalize (py_lsp.c:5004-5010), so every Method's receiver probe (py_lsp.c:5049) is a linearstrcmpscan over all types registered so far (type_registry.c:830-834). On a Python-heavy tree the build runs single-threaded for over a minute while the other workers wait. Incremental reindexes repeat it whenever the closure exceeds 8 files (src/pipeline/pipeline_incremental.c:1325). This is a different path from #1527, which is per-file resolution (py_lsp.c:5095).I have a fix that backs only the existence probe with a
cbm_htset of type QNs, as the Go (959f4de) and Rust (10b3b79) fixes did. It keeps the per-def order, so the graph is byte-identical and peak RSS is unchanged.Reproduction
To reproduce, index generated modules of 10 classes with 8 methods each, where each class inherits the previous one and calls into an imported class. My real-world case is the
.py/.pyifiles of a Python 3.14 install (26,233 files, 40,923 classes, 222,253 methods), withsite-packagesrenamed because discovery skips it and mpmath removed because it hits a separate bug.lsp_cross_prepare.builders), Python 3.14 tree, 3 runsLogs
Diagnostics trajectory (memory / performance / leak issues)
Project scale (if relevant)
No response
Confirmations