Skip to content

feat: parallel analysis and memory eviction - #43

Merged
SizzleUnrlsd merged 2 commits into
mainfrom
feat/parallel
Sep 4, 2026
Merged

SizzleUnrlsd merged 2 commits into
mainfrom
feat/parallel

Conversation

@SizzleUnrlsd

Copy link
Copy Markdown
Contributor

Summary

Second slice of Phase 10 (docs/architecture.md §29, §30): the module graph is scheduled by strongly connected components, one wave can run in several processes, and a module's intermediate results are dropped as soon as they are extracted.

  • Acceptance tests first (test: define parallel analysis and eviction behavior), implementation follows.
  • ModuleGraph.schedule(): Tarjan components, then waves by depth in the condensation. A component imports, outside itself, only components of earlier waves, so its imported summaries are final when it starts and mutually importing modules are iterated together. The global fixpoint of the previous engine becomes a per-component fixpoint in topological order, with the same least solution.
  • analyze_project(..., jobs=): components of a wave go to a spawn-based ProcessPoolExecutor when jobs is above one. A worker receives paths, the plugin roots and the imported summaries as plain data, rebuilds the configuration from the project root, and returns summaries, call sites and findings through the cache codec. With one job nothing is spawned; the result is identical either way, including the cache, the project plugins and the correlation.
  • ResultsEvicted pass: after a component is extracted, the engine drops PyIR and every derived result and keeps the semantic tables and the engine inputs; project plugins read call graphs rebuilt from the extracted call sites.
  • --jobs N on the CLI, valid with --check on a directory.

Not in this PR, still tracked on #41: incremental parsing with Tree-sitter.

Refs #41

@SizzleUnrlsd SizzleUnrlsd self-assigned this Sep 4, 2026
@SizzleUnrlsd
SizzleUnrlsd merged commit ffb49bf into main Sep 4, 2026
6 checks passed
@SizzleUnrlsd
SizzleUnrlsd deleted the feat/parallel branch September 4, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant