Repository navigation
feat: parallel analysis and memory eviction - #43
Merged
Merged
Conversation
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second slice of Phase 10 (
docs/architecture.md§29, §30): the module graph is scheduled by strongly connected components, one wave can run in several processes, and a module's intermediate results are dropped as soon as they are extracted.test: define parallel analysis and eviction behavior), implementation follows.ModuleGraph.schedule(): Tarjan components, then waves by depth in the condensation. A component imports, outside itself, only components of earlier waves, so its imported summaries are final when it starts and mutually importing modules are iterated together. The global fixpoint of the previous engine becomes a per-component fixpoint in topological order, with the same least solution.analyze_project(..., jobs=): components of a wave go to a spawn-basedProcessPoolExecutorwhenjobsis above one. A worker receives paths, the plugin roots and the imported summaries as plain data, rebuilds the configuration from the project root, and returns summaries, call sites and findings through the cache codec. With one job nothing is spawned; the result is identical either way, including the cache, the project plugins and the correlation.ResultsEvictedpass: after a component is extracted, the engine drops PyIR and every derived result and keeps the semantic tables and the engine inputs; project plugins read call graphs rebuilt from the extracted call sites.--jobs Non the CLI, valid with--checkon a directory.Not in this PR, still tracked on #41: incremental parsing with Tree-sitter.
Refs #41