Introduce SourceManager and parser-independent PyHIR

.github/workflows/ci.yml

@@ -16,5 +16,6 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - run: python -m pip install -e ".[dev]"
-      - run: ruff check .
-      - run: pytest
+      - run: python -m mypy
+      - run: python -m ruff check .
+      - run: python -m pytest

README.md

@@ -2,16 +2,26 @@
 
 A standalone, Python-specific static analysis frontend for CoreTrace.
 
-The first milestone parses Python and lowers a deliberately small language subset to a
-deterministic, Python-aware intermediate representation (PyIR). CFG construction, SSA,
-data-flow, and security rules are intentionally deferred.
+The analyzer loads source through a source manager, adapts parsed Python into a
+parser-independent high-level representation (PyHIR), and lowers a deliberately small
+language subset to deterministic PyIR. CFG construction, SSA, data-flow, and security rules
+are intentionally deferred.
+
+```text
+Python source -> SourceManager -> parser -> PyHIR -> semantic imports -> PyIR
+```
+
+The long-term engine architecture and incremental migration plan are recorded in
+[`docs/architecture.md`](docs/architecture.md).
 
 ## Development
 
 ```bash
 python -m venv .venv
 python -m pip install -e ".[dev]"
-pytest
+python -m mypy
+python -m pytest
+python -m ruff check .
 ```
 
 ## Emit PyIR

docs/architecture.md

@@ -0,0 +1,93 @@
+# CoreTrace Python Analysis Engine Architecture
+
+## Product direction
+
+CoreTrace Python Analyzer is evolving from a security-specific source scanner into an
+extensible static-analysis engine. Security detectors will be plugins built on shared,
+language-level analysis infrastructure.
+
+The target data path is:
+
+```text
+Python project
+    -> source discovery and SourceManager
+    -> parser (Tree-sitter target; stdlib ast adapter initially)
+    -> parser-independent PyHIR
+    -> scopes, imports, and canonical symbols
+    -> control-flow graph
+    -> SSA PyIR
+    -> shared analysis DAG
+    -> models, detectors, and proof/refutation plugins
+    -> normalized findings
+    -> CLI, JSON, and SARIF reporters
+```
+
+## Architectural boundaries
+
+1. Source text and locations belong to `source`; all later representations reference stable
+   source IDs and spans.
+2. Parser-specific objects stop at `frontend`. Semantic analysis and lowering consume PyHIR,
+   never `ast.*` or Tree-sitter nodes.
+3. `semantic` owns scopes, imports, and symbols. PyIR lowering consumes semantic results rather
+   than resolving names itself.
+4. CFG is built before SSA. SSA construction is a transformation over non-SSA control flow.
+5. Analyses are immutable providers managed through a dependency DAG. Results are lazy, cached,
+   and invalidated explicitly after transformations.
+6. Plugins consume versioned, typed result APIs. They do not access engine internals or rebuild
+   shared analyses.
+7. Framework plugins provide models and facts; detectors consume generic facts. The taint engine
+   is global and multi-kind.
+8. Findings store lightweight IDs. Reporters consume normalized findings and never initiate
+   analysis.
+
+## Migration strategy
+
+The migration is incremental so every merged branch leaves the CLI usable.
+
+### Phase 1: frontend foundation
+
+- Introduce `SourceManager`, `SourceId`, and `SourceSpan`.
+- Introduce parser-independent PyHIR.
+- Adapt Python's standard-library AST into PyHIR.
+- Route existing PyIR lowering through PyHIR without changing emitted output.
+
+Tree-sitter is intentionally behind the parser boundary. It can replace the initial AST adapter
+once PyHIR is stable, without affecting semantic analysis or PyIR.
+
+### Phase 2: semantic layer
+
+- Move import binding and canonical symbol resolution into `semantic`.
+- Add lexical scopes, local/global/nonlocal classification, closures, and class scopes.
+- Publish immutable semantic results keyed by stable IDs.
+
+### Phase 3: control flow and SSA
+
+- Build per-function CFGs with explicit terminators and graph integrity checks.
+- Add dominators and dominance frontiers.
+- Transform local reads and writes into SSA with phi nodes and def-use chains.
+
+### Phase 4: shared analysis infrastructure
+
+- Add typed analysis providers and dependency declarations.
+- Add lazy evaluation, caching, and transformation invalidation.
+- Validate the framework with constant propagation and abstract values.
+
+### Later phases
+
+- Plugin manifests, registry, and versioned API.
+- Global multi-kind taint and security model registry.
+- Call graph, function summaries, module graph, and interprocedural fixpoints.
+- Framework models, detectors, proof/refutation, correlation, and reporters.
+- Persistent caching, incremental parsing, parallelism, and memory eviction.
+
+## Current milestone acceptance criteria
+
+The frontend-foundation milestone is complete when:
+
+- the CLI loads files through `SourceManager`;
+- source locations use `SourceSpan` with stable `SourceId` values;
+- supported Python syntax is represented in PyHIR;
+- PyIR lowering no longer imports or accepts `ast` nodes;
+- existing `--emit-ir` output remains deterministic;
+- existing tests pass alongside focused source and PyHIR tests.
+

pyproject.toml

@@ -14,6 +14,7 @@ dependencies = []
 
 [project.optional-dependencies]
 dev = [
+  "mypy>=1.15",
   "pytest>=8.0",
   "ruff>=0.9",
 ]
@@ -32,3 +33,7 @@ testpaths = ["tests"]
 line-length = 100
 target-version = "py311"
 
+[tool.mypy]
+python_version = "3.11"
+strict = true
+files = ["src"]

src/coretrace_python/cli.py

@@ -4,10 +4,12 @@
 import sys
 from pathlib import Path
 
+from coretrace_python.frontend.ast_adapter import HIRBuildError, build_module
 from coretrace_python.frontend.imports import ImportResolutionError
 from coretrace_python.frontend.lowering import LoweringError, lower_module
-from coretrace_python.frontend.parser import ParseError, parse_file
+from coretrace_python.frontend.parser import ParseError, parse_source_file
 from coretrace_python.ir.printer import format_module
+from coretrace_python.source import SourceManager
 
 
 def build_parser() -> argparse.ArgumentParser:
@@ -31,9 +33,18 @@ def main(argv: list[str] | None = None) -> int:
         return 2
 
     try:
-        tree = parse_file(args.path)
-        module = lower_module(tree, filename=str(args.path))
-    except (OSError, ParseError, ImportResolutionError, LoweringError) as error:
+        source = SourceManager().load_file(args.path)
+        syntax_tree = parse_source_file(source)
+        hir_module = build_module(source, syntax_tree)
+        module = lower_module(hir_module)
+    except (
+        OSError,
+        UnicodeError,
+        ParseError,
+        HIRBuildError,
+        ImportResolutionError,
+        LoweringError,
+    ) as error:
         print(f"error: {error}", file=sys.stderr)
         return 1
 

src/coretrace_python/frontend/ast_adapter.py

@@ -0,0 +1,176 @@
+"""Adapter from Python's standard-library AST to parser-independent PyHIR."""
+
+from __future__ import annotations
+
+import ast
+from typing import NoReturn
+
+from coretrace_python.hir import nodes
+from coretrace_python.source import SourceFile, SourceSpan
+
+
+class HIRBuildError(Exception):
+    """A source-located failure to represent parsed syntax as PyHIR."""
+
+
+_BINARY_OPERATORS = {
+    ast.Add: "add",
+    ast.Sub: "sub",
+    ast.Mult: "mul",
+    ast.Div: "div",
+    ast.FloorDiv: "floor_div",
+    ast.Mod: "mod",
+    ast.Pow: "pow",
+    ast.LShift: "lshift",
+    ast.RShift: "rshift",
+    ast.BitOr: "bit_or",
+    ast.BitXor: "bit_xor",
+    ast.BitAnd: "bit_and",
+    ast.MatMult: "matmul",
+}
+_UNARY_OPERATORS = {ast.Invert: "invert", ast.Not: "not", ast.UAdd: "pos", ast.USub: "neg"}
+_COMPARE_OPERATORS = {
+    ast.Eq: "eq",
+    ast.NotEq: "not_eq",
+    ast.Lt: "lt",
+    ast.LtE: "lt_eq",
+    ast.Gt: "gt",
+    ast.GtE: "gt_eq",
+    ast.Is: "is",
+    ast.IsNot: "is_not",
+    ast.In: "in",
+    ast.NotIn: "not_in",
+}
+
+
+class AstHIRBuilder:
+    """Build PyHIR while containing every dependency on ``ast`` in this adapter."""
+
+    def __init__(self, source: SourceFile) -> None:
+        self._source = source
+
+    def span(self, node: ast.AST) -> SourceSpan:
+        end_line = getattr(node, "end_lineno", None)
+        end_offset = getattr(node, "end_col_offset", None)
+        return SourceSpan(
+            source_id=self._source.source_id,
+            start_line=getattr(node, "lineno", 1),
+            start_column=getattr(node, "col_offset", 0) + 1,
+            end_line=end_line,
+            end_column=end_offset + 1 if end_offset is not None else None,
+        )
+
+    def fail(self, node: ast.AST, message: str | None = None) -> NoReturn:
+        span = self.span(node)
+        detail = message or f"unsupported syntax: {type(node).__name__}"
+        raise HIRBuildError(f"{span.display()}: {detail}")
+
+    def expression(self, node: ast.expr) -> nodes.Expression:
+        span = self.span(node)
+        if isinstance(node, ast.Name):
+            return nodes.Name(node.id, span)
+        if isinstance(node, ast.Constant):
+            return nodes.Constant(node.value, span)
+        if isinstance(node, ast.BinOp):
+            operator = _BINARY_OPERATORS.get(type(node.op))
+            if operator is None:
+                self.fail(node.op)
+            return nodes.BinaryOp(operator, self.expression(node.left), self.expression(node.right), span)
+        if isinstance(node, ast.UnaryOp):
+            operator = _UNARY_OPERATORS.get(type(node.op))
+            if operator is None:
+                self.fail(node.op)
+            return nodes.UnaryOp(operator, self.expression(node.operand), span)
+        if isinstance(node, ast.Compare):
+            if len(node.ops) != 1 or len(node.comparators) != 1:
+                self.fail(node, "chained comparisons are not supported yet")
+            operator = _COMPARE_OPERATORS.get(type(node.ops[0]))
+            if operator is None:
+                self.fail(node.ops[0])
+            return nodes.Compare(
+                operator,
+                self.expression(node.left),
+                self.expression(node.comparators[0]),
+                span,
+            )
+        if isinstance(node, ast.Attribute):
+            return nodes.Attribute(self.expression(node.value), node.attr, span)
+        if isinstance(node, ast.Subscript):
+            return nodes.Subscript(self.expression(node.value), self.expression(node.slice), span)
+        if isinstance(node, ast.Call):
+            arguments = tuple(self.expression(argument) for argument in node.args)
+            keywords = tuple(
+                nodes.Keyword(keyword.arg, self.expression(keyword.value), self.span(keyword))
+                for keyword in node.keywords
+            )
+            return nodes.Call(self.expression(node.func), arguments, keywords, span)
+        self.fail(node)
+
+    def statement(self, node: ast.stmt) -> nodes.Statement:
+        span = self.span(node)
+        if isinstance(node, ast.Assign):
+            if len(node.targets) != 1 or not isinstance(node.targets[0], ast.Name):
+                self.fail(node, "only assignment to one local name is supported")
+            target = nodes.Name(node.targets[0].id, self.span(node.targets[0]))
+            return nodes.Assign(target, self.expression(node.value), span)
+        if isinstance(node, ast.Return):
+            value = self.expression(node.value) if node.value is not None else None
+            return nodes.Return(value, span)
+        if isinstance(node, ast.Expr):
+            return nodes.ExpressionStatement(self.expression(node.value), span)
+        if isinstance(node, ast.Pass):
+            return nodes.Pass(span)
+        if isinstance(node, ast.Import):
+            aliases = tuple(
+                nodes.ImportAlias(alias.name, alias.asname, self.span(alias)) for alias in node.names
+            )
+            return nodes.Import(aliases, span)
+        if isinstance(node, ast.ImportFrom):
+            aliases = tuple(
+                nodes.ImportAlias(alias.name, alias.asname, self.span(alias)) for alias in node.names
+            )
+            return nodes.ImportFrom(node.module, aliases, node.level, span)
+        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
+            return self.function(node)
+        self.fail(node)
+
+    def function(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> nodes.Function:
+        if node.decorator_list:
+            self.fail(node, "decorated functions are not supported yet")
+        arguments = node.args
+        has_advanced_arguments = (
+            arguments.posonlyargs or arguments.kwonlyargs or arguments.vararg or arguments.kwarg
+        )
+        if has_advanced_arguments or arguments.defaults or arguments.kw_defaults:
+            self.fail(arguments, "only required positional arguments are supported")
+        parameters = tuple(
+            nodes.Parameter(argument.arg, self.span(argument)) for argument in arguments.args
+        )
+        body = tuple(self.statement(statement) for statement in node.body)
+        return nodes.Function(
+            name=node.name,
+            parameters=parameters,
+            body=body,
+            is_async=isinstance(node, ast.AsyncFunctionDef),
+            span=self.span(node),
+        )
+
+    def module(self, tree: ast.Module) -> nodes.Module:
+        body = tuple(self.statement(statement) for statement in tree.body)
+        if tree.body:
+            first_span = self.span(tree.body[0])
+            last_span = self.span(tree.body[-1])
+            span = SourceSpan(
+                self._source.source_id,
+                first_span.start_line,
+                first_span.start_column,
+                last_span.end_line,
+                last_span.end_column,
+            )
+        else:
+            span = SourceSpan(self._source.source_id, 1, 1, 1, 1)
+        return nodes.Module(body, span)
+
+
+def build_module(source: SourceFile, tree: ast.Module) -> nodes.Module:
+    return AstHIRBuilder(source).module(tree)