feat: add optional bounded LSP semantic providers

.github/workflows/ci.yml

@@ -73,6 +73,12 @@ jobs:
       - name: Run tests
         run: uv run --frozen pytest tests/ -q --tb=short
 
+      - name: Check semantic provider extension
+        run: |
+          uv run --frozen ruff check graphify_semantic_providers tests/test_semantic_provider_*.py
+          uv run --frozen pyright graphify_semantic_providers
+          uv run --frozen graphify-semantic list
+
       - name: Verify install works end-to-end
         run: |
           uv run --frozen graphify --help
@@ -99,7 +105,7 @@ jobs:
 
       - name: bandit (static security analysis)
         continue-on-error: true
-        run: uv run --frozen bandit -r graphify -ll
+        run: uv run --frozen bandit -r graphify graphify_semantic_providers -ll
 
       - name: pip-audit (dependency vulnerabilities)
         continue-on-error: true

README.md

@@ -399,6 +399,11 @@ graphify prs --triage              # AI ranks your review queue (uses whatever b
 graphify prs --conflicts           # PRs sharing graph communities — merge-order risk
 ```
 
+For opt-in compiler-resolved symbols, references, implementations, and call
+hierarchies, see the [semantic provider guide](docs/SEMANTIC-PROVIDERS.md).
+Language-server evidence is bounded, provenance-tagged, and merged into a
+separate output graph; native tree-sitter extraction remains the default.
+
 See the [full command reference](#full-command-reference) below.
 
 ---

docs/SEMANTIC-PROVIDERS.md

@@ -0,0 +1,125 @@
+# Optional semantic provider guide
+
+## What this adds
+
+Graphify already parses popular web languages locally with tree-sitter. This
+extension adds optional compiler/language-server evidence. The AST graph remains
+the always-available baseline; provider facts are additive, provenance-tagged
+evidence and never replace native extraction.
+
+| Provider | Languages | Evidence level in this repository |
+|---|---|---|
+| rust-analyzer | Rust | real-tool smoke + protocol tests |
+| typescript-language-server | TypeScript, JavaScript, JSX/TSX | real-tool smoke + protocol tests |
+| Pyright | Python | real-tool smoke + protocol tests |
+| Eclipse JDT LS | Java | protocol tests; external binary not exercised in CI |
+| JetBrains Kotlin LSP | Kotlin | protocol tests; external binary not exercised in CI |
+| csharp-ls | C# | protocol tests; external binary not exercised in CI |
+| gopls | Go | protocol tests; external binary not exercised in CI |
+| Phpactor | PHP | protocol tests; external binary not exercised in CI |
+| Ruby LSP | Ruby | protocol tests; external binary not exercised in CI |
+
+Provider projects: [rust-analyzer](https://github.com/rust-lang/rust-analyzer),
+[TypeScript language server](https://github.com/typescript-language-server/typescript-language-server),
+[Eclipse JDT LS](https://github.com/eclipse-jdtls/eclipse.jdt.ls),
+[Kotlin LSP](https://github.com/Kotlin/kotlin-lsp),
+[csharp-ls](https://github.com/razzmatazz/csharp-language-server),
+[Pyright](https://github.com/microsoft/pyright),
+[gopls](https://github.com/golang/tools/tree/master/gopls),
+[Phpactor](https://github.com/phpactor/phpactor), and
+[Ruby LSP](https://github.com/Shopify/ruby-lsp).
+
+Production images should pin and verify each selected provider version. Do not
+install every language server in every image: construct domain-specific images
+from the same registry contract.
+
+Validated with real local tools during development:
+
+- `rust-analyzer 1.95.0` on a real Cargo workspace;
+- `typescript-language-server 6.0.0` with `TypeScript 6.0.3` on a real strict
+  TypeScript workspace.
+- `Pyright 1.1.409` on a real two-file Python workspace.
+
+TypeScript 7.0.2 was rejected by the tested language-server release during
+initialization, which is why container builds must pin the compiler/server pair
+rather than installing unbounded `latest` versions. Java, Kotlin, C#, Go, PHP
+and Ruby use the same protocol-tested bounded LSP contract, but remain honestly
+marked as not real-tool-tested until their optional binaries join the integration
+matrix.
+
+`auto` selection requires both a matching source file and a project marker. This
+prevents a polyglot repository from launching language servers for incidental
+examples or vendored snippets. Explicit `--provider` selection intentionally
+overrides that convenience check.
+
+## Safety and failure behavior
+
+- Native AST extraction is never disabled by a provider result.
+- Provider subprocesses use argv execution with `shell=False`.
+- Provider subprocesses receive an allowlisted toolchain environment rather
+  than an unfiltered copy of unrelated credentials.
+- The LSP client advertises no workspace-edit support and rejects edit requests.
+- The workspace, file count, source size, symbol count, RPC message size,
+  request count and timeout are bounded.
+- Source symlinks escaping the workspace are ignored.
+- Output contains symbols, locations and relationships, not source text,
+  process environment, server stderr or model chain-of-thought.
+- Missing providers return `unavailable`; exhausted limits return
+  `budget_exhausted`. Neither condition silently expands a budget.
+- Enrichment is additive and writes a separate output graph by default.
+- A semantic symbol is merged into an AST node only on one unambiguous
+  `(source_file, label)` match. Ambiguous matches stay separate.
+- Every fact includes provider kind, run ID, timestamp, confidence and source
+  range metadata. A registered profile is never described as real-tool proof.
+
+Language servers are external executables and may invoke project tooling (for
+example, compiler checks or build scripts). Run them only on trusted workspaces,
+or inside an appropriately isolated environment. Installing or running a
+language server is never part of Graphify's default extraction path.
+
+## Commands
+
+```bash
+uv sync
+uv run graphify-semantic list
+uv run graphify extract /path/to/repo --code-only
+uv run graphify-semantic run /path/to/repo \
+  --provider auto \
+  --max-files 200 \
+  --max-symbols 5000 \
+  --max-relationship-requests 500 \
+  --request-timeout 20 \
+  --out /path/to/repo/graphify-out/semantic-runs.json
+uv run graphify-semantic merge \
+  /path/to/repo/graphify-out/graph.json \
+  /path/to/repo/graphify-out/semantic-runs.json \
+  --out /path/to/repo/graphify-out/graph.semantic.json
+```
+
+Select explicit providers when `auto` is too broad:
+
+```bash
+uv run graphify-semantic run . \
+  --provider rust-analyzer \
+  --provider typescript-language-server \
+  --out graphify-out/semantic-runs.json
+```
+
+## Adding another language
+
+Custom manifests are trusted configuration because they choose an executable.
+They are strictly shape-checked and commands must be argv arrays.
+
+```json
+{
+  "name": "dart-analysis-server",
+  "languages": ["dart"],
+  "extensions": [".dart"],
+  "command": ["dart", "language-server", "--protocol=lsp"],
+  "binary_env": "GRAPHIFY_SEMANTIC_DART_BINARY",
+  "project_markers": ["pubspec.yaml"],
+  "initialization_options": {}
+}
+```
+
+No runner or merger change is needed.

graphify_semantic_providers/__init__.py

@@ -0,0 +1,18 @@
+"""Optional language-server semantic evidence for Graphify graphs.
+
+Graphify's native AST extraction remains the always-available baseline.  This
+package adds bounded, local evidence from supported language servers. Providers
+never replace or weaken native extraction.
+"""
+
+from .contracts import ProviderKind, ProviderRun, ProviderSpec, ProviderStatus
+from .registry import ProviderRegistry, builtin_registry
+
+__all__ = [
+    "ProviderRegistry",
+    "ProviderKind",
+    "ProviderRun",
+    "ProviderSpec",
+    "ProviderStatus",
+    "builtin_registry",
+]

graphify_semantic_providers/cli.py

@@ -0,0 +1,182 @@
+"""Command line interface for optional semantic provider runs."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+from graphify.paths import write_json_atomic
+
+from .contracts import ProviderKind, ProviderRun, ProviderStatus
+from .lsp import discover_files, resolve_command, run_provider
+from .merge import merge_runs
+from .registry import ProviderRegistry, builtin_registry
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="graphify-semantic")
+    parser.add_argument(
+        "--manifest",
+        action="append",
+        type=Path,
+        default=[],
+        help="operator-trusted custom provider JSON manifest",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    list_parser = sub.add_parser("list", help="list built-in and custom providers")
+    list_parser.add_argument("--json", action="store_true")
+
+    run_parser = sub.add_parser("run", help="collect bounded local semantic evidence")
+    run_parser.add_argument("path", type=Path)
+    run_parser.add_argument("--provider", action="append", default=[])
+    run_parser.add_argument("--out", type=Path, required=True)
+    run_parser.add_argument("--max-files", type=int, default=200)
+    run_parser.add_argument("--max-symbols", type=int, default=5_000)
+    run_parser.add_argument("--max-relationship-requests", type=int, default=500)
+    run_parser.add_argument("--request-timeout", type=float, default=20.0)
+
+    merge_parser = sub.add_parser("merge", help="add provider results to a separate graph file")
+    merge_parser.add_argument("graph", type=Path)
+    merge_parser.add_argument("runs", type=Path)
+    merge_parser.add_argument("--out", type=Path, required=True)
+
+    args = parser.parse_args(argv)
+    try:
+        registry = _registry(args.manifest)
+        if args.command == "list":
+            return _list(registry, args.json)
+        if args.command == "run":
+            return _run(args, registry)
+        if args.command == "merge":
+            return _merge(args)
+    except (KeyError, OSError, ValueError, json.JSONDecodeError) as exc:
+        print(f"graphify-semantic: {exc}", file=sys.stderr)
+        return 2
+    return 2
+
+
+def _registry(manifests: list[Path]) -> ProviderRegistry:
+    registry = builtin_registry()
+    for path in manifests:
+        registry.load_manifest(path.resolve())
+    return registry
+
+
+def _list(registry: ProviderRegistry, as_json: bool) -> int:
+    rows = [
+        {
+            "name": spec.name,
+            "languages": list(spec.languages),
+            "extensions": list(spec.extensions),
+            "available": resolve_command(spec) is not None,
+            "description": spec.description,
+        }
+        for spec in registry.all()
+    ]
+    if as_json:
+        print(json.dumps(rows, indent=2, sort_keys=True))
+    else:
+        for row in rows:
+            availability = "available" if row["available"] else "not installed"
+            print(f"{row['name']}: {', '.join(row['languages'])} ({availability})")
+    return 0
+
+
+def _run(args: argparse.Namespace, registry: ProviderRegistry) -> int:
+    root = args.path.resolve()
+    if not root.is_dir():
+        raise ValueError(f"workspace does not exist: {root}")
+    requested = args.provider or ["auto"]
+    if "auto" in requested:
+        specs = [spec for spec in registry.for_workspace(root) if discover_files(root, spec, 1)]
+    else:
+        specs = [registry.get(name) for name in requested]
+    runs = [
+        run_provider(
+            spec,
+            root,
+            max_files=_positive_int(args.max_files, "max-files"),
+            max_symbols=_positive_int(args.max_symbols, "max-symbols"),
+            max_relationship_requests=_positive_int(
+                args.max_relationship_requests, "max-relationship-requests"
+            ),
+            request_timeout=_positive_float(args.request_timeout, "request-timeout"),
+        )
+        for spec in specs
+    ]
+    payload = {
+        "contract": "graphify-semantic-providers/v1",
+        "workspace": root.name,
+        "runs": [run.to_dict() for run in runs],
+    }
+    args.out.parent.mkdir(parents=True, exist_ok=True)
+    write_json_atomic(args.out.resolve(), payload, indent=2)
+    completed = sum(
+        run.status in {ProviderStatus.COMPLETED, ProviderStatus.BUDGET_EXHAUSTED} for run in runs
+    )
+    print(f"wrote {args.out}: {completed}/{len(runs)} provider runs produced bounded evidence")
+    return 0 if completed or not runs else 1
+
+
+def _merge(args: argparse.Namespace) -> int:
+    graph = _read_json(args.graph, 512 * 1024 * 1024)
+    payload = _read_json(args.runs, 256 * 1024 * 1024)
+    raw_runs = payload.get("runs", []) if isinstance(payload, dict) else []
+    runs: list[ProviderRun] = []
+    for raw in raw_runs:
+        if not isinstance(raw, dict):
+            continue
+        try:
+            status = ProviderStatus(raw.get("status"))
+        except ValueError:
+            continue
+        try:
+            provider_kind = ProviderKind(raw.get("provider_kind", "semantic"))
+        except ValueError:
+            continue
+        runs.append(
+            ProviderRun(
+                provider=str(raw.get("provider", "unknown")),
+                status=status,
+                provider_kind=provider_kind,
+                version=str(raw.get("version", "unknown")),
+                run_id=str(raw.get("run_id", "")) or "legacy-unknown",
+                timestamp=str(raw.get("timestamp", "")) or "unknown",
+                nodes=raw.get("nodes", []) if isinstance(raw.get("nodes"), list) else [],
+                edges=raw.get("edges", []) if isinstance(raw.get("edges"), list) else [],
+            )
+        )
+    merged = merge_runs(graph, runs)
+    args.out.parent.mkdir(parents=True, exist_ok=True)
+    write_json_atomic(args.out.resolve(), merged, indent=2)
+    print(f"wrote additive semantic graph: {args.out}")
+    return 0
+
+
+def _read_json(path: Path, max_bytes: int) -> Any:
+    resolved = path.resolve()
+    if not resolved.is_file():
+        raise ValueError(f"file does not exist: {resolved}")
+    if resolved.stat().st_size > max_bytes:
+        raise ValueError(f"file exceeds size limit: {resolved}")
+    return json.loads(resolved.read_bytes())
+
+
+def _positive_int(value: int, name: str) -> int:
+    if value <= 0:
+        raise ValueError(f"{name} must be positive")
+    return value
+
+
+def _positive_float(value: float, name: str) -> float:
+    if value <= 0:
+        raise ValueError(f"{name} must be positive")
+    return value
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())