feat(parser): add Rust support (roadmap #16b) — reland of #18 + #29

.github/skills/harden-tree-sitter-parsers/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: harden-tree-sitter-parsers
+description: Harden new or changed tree-sitter parsers against adversarial nesting, malformed files, partial-state leaks, and stale non-source cache inputs.
+reliability: validated-2x
+---
+
+# Harden Tree-sitter Parsers
+
+Use this skill for new parser implementations, parser reviews, recursion failures,
+or changes to AST traversal and linking.
+
+## Required workflow
+
+1. Inventory every AST traversal helper and classify it as iterative or recursive.
+2. Replace source-depth recursion with an explicit stack or queue. Preserve traversal
+   order deliberately and avoid repeated tuple/list copying where practical.
+3. Extract each file into isolated temporary state. Merge entities, edges, imports,
+   packages, and pending links only after the file succeeds.
+4. Log skipped files with the failure class; do not silently discard valid siblings.
+5. Add adversarial fixtures deeper than `sys.getrecursionlimit()` for every distinct
+   traversal shape. Each fixture must be parsed beside a valid sibling file.
+6. Test cache invalidation for manifests or configuration that changes graph identity.
+7. Time the parser on one large generated single-package file before declaring it done.
+   A de-duplicated ordered collection guarded by `if x not in list` is quadratic and only
+   shows up at scale (see `docs/BUG_CATALOG.md` #3).
+8. Run, in order:
+   - focused parser and cache tests;
+   - Ruff and the full test suite;
+   - a large real repository for the target language, reporting wall-clock time;
+   - arcade-agent self-analysis before/after, reporting metric and smell deltas.
+9. Record any newly discovered reusable failure class in `docs/BUG_CATALOG.md`.
+
+## Acceptance invariants
+
+- No `RecursionError` for valid tree-sitter AST depth within the configured file limit.
+- One malformed or adversarial file cannot erase healthy sibling entities.
+- No partial entities from a failed file enter the final graph.
+- No dangling edges, missing method owners, or duplicate package membership.
+- Parse time grows linearly, not quadratically, with entity count.
+- Relevant non-source inputs invalidate cached graphs, and so do flags such as
+  `exclude_tests` that change the graph for an identical file list.
+- Test exclusion covers every syntactic shape the language offers, imports included.
+- Correctness and explicit failure behavior take precedence over cosmetic metric gains.
+
+## Evidence
+
+- Kotlin follow-up `b7effc5`: iterative deep-expression traversal and sibling survival.
+- Rust PR #18: iterative path/use/module/type traversal, transactional file extraction,
+  Cargo-aware cache invalidation, and adversarial regression matrix.
+- Rust reland: linear package membership (70.2 s -> 3.0 s on a 5.2 MB generated file, with
+  identical entity and edge counts) and full `#[cfg(test)]` shape coverage.

.github/workflows/architecture-analysis-reusable.yml

@@ -24,7 +24,7 @@ on:
         type: string
         default: "."
       language:
-        description: Optional language override (java, python, typescript, c, go, kotlin, multi).
+        description: Optional language override (java, python, typescript, c, go, kotlin, rust, multi).
         required: false
         type: string
         default: ""

README.md

@@ -135,6 +135,16 @@ smell burden, or another architectural pressure.
 - TypeScript/JavaScript (full support)
 - Go (full support)
 - Kotlin (structural support via optional `[languages]` extra; import + inheritance graph)
+- Rust (structural support via optional `[languages]` extra; structs, enums, unions, traits,
+  type aliases, functions, methods, imports, qualified references, trait
+  inheritance/implementations, and Cargo workspaces)
+
+Rust unit tests live *inline* in production files, so path-based test exclusion
+cannot see them. With `exclude_tests=True` (the default) the Rust parser also
+drops `#[cfg(test)]` items — including `cfg(all(test, ...))` / `cfg(any(test, ...))`,
+inner `#![cfg(test)]` files and module bodies, the file behind an out-of-line
+`#[cfg(test)] mod helpers;`, and `#[cfg(test)] use ...` dev-dependency imports.
+Pass `exclude_tests=False` to `ingest`/`parse`/`analyze` to keep them.
 
 ## Example: ARCADE Core
 
@@ -381,7 +391,7 @@ arcade-agent ports and extends the capabilities of the original [ARCADE](https:/
 | 6 quality metrics | Done | RCI, TurboMQ, BasicMQ, IntraConnectivity, InterConnectivity, TwoWayPairRatio |
 | Balanced architecture score | Done | Derived reporting score combining core metrics, principle signals, and smell burden |
 | A2A architecture comparison | Done | Hungarian algorithm on Jaccard similarity |
-| Multi-language parsing | Done | Java, Python, C/C++, TypeScript/JavaScript, Go (full); Kotlin (structural); polyglot merge+relink via `languages=[...]` / `language="multi"` (cross-language edges within the JVM family only) |
+| Multi-language parsing | Done | Java, Python, C/C++, TypeScript/JavaScript, Go (full); Kotlin, Rust (structural); polyglot merge+relink via `languages=[...]` / `language="multi"` (cross-language edges within the JVM family only) |
 | 5 export formats | Done | HTML, DOT, JSON, RSF, Mermaid |
 | LLM concern extraction | Done | Claude CLI for semantic BCO/SPF detection |
 | MCP server | Done | Expose tools to AI agents via Model Context Protocol with session store |

ROADMAP.md

@@ -43,7 +43,9 @@ Handle real-world polyglot monorepos.
 - [x] **15. TypeScript/JS parser** — Shipped in #8 (`parsers/typescript.py`).
 - [x] **16a. Go parser** — Shipped alongside TS/JS in #8 (`parsers/go.py`).
 - [x] **16a2. Kotlin parser** — Shipped (`parsers/kotlin.py`) for JVM/Kotlin-first repos (e.g. embabel-agent).
-- [ ] **16b. Rust parser** — Still open. High-demand language for agent-assisted development.
+- [x] **16b. Rust parser** — Shipped (`parsers/rust.py`): modules, types, traits, functions,
+  methods, imports, qualified references, trait relationships, Cargo workspaces, and inline
+  `#[cfg(test)]` exclusion.
 - [x] **17. Incremental parsing** — Content-hash extract cache shipped in #9 (`incremental.py`), wired for the Python parser only; extending to the other two-pass parsers is follow-up.
 - [x] **18. Cross-language dependency tracking** — MVP: multi-language ingest/parse (`languages=[...]` / `language="multi"`) merges per-language graphs and relinks import/extends/implements across FQN space. Relinking is **family-scoped**: the `jvm` family (Java↔Kotlin) is the supported and validated pair; every other language is its own family and is merged without cross-language edges. Extending relinking to further families (and broader RPC/IDL bridges — gRPC stubs, OpenAPI) remains follow-up.
 
@@ -60,6 +62,6 @@ Work everywhere agents work.
 
 | Priority | Items | Rationale |
 |----------|-------|-----------|
-| **Done** | 1–10, 12, 13, 14, 15, 16a, 16a2, 17, 18 (MVP) | Phases 1–2 + TS/JS & Go & Kotlin parsers, incremental parsing (Python), `diff_impact`, `context_for_task`, `api_surface`, polyglot merge+relink, `changelog_architecture` |
-| **Now** | 11, 16b | Component ownership, Rust parser |
+| **Done** | 1–10, 12, 13, 14, 15, 16a, 16a2, 16b, 17, 18 (MVP) | Phases 1–2 + TS/JS & Go & Kotlin & Rust parsers, incremental parsing (Python), `diff_impact`, `context_for_task`, `api_surface`, polyglot merge+relink, `changelog_architecture` |
+| **Now** | 11 | Component ownership |
 | **Then** | 19–22 | Ecosystem breadth (OpenAI / LangChain / Claude SDK / IDE) |

actions/analyze/action.yml

@@ -24,7 +24,7 @@ inputs:
     default: "."
   language:
     description: >
-      Optional language override (java, python, typescript, c, go, kotlin, or multi
+      Optional language override (java, python, typescript, c, go, kotlin, rust, or multi
       for every detected language with cross-language edge relinking).
     required: false
     default: ""

docs/BUG_CATALOG.md

@@ -0,0 +1,100 @@
+# Parser Robustness Bug Catalog
+
+Reliability: `validated-2x`
+
+This is the living catalog for parser failure classes that can abort or distort
+whole-repository analysis. Entries use reproducible fixtures and design-time
+prevention rules so the same defect is not rediscovered language by language.
+
+## Design-time checklist
+
+- Traverse untrusted AST depth with explicit stacks or queues, never Python recursion.
+- Put every file extraction behind a transactional boundary: publish its entities only
+  after extraction succeeds.
+- Test nesting deeper than `sys.getrecursionlimit()` for every traversal shape.
+- Pair each poisoned input with a healthy sibling file and assert the sibling survives.
+- Track non-source inputs such as manifests when they affect graph identity or cache keys.
+- Do not add per-parser input caps (file size, node counts) as a stand-in for robustness:
+  they drop real code silently and never fix the traversal or complexity defect they appear
+  to mitigate. The per-file exception boundary is the backstop. Caps are a legitimate
+  *performance* tool for input that is genuinely not human-authored — `parsers/go.py` and
+  `parsers/typescript.py` both keep a 1 MB `_MAX_FILE_BYTES` for vendored and minified
+  bundles — but adopt one only after the underlying algorithm is linear, and say so
+  explicitly rather than claiming other parsers have no cap.
+- Run focused tests, the full suite, a large real repository, and arcade-agent's own
+  self-analysis before publishing parser changes.
+
+## 1. Kotlin deep-expression traversal aborted repository analysis
+
+- **Symptom:** A machine-generated expression with thousands of nested parentheses
+  raised `RecursionError`; valid sibling files disappeared because parsing aborted.
+- **Root cause:** Recursive AST descent treated source nesting as trusted call-stack depth.
+- **Detection:** Parse a deeply nested Kotlin file beside a valid file and assert the
+  valid entity remains in the graph.
+- **Fix:** Replace recursive descent with explicit stacks and isolate failures per file.
+- **Prevention:** Apply the parser hardening skill to every new or materially changed
+  tree-sitter traversal.
+- First encountered: Kotlin parser follow-up `b7effc5`.
+- **Pattern note:** First confirmed instance of cross-language AST depth fragility.
+
+## 2. Rust path/use/module/type traversals repeated the recursion defect
+
+- **Symptom:** Roughly 1,000 nested path segments, use groups, inline modules, or type
+  wrappers raised `RecursionError` and killed analysis for healthy sibling files.
+- **Root cause:** Four helpers used recursive descent even though `_references` already
+  demonstrated the safe iterative pattern; extraction also ran outside the file-level
+  exception boundary.
+- **Detection:** Parameterize all four AST shapes above the Python recursion limit and
+  parse each beside a valid Rust file.
+- **Fix:** Use explicit LIFO worklists, publish per-file extraction state transactionally,
+  and log-and-skip unexpected file-level failures.
+- **Prevention:** Require the shared adversarial matrix and self-dogfood before parser PRs.
+- First encountered: Rust parser PR #18 review, 2026-07-21.
+- **Pattern note:** Second confirmed cross-language instance. Keep the class on the
+  design checklist; wait for a third instance before naming a broader meta-pattern.
+
+## 3. Rust package membership index was quadratic in entities per package
+
+- **Symptom:** A 5.2 MB generated `.rs` file (79,500 entities) took 70 s to parse. Parse
+  time grew with the square of the entity count, so large real crates looked like hangs.
+- **Root cause:** `add_entity` guarded the per-package entity list with `if fqn not in
+  package_entities`, a linear scan of a list that grows to tens of thousands of entries;
+  the cross-file merge repeated the same `not in` test inside a generator expression.
+- **Detection:** Generate one large single-package source file and time the parse; the
+  entity count is a fine proxy for the input size the cap was hiding.
+- **Fix:** Keep a companion `set` next to each ordered list purely for membership, and
+  reset it wherever the list it shadows is reset (the Rust parser clears per-file state
+  each iteration — a stale set would leak entities across files).
+- **Prevention:** For any de-duplicated *ordered* collection, pair the list with a set at
+  the moment it is introduced. Treat "an input cap makes this fast enough" as a signal
+  that a container is being scanned linearly.
+- First encountered: Rust parser reland, 2026-08-10 (70.2 s → 3.0 s, 23x, identical
+  entity and edge counts).
+- **Pattern note:** The mirror image of class 1/2 — not a crash, a silent complexity cliff
+  that an input cap conceals instead of fixing.
+
+## 4. Conditional-compilation test gating leaks through its less common shapes
+
+- **Symptom:** With `exclude_tests=True`, a Rust probe crate with 2 production structs and
+  10 test-only entities still yielded 10 entities, 8 of them test-only. Every production
+  entity also carried a phantom `mockall` import, inflating fan-out and inventing coupling
+  to dev-only crates.
+- **Root cause:** The exclusion matched one syntactic shape (`#[cfg(test)]` normalizing to
+  exactly that text, on an outer `attribute_item`, attached to an item with a body). Four
+  other shapes bypassed it: compound predicates (`cfg(all(test, ...))`, `cfg(any(test, ...))`),
+  inner `#![cfg(test)]` on a file or module body, out-of-line `#[cfg(test)] mod x;` whose
+  backing file was later parsed as independent production source, and `#[cfg(test)] use ...`
+  which was collected by an import pass that ran before attributes were inspected.
+- **Detection:** Build one probe crate containing *every* shape and assert the production
+  entity set exactly, not just the absence of one fixture name. Assert on `entity.imports`
+  too — import leaks are invisible in entity counts.
+- **Fix:** Evaluate the cfg predicate tree rather than string-matching (while leaving
+  `not(test)` alone, which marks production-only code), inspect inner attributes, make the
+  import pass attribute-aware, and record out-of-line test module paths so their files are
+  skipped — which requires visiting a module's declaring file before the module's own file.
+- **Prevention:** When a language gates test code by annotation rather than by path,
+  enumerate the annotation's full grammar before implementing the filter; a single
+  normalized string comparison is a smell.
+- First encountered: Rust parser reland, 2026-08-10.
+- **Pattern note:** Applies to any annotation-gated exclusion (Go build tags,
+  C/C++ `#ifdef`), not only Rust.

pyproject.toml

@@ -45,6 +45,7 @@ languages = [
     "tree-sitter-c>=0.23.0",
     "tree-sitter-cpp>=0.23.0",
     "tree-sitter-kotlin>=1.1.0",
+    "tree-sitter-rust>=0.23.0",
 ]
 mcp = [
     "mcp[cli]>=1.0,<2",

src/arcade_agent/cache.py

@@ -20,7 +20,18 @@ def _cache_dir(project_root: Path) -> Path:
     return project_root / _CACHE_DIR
 
 
-def cache_key(source_path: str, language: str | None, files: list[str] | None) -> str:
+_SOURCE_SUFFIXES = {
+    ".java", ".py", ".c", ".cpp", ".h", ".hpp", ".ts", ".tsx", ".js", ".jsx",
+    ".go", ".kt", ".kts", ".rs",
+}
+
+
+def cache_key(
+    source_path: str,
+    language: str | None,
+    files: list[str] | None,
+    exclude_tests: bool = True,
+) -> str:
     """Compute a cache key from source path, language, and file mtimes.
 
     The key is a SHA-256 hash of the sorted file paths and their modification
@@ -31,6 +42,9 @@ def cache_key(source_path: str, language: str | None, files: list[str] | None) -
         source_path: Root directory of the project.
         language: Language being parsed (or None for auto-detect).
         files: Specific files to parse, or None to discover all.
+        exclude_tests: Whether inline test code is excluded. Parsers that honor
+            it (Rust) produce a different graph for the same file list, so it
+            must take part in the key.
 
     Returns:
         A hex digest string usable as a cache filename.
@@ -39,17 +53,27 @@ def cache_key(source_path: str, language: str | None, files: list[str] | None) -
     hasher = hashlib.sha256()
     hasher.update(str(root).encode())
     hasher.update((language or "auto").encode())
+    hasher.update(b"tests:excluded" if exclude_tests else b"tests:included")
 
     if files:
-        file_paths = sorted(files)
+        file_paths = set(files)
     else:
         # Hash all source-like files under root
-        file_paths = sorted(str(f) for f in root.rglob("*") if f.is_file() and f.suffix in {
-            ".java", ".py", ".c", ".cpp", ".h", ".hpp", ".ts", ".tsx", ".js", ".jsx",
-            ".go", ".kt", ".kts",
-        })
-
-    for fp in file_paths:
+        file_paths = {
+            str(f) for f in root.rglob("*") if f.is_file() and f.suffix in _SOURCE_SUFFIXES
+        }
+
+    # Rust module layout and crate names come from Cargo manifests, so editing
+    # one changes the graph without touching any .rs file.
+    tracks_rust = language is None or "rust" in language or any(
+        fp.endswith(".rs") for fp in file_paths
+    )
+    if tracks_rust:
+        file_paths.update(
+            str(manifest) for manifest in root.rglob("Cargo.toml") if manifest.is_file()
+        )
+
+    for fp in sorted(file_paths):
         p = Path(fp)
         hasher.update(fp.encode())
         if p.exists():

src/arcade_agent/ci/run_self_analysis.py

@@ -48,7 +48,7 @@ def main() -> None:
         "--language",
         default="",
         help=(
-            "Optional language override (java, python, typescript, c, go, kotlin, "
+            "Optional language override (java, python, typescript, c, go, kotlin, rust, "
             "or multi for every detected language)"
         ),
     )

src/arcade_agent/parsers/__init__.py

@@ -26,6 +26,11 @@
 except ImportError:
     pass
 
+try:
+    import arcade_agent.parsers.rust  # noqa: F401
+except ImportError:
+    pass
+
 from arcade_agent.parsers.base import LanguageParser, get_parser
 
 __all__ = ["LanguageParser", "get_parser"]

src/arcade_agent/parsers/base.py

@@ -9,6 +9,18 @@
 class LanguageParser(ABC):
     """Abstract base class for language-specific parsers."""
 
+    def __init__(self, exclude_tests: bool = True) -> None:
+        """Create a parser.
+
+        Args:
+            exclude_tests: Whether test code should be kept out of the graph.
+                File-level exclusion happens during discovery; this flag lets a
+                parser additionally drop *inline* test constructs that live in
+                production files (e.g. Rust's ``#[cfg(test)] mod tests``).
+                Parsers for languages without inline tests ignore it.
+        """
+        self.exclude_tests = exclude_tests
+
     @property
     @abstractmethod
     def language(self) -> str: