feat(parser): add Rust support (roadmap #16b) — reland of #18 + #29 - #39
Merged
Conversation
Tree-sitter parser for Rust modules, types, traits, functions, methods,
imports, qualified references, trait inheritance/implementations, and Cargo
workspaces. All AST traversal is iterative and each file is extracted
transactionally, so one adversarial file cannot erase healthy siblings.
Two defects found while relanding the original contribution are fixed here.
Linear entity indexing. `add_entity` guarded the per-package entity list with
`if fqn not in package_entities` — a linear scan of a list that grows to tens
of thousands of entries — and the cross-file merge repeated the same test in a
generator expression. A companion `set` per package makes both O(1). On a
5.2 MB generated single-package file this is 70.2s -> 3.0s (23x) with
identical entity (79,500) and edge (63,600) counts. The membership set is
reset alongside the per-file `packages` dict so no state leaks between files.
Complete `#[cfg(test)]` exclusion. Rust unit tests live inline, so path-based
exclusion never sees them, and matching only the literal `cfg(test)` text on
an outer attribute of an item with a body left four shapes leaking. A probe
crate with 2 production structs and 10 test-only entities yielded 10 entities
(8 test-only) before and yields exactly the 2 production structs now:
* compound predicates — the cfg predicate tree is evaluated rather than
string-compared, so `cfg(all(test, ...))` and `cfg(any(test, ...))` match
while `cfg(not(test))` correctly stays production;
* inner `#![cfg(test)]` on a file or module body, previously invisible
because only `attribute_item` was inspected;
* out-of-line `#[cfg(test)] mod helpers;`, whose backing `helpers.rs` was
later parsed as an independent production file — files are now visited
in declaring-module-first order so the module path can be excluded;
* `#[cfg(test)] use ...`, which gave every production entity in the file a
phantom import of mockall/proptest/rstest and inflated its fan-out.
No input size cap is added. `go.py` and `typescript.py` both keep a 1 MB
`_MAX_FILE_BYTES` for minified and vendored bundles; Rust does not need one
now that the quadratic index is gone.
Co-Authored-By: Tony Nguyen <tuannx87@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rkspaces Three wiring gaps kept the Rust parser's inline test exclusion from ever taking effect through the public tools. `exclude_tests` never reached a parser. It only drove *path* filtering during discovery, so `_parse_one` left every parser on its default and inline `#[cfg(test)]` exclusion was dead on arrival. `get_parser` returns a fresh instance per call, so setting the flag there cannot leak between calls. `analyze` now forwards its own `exclude_tests` to `parse` as well. `cache_key` ignored `exclude_tests`. Inline exclusion changes the graph for an *identical* file list, so the explicit `files=` path could return a cached graph of the wrong shape. The flag now takes part in the key, `.rs` joins the tracked suffixes, and Cargo manifests are hashed for Rust parses — crate names and module layout come from `Cargo.toml`, which no `.rs` mtime reflects. The manifest probe matches composite language keys such as `rust|<exclusions>`, which is what `source.parse` actually passes. `_detect_source_root` narrowed Cargo workspaces to the root crate's `src`. On the new three-crate fixture that ingested 1 of 3 files; the `[workspace]` probe now runs before the generic source-root candidates and returns the workspace root, ingesting all 3. Co-Authored-By: Tony Nguyen <tuannx87@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds Rust to the supported-language lists across the README, ROADMAP, the reusable analysis workflow, the analyze action, the MCP tool docstrings, and the self-analysis CLI help. Ports the contributor process files from the original Rust contribution, with one claim corrected: `docs/BUG_CATALOG.md` asserted that per-parser input caps "diverge from the other parsers", but `parsers/go.py` and `parsers/typescript.py` both define `_MAX_FILE_BYTES = 1_000_000`. The rule now says what it means — a cap is a legitimate performance tool for input that is not human-authored, but never a substitute for fixing the underlying algorithm. Records two new reusable failure classes: quadratic membership tests on de-duplicated ordered collections, and annotation-gated test exclusion leaking through its less common syntactic shapes. Co-Authored-By: Tony Nguyen <tuannx87@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🤖 Architecture Analysis SummaryPowered by arcade-agent — automatic architectural self-analysis 📈 Metric EvolutionBaseline commit: Legend: 🟢 better · 🔴 worse · 🟡 low impact · ⚪ no change
🏛️ Current Architecture
🧭 Principle Signals
🎯 Score DriversBiggest risks
Strongest areas
🕸️ High-Level Designgraph TD
Algorithms["Algorithms\n47 entities\n10 classes / 3 methods"]
Budget["Budget\n3 entities\n0 classes / 0 methods"]
Cache["Cache\n4 entities\n0 classes / 0 methods"]
Ci["Ci\n5 entities\n0 classes / 0 methods"]
Display["Display\n1 entities\n0 classes / 0 methods"]
Exporters["Exporters\n17 entities\n1 classes / 0 methods"]
Incremental["Incremental\n2 entities\n1 classes / 2 methods"]
Parsers["Parsers\n26 entities\n16 classes / 32 methods"]
Serialization["Serialization\n8 entities\n0 classes / 0 methods"]
Source["Source\n7 entities\n1 classes / 1 methods"]
Tools["Tools\n23 entities\n3 classes / 1 methods"]
Cache --> Serialization
Ci --> Algorithms
Ci --> Display
Ci --> Exporters
Ci --> Serialization
Ci --> Tools
Serialization --> Algorithms
Serialization --> Parsers
Source --> Cache
Source --> Parsers
Tools --> Algorithms
Tools --> Exporters
Tools --> Source
🏗️ Components breakdown
🚨 Architectural Smells
📈 Evolution vs BaselineBaseline commit: Architecture-to-Architecture (A2A) Comparison
Component matching detailsMatched:
High-level component statistics
Before/After Mermaid diagramsBaseline graph TD
Algorithms["Algorithms\n47 entities\n10 classes / 3 methods"]
Budget["Budget\n3 entities\n0 classes / 0 methods"]
Cache["Cache\n4 entities\n0 classes / 0 methods"]
Ci["Ci\n5 entities\n0 classes / 0 methods"]
Display["Display\n1 entities\n0 classes / 0 methods"]
Exporters["Exporters\n17 entities\n1 classes / 0 methods"]
Incremental["Incremental\n2 entities\n1 classes / 2 methods"]
Parsers["Parsers\n21 entities\n11 classes / 27 methods"]
Serialization["Serialization\n8 entities\n0 classes / 0 methods"]
Source["Source\n7 entities\n1 classes / 1 methods"]
Tools["Tools\n23 entities\n3 classes / 1 methods"]
Cache --> Serialization
Ci --> Algorithms
Ci --> Display
Ci --> Exporters
Ci --> Serialization
Ci --> Tools
Serialization --> Algorithms
Serialization --> Parsers
Source --> Cache
Source --> Parsers
Tools --> Algorithms
Tools --> Exporters
Tools --> Source
Current graph TD
Algorithms["Algorithms\n47 entities\n10 classes / 3 methods"]
Budget["Budget\n3 entities\n0 classes / 0 methods"]
Cache["Cache\n4 entities\n0 classes / 0 methods"]
Ci["Ci\n5 entities\n0 classes / 0 methods"]
Display["Display\n1 entities\n0 classes / 0 methods"]
Exporters["Exporters\n17 entities\n1 classes / 0 methods"]
Incremental["Incremental\n2 entities\n1 classes / 2 methods"]
Parsers["Parsers\n26 entities\n16 classes / 32 methods"]
Serialization["Serialization\n8 entities\n0 classes / 0 methods"]
Source["Source\n7 entities\n1 classes / 1 methods"]
Tools["Tools\n23 entities\n3 classes / 1 methods"]
Cache --> Serialization
Ci --> Algorithms
Ci --> Display
Ci --> Exporters
Ci --> Serialization
Ci --> Tools
Serialization --> Algorithms
Serialization --> Parsers
Source --> Cache
Source --> Parsers
Tools --> Algorithms
Tools --> Exporters
Tools --> Source
Component dependency delta
💡 CI/CD Insights
📄 View HTML reports and artifacts This comment is auto-generated by the self-dogfooding CI job. It updates on every push to this PR. |
Contributor
Architecture Drift ReportAlgorithm: PKG | Entities: 143 | Components: 11 Drift from Baseline
Architectural changes —
|
| Component | Entities | Responsibility |
|---|---|---|
| Algorithms | 47 | Entities in algorithms |
| Parsers | 26 | Entities in parsers |
| Tools | 23 | Entities in tools |
| Exporters | 17 | Entities in exporters |
| Serialization | 8 | Entities in serialization |
| Source | 7 | Entities in source |
| Ci | 5 | Entities in ci |
| Cache | 4 | Entities in cache |
| Budget | 3 | Entities in budget |
| Incremental | 2 | Entities in incremental |
| Display | 1 | Entities in display |
Architecture Diagram
graph LR
Algorithms["Algorithms"]
Budget["Budget"]
Cache["Cache"]
Ci["Ci"]
Display["Display"]
Exporters["Exporters"]
Incremental["Incremental"]
Parsers["Parsers"]
Serialization["Serialization"]
Source["Source"]
Tools["Tools"]
Cache --> Serialization
Ci --> Algorithms
Ci --> Display
Ci --> Exporters
Ci --> Serialization
Ci --> Tools
Serialization --> Algorithms
Serialization --> Parsers
Source --> Cache
Source --> Parsers
Tools --> Algorithms
Tools --> Exporters
Tools --> Source
Smells (1)
- Concern Overload: Tools
- 💡 Split the overloaded component into smaller, focused units. Consider extracting sub-packages or introducing an interface layer.
Generated by arcade-agent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relands the Rust parser from #18 and the
cfg(test)hardening from #29 on top ofcurrent
main, with the reviewed defects fixed. Rebuilt on a fresh branch ratherthan merged: #18's
tools/parse.pydiff predates the polyglot rewrite in #19 andwould revert it.
Original contribution by @tuannx (
Tony Nguyen), carried as a co-author traileron every commit.
What landed
The parser —
src/arcade_agent/parsers/rust.pyand its registration, thetree-sitter-rustdependency in the[languages]extra, and 36 tests. All ASTtraversal is iterative and each file is extracted transactionally, so one
adversarial file cannot erase healthy siblings.
The O(n²) fix (blocker).
add_entityguarded the per-package entity list withif fqn not in package_entities— a linear scan of a list that grows to tens ofthousands of entries — and the cross-file merge repeated the same test in a
generator expression. A companion
setper package makes both O(1). The set isreset alongside the per-file
packagesdict, so no state leaks between files..rsCounts are identical — this is a pure complexity fix.
Complete
#[cfg(test)]exclusion. Matching only the literalcfg(test)texton an outer attribute of an item with a body left four shapes leaking. A probe
crate with 2 production structs and 10 test-only entities:
Prod.imports['std::collections::HashMap', 'mockall::predicate::Eq']['std::collections::HashMap']The four newly-closed shapes, each with a test:
string-compared, so
cfg(all(test, ...))andcfg(any(test, ...))match.cfg(not(test))correctly stays production (covered by its own test).#![cfg(test)]on a file or module body — previously invisiblebecause only
attribute_itemwas inspected. Such a file now yields zeroentities.
#[cfg(test)] mod helpers;—helpers.rswas later parsed asan independent production file. Files are now visited in
declaring-module-first order so the module's backing path can be excluded.
#[cfg(test)] use ...— gave every production entity in the file aphantom import of mockall/proptest/rstest and inflated its fan-out.
Wiring that
main's restructure needs. All three gaps were confirmed real:exclude_testsnever reached a parser — it only drove path filtering, so_parse_oneleft every parser on its default and inline exclusion was dead onarrival.
analyzenow forwards its flag toparsetoo.cache_keycall sites now passexclude_tests, which changes the graphfor an identical file list.
.rsjoins the tracked suffixes and Cargomanifests are hashed for Rust parses (crate names and module layout come from
Cargo.toml, which no.rsmtime reflects). The manifest probe matches thecomposite keys
source.parseactually passes, e.g.rust|<exclusions>.[workspace]probe now runs before_LANG_PREFERRED_ROOTSin_detect_source_root. On the new three-crate fixture(
tests/fixtures/rust_workspace/: root +crates/alpha+crates/beta),mainingested 1 of 3.rsfiles; it now ingests 3 of 3.Docs and process files. Rust added to the language lists in the reusable
workflow, the analyze action, the MCP docstrings, the self-analysis CLI help, and
README/ROADMAP prose. Version defaults are
main's — no0.1.1→0.2.0bump.Deliberately left out
The
recover.pyscope creep (commit86fb591oncodex/rust-parser, whichchanges
_refine_facade_groupsand importsCONCERN_OVERLOAD_ENTITY_THRESHOLDfrom
algorithms/concern.py). Its blast radius is 9/11,419 entities (0.08%)across six non-Rust corpora, and its only real effect is silencing arcade-agent's
own concern-overload smell by relocating five tool entry points, while shifting
this repo's own RCI +9.9% and TurboMQ +10.9%. Rust does not depend on it. It
remains unlanded on
codex/rust-parserfor a separate decision.No input size cap. #18's commit message claimed "no other parser has a file
size cap" — that is false:
parsers/go.py:20andparsers/typescript.py:26bothdefine
_MAX_FILE_BYTES = 1_000_000. Rust ships without one because it no longerneeds one, not because the others lack one.
docs/BUG_CATALOG.mdcarried the samewrong claim and is corrected here; it now says a cap is a legitimate performance
tool for genuinely non-human-authored input but never a substitute for fixing the
algorithm. Two new reusable failure classes are recorded (quadratic membership
tests; annotation-gated test exclusion leaking through its rarer shapes).
Verification
pytest— 517 passed.ruff check src/ tests/— clean. Notests/test_mcp_e2e.pyerrors (branching offmainpicks up themcp[cli]<2cap).2,623 entities. Tokio (a Cargo workspace): 505 files, 0.8 s, 4,345 entities.
Against the old parser those were 2,662 and 4,516 — the 39 and 171 differences
are the
cfg(test)entities that used to leak.arcade-arch-diff— similarity 0.98, "No architectural changes since thebaseline", RCI +0.02 and TurboMQ +0.01. The
Toolsconcern-overload smellstays, as expected given
recover.pywas excluded. The baseline is refreshedautomatically by
arch-drift.ymlon push tomain.Closes #29. #18 left open for the maintainer to close with a pointer here.
🤖 Generated with Claude Code