fix(parser): harden Rust cfg(test) exclusion and remove file size cap - #29
Closed
tuannx wants to merge 1 commit into
Closed
fix(parser): harden Rust cfg(test) exclusion and remove file size cap#29tuannx wants to merge 1 commit into
tuannx wants to merge 1 commit into
Conversation
Red-team review findings against the Rust parser: 1. Comment/doc-comment between #[cfg(test)] and its item broke test exclusion — tree-sitter emits comments as named children which cleared pending_attributes before the mod_item was reached. Fix: skip line_comment/block_comment nodes without clearing. 2. #[cfg(test)] on non-mod items (functions, structs, impls) was silently ignored — is_cfg_test was computed but only checked for mod_item. Fix: skip ALL items annotated with #[cfg(test)]. 3. Removed _MAX_FILE_BYTES (1MB) cap — no other parser has it, it silently drops real code (generated protobuf/diesel schemas), and the per-file except boundary is the actual backstop. Regression tests cover all three fixes: comment-between-attribute, cfg(test) on fn/struct/impl, and >1MB file survival.
tuannx
force-pushed
the
fix/rust-parser-red-team-findings
branch
from
July 23, 2026 14:48
8b7e248 to
9dd08b3
Compare
Contributor
Architecture Drift ReportAlgorithm: PKG | Entities: 722 | Components: 7 Drift from Baseline
Changes
Smells (1)
Generated by arcade-agent |
This was referenced Aug 10, 2026
lemduc
added a commit
that referenced
this pull request
Aug 10, 2026
) * feat(parser): add Rust support (roadmap #16b) Tree-sitter parser for Rust modules, types, traits, functions, methods, imports, qualified references, trait inheritance/implementations, and Cargo workspaces. All AST traversal is iterative and each file is extracted transactionally, so one adversarial file cannot erase healthy siblings. Two defects found while relanding the original contribution are fixed here. Linear entity indexing. `add_entity` guarded the per-package entity list with `if fqn not in package_entities` — a linear scan of a list that grows to tens of thousands of entries — and the cross-file merge repeated the same test in a generator expression. A companion `set` per package makes both O(1). On a 5.2 MB generated single-package file this is 70.2s -> 3.0s (23x) with identical entity (79,500) and edge (63,600) counts. The membership set is reset alongside the per-file `packages` dict so no state leaks between files. Complete `#[cfg(test)]` exclusion. Rust unit tests live inline, so path-based exclusion never sees them, and matching only the literal `cfg(test)` text on an outer attribute of an item with a body left four shapes leaking. A probe crate with 2 production structs and 10 test-only entities yielded 10 entities (8 test-only) before and yields exactly the 2 production structs now: * compound predicates — the cfg predicate tree is evaluated rather than string-compared, so `cfg(all(test, ...))` and `cfg(any(test, ...))` match while `cfg(not(test))` correctly stays production; * inner `#![cfg(test)]` on a file or module body, previously invisible because only `attribute_item` was inspected; * out-of-line `#[cfg(test)] mod helpers;`, whose backing `helpers.rs` was later parsed as an independent production file — files are now visited in declaring-module-first order so the module path can be excluded; * `#[cfg(test)] use ...`, which gave every production entity in the file a phantom import of mockall/proptest/rstest and inflated its fan-out. No input size cap is added. `go.py` and `typescript.py` both keep a 1 MB `_MAX_FILE_BYTES` for minified and vendored bundles; Rust does not need one now that the quadratic index is gone. Co-Authored-By: Tony Nguyen <tuannx87@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(source): honor exclude_tests in parsers, cache keys, and Cargo workspaces Three wiring gaps kept the Rust parser's inline test exclusion from ever taking effect through the public tools. `exclude_tests` never reached a parser. It only drove *path* filtering during discovery, so `_parse_one` left every parser on its default and inline `#[cfg(test)]` exclusion was dead on arrival. `get_parser` returns a fresh instance per call, so setting the flag there cannot leak between calls. `analyze` now forwards its own `exclude_tests` to `parse` as well. `cache_key` ignored `exclude_tests`. Inline exclusion changes the graph for an *identical* file list, so the explicit `files=` path could return a cached graph of the wrong shape. The flag now takes part in the key, `.rs` joins the tracked suffixes, and Cargo manifests are hashed for Rust parses — crate names and module layout come from `Cargo.toml`, which no `.rs` mtime reflects. The manifest probe matches composite language keys such as `rust|<exclusions>`, which is what `source.parse` actually passes. `_detect_source_root` narrowed Cargo workspaces to the root crate's `src`. On the new three-crate fixture that ingested 1 of 3 files; the `[workspace]` probe now runs before the generic source-root candidates and returns the workspace root, ingesting all 3. Co-Authored-By: Tony Nguyen <tuannx87@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: document Rust support and the parser hardening workflow Adds Rust to the supported-language lists across the README, ROADMAP, the reusable analysis workflow, the analyze action, the MCP tool docstrings, and the self-analysis CLI help. Ports the contributor process files from the original Rust contribution, with one claim corrected: `docs/BUG_CATALOG.md` asserted that per-parser input caps "diverge from the other parsers", but `parsers/go.py` and `parsers/typescript.py` both define `_MAX_FILE_BYTES = 1_000_000`. The rule now says what it means — a cap is a legitimate performance tool for input that is not human-authored, but never a substitute for fixing the underlying algorithm. Records two new reusable failure classes: quadratic membership tests on de-duplicated ordered collections, and annotation-gated test exclusion leaking through its less common syntactic shapes. Co-Authored-By: Tony Nguyen <tuannx87@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Tony Nguyen <tuannx87@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Owner
|
Merged into the reland, #39 (squash commit Everything real from this branch is in: the Two notes on the rest:
Thanks — the hardening work here carried over intact, with co-author credit on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Red-team review findings against the Rust parser in #18. Three fixes:
1. Comment between
#[cfg(test)]and item broke test exclusionTree-sitter emits
line_comment/block_commentas named children. A comment between the attribute and its item clearedpending_attributesbefore themod_itemwas reached, so#[cfg(test)] mod testswas NOT skipped.Fix: Skip comment nodes without clearing
pending_attributes.2.
#[cfg(test)]on non-mod items was silently ignoredis_cfg_testwas computed for ALL node types but only checked in themod_itembranch. Functions, structs, and impls annotated with#[cfg(test)]were extracted as production entities.Fix: Skip ALL items when
is_cfg_testis true (earlycontinuebefore the type dispatch).3. Removed
_MAX_FILE_BYTES(1MB) capexcept Exceptionboundary is the actual backstopRegression tests
test_rust_parser_skips_cfg_test_with_comment_between_attribute_and_itemtest_rust_parser_skips_cfg_test_on_non_mod_itemstest_rust_parser_handles_large_files_without_capValidation