Read NuGet and Cargo crawler project files through the FIFO-safe reader (#592)

[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #592

Summary

A FIFO in a checkout at obj/project.assets.json (NuGet) or vendor/<crate>/Cargo.toml (Cargo) blocked open(2) forever, so scan, get and apply hung on that repository. Those reads, and the Pipenv .venv read, now go through the shared FIFO-safe readers in utils::fs. A new architecture test stops crawler production code from using a bare file read again.

Why (leverage)

• Issue #592, register row E06 (register), living document Part 6.6 "FIFO safety" (doc/06-discovery-vex.md).
• B=1, U=0, D=0, R=L, score 3. This is the top eligible candidate. The higher-scored ones all touch files that open PRs change: Read and splice nuget.config through formats::nuget in hosted, vendored and restore #594/Hosted NuGet rewrites packages.lock.json entries at other versions of the patched id #593 (redirect/mod.rs, Unify hosted NuGet routing and XML splice anchors #597), Vendored→hosted takeover drops the vendored ledger entry when the revert drift-keeps the artifact #568 (scan/hosted.rs, Fix hosted scan from a workspace member pinning nothing or the wrong files (#590, #417) #598) and E37 (vendor/cargo.rs, Fix hosted scan from a workspace member pinning nothing or the wrong files (#590, #417) #598).

What changed

• crawlers/nuget_crawler.rs parse_project_assets_package_folders: tokio::fs::read_to_string → utils::fs::read_regular_to_string.
• crawlers/cargo_crawler.rs verify_crate_at_path (find_by_purls): tokio::fs::read_to_string → read_regular_to_string. read_crate_cargo_toml (blocking-pool crawl_all): std::fs::read_to_string → read_regular_to_string_sync.
• crawlers/python_crawler.rs Pipenv .venv: std::fs::read_to_string → read_regular_to_string_sync. Before, only the is_file() check guarded this read, which left a TOCTOU window. The issue called this read out of scope, but the new guard needs it fixed; it is one line.
• crawlers/mod.rs: architecture_tests::crawlers_read_project_files_fifo_safely fails on any fs::read_to_string(, fs::read( or File::open( in crawler production code. The only allowlisted reads are the two machine-repository Maven POM reads in maven_crawler.rs, which NuGet and Cargo crawlers hang on a FIFO at obj/project.assets.json or vendor/<crate>/Cargo.toml #592 left out of scope.

Deleted / diff

git diff --stat origin/main...HEAD: 4 files, +245 / −4.

• Production: +4 / −4. These are call-site swaps; no new production code.
• Tests: +241. That covers two FIFO regression tests, their helpers and the architecture guard.

Behavior

None for regular files. Unreadable entries are still skipped. A non-regular file (FIFO, device, directory) at one of these paths is now skipped right away instead of hanging.

Test evidence

• Red on main's reads (the three call sites temporarily reverted on this branch):
  • fifo_crate_manifest_is_skipped_not_blocked_on FAILED (5 s deadline);
  • fifo_project_assets_is_skipped_not_blocked_on FAILED (5 s deadline);
  • crawlers_read_project_files_fifo_safely FAILED.
• Green on the branch: all three pass. cargo test -p socket-patch-core --lib -- crawlers: 516 passed.
• cargo test -p socket-patch-core --lib: 4843 passed. The 4 failures are the known root-sandbox ones that also fail on main: copy_tree::relax_loop_must_not_traverse_symlinked_root, vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, pypi_poetry::wire_write_failure_maps_error_and_leaves_lock_untouched and pypi_requirements::wire_failure_rolls_back_already_written_files. They pass in CI.
• cargo clippy --workspace --all-features -- -D warnings: clean.
• cargo test -p socket-patch-cli --all-features with --test in_process_scan, in_process_cargo_apply, crawl_fd_limit_e2e, e2e_nuget and e2e_cargo: all green.
• CI: all 341 checks green on a024719. Before that, ac9bece failed only test (windows-latest). On a CRLF checkout the guard's #[cfg(test)]\nmod tests marker never matched, so it counted a test-only Maven read as production code. The fix is a024719, which normalizes CRLF first. Reproduced locally with a CRLF maven_crawler.rs: before the fix it reports 3 reads against 2 allowed and fails; after, it passes.

Bugbot round 1 found that the test watchdog's blocking writer open could hang the suite. That is fixed in ac9bece (O_NONBLOCK), and later rounds, including one on a024719, are clean.

Risk

Low. Four one-line call-site swaps onto readers that every other crawler already uses. The FIFO regression tests are #[cfg(unix)]; the architecture guard runs on every platform.

🤖 Generated with Claude Code

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a024719. Configure here.}

[Mikola Lysenko (mikolalysenko)]

[burn-down agent] Labeled Ready for review at a024719ab281f48223153f1ec641e07aa450d585.

• CI: 335/341 check runs green on this head (6 skipped by path/matrix filters), 0 failing.
• Bugbot: reviewed this exact head, no findings; its one earlier thread (cargo_crawler.rs) is resolved.
• Mergeable; 1 commit behind main (045d7ec, API client only, no overlap with these crawler files).
• Reviewer focus: the new guard test in crawlers/mod.rs that forbids bare read_to_string in crawlers, and CRLF marker matching.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Reviewed a024719ab281f48223153f1ec641e07aa450d585: ready to merge as-is; no actionable findings.

The four call-site changes preserve regular-file and symlink behavior while using the existing descriptor-based guard to reject FIFOs and other nonregular inputs. Errors still skip the affected entry, and discovery continues for readable siblings. The watchdog cleanup and CRLF-aware architecture guard also look correct.

Validation: 517 crawler tests passed. Independent public-API tests passed 33 file-kind scenarios across Cargo, NuGet, and Pipenv; the CRLF guard control and two shared-reader contract tests also passed. These local checks ran on macOS; I did not rebuild the full CLI/network matrix.

The exact head has 335 successful and seven skipped checks, a clean Bugbot review, and no unresolved threads. It merges cleanly with current main 045d7ec783d788bf3c5a1310724b51e09fb6505d.