Speed up hosted and vendored scans: concurrent API requests, parallel crawl, single-pass rewriters - #257
Speed up hosted and vendored scans: concurrent API requests, parallel crawl, single-pass rewriters#257Mikola Lysenko (mikolalysenko) wants to merge 236 commits into
Conversation
`ordered_concurrent` / `map_ordered_concurrent` wrap `stream::iter(..).map(f).buffered(limit)`: at most `limit` requests in flight, results yielded in input order, nothing started until polled. The serial patch-API loops can adopt it and fold results exactly as before. `API_CONCURRENCY` (8) and `PROXY_API_CONCURRENCY` (4) carry the per-client caps. futures-util was already in the lock; it is now a direct dependency of core and the CLI. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New subprocess suite for the three patch-API loops `scan` drives (batch POSTs, per-package detail GETs, hosted record views). Every mock answers later requests first, so an implementation that folds in completion order, or lets a discarded response leak in, changes the output: - batch: a 401 on the first chunk sends that chunk and all later ones to the proxy with one auth request and one warning; a 401 on chunk 3 of 6 folds 0-2 from the auth API and replays 3-5 on the proxy; per batch 500 warnings print in chunk order; the all-failed error carries the last chunk's error. - details: partial-failure warnings print in package order and the whole human preview equals a zero-latency run; the all-failed error names the last package. - hosted wet run: record_fetch_failed warnings keep confirmed order and stdout, lockfile and ledger equal a zero-latency run. The suite passes against the current serial loops (checked with the baseline binary) and is the oracle for making them concurrent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`fetch_patch_details` awaited one `by-package` GET per package with patches (74 on depscan, ~10 s of serial round trips). The queries now run through `ordered_concurrent` (8 in flight, 4 on the public proxy) and are consumed in `packages` order, so `results`, `failures`, the warn-after loop and the all-failed rule see exactly what the serial loop produced. `ApiClient::uses_public_proxy` picks the cap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The batch loop POSTed one chunk at a time (56 chunks on depscan). Chunks now run through `ordered_concurrent` and are consumed strictly in chunk order, so per-batch warnings, `batch_error_count`, `last_batch_error` and the paid-access flag fold as before. The authenticated-to-proxy downgrade keeps the serial loop's exact sequence: the first chunk goes alone (a stale token still costs the auth API one request), and at the first consumed chunk k whose error is a fallback candidate — any index — the window is dropped, responses for chunks past k are discarded unfolded, the same warning prints, chunk k is retried on the proxy and the rest continue there (4 in flight). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A wet hosted run fetched `patches/view/{uuid}` for every confirmed
redirect one at a time (74 on depscan, ~9 s). The views now run through
`ordered_concurrent` and are consumed in `confirmed` order, so `records`
(newest wins) and the `record_fetch_failed` warnings are unchanged. The
ledger re-fetch on idempotent re-runs is deliberately kept.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scan awaited each telemetry POST inline (150-300 ms typical, up to the 5 s budget on a bad network) before carrying on. Its three events now go through `spawn_patch_scanned` / `spawn_patch_scan_failed`: the event is built and its endpoint resolved where it fires (same body, timestamp, env reads and "Sending telemetry" debug line), and only the POST runs in a background task. `scan::run` awaits `PendingTelemetry::flush` before returning, so every event is still delivered, or given up on within the same 2 s connect / 5 s request budget, before the process exits. The inline trackers and every other command are unchanged. Tests: core unit tests pin that a background send posts the same bytes and headers as an inline one and that flush waits for it; telemetry_e2e pins that each scan terminal (success, empty crawl, all batches failed) delivers its one event and stays alive until the slow endpoint answers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The background send was only awaited after run_scan returned, so a process killed after the event fired but before that flush lost it: `scan | head` / `scan | true` dies of SIGPIPE on its first result write (main restores SIG_DFL), and a Ctrl-C at a confirm prompt or a CI SIGTERM had the same effect. The inline send it replaced had always landed before any output. `PendingTelemetry::flush` now drains (`&mut self`), and scan flushes at the first output point after each event fires: right after the send on the empty-crawl and all-batches-failed terminals (they print at once), at the start of the human section (before the table, prompts and every human exit), before the plain `--json` envelope, and inside `discover_selected` right after the detail fetches (before its error line and whatever the `--apply`, hosted and vendored `--json` arms print next). The send still overlaps the by-package detail fetches on those arms; the flush at the end of `run` stays as the exit backstop. Under `--debug` this also puts the human path's "Telemetry sent" line back ahead of the per-package detail warnings, as in the inline order. Tests: telemetry_e2e runs each JSON terminal with stdout closed before the child writes and requires the event delivered (red on the previous commit: SIGPIPE, 0 events); a core unit test pins that flush drains and that sends started after it join the next flush. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`track_patch_scan_failed` and `spawn_patch_scan_failed` each spelled out
the `{"fallback_to_proxy": ...}` literal; build it in one place, as
`patch_scanned_metadata` already is for the success event, so the inline
and background paths cannot drift. The inline trackers stay: they are
public API of the published core crate.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No production caller used `map_ordered_concurrent`: every API loop consumes `ordered_concurrent` directly. Move it into the tests module so it no longer ships as unused public API. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… + ordered merge `crawl_all` and the workspace roots walk made one `spawn_blocking` round trip per readdir, stat and package.json read, strictly in sequence. Both now run as one blocking-pool task: directory I/O is gathered in parallel (rayon, already in the dependency graph via qbsdiff) into per-root event trees that record the sequential visit order, and a single-threaded merge replays them so the order-dependent `seen` dedup and the store entries' `identity_seen` decisions see exactly the state the old walk saw — same packages, same paths, same order. Two probes are answered from listings the walk reads anyway, only where that is provably the same answer: - the roots walk skips the `is_dir(child/node_modules)` stat when the child's complete listing holds nothing that could alias `node_modules` on a case-insensitive filesystem (a listed dir still stats: a readable-but-unsearchable parent lists kinds while stats fail); - a store entry's `node_modules` existence probe is the readdir the scan needs next; a dir that does not open falls back to the stat. FIFO-safe package.json reads (read_regular_to_string_sync), the NESTED_STORE depth/dir caps (kept sequential: the budget order decides survivors), symlink-not-traversed rules and lossy-vs-raw name joins are unchanged. The previous async implementation is kept verbatim as a #[cfg(test)] oracle; a randomized fixture test (flat/nested/legacy stores, scoped, live/dangling/store symlinks, duplicate identities, aliases, broken/BOM/FIFO/dir package.json, unreadable and unsearchable dirs, node_modules case variants) plus a kitchen-sink tree assert identical roots, crawl output, find_by_purls results and store enumeration. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ly listed names The resolver opened `<nm>/<target>/package.json` for every pending target in every visited node_modules — targets × dirs failed opens, each its own spawn_blocking hop — then listed the same dir again for the descent. Both passes now run as one blocking-pool task: each dequeued dir is listed once, a target is probed there only when the listing could hold its first path component, the surviving probes run in parallel and fold back in target order, and the same listing drives the descent (whose per-entry stats also run in parallel, appended in listing order). The name filter is a strict superset: it only engages for a complete, all-ASCII listing and matches ASCII-case-insensitively (APFS/NTFS), and components a filesystem can resolve to a differently spelled entry (non-ASCII, `~` 8.3 aliases, trailing dot/space) are always probed. BFS root-first order, every-copy collection, the name+version identity check, the pass-2 fallback and the store-entry name filter are unchanged. `.pnpm` entry names are still filtered after the `node_modules` stat, not before: an entry without one is a nested host whose synthesized children can match, so the stat decides the result. The oracle equivalence suite (now also covering case-variant package and scope dirs) asserts identical find_by_purls maps on every generated root. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`crawl_all_ecosystems` awaited each crawler in turn, and the crawlers that block (maven's walkdir walk + POM reads, `gem env`, the python site-packages probe, `composer global config home`) did so inline on the async task. The crawlers are independent — none prints, none mutates shared state — so they are now joined, with every blocking walk or subprocess moved onto the blocking pool, and their results are consumed in the fixed Npm, Pypi, Cargo, Gem, Golang, Maven, Composer, Nuget, Deno order, so packages and counts are exactly the serial run's. The joined futures are heap-allocated from a non-async constructor so the caller's poll frame does not grow by their combined size (Windows main-stack budget). `gem env gemdir` and `gem env gempath` run concurrently but are still two calls consumed gemdir-then-gempath (no single-call merge: platform path separators). A polyglot `--global-prefix` test pins the joined output against the serial sequence. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The previous commit fanned each visited dir's probes and descent stats out to the rayon pool separately, one injection per dir: on a deep pnpm tree the per-dir handoff latency outweighed the parallelism, and `apply --dry-run` on a large monorepo ran slower than the async walk. A visit's reads depend only on the dir and the fixed target list, never on what earlier dirs resolved, so the walk now proceeds level by level (exactly the FIFO queue's order: everything a dir enqueues lands behind the rest of its level). Each level's visits — listing, filtered probes, nested-dir discovery with the virtual store's entries returned whole — are gathered in one parallel pass, then the order-dependent part (folding matches into the result, the unmatched-name store filter, next-level order) is replayed sequentially in queue order. Output is unchanged; the oracle equivalence suite still covers it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ix targets Symlinks, FIFOs and permission stripping are generated on Unix only, so the fields that record them are never read elsewhere. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…'s use block Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The parallel roots walk recursed once per directory level on rayon and blocking-pool threads (2 MiB stacks), where the old async walk recursed through boxed futures on the 8 MiB main thread. A deep enough directory chain (reachable under Linux's 4096-byte PATH_MAX, and deeper on Windows long paths) aborted the scan with a stack overflow the old walk survived. Read the tree one level at a time, each level's dirs in parallel, record each dir's child range, then emit with an explicit stack in the same depth-first order. Stack use no longer grows with depth; a new test runs a 400-deep chain on 256 KiB walk threads (the recursive walk overflowed there). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… budget Two properties of the old sequential async walk did not survive the move to parallel sync walks on rayon's global pool: - Stack: the recursive node_modules gather ran on 2 MiB worker threads instead of the 8 MiB main thread. The npm walks now run on a dedicated walk pool whose threads get the main thread's 8 MiB. - Descriptors: every walker treats a failed read_dir/open, EMFILE included, as an absent dir, and the old crawl held one descriptor at a time with the nine crawlers run back to back. With one walk thread per CPU plus concurrent crawlers, depscan lost packages silently below `ulimit -n 24` (5349 of 5520 at 20) where the old crawl was intact down to 14. Under a soft RLIMIT_NOFILE below 128 the walk pool now gets one thread and the crawlers run one at a time (the old descriptor profile); above it the pool is capped at half of what is left after a 64-descriptor reserve. depscan now matches the baseline byte-for-byte at every limit from 16 to 256. New tests: pool sizing, a 4 MiB frame fitting on a walk thread, and an e2e scan under `ulimit -n 16` that must match the ample-limit JSON (the one-thread-per-CPU pool lost most of that tree there). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The hosted pnpm rewriter re-parsed every lock (entries, the early shrinkwrap sniff, the residual gate) and rebuilt the whole lock string once per dep: O(deps x lock) work that cost ~430 ms of critical-path CPU on depscan's 2 MB lock with 74 redirected deps. Each lock is now parsed and key-indexed once; a dep's instances are found by binary search, the residual gate judges each instance on its post-splice body, and committed splices are applied in one pass at the end. A later dep that hits an already-spliced entry (a duplicate name@version override) folds the pending splices in and re-indexes first, so it re-reads the rewritten text exactly as before, and the vendored-marker scan runs over the post-splice text the same way. Output bytes, the FileEdit list (order and original fragments), warnings and refusals are unchanged: the previous implementation is kept as a test oracle and compared on a depscan-sized synthetic lock set, on 300 randomized mixes of every lock flavor, and on duplicate-override and peer-suffixed multi-instance cases. depscan wet run: pnpm-lock.yaml and redirect-state.json byte-identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rently Deciding which pypi deps need hosted wheel metadata ran a full `rewrite_python_lock` (parse, a second parse for the source-scope check on script locks, mutate, serialize) per dep per lock, only to test the result for `Some`. The rewrite's refusal and not-applicable checks now live in one planning step that `rewrite_python_lock` and a new `PythonLockProbe` share: the probe parses each lock once and answers exactly `matches!(rewrite_python_lock(..), Ok(Some(_)))` per dep, and the rewrite no longer re-parses the lock for the scope check. The qualifying wheels are then downloaded through an ordered `buffered(8)` stream and folded in dep order, so `python_metadata`, the withheld artifacts and the `python_metadata_unavailable` skips are unchanged. The stream is inlined here (futures-util added with the same workspace spec as the scan-concurrency branch); it moves onto the shared ordered-concurrency helper once that lands. New tests: a probe/rewrite equivalence sweep over every lock shape and outcome, and a hosted scan whose slow first failure must still be reported before a fast second one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… lock The npm package-lock rewriter re-derived every `packages` entry's identity (the `node_modules/` key split plus the `name`/`version` lookups) for every dep, and the classic yarn.lock rewriter re-split every block's key patterns for every dep: O(deps x entries) work that dominated both rewriters' CPU. Each identity is now computed once per lock. npm entries keep theirs by map position (a rewrite only touches `resolved`/`integrity`, never a key, `name` or `version`); a yarn block's key and sole real package are recomputed whenever this run rewrites that block, so later deps still see its current text. Output bytes, FileEdits and warnings are unchanged: both previous implementations are kept as test oracles and compared on 400 randomized locks each (aliases, links, bundled copies, workspaces, v1/v2 dependency trees, alias-only and fork-substitution yarn keys, CRLF and mixed line endings, duplicate overrides). Rewrite-phase CPU on the lockfile-only benches: npm-socket 103 -> 61 ms, yarn-strapi 49 -> 32 ms (whole-process medians). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The indexed rewriter judges residuals inline, so the boundary test over `pnpm_unrewritten_instances` now covers only the test-only reference. Feed the same boundary locks through `rewrite_registry_redirect`: hosted, longer-version, scoped and snapshot keys never count, v6 nested-paren and v5 `_` instances are repointed, and only the unparseable instance is named in the refusal. The helper's doc comment now says what it is. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each in-flight wheel download buffers the whole wheel under its own body timeout and retry budget, so memory and link sharing scale with the limit; 4 keeps the overlapped round trips while halving that. The comment records what concurrency changes that output cannot see (status line names the awaited dep, debug lines interleave, Retry-After pauses one fetch). The order test now also records request arrivals and fails if `bbb` is not requested before `aaa`'s delayed response is due, so a regression to serial fetching is caught. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scan time is dominated by API round trips, so live timings are noisy and unrepeatable. scripts/perf/replay.py stands in for api.socket.dev, patch.socket.dev and the public proxy: `record` forwards and stores every response, `replay` serves only from the store with a fixed or recorded per-request latency (plus optional per-connection latency) and reports request counts per endpoint, max in-flight, connections and network span. Batch POSTs replay per purl, so a build that changes chunking or order still gets identical answers. The listener skips HTTPServer's getfqdn(), which stalls ~35 s under the macOS sandbox. scripts/perf/bench.sh drives it: `record`, `replay`, and `ab`, which runs BASE and NEW interleaved against one store and fails unless every run's stdout sha256 and exit code match the first BASE run. Stores hold real API responses (possibly paid-patch data), so bench.sh refuses a store path inside the repository. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Syntax-checks replay.py (py_compile) and bench.sh (bash -n), then drives the harness against a local upstream stub: record-then-replay with batch re-assembly across chunks and orders, miss/unknown-purl accounting, --fill, a 502 (never stored) for an unreachable upstream, per-request latency with max in-flight, the getfqdn-free bind, the in-repo store refusal, and an end-to-end `bench.sh ab` pass and sha-mismatch failure with fake CLI binaries. Picked up by the existing `unittest discover -s scripts/tests` CI step. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…current helper The inline stream::iter().buffered() from the wheel-metadata fan-out predates utils::concurrent landing; route it through ordered_concurrent with the same limit (4) and the same in-order fold. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… too 0ef24902 flushed the scan event before the first stdout write after it fires, but stderr raises SIGPIPE just as well (main restores SIG_DFL). Two stderr writers could run in that window with the send still in flight: the lenient redirect-ledger load's "Warning: <corrupt ledger>" (non-hosted JSON and human paths, before discover_selected or the human flush) and, on the report-only JSON arm, the GC and VEX build ahead of the envelope. The inline send it replaced was always delivered first. The ledger load is inlined at its scan call site so the send is flushed right before its warning (only when it warns, so the overlap with the detail fetches is kept), and the JSON arm flushes before the GC/VEX step instead of just before the envelope. The --apply arm's warnings already follow discover_selected's flush. Test: telemetry_e2e runs a scan over a malformed redirect ledger with stderr closed and requires the event delivered (red before: SIGPIPE, 0 events). It uses a well-shaped token so the token-shape warning does not kill the child before the event fires. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scan now sends patch_scanned / patch_scan_failed through the spawn_* variants, which leaves the inline trackers without an in-tree caller. They stay: socket-patch-core is published to crates.io, removing a pub fn is a breaking change there, and every other event keeps its inline tracker. The doc comments now say so, so a later cleanup does not read them as leftovers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
batch_fallback_mid_run_replays_from_the_failing_chunk checked the folded uuids and the proxied tail, but not that chunks 4-5 were ever sent to the authenticated API. A serial loop (or a window of 1) would never request them and still pass. Assert all 6 authenticated requests: chunk 0 alone, then the whole 1..6 window in flight, so the discard path really runs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the public proxy scan runs up to PROXY_API_CONCURRENCY (4) batch windows at once. Each window's search_patches_batch degrades to the legacy per-package GETs (10 at a time) when /patch/batch rejects the chunk: a 400 from one exotic purl such as pkg:jsr, or an old proxy with no batch route. So a polyglot project on the proxy could put 4 x 10 by-package GETs in flight where the serial loop peaked at 10. That path swallows per-purl errors as "no patches", so extra load that saturates the proxy could change which packages come back. The client now holds a semaphore of PROXY_BATCH_PATH_CONCURRENCY (10) slots, shared by clones. Every proxy /patch/batch POST and every legacy per-package GET takes a slot, so all concurrent batch calls on one client stay within the old peak. A single call never waits: its groups of 10 fit the cap exactly as before. The authenticated API is untouched. Test: four concurrent batch calls of 10 purls each, all rejected with 400, keep at most 10 by-package GETs in flight and still reach 10 (red without the slots: 40). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…can't spawn When the walk pool could not be built (the OS refusing threads: a tight RLIMIT_NPROC or cgroup pids.max, or a huge RAYON_NUM_THREADS), run_walk ran the walk on the calling thread and the first parallel iterator then tried to build rayon's global pool, which needs the same refused threads and panics (exit 101) where the sequential walk succeeded. - Retry the pool build with half the threads on each failure, down to 1. - Route every parallel gather through walk_pool::par_map, which maps sequentially (in order) on a thread outside any rayon pool, so the no-pool fallback never reaches the global pool. - RAYON_NUM_THREADS can lower the walk thread count but no longer raise it past available_parallelism. Tests: halving build, par_map's sequential/ordered contract, run_walk's no-pool path, and the randomized oracle comparison with the pool off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adds `api_concurrency_for(use_public_proxy, requests)`, the in-flight cap for one window of `requests` patch-API requests. On the authenticated API it is `requests / REQUESTS_PER_SLOT` (4) clamped between `API_CONCURRENCY_FLOOR` and `API_CONCURRENCY`, so a large window can fill the cap while a short one stays at the floor and each connection still pipelines several requests. The public proxy, `SOCKET_API_CONCURRENCY` and a tight descriptor limit decide exactly as `api_concurrency` does. The windows whose length is known up front now ask for it: scan's batch and per-package queries, the hosted and vendored record views, discovery's baseline views and `get`'s variant and record views. The fixed-ceiling windows (vex record fetch, wheel metadata, vendor archive prefetch) keep `api_concurrency`. No behavior change: the floor and the ceiling are both 8, so every window still runs 8 in flight on the authenticated API. New tests pin the clamp and that the per-window cap defers to the proxy cap, the env override and the descriptor limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The walk pool took one thread per logical CPU up to 16, which on a 14-core machine meant 14 threads contending in the kernel's directory cache. On the polyglot monorepo fixture's npm crawl (21k walked dirs), 4 threads are the fastest or tied on macOS APFS (0.71 s vs 0.83 s at 14) and on Linux ext4 in docker (0.033 s vs 0.051 s at 10), at a quarter of the system time; the table is in walk_pool.rs. The default is now 4, lowered to the performance-core count (`hw.perflevel0.logicalcpu` on Apple silicon, the logical CPUs elsewhere) when that is smaller. `SOCKET_WALK_THREADS=<n>` replaces it, clamped to the existing hard ceiling of 16; the descriptor budget and the CPU count still bind, and a tight RLIMIT_NOFILE still means one thread. `par_map` keeps input order, so what the crawl finds and its order are unchanged. Documented in README and CHANGELOG; new tests pin the default, the performance-core bound and the override. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
User-approved operation change (H2-const). `API_CONCURRENCY` goes from 8 to 32. With the per-window sizing from the previous commit, each authenticated window runs a quarter of its requests at once, between 8 and 32: a step with 32 or fewer requests still runs 8, one with 128 or more runs 32. The fixed-ceiling windows follow the cap up to their own ceilings: vex / scan --vex record fetches now run up to 10 (was 8); wheel metadata and the vendored archive prefetch stay at 4. The public proxy stays at 4, and `SOCKET_API_CONCURRENCY` still replaces everything (1-32, proxy can only lower). Every window folds in request order, so output is unchanged. On the polyglot monorepo at 100 ms of latency the hosted scan ran the network at ~7 of 8 slots; at 32 the wall dropped from ~20 s to ~9 s with identical stdout. Existing test edits, all pinning this change: - vex_sources.rs `socket_api_concurrency_paces_the_record_fetch`: the default authenticated record-fetch window is now FETCH_CONCURRENCY (10), not 8. - hosted.rs wheel-metadata test: comment only (the cap it names is 32). Documented in CHANGELOG [Unreleased] and README's pacing paragraph. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… walk User-approved behavior change (DC-2/B5). The workspace `node_modules` walk descended into every untagged-by-name directory, including cargo `target/` trees and other tool caches: 12k of the 21k directories it walked on the polyglot monorepo. A visited directory whose listing (already in hand) names a regular `CACHEDIR.TAG` that begins with the Cache Directory Tagging signature is now pruned whole — its own `node_modules` and everything below it. Only a directory whose listing carries the name costs a read, of the signature's 43 bytes, through the FIFO-safe open. The one semantic change: a `node_modules` under a tagged cache directory is no longer crawled. The scan root is always crawled even when tagged, and a `CACHEDIR.TAG` without the signature, that is a directory, or that is a symlink prunes nothing. New test pins both the pruned and the kept cases and that removing the tag un-prunes. No existing test changed; the legacy walk oracle and its randomized fixtures never create a tag. Documented in CHANGELOG [Unreleased] and docs/ecosystems.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Clippy's type_complexity flagged the per-ecosystem map's element type. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…oup oracle The H4 equivalence test only ran the golden fixtures, none of which is a hatch project confirmed through a pinned requirements.txt (hatch reads the requirements rewriter's confirmations) or carries wheel metadata, so moving hatch into a group of its own still passed it. Add those projects (hatch + requirements.txt, native uv with and without wheel metadata, a PEP 723 script lock), thread python_metadata through the oracle, and assert the oracle tells a hatch/requirements split from the serial chain. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…read Scope::spawn panics when thread creation fails (a pids cgroup or RLIMIT_NPROC cap), which turned the hosted rewrite -- serial and thread-free before the groups existed -- into an exit-101 abort. Spawn through Builder::spawn_scoped and run a refused group on the calling thread in its place in group order. Every group starts from its own prefix clone, so where it runs cannot change the result; a test hook that refuses every thread pins the result and the first-panic order. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ence test No golden text carries the `-socket.<hex8>` suffixed version (or the registry index URL) without the artifact URL beside it, so dropping the suffixed-version needle from candidate_presence_needles passed every test. Add single-needle texts for each registry override and require a lone suffixed version to confirm its maven override. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The H3 snapshot test's identity-only copy was found by the alias walk, so the fallback never ran and answering it from an empty (or partial) snapshot still passed. Add two symlinked alias installs -- which the targeted lookup and the alias walk both skip -- among other crawled packages, and assert the snapshot answer equals a fresh crawl's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ompt The interactive hosted `scan --vex` handed its pre-prompt crawl to the embedded VEX step even when the confirm prompt waited on a person, so a tree changed while it waited was attested from the old roots and packages. Pass the snapshot only when the prompt answered at once (--yes, --json, non-terminal stdin, or no prompt), as the vendored path already does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- B5: every command that looks for installed npm copies walks the same trees, so `scan --prune`/`--sync` garbage-collect entries for packages installed only under a tagged directory (unless a lockfile resolves them) and apply/rollback/remove/repair/vendor/vex no longer find them. - DC-1: the walk pool also sizes the Maven repository walk and POM parse; only the npm crawl was measured. - H2-const: SOCKET_API_CONCURRENCY overrides the adaptive cap, but the fixed windows keep their own ceilings on top of it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
BEHAVIOR CHANGE. Every patch-API JSON call (get_json / post_json and the proxy batch POST: batch search, by-package, patch view / VEX records, package references) now retries an HTTP 429 or 503 up to 3 times (SOCKET_API_MAX_RETRIES, 0-10; 0 = the old single attempt). Waits honor Retry-After (delta-seconds or HTTP-date, capped at 30 s), otherwise 0.5 s / 1 s / 2 s steps (capped at 8 s) with deterministic equal jitter; all waits in a process draw from one 60 s budget. Nothing else is retried: 401/403 still classify on the first answer for the proxy fallback, the proxy's permanent "Patch API is not configured" 503 still degrades at once. An exhausted retry keeps its classification (RateLimited / Other) and names the retries in the message. The legacy per-package proxy path no longer swallows a throttled package: its error fails the batch (unresolvable PURLs are still skipped). Tests run on an injected virtual clock (RetryHooks). Two existing proxy_batch_e2e tests pin single-answer classification with expect(1); they now opt out of retries with ApiRetryPolicy::none(). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ings
BEHAVIOR CHANGE. A batch query that failed while others succeeded
vanished from the --json envelope with exit 0 (the human run warned on
stderr). It is now a run-level warnings[] entry {code: "api_batch_failed",
detail: "API batch <n> of <total> failed: <error>"}, in chunk order. The
agent / hosted / vendored flows' partial patch-list failures likewise
add {code: "patch_details_failed", detail: "could not fetch details for
<purl>: <error>"}. Status and exit are unchanged while some query
succeeded; the all-failed error rules are unchanged.
Docs: README (retry policy + SOCKET_API_MAX_RETRIES), CLI_CONTRACT.md,
CHANGELOG [Unreleased].
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…igured"
Review fixes on the 429/503 retry:
- The 60 s budget summed every request's wait, so 32 throttled requests in
flight exhausted it after ~12 waits (~5 s of wall time) and the other 20
got no retry at all. It is now a run-wide WALL-CLOCK window
(ApiRetryPolicy::retry_window, 60 s) opened by the run's first retry: a
retry whose wait would end after the window closes is refused. Parallel
waits overlap instead of adding up, so every request keeps its 3
retries while the run adds at most about 60 s. RetryHooks gains a
monotonic clock hook for tests.
- A Retry-After over the 30 s cap is no longer slept for 30 s and retried
early: the answer is final at once ("Retry-After <n> s exceeds the 30 s
retry cap").
- Retry-After: 0 or a past HTTP-date waits the jittered first backoff step
instead of retrying at once.
- The proxy's permanent 503 "Patch API is not configured" is now vetoed on
get_json / post_json too (per-package lookups, patch views), not only on
the proxy batch: answered once, the pre-retry ApiError::Other, so the
legacy per-package path skips it exactly as before.
- Throttle classification is typed: an over-capacity 503 left final by the
loop is the new ApiError::ServiceUnavailable (same text), and
is_throttle_error matches RateLimited | ServiceUnavailable instead of an
error-message prefix.
Tests: 32 concurrent persistently throttled requests each get 3 retries;
the window refuses a wait ending after it closes (sequential advancing
clock + unit test with a manual clock); Retry-After over the cap gives up
with 1 request; 0 / past date is floored; "not configured" on by-package
and view is answered once and skipped; a persistent over-capacity 503 and
429 on the proxy /patch/batch error with ZERO per-package GETs under the
default policy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The retry changes error text, adds waiting before a throttled run fails and makes a throttled package fail its legacy-proxy batch, so it moves to CHANGELOG "Changed (BREAKING)" (and the semver note); the --json api_batch_failed / patch_details_failed warnings are additive and move to "Added". README, CLI_CONTRACT.md and the CHANGELOG describe the wall-clock window, the over-cap / floor Retry-After rules, the never-retried "not configured" 503 and the legacy-path batch failure accurately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
WP-P2 pushed:
|
| run | before | after |
|---|---|---|
| polyglot hosted dry Dmitri Iouchtchenko (@0) ms | 3.61 s / 40.2 G instr | 2.03 s / 24.9 G |
| polyglot hosted wet Dmitri Iouchtchenko (@0) ms | 3.45 s / 40.6 G | 1.56 s / 25.2 G |
| polyglot vendored fresh Dmitri Iouchtchenko (@0) ms | 8.74 s / 107.1 G | 7.87 s / 100.8 G |
| polyglot vendored rerun Dmitri Iouchtchenko (@0) ms | 3.69 s / 40.0 G | 2.67 s / 33.9 G |
| polyglot hosted wet Devin Soni (@100) ms | 18.42 s | 5.82 s (concurrency 32) |
| depscan hosted dry ×2 Dmitri Iouchtchenko (@0) ms | 1.77 / 1.46 s (11.7 / 10.5 G) | 1.49 / 1.46 s (10.0 / 10.0 G) |
| depscan vendored fresh ×2 Dmitri Iouchtchenko (@0) ms | 1.70 / 1.60 s | 1.71 / 1.59 s (flat) |
Equivalence gate. In every polyglot step at 0 ms (hdry, hwet, vfresh, vrerun), canonical stdout, the full work-tree digest, request count and per-ecosystem purls are identical to cfeafb0c. Depscan is identical too: hosted dry stdout, and vendored fresh stdout, lock, workspace, package.json, state.json, files and markers. The polyglot tree has 5 signed CACHEDIR.TAG dirs (target/, build/, tools/patcher/target/, two .venv/). None contains a node_modules, so the pruning is not expected to change output here, and it does not.
⚠️ Behavior changes (approved)
- API concurrency 8 → 32. The window is adaptive: n/4 of a step's requests, clamped to 8..32. VEX record fetches go up to 10. The public proxy stays at 4.
SOCKET_API_CONCURRENCYoverrides the cap (1-32; on the proxy it can only lower it). - CACHEDIR.TAG pruning. A
node_modulesinside or below a directory with a signedCACHEDIR.TAGis no longer crawled. The scan root is always crawled. This reachesscan,scan --prune/--sync(such a package counts as not installed and is garbage-collected unless a lockfile resolves it), and the discovery ofapply,rollback,remove,repair,vendor,getandvex. - Bounded 429/503 retry.
- Policy: every patch-API JSON call retries up to 3 times. A
Retry-Afteris honored; one over 30 s gives up at once, and one under the jittered first step (0, a past date) is floored to it. Without one, backoff is 0.5 / 1 / 2 s (steps capped at 8 s) with equal jitter. All retries in a run must end within a 60 s wall-clock window opened by the first retry, so parallel waits overlap instead of adding up. SOCKET_API_MAX_RETRIES(0-10;0= old single attempt).- Never retried: 401/403, which still drive the proxy fallback, and the proxy's permanent
503 "Patch API is not configured"on any path. - New error text names why retrying stopped:
(gave up after 3 retries),(Retry-After <n> s exceeds the 30 s retry cap),(the run's 60 s retry window has closed). An over-capacity 503 is the typedApiError::ServiceUnavailable. - New additive
--jsonrun-level warnings:api_batch_failedandpatch_details_failed. - Legacy proxy batch: on a token-less proxy without
POST /patch/batch, a package still throttled after its retries fails its whole batch query instead of being skipped silently. - Listed under CHANGELOG "Changed (BREAKING)" and the semver note; the
--jsonwarnings are under "Added".
- Policy: every patch-API JSON call retries up to 3 times. A
Tests
cargo clippy --workspace --all-targets -D warnings is clean. cargo test --workspace --no-fail-fast on the PR head: 9273 passed, 0 failed, 136 ignored.
Review notes
- pp2: 3 adversarial reviewers, then a fixer.
- The retry got a dedicated adversarial reviewer, then a fixer. Its fixes landed in
fix(api): bound retries by a wall-clock window; never retry "not configured":- The summed 60 s budget starved concurrent requests; it is now a wall-clock window.
- A
Retry-Afterover the cap now gives up at once, and 0 / past dates are floored. - The "not configured" 503 is no longer retried or treated as throttling on per-package and view calls.
- Throttle classification is typed instead of matched on a message prefix.
- New test: a persistent 503 / 429 on the proxy batch makes zero per-package GETs.
- Pushed commit subjects keep their original
!markers.fix(scan)!(the--jsonwarnings) is additive and is documented under "Added"; history was not rewritten. - Known pre-existing follow-up, unchanged here: two paths still drop failures silently, the
scan/discovery.rsbaseline view andrepair_vendor.rs's.ok()??.
🤖 Generated with Claude Code
#256 replaced the cargo manifest regex readers (CARGO_TOML_PACKAGE_RE, CARGO_TOML_VERSION_VAL_RE, and the capturing CARGO_TOML_REGISTRY_VAL_RE) with TOML-parsed values (cargo_toml_string / cargo_toml_inline_string), so single-quoted literals, quoted keys, and trailing comments read the same way everywhere. The textual merge was clean, but #257's CargoRegistryPins (the one-scan-per-manifest pin extractor behind the vendored cargo PIN_MEMO) still used the removed package regex and the old capture group of the registry regex. Resolutions: - CargoRegistryPins::of now reads values exactly as #256's cargo_socket_registry_pin does: header-table `package`/`registry` via cargo_toml_string, dotted `<crate>.registry` via cargo_toml_string, inline tables via cargo_toml_inline_string (registry first, package parsed only when a Socket pin is present). cargo_socket_registry_pin (now #256's semantics) stays its #[cfg(test)] equivalence oracle. - The V-7 pin-equivalence corpus gains #256's spellings (literal strings, quoted inline keys, commented header lines) so the oracle pins the new semantics. - Everything else from #256 (workspace-inheritance resolution, (name, version) lock owners, validate_cargo_toml_pins, annotated registry headers in plan_cargo_config) merged untouched; #257's lock_block_end bounded search + oracle, parse memos, presence probe and parallel rewriter groups needed no change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Merged origin/main (#256) into this branch as 2b380e3. It's a merge commit, not a rebase or force-push. CI was red because GitHub builds the PR merged with main. #256 removed Resolution:
Verification:
🤖 Generated with Claude Code |
…relude and a plan gate The vendor loop only prefetched npm's service downloads, because only the npm flavors could say, ahead of the loop, which packages they would ask the patch service for. Every other backend decided that inline: cargo, composer, gem, golang, maven, nuget and pypi each ran their coordinate guards, lock/manifest reads, refusals and in-sync hot-path tests at the top of `vendor_*`, interleaved with the service call. Each of those prefixes is now one function the entry point runs first (`cargo_prelude` + `cargo_wet_preflight`, `composer_prelude`, `gem_prelude` + `gem_edits`, `go_prelude`, `maven_prelude`, `nuget_prelude`, `pypi_prelude`), returning either the outcome the old inline code returned (a refusal, the empty-patch no-op, the in-sync result) or the values the rest of the body reads. Nothing moved relative to anything observable: the same checks run in the same order with the same outcomes; only the code is shaped so the same checks can be asked without the rest of the call. `service_preflight` per backend, and `vendor::service_preflight` over them, answer "would a wet run with the service enabled ask the service for this record?" by calling exactly those functions: past every refusal raised before the first service call, and not answered by the hot path or (pypi) a committed-wheel reuse. A download plan built from it never names a package the loop refuses first. Tests: per backend, an oracle over a mix of good, refused and no-op records — the gate's verdicts, computed first like the plan, must equal the grants the backend then requests, in order; and once vendored the re-run must plan (and ask) nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…npm's The vendored loop wires one package at a time, and only npm's grant + archive round trips were fetched ahead of it. Every other ecosystem paid them back to back inside the loop, ecosystem after ecosystem: on the polyglot monorepo at 100 ms RTT, 408 non-npm downloads at 0.22-0.35 s each were most of a ~120 s vendored scan. The loop's download plan now covers every service-backed record, in loop order. `plan_service_downloads` walks the sorted packages exactly as the loop does — variant bases fanned out once, each variant through the same installed-variant probe (a probe that would need a download not made yet leaves the variant unplanned), the Bun refusal, the hosted-takeover gate — and keeps only the records whose backend gate says they reach the service: npm's one-read `preflight_packages` as before, every other ecosystem's `service_preflight`. Records the ledger already holds at the record's uuid are left out (the backends' in-sync hot paths answer them, and proving that ahead of the loop would verify every committed artifact twice); an unplanned record is simply fetched live. Every doubt resolves to "not planned", never to a grant the loop does not ask for. The gates run one at a time: several at once each hold their own parse of the project's locks (a cargo gate clones the whole Cargo.lock document). The prefetch itself: - the window is the API's in-flight cap (was min(cap, 4)); a byte budget (128 MiB of fetched archives waiting for the loop) bounds memory: past it only the position the loop is at may start; - a planned download may name a secondary artifact — gem's stub gemspec rides its `.gem` — fetched along with the archive only when the archive is ready and passes the same integrity checks `fetch_verified_archive` runs, and taken by `fetch_verified_secondary` in place of the live request, its debug lines held back until then. Consumption stays serial in record order and the breaker is still folded at the loop's own call, so outcomes, events and stderr are unchanged. Polyglot (vs PR #257 head): vfresh/vrerun at 0 ms byte-identical stdout and work-tree digest, 479 grants either way, replay misses 0; vfresh at 100 ms ~120 s -> 16.0 s. Tests: budget and secondary-prefetch oracles against the serial loop (same outcomes, same requests; no secondary for a tampered archive), and a composer twin of the zero-grant e2e. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With every service download prefetched, the vendored loop's next floor is its own local work, and for the directory-shaped backends most of it is one step nothing overlapped: extracting the verified prebuilt archive into the stage (cargo `.crate`, composer dist zip, golang module zip, the gem's data.tar.gz) — about a second of the polyglot monorepo's loop at 0 ms, serial between packages. Each backend's plan gate now also names what to do with its archive once it lands (`vendor::prestage::PrestageRecipe`, built from the same copy dir its stage derives from), and the prefetch task runs it on a small bounded blocking pool (4) as the download completes — only for an archive that passed the integrity checks `fetch_verified_archive` runs, exactly like the gem stub. The tree lands in `<copy>.socket-prestage`, a sibling of the backend's stage, never the stage itself; the backend claims it with one rename right where it would have extracted (replacing stage litter as its `remove_tree` would), and everything after — the tree check, the tag, the swap, the wiring, the marker and the ledger — runs as before, serially in record order. maven, nuget and pypi get the one pure step they run on the archive bytes instead: the afterHash check of the patched members, whose verdict rides the archive and is used only for the exact file set it was computed against. Nothing observable changes: - a failed pre-stage (the extractor's own refusal included) stages nothing and hands the bytes back: the backend extracts live and reports its own words; - a backend that runs without its pre-stage (its call skipped by the breaker, its plan position passed over) builds in its own stage undisturbed; - an unclaimed tree is queued and removed, with the vendor levels it created, only by `prestage::settle` after the loop and the plan are done — never concurrently with the loop's own unwinds; - dry runs build no plan, so nothing is staged. Composer's intra-archive parallel inflate (CMP-1(2)) already exists. Polyglot (vs PR #257 head): vfresh/vrerun at 0 ms byte-identical stdout and work-tree digest, 479 grants, replay misses 0; vfresh at 0 ms 8.2 -> 7.8 s, at 100 ms 16.0 -> 15.1 s (on top of the prefetch). Tests: the pre-stage lifecycle (claim, refused extraction, unclaimed tree swept only by settle, verdict scoping), and cargo/composer oracles: planned and unplanned runs leave the same outcome and the same tree, for a good archive and one the extractor refuses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…and sources (approved behavior change)
USER-APPROVED BEHAVIOR CHANGE (perf plan V-4a / JS-2 generalized).
A vendored run fetched every selected patch's view, and the pristine
source of every lockfile-only package, before the vendor loop refused the
packages whose backend can never wire them — on the project's lock text
alone. On the polyglot monorepo that is 80 of 560 packages, refused again
on every re-run, each paying a view round trip (and 3 of them a registry
tarball) for a refusal that needed no network.
`vendor::lock_text_refusals` answers, with no view, which purls a backend
refuses on lock and manifest text alone, with the backend's exact code and
detail:
- npm in a pnpm / yarn classic / yarn berry project: the flavor's own
pre-flight (V-1's gate) names the refused packages, and the backend
itself — dry-run, no source, no patch content — supplies the words; its
code must equal the pre-flight's, or nothing is refused early. Those
backends evaluate every such gate before they read the package, dry run
or not.
- cargo: the prelude's `locked_version_mismatch`, only when it is the
crate's first refusal (an in-tree `cargo vendor` copy refuses first and
stays the loop's).
What changes (documented in CHANGELOG [Unreleased] and CLI_CONTRACT.md):
- `scan --mode vendored` / `get --mode vendored`: such a package is
refused in the download phase, after the Bun refusal and the ledger's
already-vendored skip — `download.patches[]` records it `action:
"failed"` with the backend's `errorCode`/`error`; `download.downloaded`
drops and `download.failed` rises by their number; `vendor.events` loses
their `failed` events (`vendor.summary.failed` drops by the same number)
and, for lockfile-only packages, their `vendor_fetched_missing` events;
no view, no pristine fetch. Exit code and `status` are unchanged. Purls
the hosted redirect ledger claims (their takeover rewrites the lock) keep
the loop's refusal, as does everything when that ledger is malformed.
- `vendor` (manifest-driven): the per-package `failed` events stay, but a
lockfile-only package refused this way is deferred instead of fetched —
no `vendor_fetched_missing` event, no registry request (with the registry
unreachable, the gate's code replaces `vendor_fetch_failed`).
Observed on the polyglot fixture (vs the V-1+V-3 build): vfresh — download
downloaded 560->480, failed 0->80 (74 vendor_lock_entry_not_found, 4
vendor_override_conflict, 2 vendor_lock_entry_unsupported), vendor
summary.failed 81->1, 80 failed + 3 vendor_fetched_missing events gone,
requests 2250->2167 (80 views, 3 registry tarballs); vrerun — downloaded
81->1, failed 0->80, the same events gone, requests 791->708. Work-tree
digest byte-identical in both.
Tests updated (they pinned the old ordering):
- scan_vendor_e2e `exact_download_plan::a_package_the_loop_refuses_costs_zero_grants`
(pkg-b's refusal is now a failed download record, and costs no view);
- e2e_yarn_legacy_cachekey_refusal_build `yarn{2,3}_{default,compression0}_*`
(the berry cacheKey refusal fires in the download phase).
New: core oracles (the early refusal is exactly the backend's wet-run
refusal; package-lock refuses nothing early; cargo's refusal only when it
is the first), and a `vendor` e2e proving a refused lockfile-only package
is never fetched.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-installed skip V-4a moved every lock-text refusal ahead of the view fetch, including packages the vendor loop never handed to a backend at all: a package absent from the lock and not installed has no pristine source, so the loop used to report it `skipped` / `package_not_installed` — and V-4a turned it into a failed `vendor_lock_entry_not_found` (pnpm) or `locked_version_mismatch` (cargo) download record, a new failure for a package the user never had. The early refusal now applies only where the loop would hand the package to its backend (`vendor::lock_refusals_reaching_backend`): an installed copy (the loop's own resolver + npm identity lookup, reusing scan's npm crawl), or a verifiable registry resolution the pristine-source ladder fetches (`pristine_fetch_is_verifiable`, the renamed cargo check — the same entry choice as `fetch_pristine_package`). Everything else keeps today's outcome and counts, on `scan`/`get --mode vendored` (both get paths) and on manifest-driven `vendor`. CHANGELOG and CLI_CONTRACT describe the final scope, and that a lock-text refusal now takes precedence over view-derived outcomes (paid-access 403, failed view, no applicable files). Tests (fail with the narrowing bypassed): scan and exact-purl get over a pnpm project (installed-unlocked and locked-uninstalled refused early, absent-from-both skipped as before) and a cargo project (only a crate installed at an unlocked version refused early), and `vendor` keeping the not-installed skip for pnpm and cargo. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… path npm_flavor::lock_text_refusals handed the backend an empty path as the installed folder; an empty path resolves against the process cwd if anything ever read it. Use a path inside a fresh private temp dir, and skip the dry runs entirely when the pre-flight refuses nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…se refusal The V-4a edit of the berry cacheKey refusal cases dropped the vendor-step assertion along with the old per-package failed event. Restore it in its new form: no vendor event for the refused package and a vendor summary that does not count it again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e plan itself The composer twin of the zero-grant e2e put the refused package first in the loop order, where the prefetch passes it over before any request whether the plan named it or not — it passed with the gate bypassed. The refused package (now psr/http-message) sorts between the two granted ones, behind a position the service answers, so a bypassed gate grants it. A direct unit test of plan_service_downloads pins the plan too. Both fail with the non-npm gate replaced by an unconditional plan entry (3/3 e2e runs). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… loop A run that crashed or was interrupted between pre-staging an archive and prestage::settle left <copy>.socket-prestage trees under .socket/vendor that only a later pre-stage of the same package replaced. Every wet vendor loop (all sources, --offline included) now sweeps them first — under the apply lock and before anything of its own is staged, so every one on disk is stale — with the vendor levels only they kept alive. The sweep never descends into a copy dir, and a dry run deletes nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…udget bounds V-1 opened the prefetch reach straight to the API's in-flight cap (32) for every ecosystem once the service answered, where the npm-only prefetch had used 4 — so a service failing after a good start could face up to 31 extra retry ladders. The reach now slow-starts: 4 after the first answer, one more per good answer up to the cap, and back to 4 on an availability failure (the barrier and the task breaker are unchanged). Outcomes, events and stderr are unchanged: the plan stays advisory. The module docs (and ARCHIVE_PREFETCH_BYTES) now say that the 128 MiB budget bounds starting new downloads — the in-flight ones still land and can carry the held bytes past it — and that pre-staged trees live on disk, bounded by the plan and the pre-stage pool, not by size. Tests: the reach arithmetic; a 32-wide window opens only 4 positions while they are unanswered; good answers grow it to exactly the window. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…y lock refusal The depscan vendored gate showed the one envelope difference the V-4a docs did not name: with its 3 lock-text refusals moved to the download phase, the vendor step has no failures left, so vendor.status goes partialFailure -> success (the top-level status and the exit code stay partial_failure/1). Say so, and name the human arm's message when every selected package is refused this way. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
WP-P3 pushed: vendored-loop prefetch across ecosystems, pre-staging, early lock refusals (2b380e3 → 35a42b0, fast-forward)
Before → after (replay harness, polyglot monorepo and depscan)
Output equality
|
Hosted and vendored scans spent most of their time in two places:
This PR makes the network phases concurrent and the crawlers and rewriters single-pass. It also adds lazy vendored sources, in-memory repacking and per-run parse memos, and ends with a small set of approved behavior changes.
Up to the approved batch, stdout, stderr, lockfiles, ledgers and exit codes are byte-identical to
main. The approved changes are listed below and detailed in this comment.Results
"Before" is
mainwithout this PR and "after" is this branch's head. Both use a replay proxy (recorded API responses served locally) at 100 ms/request, 2 interleaved reps, medians. Instructions retired are in parentheses.The "before" binary predates #251–#254. #254 made cargo vendoring heavier, so the cargo rows understate the gain. Measured against the pre-batch head of this branch, cargo vendored fresh drops 49.6 → 38.9 G.
scan --mode hosted --dry-run(live API)¹¹ Measured on the first slice of this PR and not re-run since.
main's 71 grants / 344 requests.What changed
Network (all ecosystems)
utils/concurrent.rsis an ordered-concurrency helper (buffered, neverbuffer_unordered). Results, warnings and errors are folded in input order.GET /patches/view, download views and vex record fetches run concurrently: 8 in flight on the API, 4 on the public proxy.SOCKET_API_CONCURRENCY:1restores strictly serial requests. A tight descriptor limit falls back to serial automatically.Crawl
gem envand composer home are asked once per process.Rewriters and vendoring
[[package]]). The old rewriters are kept as oracles.Tooling:
scripts/perf/holds the record/replay benchmark harness.Behavior changes (approved)
Details and measurements are in the batch comment.
{snapshot, ops}edits of the record's ownoriginal. Version 1 ledgers are still read and revert byte for byte. Other ledgers stay byte-identical v1.state.jsonshrinks: maven 29.8 → 15.1 MB, pylock 43.9 → 22.0 MB, nuget 7.2 → 3.9 MB.--offline/ network-less re-run now succeeds (already_vendored, exit 0), and re-runs no longer reportvendor_fetched_missing.-e <eco>crawl scope (MVN-4): without--prune/--sync, only the named ecosystems are crawled.lockfileOnlyPackagescounts only them (depscan+zed-e npm: 6920 → 5347).~/.m2path (MVN-1): canonical POMs are not opened once their top-level group directory is confirmed. A hand-placed POM that disagrees with its path reports the path's coordinates..socket/vendor/.commit-journal.json. A crash leaves the pre-run wiring, or is rolled forward by the next locked command. A failed commit write now fails the run with the new top-level errorvendor_commit_failed(exit 1), replacing the per-purlvendor_state_write_failed/redirect_ledger_write_failedevents; locks and ledgers stay pre-run, and artifacts written in the loop remain until the next run.vendor_stale_artifact_removedevents now follow the per-package events.Resolved: exact plan, 71 grants
The earlier prefetch planned download grants before each backend's pre-flight checks ran, so depscan sent 74 grant POSTs where
mainsends 71. The plan now calls each backend's own pre-flight gates, so fresh depscan vendoring sends exactly 71 grants and 344 requests, with 0 replay misses. Parity tests pin each npm flavor's plan against its loop, refusal for refusal.Other observable differences
--debug, the[socket-patch debug] GET/POST <url>lines for concurrent requests interleave. Normal warnings keep serial order, and tests pin that.Test plan
cargo clippy --workspace --all-targets -- -D warningsis clean.cargo test --workspace --no-fail-fast: 9,218 passed, 0 failed, 136 ignored. No existing test was ignored or deleted. All existing test edits pin an approved behavior change: the two ledger-write-failure tests incovgap_commands_vendor.rsnow expect the newvendor_commit_failedenvelope error instead of a per-purlfailedevent (V2), and the redirect one no longer asserts that the artifact is absent, since it is written before the commit fails. It asserts instead that nothing is committed or rewired and no journal remains. The v1 legacy-ledger fixture for cargo was regenerated with the pre-batch binary to matchmain's Wire vendored cargo patches through Cargo.toml and tag the copy #254 cargo wiring. Every other change before the approved-behavior batch kept all existing assertions unchanged.SOCKET_API_CONCURRENCY=1and tight fds;The commits are the original reviewed commits, replayed onto current
main(#251–#254). Where a conflict or an adaptation tomainwas needed, it is folded into the commit that needed it, and each batch comment lists them.🤖 Generated with Claude Code
Note
High Risk
Changes vendoring crash semantics, ledger schema, offline/idempotent vendor behavior, Maven discovery coordinates, and concurrent API failure modes—areas that affect committed project state and patch correctness.
Overview
This PR speeds up hosted and vendored workflows by running patch-API traffic concurrently (default 8 authenticated / 4 on the public proxy) via
ordered_concurrent, withSOCKET_API_CONCURRENCY(1–32) to throttle when a proxy or WAF limits in-flight requests. The same pacing applies tovex/scan --vexrecord fetches and hosted wheel-metadata downloads;hold_back_debugkeeps per-request--debuglines in the same order as the old serial loop.Beyond networking, it lands several v5.0 contract changes documented in CHANGELOG / CLI_CONTRACT: deferred pristine fetch so already-vendored / service-backed cargo re-runs can succeed offline without upfront registry pulls;
scan -ecrawls only selected ecosystems (except prune/sync); Maven derives coordinates from canonical~/.m2paths when the repo root is confirmed; vendor ledger v2 stores large whole-file wiring as snapshot+ops edits; one group commit per vendored run through.socket/vendor/.commit-journal.json(new top-levelvendor_commit_failedon commit I/O failure); and batched fsync for artifacts before durable lockfile/ledger writes.Smaller fixes include
vexapi_auth_fallbackquoting the first refusal in pending order (not completion order), batched gem/Python stale-install probes usingjudge_installed_record, andconfirm_waitsso scan knows when a hosted confirm prompt will block.Reviewed by Cursor Bugbot for commit f9fc676. Configure here.