diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 107b5a59..3d6155e7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1123,7 +1123,7 @@ jobs: ruby-version: ${{ matrix.ruby || '3.2.10' }} # The legs pin their bundler (`bundler:`) so a capstone never rides # whatever bundler the runner's Ruby ships; bundler 1.x is not - # installable through setup-ruby, see the next step. e2e_gem keeps + # installable through setup-ruby, see the next steps. e2e_gem keeps # the 2.5 floor. bundler: ${{ startsWith(matrix.bundler, '1.') && 'none' || matrix.bundler || '2.5' }} bundler-cache: false @@ -1131,11 +1131,26 @@ jobs: - name: Install Bundler 1.x if: startsWith(matrix.bundler, '1.') shell: bash + env: + BUNDLER_TEST_VERSION: ${{ matrix.bundler }} + run: gem install bundler -v "$BUNDLER_TEST_VERSION" --no-document + + # Installing a bundler does not make `bundle` run it: with no lockfile + # to read, RubyGems' binstub activates the HIGHEST installed bundler, + # so a leg pinned BELOW its Ruby's default gem (2.1.4 / 2.2.33 on Ruby + # 3.1, whose default is 2.3.27) silently runs the default instead and + # tests/common/bundler_e2e.rs rightly panics. BUNDLER_VERSION makes + # the binstub select exactly the pinned release in every process (and + # turns off bundler >= 2.3's lockfile-driven self-switch), the same + # knob tests/docker/Dockerfile.gem-b1 sets. + - name: Select the pinned Bundler + if: matrix.bundler != '' + shell: bash env: BUNDLER_TEST_VERSION: ${{ matrix.bundler }} run: | - gem install bundler -v "$BUNDLER_TEST_VERSION" --no-document echo "BUNDLER_VERSION=$BUNDLER_TEST_VERSION" >> "$GITHUB_ENV" + BUNDLER_VERSION="$BUNDLER_TEST_VERSION" bundle --version - name: Setup PHP if: matrix.composer != '' @@ -1420,6 +1435,10 @@ jobs: - name: Install Rust run: rustup show - name: Install the cargo under test + # bash, not the Windows default pwsh: in PowerShell + # "$CARGO_TEST_TOOLCHAIN" is an (unset) PowerShell variable, so the + # windows-latest leg ran `rustup toolchain install ""`. + shell: bash env: CARGO_TEST_TOOLCHAIN: ${{ matrix.toolchain }} run: rustup toolchain install "$CARGO_TEST_TOOLCHAIN" --profile minimal diff --git a/CHANGELOG.md b/CHANGELOG.md index d091a5ab..8540e431 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -150,6 +150,18 @@ into the new version's section — see docs/releasing.md. ### Added +- **`redirect_yarn_berry_mixed_line_endings` and + `vendor_yarn_berry_mixed_line_endings`.** A `yarn.lock` (or, vendored, a + root `package.json`) that mixes CRLF and LF line endings — or holds a bare + CR — has no single ending to keep, and yarn itself rejects such a lock + under `--immutable` (YN0028) and rewrites it wholesale on its next plain + install. Both modes now refuse it before any write with a code naming the + line endings and the `yarn install` remedy; a revert never refuses on line + endings (a restored lock entry takes the terminator of the entry it + replaces). The real-yarn berry suites gained `SOCKET_PATCH_YARN_BERRY_EOL=crlf` + to run on CRLF files on macOS / Linux as yarn writes them on Windows, and + print a `BERRY-EOL||||yarn=…|flow=…` line per fixture + file — see [yarn berry compatibility](docs/testing/yarn-berry-compatibility.md). - **Hosted npm redirects configure npm 12's `allow-remote` for you.** npm 12 defaults to `allow-remote=none` and refuses (EALLOWREMOTE) a lock that resolves patched packages from the Socket patch host. When `scan --mode @@ -559,6 +571,49 @@ into the new version's section — see docs/releasing.md. proxy, sending private module paths off the machine. It is now refused (`vendor_fetch_unverifiable`, then the usual `package_not_installed` skip) unless `SOCKET_GOPROXY` names a proxy. +- **yarn berry projects on Windows (CRLF files) are redirected and vendored + instead of refused.** yarn berry (2.x–4.x) writes a file it creates with + the OS line ending (`os.EOL`) and keeps an existing file's majority ending + on every later write (`normalizeLineEndings` in yarnpkg-fslib's + `FakeFS.ts`, used by `Project.persistLockfile` and + `Workspace.persistManifest`) — so on Windows a fresh `yarn.lock` and the + `package.json` yarn first pretty-prints are CRLF, and a `core.autocrlf` + checkout makes them CRLF on any OS. `scan --mode hosted` / `get --mode + hosted` refused every such lock (`redirect_yarn_berry_crlf_unsupported`, + redirected 0); a CRLF lock is now rewritten in its own line ending — every + untouched byte, a leading BOM included, round-trips — and the ledger records + the lock's on-disk CRLF fragments, so `rollback`, `remove` and the hosted → + vendored takeover restore it byte-for-byte (they also replay a ledger + recorded on a checkout whose uniform line ending has since flipped, LF ↔ + CRLF). `vendor` / `scan --mode vendored` now keep `package.json`'s layout + (BOM, indent, line ending, trailing-newline shape) on both the wiring and + the revert: `vendor --revert` wrote a CRLF manifest back LF, never + byte-identical to the pre-vendor file. A BOM'd `package.json` (and a + BOM'd `.yarnrc.yml`, whose first-line `compressionLevel` was read as + unset) no longer fails the vendored backend, and every berry reader skips + a BOM in front of a header-less `__metadata:`. Verified on real yarn + 4.12.0 (hosted, vendored, workspaces, pnpm linker, both mode takeovers) + with the fixtures re-spelled CRLF, and on yarn 2.4.3 / 3.8.7 (still + refused for their cacheKey, never for their endings). +- **A yarn berry mode takeover no longer strips the old mode's patch before + the new mode refuses the project.** `scan` / `get --mode hosted` over a + vendored berry purl reverted its vendored wiring, ledger entry and + artifact (`redirect_takeover_reverted_vendored`: "now fully hosted") and + only then ran the rewriter, which refused a lock with mixed line endings + (or an unsupported `cacheKey` / `.yarnrc.yml` `compressionLevel`) — + `redirected: 0`, and the next `yarn install` pulled the unpatched registry + package. `vendor` / `scan --mode vendored` over a hosted berry purl did the + same in reverse (`vendor_takeover_reverted_redirect`, then `failed` + `vendor_yarn_berry_mixed_line_endings`). Both takeovers now run the new + mode's berry gates first — wet and `--dry-run` alike — and a refused purl + keeps the old mode's wiring byte-identical. +- **`setup` keeps a CRLF `package.json` CRLF.** `setup` and `setup --remove` + re-serialized `package.json` with bare LF and dropped a leading BOM, so on + a Windows yarn berry project (yarn pretty-prints the manifest with CRLF) a + two-key script edit became a whole-file diff that yarn then kept, and + `setup --remove` could not land byte-identical on the pre-setup file. + `package.json` is now written in its own layout (BOM, indent, line ending, + trailing-newline shape), the same helper the vendored backends use. - **A vendoring-service outage no longer re-vendors packages.** An npm re-run (every lock flavor, `bun.lockb` included) re-acquired its tarball from whichever source answered — the service's prebuilt, or a local pack diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index f722e4f6..e4b72f53 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -123,7 +123,7 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc **Vendored entries and the rest of the CLI.** Because nothing is in the manifest, vendored patches are invisible to `apply` (nothing to apply in place) but fully visible to `list` (listed from the ledger, labeled `Mode: vendored (recorded in .socket/vendor/state.json)` in human mode, exit 0 on a vendored-only project), `vex` (attested from the embedded records while a lockfile still wires the artifact — see "Manifest-less VEX"), `repair` (health-checked and rebuilt from the ledger), `scan --prune` (lockfile-driven reconcile) and `setup --check`'s patch-consistency property (consulted from the embedded records). They are exempt from standalone `vendor`'s manifest reconcile (`reconcile_dropped` never touches `detached` entries) and exit via `remove ` (which reverts them), `vendor --revert`, or `rollback`, whose vendored leg reverts every in-scope ledger entry (unscoped and identifier-scoped runs; path-scoped runs reach them only when an installed copy matches). The hidden `--detached` flag (`scan --vendor --detached`) names exactly this — the only — vendored posture and is accepted as a no-op for compatibility. -`scan --mode hosted` (== `--redirect`) swaps the in-place apply for the registry-redirect pipeline: discover → resolve hosted-patch references (grant token + integrity + per-dep registry override) → rewrite ONLY the patched dependencies' lockfile / registry-config entries to point at the hosted packages. A dep counts as **redirected** only when its hosted-artifact URL (or per-dep registry index URL) actually landed in a project file — a granted reference whose rewriter found nothing to edit is neither recorded nor attested. Cargo and golang are confirmed only by their rewriter's own report (`confirmed_cargo_uuids` / `confirmed_golang_uuids`): a golang dep counts only when its go.mod `replace M V => patch.socket.dev/gopatch/ ` and both go.sum lines are in place, never because the patch-server origin or leftover go.sum lines appear somewhere. A golang module that go.mod does not require and go.sum does not list at the patched version is outside the build graph and is refused with `redirect_golang_not_in_module_graph` (nothing written). Only the exact module `patch.socket.dev/gopatch/` is socket-owned; any other module path is refused with `redirect_golang_untrusted_module_path`. A vendored golang module is taken over like cargo and the npm family: its vendor wiring, committed copy and ledger entry are reverted first (`redirect_takeover_reverted_vendored`). Re-runs over already-rewritten output record zero new edits. **Lock (v5.0)**: the hosted engine acquires `<.socket>/apply.lock` around its first wet write (the takeover pre-reverts) — not on `--dry-run`, and not when the run would write nothing (zero redirects, all skipped) — so previews and no-op runs never create `.socket/` (and never quarantine: a `--dry-run` or a zero-grant wet run that finds a malformed `redirect-state.json` reports it as the hard error it is — exit 1, the repair-or-move-aside remedy — but moves nothing; only a run holding the lock moves it aside to `redirect-state.json.corrupt`); contention is `lock_held` and a lock-file I/O fault (a read-only project root, a file squatting on `.socket/`) is `lock_io` — both exit 1, refused BEFORE the redirect ledger is read or written, and rendered like every other lock holder: human `Error (): ` on stderr (+ the `--lock-timeout` hint for a live holder); JSON keeps the hosted shape — top-level `status: "error"`, `errorCode: "lock_held" | "lock_io"`, a string `error`, and `redirect: {mode: "hosted"}` retained (NOT the vendored `error: {code, message}` object). **Takeover symlink pre-check (v5.0)**: a vendored→hosted takeover whose recorded wiring file is a symlink is refused up front with `redirect_symlinked_file_unsupported` — wet and `--dry-run` alike, before any revert — so "nothing was written" holds. **Human mode (v5.0)**: `scan --mode hosted` prints the results table and update detection like the other modes and confirms once — `Redirect N packages to the hosted patch server?` (singular for one), default yes, skipped by `--yes`/`--json`, on `--dry-run` (the engine honors the preview itself; nothing mutates), and when the detail fetch leaves nothing to redirect (that run enters the engine as a no-op — `Redirected 0 packages; rewrote 0 files.`, no lock, no `.socket/` — without prompting); without `--yes` on a non-TTY stdin the shared prompt prints `Non-interactive mode detected, proceeding automatically.` to stderr (unless `--silent`) and proceeds — before rewriting anything (parity with the agent/vendored arms and with `get --mode hosted`). The detail fetch prints the same progress counter and per-package `Warning: could not fetch details for …` lines as the agent arm. An EMPTY hosted discovery prints `No patches available for installed packages.` and exits 0 without entering the engine (previously `Redirected 0 packages; rewrote 0 files.`); a discovery whose every offer is paid-tier for an org without paid access prints the table's paid nudge, then `No downloadable patches (paid subscription required).`, and exits 0 without entering the engine (parity with the agent/vendored arms). A malformed redirect ledger on a human hosted run that returns before the engine (empty discovery, nothing downloadable, a detail-fetch failure, a declined confirm) is surfaced there as the read-only `Warning: the redirect ledger … is malformed …` advisory (muted by `--silent`), never moved; the `--json` arm always enters the engine and hard-errors instead. JSON output gains a `redirect` sub-object: `{ mode: "hosted", redirected, rewrittenFiles, skipped, warnings, dryRun }` (`mode` is additive so consumers can dispatch without inferring it). Rewriter warnings carry stable `redirect_*` codes (e.g. `redirect_npm_no_lockfile`, `redirect_gradle_manual_snippet`, `redirect_golang_unsupported`); new codes are additive (MINOR). v5.0 additive codes: `redirect_composer_no_lockfile` / `redirect_gem_no_gemfile` (composer / gem: neither manifest nor lock present — once per run, after the intake gates), `redirect_maven_no_pom` (no `pom.xml` and no Gradle build), `redirect_nuget_lock_unparseable` (a present-but-corrupt `packages.lock.json` — warned once, nothing mutated; an absent lock still proceeds), `redirect_cargo_lock_pkg_ambiguous` (several same-name+version `[[package]]` blocks and none carries the index `source` — transactional skip). Also v5.0: a registry override of the wrong kind (or none at all) warns the arm's missing-override code for nuget/gem/golang where it used to skip silently, and the ledger's `redirect_nuget_source` edit records `action: "added"` when `nuget.config` was authored from scratch (`rewritten` otherwise). Refusals stay fail-closed with a diagnosis that names the actual cause: a yarn-berry lock entry resolving through a non-`npm:` protocol keeps `redirect_yarn_berry_unsupported_protocol` with the entry's ACTUAL protocol in the detail — except socket-patch's OWN vendored wiring (a `file:` range into `.socket/vendor/`), which gets the distinct `redirect_yarn_berry_vendored_entry` code whose detail names the retirement path (`remove ` per package, or `vendor --revert` which unwinds every vendored package, then re-run `scan --mode hosted`). Both leave the entry byte-identical; neither changes exit code or status. +`scan --mode hosted` (== `--redirect`) swaps the in-place apply for the registry-redirect pipeline: discover → resolve hosted-patch references (grant token + integrity + per-dep registry override) → rewrite ONLY the patched dependencies' lockfile / registry-config entries to point at the hosted packages. A dep counts as **redirected** only when its hosted-artifact URL (or per-dep registry index URL) actually landed in a project file — a granted reference whose rewriter found nothing to edit is neither recorded nor attested. Cargo and golang are confirmed only by their rewriter's own report (`confirmed_cargo_uuids` / `confirmed_golang_uuids`): a golang dep counts only when its go.mod `replace M V => patch.socket.dev/gopatch/ ` and both go.sum lines are in place, never because the patch-server origin or leftover go.sum lines appear somewhere. A golang module that go.mod does not require and go.sum does not list at the patched version is outside the build graph and is refused with `redirect_golang_not_in_module_graph` (nothing written). Only the exact module `patch.socket.dev/gopatch/` is socket-owned; any other module path is refused with `redirect_golang_untrusted_module_path`. A vendored golang module is taken over like cargo and the npm family: its vendor wiring, committed copy and ledger entry are reverted first (`redirect_takeover_reverted_vendored`). Re-runs over already-rewritten output record zero new edits. **Lock (v5.0)**: the hosted engine acquires `<.socket>/apply.lock` around its first wet write (the takeover pre-reverts) — not on `--dry-run`, and not when the run would write nothing (zero redirects, all skipped) — so previews and no-op runs never create `.socket/` (and never quarantine: a `--dry-run` or a zero-grant wet run that finds a malformed `redirect-state.json` reports it as the hard error it is — exit 1, the repair-or-move-aside remedy — but moves nothing; only a run holding the lock moves it aside to `redirect-state.json.corrupt`); contention is `lock_held` and a lock-file I/O fault (a read-only project root, a file squatting on `.socket/`) is `lock_io` — both exit 1, refused BEFORE the redirect ledger is read or written, and rendered like every other lock holder: human `Error (): ` on stderr (+ the `--lock-timeout` hint for a live holder); JSON keeps the hosted shape — top-level `status: "error"`, `errorCode: "lock_held" | "lock_io"`, a string `error`, and `redirect: {mode: "hosted"}` retained (NOT the vendored `error: {code, message}` object). **Takeover symlink pre-check (v5.0)**: a vendored→hosted takeover whose recorded wiring file is a symlink is refused up front with `redirect_symlinked_file_unsupported` — wet and `--dry-run` alike, before any revert — so "nothing was written" holds. **Human mode (v5.0)**: `scan --mode hosted` prints the results table and update detection like the other modes and confirms once — `Redirect N packages to the hosted patch server?` (singular for one), default yes, skipped by `--yes`/`--json`, on `--dry-run` (the engine honors the preview itself; nothing mutates), and when the detail fetch leaves nothing to redirect (that run enters the engine as a no-op — `Redirected 0 packages; rewrote 0 files.`, no lock, no `.socket/` — without prompting); without `--yes` on a non-TTY stdin the shared prompt prints `Non-interactive mode detected, proceeding automatically.` to stderr (unless `--silent`) and proceeds — before rewriting anything (parity with the agent/vendored arms and with `get --mode hosted`). The detail fetch prints the same progress counter and per-package `Warning: could not fetch details for …` lines as the agent arm. An EMPTY hosted discovery prints `No patches available for installed packages.` and exits 0 without entering the engine (previously `Redirected 0 packages; rewrote 0 files.`); a discovery whose every offer is paid-tier for an org without paid access prints the table's paid nudge, then `No downloadable patches (paid subscription required).`, and exits 0 without entering the engine (parity with the agent/vendored arms). A malformed redirect ledger on a human hosted run that returns before the engine (empty discovery, nothing downloadable, a detail-fetch failure, a declined confirm) is surfaced there as the read-only `Warning: the redirect ledger … is malformed …` advisory (muted by `--silent`), never moved; the `--json` arm always enters the engine and hard-errors instead. JSON output gains a `redirect` sub-object: `{ mode: "hosted", redirected, rewrittenFiles, skipped, warnings, dryRun }` (`mode` is additive so consumers can dispatch without inferring it). Rewriter warnings carry stable `redirect_*` codes (e.g. `redirect_npm_no_lockfile`, `redirect_gradle_manual_snippet`, `redirect_golang_unsupported`); new codes are additive (MINOR). v5.0 additive codes: `redirect_composer_no_lockfile` / `redirect_gem_no_gemfile` (composer / gem: neither manifest nor lock present — once per run, after the intake gates), `redirect_maven_no_pom` (no `pom.xml` and no Gradle build), `redirect_nuget_lock_unparseable` (a present-but-corrupt `packages.lock.json` — warned once, nothing mutated; an absent lock still proceeds), `redirect_cargo_lock_pkg_ambiguous` (several same-name+version `[[package]]` blocks and none carries the index `source` — transactional skip). Also v5.0: a registry override of the wrong kind (or none at all) warns the arm's missing-override code for nuget/gem/golang where it used to skip silently, and the ledger's `redirect_nuget_source` edit records `action: "added"` when `nuget.config` was authored from scratch (`rewritten` otherwise). Refusals stay fail-closed with a diagnosis that names the actual cause: a yarn-berry lock entry resolving through a non-`npm:` protocol keeps `redirect_yarn_berry_unsupported_protocol` with the entry's ACTUAL protocol in the detail — except socket-patch's OWN vendored wiring (a `file:` range into `.socket/vendor/`), which gets the distinct `redirect_yarn_berry_vendored_entry` code whose detail names the retirement path (`remove ` per package, or `vendor --revert` which unwinds every vendored package, then re-run `scan --mode hosted`). Both leave the entry byte-identical; neither changes exit code or status. **yarn berry line endings (v5.0)**: yarn writes a NEW `yarn.lock` with the OS line ending (`os.EOL` — CRLF on Windows) and keeps an existing lock's majority ending on every later write, and a `core.autocrlf` checkout turns an LF lock CRLF on any OS — so a uniformly CRLF lock is rewritten in its own ending: every untouched byte (a leading BOM included) round-trips, and the `redirect_yarn_berry_entry` ledger edits record the lock's ON-DISK (CRLF) fragments, which the reverts match byte-exactly. A lock that MIXES CRLF and LF (or holds a bare CR) has no single ending to keep — yarn's own `--immutable` check rejects it too (YN0028) — so it is refused untouched with `redirect_yarn_berry_mixed_line_endings` (the detail names `yarn install`, which normalizes it). This replaces v4's `redirect_yarn_berry_crlf_unsupported`, which refused every CRLF lock and is no longer emitted. A vendored→hosted takeover runs these berry gates (mixed line endings, unsupported `cacheKey`, a non-zero `.yarnrc.yml` `compressionLevel`) BEFORE reverting a vendored berry purl — wet and `--dry-run` alike — so a refused purl keeps its vendored wiring, ledger entry and artifact byte-identical and is skipped with the gate's code (never announced as `redirect_takeover_reverted_vendored` and then left unpatched in both modes). The rewriter reads a fixed set of candidate files from the project root: the npm-family locks (`package-lock.json`, `npm-shrinkwrap.json`, `pnpm-lock.yaml`, `shrinkwrap.yaml`, `yarn.lock`, plus `.yarnrc.yml` for the berry cache-config gate and `bun.lock` / `bun.lockb`), `requirements.txt` / `uv.lock` / `Pipfile.lock` (pipfile-spec 6; see the Pipenv section below) / `poetry.lock` (every Poetry lock generation from 1.0 on — the 0.12 `[metadata.hashes]` layout is refused because that installer ignores URL sources; a Poetry < 1.4 writer additionally gets `redirect_poetry_stale_install_risk`, see `docs/testing/poetry-compatibility.md`) / `pdm.lock` (PDM lock formats `2` and `4.3`–`4.5.1`; the identity-losing `3.1` / `4.0`–`4.2` formats and unknown future formats are refused with `redirect_pdm_refused`, and a lock-format-`2` writer additionally gets `redirect_pdm_legacy_sync_required`, see `docs/testing/pdm-compatibility.md`; when `uv.lock` or `poetry.lock` sits beside it they drive and `pdm.lock` is left alone), `Cargo.toml` / `Cargo.lock` / `.cargo/config.toml` (plus the legacy extensionless `.cargo/config` — cargo reads that spelling in preference when both exist, so the managed `[registries.…]` block is written into whichever one is present), `composer.lock`, `nuget.config` / `packages.lock.json`, `Gemfile` / `Gemfile.lock`, `pom.xml` (+ `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` for maven Trusted Checksums merge, and the Gradle build scripts read only to trigger the manual-snippet warning). **npm-family flavor coverage**: package-lock / npm-shrinkwrap, pnpm (root OR any nested `*/pnpm-lock.yaml`), yarn classic, **yarn berry** (`yarn.lock` entry only — `resolution: ::__archiveUrl=` + `yarnBerry10c0` checksum; cacheKey `10c0` and `.yarnrc.yml compressionLevel 0` gated by `redirect_yarn_berry_cache_unsupported`), and **bun** (text `bun.lock` lockfileVersion 0, 1 or 2 — 0 is the `--save-text-lockfile` opt-in lock of Bun 1.1.39–1.1.45, 1 the 1.2–1.3 default, 2 the 1.4+ default; all three emit one `packages` grammar, so the registry 4-tuple → URL 3-tuple rewrite is version-independent and the lock's own version line is kept. Any other or missing version, or a `packages` section outside bun's single-line grammar, is refused `redirect_bun_lock_unsupported` — the detail is the shared version gate's text (a newer version: update socket-patch, re-locking would reproduce it; no integer: re-lock with Bun ≥ 1.2), identical to the vendored refusal. A version-0 lock holding `workspace:` packages is refused `redirect_bun_workspace_unsupported` (its 2-tuple workspace grammar cannot keep the hosted tuple through a frozen install); the remedy is to delete `bun.lock` and re-run `bun install` with Bun ≥ 1.2, which writes lockfileVersion 1 (accepted). A plain in-place `bun install` bumps the version only when a workspace depends on another workspace (e.g. root → member — the shape the matrix measured); otherwise Bun 1.2.0 keeps version 0 and Bun 1.2.23+ fail to resolve, so the in-place bump is not the documented remedy. Bun lock version, grammar and workspace compatibility are checked before a vendored takeover, including during dry-run: these refusals preserve the existing lock, artifact and vendor ledger. Version-1 and version-2 workspace locks are rewritten, nested versions included. A granted dep with no rewritable entry warns `redirect_bun_entry_not_found`, a grant without a sha512 `redirect_bun_missing_sha512`; a CRLF lock keeps `\r\n` on the rewritten line, and a hosted URL left by an earlier grant of the same `name@version` is re-pinned in place. **Digest-less re-saves (Bun 1.1.39–1.3.9)**: every text-lock Bun below 1.3.10 re-saves a URL tuple WITHOUT its `sha512` whenever the lock is re-saved for another reason (`bun add`, `bun install` after a package.json or workspace change), leaving the 2-tuple `["name@", {meta}]` — the spec Bun installs from is intact. The CLI treats that spelling as its own wiring: a repeat hosted run counts the dep as redirected (no `redirect_bun_entry_not_found`) and HEALS the line back to the 3-tuple with the current `sha512`, recording the heal as a further `redirect_bun_lock_package` edit whose `original` is the 2-tuple (a stale URL is re-pinned from either spelling); `rollback`, scoped `rollback ` / `remove ` and the vendored takeover accept the digest-less spelling of a recorded `new` line (same key, spec and meta, only the trailing `"sha512-…"` missing) and restore the recorded original over it, so the chain always unwinds to the pristine registry line. Anything else — another uuid/token, another version, a re-laid meta object — is still drift. **Native `bun.lockb`**: when no text `bun.lock` exists, binary format versions 1, 2 and 3 are read and rewritten directly. Socket Patch does not invoke Bun or convert the project to a text lockfile. Exact matching package records are rewritten to hosted tarballs with the granted integrity, preserving dependency resolution IDs, workspace/dependency topology and unrelated package metadata; binary pointers and the package metadata hash are updated. Per-package `redirect_bun_lockb_package` snapshots support scoped rollback, repeat runs, superseding grants and hosted ↔ vendored takeover. A regular binary lock is discoverable even with no Bun runtime or `node_modules`; a dry run previews the same binary edits without writing them. A malformed, unreadable, unsupported or unverified binary structure is `redirect_bun_lockb_invalid` (exit 0, `redirected: 0`), and it refuses the npm rewrite before any takeover or sibling npm-family lock mutation. A symlinked binary write target is `redirect_symlinked_file_unsupported` (exit 1, including dry-run). `bun.lock` wins when both spellings exist. Binary-only projects do not receive `redirect_npm_no_lockfile`. Measured boundaries and the real-Bun matrix: `docs/testing/bun-compatibility.md`). **Rush monorepos**: when `rush.json` is present the rewriter also reads `common/config/rush/pnpm-lock.yaml` and each `common/config/subspaces//pnpm-lock.yaml` (sorted for determinism) under their repo-relative keys and repoints them in place; editing them emits `redirect_rush_repo_state_stale` when `common/config/rush/repo-state.json` exists (the `pnpmShrinkwrapHash` desync is refreshed by `rush update`, which the redirect survives). **maven** is fail-closed via version suffixing: a `mavenSuffixedVersion` + `mavenPomSha256` override pins the Socket-only `-socket.` by rewriting the literal `` (`redirect_maven_dep_version`) or adding a `` entry (`redirect_maven_dep_management_added`), plus optional Trusted Checksums (`redirect_maven_trusted_checksums`, conflicts as `redirect_maven_trusted_checksums_conflict`); a `${property}` version is refused (`redirect_maven_dep_unpinned`), a non-matching literal skipped (`redirect_maven_dep_version_mismatch`), and an override without a suffixed version falls back to same-GAV repository injection (`redirect_maven_same_gav_fallback`, NOT fail-closed). @@ -334,7 +334,9 @@ in particular, are behavior changes that gate a version bump when implemented). (v5.0) prunes an emptied `.socket/` (non-recursive `remove_dir` — a `.socket/` still holding a manifest, blobs, vendored state or a user-authored `.gitignore` is kept), so a project that never ran `apply` is back to its pre-setup tree. *(Implemented for the manifest edits — npm - `package.json` and Python deps round-trip byte-for-byte.)* + `package.json` and Python deps round-trip byte-for-byte. `package.json` is re-serialized in its + own layout — BOM, indent, line ending and trailing-newline shape (v5.0) — so a Windows manifest + (yarn berry pretty-prints it with CRLF) keeps CRLF through `setup` and `setup --remove`.)* 9. **Nested workspaces, with exclude.** Setup applies to every subproject below the repo root: npm / yarn / pnpm / bun workspace members are all discovered and configured (pnpm is root-package-only by @@ -689,7 +691,7 @@ to **six flavors**. |---|---|---|---| | npm (package-lock) | deterministic patched tarball `[@scope/]-.tgz` | `package-lock.json` only (`npm-shrinkwrap.json` wins when present): every entry matching name+version gets `resolved: "file:…"` + recomputed `integrity`. `package.json` untouched | `npm ci` (integrity-verified). Plain `npm install` preserves the entry; `npm update ` re-resolves and drops it | | npm / yarn classic | (same tarball) | `yarn.lock` only: matching blocks get `resolved "file:./…#"` + `integrity` (both checksums recomputed; merged-key & `npm:`-alias blocks covered) | `yarn install --frozen-lockfile --offline` (sha1 fragment + sha512 SRI both enforced; byte-stable lock) | -| npm / yarn berry (node-modules linker) | (same tarball) | root `package.json` `resolutions` + `yarn.lock` entry with `checksum: 10c0/` of the berry cache-zip (reproduced from the tarball offline). **PnP is refused** (`.pnp.*` → different artifact pipeline) | `yarn install --immutable --check-cache`, cold cache. Refused if `__metadata.cacheKey ≠ 10c0` or a non-default `compressionLevel` | +| npm / yarn berry (node-modules linker) | (same tarball) | root `package.json` `resolutions` + `yarn.lock` entry with `checksum: 10c0/` of the berry cache-zip (reproduced from the tarball offline). **PnP is refused** (`.pnp.*` → different artifact pipeline) | `yarn install --immutable --check-cache`, cold cache. Refused if `__metadata.cacheKey ≠ 10c0` or a non-default `compressionLevel`. Both files keep their own layout — a CRLF lock (yarn's output on Windows) is spliced in CRLF, `package.json` is re-serialized with its BOM, indent, line ending and trailing-newline shape — so vendor + `--revert` round-trip byte-exactly; a lock or `package.json` MIXING CRLF and LF is refused before any write (`vendor_yarn_berry_mixed_line_endings`) | | npm / pnpm (lockfileVersion 9) | (same tarball) | root `package.json` `pnpm.overrides` (versioned selector) **+** `pnpm-lock.yaml` surgery (overrides / importer version / packages `resolution.integrity` / snapshots) | `pnpm install --frozen-lockfile --offline`, cold store (integrity-verified; byte-stable on pnpm 9 & 10). Other lockfileVersions: 5.4/6.0 route to the legacy backend below; anything else refused | | npm / pnpm LEGACY (lockfileVersion 5.4 = pnpm 7, 6.0 = pnpm 8; flavor `pnpm-legacy`) | (same tarball) | root `package.json` `pnpm.overrides` **+** legacy lock surgery (overrides / root dep + specifiers / packages rekey to a bare `file:` key with recomputed integrity / in-package dep refs). **No `pnpm-workspace.yaml` is written** (pnpm ≤ 8 reads overrides only from package.json). The lock's SPECIFIER is machine-ABSOLUTE — pnpm ≤ 8 absolutizes `file:` overrides itself — surfaced as `vendor_pnpm_legacy_absolute_specifier`. Legacy WORKSPACE locks (`importers:`) refused | same-path `pnpm install --frozen-lockfile --offline`, cold store (byte-stable on pnpm 7.33.5 / 8.15.9). A checkout at a DIFFERENT path fails the frozen check (path-bound specifier) and must run `pnpm install --offline --no-frozen-lockfile` once (the flag matters on CI, where pnpm defaults frozen on), which installs the vendored tarball and re-resolves only the specifier line | | npm / bun (`bun.lock`, lockfileVersion 0, 1 or 2 — `vendor_lockfile_version_unsupported` otherwise) | (same tarball) | `bun.lock` only: the packages entry's registry 4-tuple → local 3-tuple with recomputed `sha512`; the entry's `{deps}` meta, the lock's version line and its line endings are preserved. A lock holding `workspace:` packages is refused `vendor_bun_workspace_unsupported` unless lockfileVersion is 2 — Bun 1.2–1.3 resolve a workspace member's local-tarball path relative to the MEMBER (ENOENT on our root-relative path), 1.4 relative to the lockfile, and a committed version-2 lock is the only proof every consumer runs Bun ≥ 1.4 (a deliberate over-approximation: a package declared only by the workspace root would install on version 1 too). The gate fires only on a run that would WRITE a new local tuple, so in-sync re-runs, `already_vendored` skips and `repair` rebuilds on such a lock pass. The detail names the version and the remedy: delete `bun.lock` and re-lock with Bun ≥ 1.4 (an in-place `bun install` keeps the existing lockfileVersion), or `--mode hosted`. Native binary support is described in the next row. `scan`/`get --mode vendored` apply all four refusals BEFORE downloading (see the `get --mode vendored` bullet). Bun 1.1.39–1.3.9 re-save the local tuple WITHOUT its `sha512` on any later lock re-save (`bun add`, `bun install` after a manifest change); the digest-less 2-tuple is recognised as the same wiring — an in-sync re-run stays `already_vendored` and re-pins the digest on disk (no new wiring record) when the committed artifact still holds the bytes the lock was written from — otherwise, as for any stale tuple of ours, the line is re-pinned and the fresh entry carries the new fingerprint — `repair` rebuilds through it, and `vendor --revert` / `rollback` restore the registry line over it (a 2-tuple at ANOTHER uuid is still `vendor_lock_entry_drifted`) | `bun install --frozen-lockfile`, cold cache (the local tarball's sha512 is enforced by Bun ≥ 1.3.10; 1.1.39–1.3.9 install it unverified — the committed artifact is the protection there) | @@ -898,7 +900,7 @@ Restore the system but keep the local patch state for a later re-apply: manifest ### Hosted unwind coverage -* **Per-purl reverts** exist for **cargo, golang and the npm family** (`redirect_revert_supported`): staged, fail-closed on drift, and honoring `dry_run` (every inverse and drift check resolves like a wet run; nothing flushes and the ledger is untouched). npm purls on projects with bun-lock edits DEFER to the whole-ledger replay (below) whenever it will run — the scope covers every record, and the replay stages the bun group all-or-nothing. A SCOPED unwind (`rollback `, or `remove ` while other hosted records remain) takes the per-purl revert instead: it claims that purl's `redirect_bun_lock_package` edits by the recorded line's spec (`@` registry spec, or a hosted URL whose tarball leaf is `-.tgz`) and replays them like the yarn/pnpm text kinds (whole-line fragments, CRLF-exact); a sibling version's edit is neither claimed nor a refusal; an edit that mentions the package but is not a bun packages-entry line refuses with the unscoped-`rollback` remedy. Pinned by `tests/in_process_vendor_bun_takeover.rs` (`bun_scoped_rollback_of_one_of_two_hosted_records_unwinds_only_that_purl` and the `remove` twin). Native binary `redirect_bun_lockb_package` snapshots follow the same scoped ownership rule and restore only the claimed package records; unrelated binary resolutions stay intact. +* **Per-purl reverts** exist for **cargo, golang and the npm family** (`redirect_revert_supported`): staged, fail-closed on drift, and honoring `dry_run` (every inverse and drift check resolves like a wet run; nothing flushes and the ledger is untouched). npm purls on projects with bun-lock edits DEFER to the whole-ledger replay (below) whenever it will run — the scope covers every record, and the replay stages the bun group all-or-nothing. A SCOPED unwind (`rollback `, or `remove ` while other hosted records remain) takes the per-purl revert instead: it claims that purl's `redirect_bun_lock_package` edits by the recorded line's spec (`@` registry spec, or a hosted URL whose tarball leaf is `-.tgz`) and replays them like the yarn/pnpm text kinds (whole-line fragments, CRLF-exact); a sibling version's edit is neither claimed nor a refusal; an edit that mentions the package but is not a bun packages-entry line refuses with the unscoped-`rollback` remedy. Pinned by `tests/in_process_vendor_bun_takeover.rs` (`bun_scoped_rollback_of_one_of_two_hosted_records_unwinds_only_that_purl` and the `remove` twin). Native binary `redirect_bun_lockb_package` snapshots follow the same scoped ownership rule and restore only the claimed package records; unrelated binary resolutions stay intact. yarn lock blocks (`redirect_yarn_berry_entry` / `redirect_yarn_classic_entry`) are recorded in the lock's on-disk line endings and replayed byte-exactly; when a `core.autocrlf` checkout has since flipped the lock's UNIFORM ending (LF ↔ CRLF — the committed ledger keeps its fragments verbatim), this per-purl revert and the whole-ledger replay below match the recorded blocks respelled in the live ending and restore in that ending (v5.0). A lock with mixed endings proves nothing and still refuses as drift. * **Whole-ledger reverse replay** (`revert_remaining_redirect_edits`, core `patch/redirect/replay.rs`) runs whenever the in-scope hosted record set equals the FULL ledger record set — however the scope was spelled (bare `rollback`, `rollback '**'`, an identifier set covering every record; `remove` reuses the same eligibility rule). It walks every remaining ledger edit in reverse write order through a **per-kind inverse table**, staged and committed **per ecosystem group, all-or-nothing**: one drifted, ambiguous (a fragment appearing more than once), or unhandled edit refuses the whole group byte-untouched while other groups proceed. This covers **gem, golang, pypi, composer, bun**, the yarn/pnpm text kinds (normally claimed by the per-purl npm revert first), and the **non-package rideshare edits** — the pnpm `trustLockfile` auto-config (a pristine created scaffold is deleted; a user-modified one keeps the file and loses only the `trustLockfile: true` line, warned as `redirect_pnpm_trust_scaffold_modified`) — plus a "last one out turns off the lights" pass: when the record map empties but non-package edits remain, they are replayed in the same persist, so the trust edit never strands. The npm `.npmrc` `allow-remote=all` auto-config (`redirect_npmrc_allow_remote`) replays in the `npm` group (a pristine created file is deleted; otherwise only the line is removed, warned as `redirect_npmrc_allow_remote_modified` for a modified created file) and is ALSO claimed by the per-purl npm revert of the last package-lock entry, so a scoped unwind never strands it. * **maven and nuget fail closed**: their structured-metadata kinds (`redirect_maven_repository` / `redirect_maven_dep_management` / `redirect_maven_config` / `redirect_maven_trusted_checksums`, `redirect_nuget_source` / `redirect_nuget_lock`) have no revert implementation, so any such edit refuses its whole group (the maven `` suffix rewrite alone IS invertible, but it rides the same all-or-nothing group). The refusal keeps their records + edits in the ledger and names the remedy: re-run `scan --mode hosted` to normalize, or restore the lockfiles from version control. Unknown future kinds refuse the same way (forward-compat). * **Scoped runs** (paths / identifiers / `--ecosystems`) that do NOT cover the full record set get per-purl reverts only; in-scope hosted purls of ecosystems without one fail closed — `rollback` reports them in `hosted.unsupported` (exit 1), `remove` as the top-level `hosted_revert_unsupported` error — with the remedy "run an unscoped `socket-patch rollback` to unwind ALL hosted redirects, or re-run `scan --mode hosted`". @@ -1218,6 +1220,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `vendor_would_revert_redirect` / `vendor_takeover_reverted_redirect` | `skipped` (advisory event) | vendor / scan / get `--mode vendored` over a hosted-redirected purl (cargo and the npm family, bun included): dry run — the per-purl hosted revert was PROBED and would succeed (for bun, only after the Bun vendored preflight accepted the lock; a refused lock is previewed as the wet run's `failed ` instead) / wet run — the hosted lockfile edits were reverted to their pre-redirect registry values and the redirect-ledger record dropped before vendoring (mode takeover). Fires on the run that takes over, not on re-runs. | | `redirect_revert_failed` | `failed` | vendor / scan / get `--mode vendored` (dry and wet): the per-purl hosted revert refused (drifted lock, missing original fragment, an undecidable ledger edit) — nothing vendored for the purl, hosted wiring left in place, exit 1 `partial_failure`; the detail names the remedy (for bun: an unscoped `socket-patch rollback`). | | `vendor_yarn_berry_cache_unsupported` | `failed` | vendor (yarn berry): lock `cacheKey ≠ 10c0` or non-default `.yarnrc.yml` `compressionLevel` — the cache-zip checksum is not reproducible. | +| `vendor_yarn_berry_mixed_line_endings` | `failed` | vendor (yarn berry): `yarn.lock` or the root `package.json` mixes CRLF and LF line endings (or holds a bare CR) — no single ending can be kept, and yarn rewrites such a file wholesale on its next install (a mixed lock also fails `--immutable`, YN0028). Refused before any write; `yarn install` normalizes the files. A uniformly CRLF pair is vendored in CRLF. A hosted→vendored takeover (`vendor`, `scan`/`get --mode vendored`) raises this — and the berry `vendor_yarn_berry_cache_unsupported` gates — BEFORE reverting the hosted redirect (dry run too), so a refused purl stays hosted. | | `vendor_override_conflict` | `failed` | vendor (pnpm/yarn-berry): a user-authored override/resolution for the package already exists. | | `vendor_integrity_unverified` | `skipped` (warning) | vendor (pipenv): the lockfile format does not hash-check file entries; the committed wheel bytes are the protection. | | `vendor_content_mismatch_overwritten` | `skipped` (warning) | vendor: a staged file matched NEITHER beforeHash nor afterHash (patch built against different bytes, or local edits); the stage was overwritten with the verified patched content and the vendor succeeded. | diff --git a/crates/socket-patch-cli/src/commands/repair_vendor.rs b/crates/socket-patch-cli/src/commands/repair_vendor.rs index ef809f27..c880ff65 100644 --- a/crates/socket-patch-cli/src/commands/repair_vendor.rs +++ b/crates/socket-patch-cli/src/commands/repair_vendor.rs @@ -310,8 +310,14 @@ async fn detect_reference_flavor(project_root: &Path, eco: &str, uuid: &str) -> } if let Some(text) = read("yarn.lock").await { if text.contains(&needle) { - // Same head sniff as core's `sniff_yarn_lock`; berry wins. - let head: Vec<&str> = text.lines().take(30).collect(); + // Same head sniff as core's `sniff_yarn_lock` (BOM skipped, + // CRLF-tolerant); berry wins. + let head: Vec<&str> = text + .strip_prefix('\u{feff}') + .unwrap_or(&text) + .lines() + .take(30) + .collect(); return if head.iter().any(|l| l.starts_with("__metadata:")) { Some("yarn-berry".to_string()) } else if head.iter().any(|l| l.trim() == "# yarn lockfile v1") { diff --git a/crates/socket-patch-cli/src/commands/scan/hosted.rs b/crates/socket-patch-cli/src/commands/scan/hosted.rs index 572d5b47..f9f76674 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted.rs @@ -1414,10 +1414,62 @@ pub(crate) async fn run_redirect_selected( } else { None }; - // A bun-refused npm purl is never dispatched (see the loop), so its - // wiring is not a write target here. - let bun_refused = - |c: &Candidate| bun_takeover_refusal.is_some() && c.purl.starts_with("pkg:npm/"); + // Yarn berry twin of the bun gate: the berry rewriter's project-level + // refusals (mixed line endings, cacheKey, `.yarnrc.yml` + // compressionLevel) must be known before the takeover reverts a + // vendored berry purl — the vendored revert keeps a mixed lock mixed + // (it never refuses on line endings), so reverting first stripped + // the live vendored patch and then the rewriter refused the lock, + // leaving the package unpatched in both modes. Only entries the + // vendor ledger wired through the yarn-berry backend are gated (the + // lock is read only when one exists); an unreadable lock is left to + // the revert's own diagnostics. + let berry_entry = |entry: &socket_patch_core::vendor::VendorEntry| { + entry.ecosystem == "npm" && entry.flavor.as_deref() == Some("yarn-berry") + }; + let berry_takeover_refusal = if takeover + .iter() + .any(|(_, entry)| entry.as_ref().is_some_and(berry_entry)) + { + match socket_patch_core::utils::fs::read_regular_to_string( + &common.cwd.join("yarn.lock"), + ) + .await + { + Ok(lock) => { + let yarnrc = socket_patch_core::utils::fs::read_regular_to_string( + &common.cwd.join(".yarnrc.yml"), + ) + .await + .ok(); + socket_patch_core::patch::redirect::preflight_yarn_berry_hosted( + &lock, + yarnrc.as_deref(), + ) + .err() + } + Err(_) => None, + } + } else { + None + }; + // The takeover refusal (if any) for one candidate: bun gates every + // npm purl, berry only its vendored-berry entries. A refused purl is + // never dispatched (see the loop), so its wiring is not a write + // target here. + let takeover_refusal = + |c: &Candidate, + entry: Option<&socket_patch_core::vendor::VendorEntry>| + -> Option<&socket_patch_core::patch::redirect::RewriteWarning> { + if !c.purl.starts_with("pkg:npm/") { + return None; + } + bun_takeover_refusal.as_ref().or_else(|| { + berry_takeover_refusal + .as_ref() + .filter(|_| entry.is_some_and(berry_entry)) + }) + }; // SYMLINK PRE-CHECK for the takeover reverts — the same rule as the // SYMLINK GUARD below, applied to the files the reverts rewrite // (each ledger entry's recorded wiring): the revert backends stage @@ -1428,7 +1480,11 @@ pub(crate) async fn run_redirect_selected( // (and under --dry-run too) so "nothing was written" stays true. let revert_targets = takeover .iter() - .filter_map(|(c, entry)| entry.as_ref().filter(|_| !bun_refused(c))) + .filter_map(|(c, entry)| { + entry + .as_ref() + .filter(|e| takeover_refusal(c, Some(e)).is_none()) + }) .flat_map(|entry| entry.wiring.iter().map(|w| w.file.as_str())); if let Some(linked) = socket_patch_core::utils::fs::first_symlink(&common.cwd, revert_targets).await @@ -1442,10 +1498,7 @@ pub(crate) async fn run_redirect_selected( let purl = &candidate.purl; let uuid = &candidate.dep.patch_uuid; if let Some(entry) = ledger_entry { - if let Some(warning) = bun_takeover_refusal - .as_ref() - .filter(|_| bun_refused(candidate)) - { + if let Some(warning) = takeover_refusal(candidate, Some(entry)) { refused.push(purl.clone()); if !takeover_pre_warnings .iter() @@ -1581,10 +1634,8 @@ pub(crate) async fn run_redirect_selected( } } for purl in &refused { - if let Some(c) = candidates.iter().find(|c| &c.purl == purl) { - let reason = bun_takeover_refusal - .as_ref() - .filter(|_| bun_refused(c)) + if let Some((c, entry)) = takeover.iter().find(|(c, _)| &c.purl == purl) { + let reason = takeover_refusal(c, entry.as_ref()) .map_or("vendored_revert_failed", |w| w.code.as_str()); skipped.push(serde_json::json!({ "purl": purl, "uuid": c.dep.patch_uuid, "reason": reason, diff --git a/crates/socket-patch-cli/src/commands/vendor.rs b/crates/socket-patch-cli/src/commands/vendor.rs index 1fb00bf3..76f43262 100644 --- a/crates/socket-patch-cli/src/commands/vendor.rs +++ b/crates/socket-patch-cli/src/commands/vendor.rs @@ -1459,6 +1459,16 @@ pub(crate) async fn vendor_records( Err(corrupt) => (None, Some(corrupt)), }; + // Yarn berry takeover preflight (see + // `socket_patch_core::vendor::yarn_berry_vendor_preflight`): the berry + // backend's project-level refusals (mixed line endings in yarn.lock or + // package.json, cacheKey, `.yarnrc.yml` compressionLevel), computed at + // most once per run and only when a hosted-claimed npm purl reaches the + // takeover below — which must refuse such a purl BEFORE reverting its + // hosted edits: a hosted revert keeps a mixed lock mixed, so the backend + // then refused it with the redirect already gone. + let berry_takeover_refusal: tokio::sync::OnceCell> = + tokio::sync::OnceCell::new(); let pipenv_version = tokio::sync::OnceCell::new(); let mut dry_in_sync: u32 = 0; // Sorted, so per-package lines print in the same order every run. @@ -1575,6 +1585,26 @@ pub(crate) async fn vendor_records( .keys() .any(|k| canonical_purl(k) == canonical_purl(candidate)) }); + // The refusal the berry backend would raise after the + // revert, raised HERE instead — the same `failed` event, + // code and detail, in the dry run and the wet run alike — + // so the hosted wiring and redirect ledger stay untouched. + if claimed && candidate.starts_with("pkg:npm/") { + let refusal = berry_takeover_refusal + .get_or_init(|| { + socket_patch_core::vendor::yarn_berry_vendor_preflight(&common.cwd) + }) + .await; + if let Some((code, detail)) = refusal { + has_errors = true; + env.record( + PatchEvent::new(PatchAction::Failed, candidate.clone()) + .with_error(*code, detail.clone()), + ); + report_vendor_failure(common, candidate, detail); + continue; + } + } if claimed && common.dry_run { // Probe the takeover exactly as the wet run would — the // per-purl revert's dry run resolves every inverse and diff --git a/crates/socket-patch-cli/tests/common/bundler_e2e.rs b/crates/socket-patch-cli/tests/common/bundler_e2e.rs index 6f603047..fe1701cf 100644 --- a/crates/socket-patch-cli/tests/common/bundler_e2e.rs +++ b/crates/socket-patch-cli/tests/common/bundler_e2e.rs @@ -2,8 +2,10 @@ //! (`e2e_redirect_gem_build`, `e2e_vendor_gem_build`). //! //! The suites shell out to whatever `bundle` is first on `PATH` — in CI the -//! one `ruby/setup-ruby`'s `bundler:` input installed; locally the host's, -//! or a per-version wrapper dir prepended to `PATH`: +//! one `ruby/setup-ruby`'s `bundler:` input installed, selected by exporting +//! `BUNDLER_VERSION` (without it RubyGems' binstub runs the highest installed +//! bundler, i.e. the Ruby's newer default gem on a leg pinned below it); +//! locally the host's, or a per-version wrapper dir prepended to `PATH`: //! //! ```text //! gem install bundler -v 2.4.22 --install-dir "$D/2.4.22" --no-document @@ -143,7 +145,8 @@ pub fn gate( assert!( want.is_empty() || version_matches(&bundler.version, want), "{suite} ({tag}): {VERSION_ENV}={want} but `bundle` on PATH is {} — the matrix \ - leg is not running the bundler it is named after", + leg is not running the bundler it is named after (a bundler older than the \ + Ruby's default gem is only selected with BUNDLER_VERSION={want})", bundler.version ); } diff --git a/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs b/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs index eac23064..69c1fd40 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_scan_mod.rs @@ -1583,6 +1583,18 @@ mod pty { cmd.env_remove(&key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd.env("SOCKET_TELEMETRY_DISABLED", "1"); cmd.env("SOCKET_NO_UPDATE_CHECK", "1"); for (k, v) in env { diff --git a/crates/socket-patch-cli/tests/covgap_commands_vendor.rs b/crates/socket-patch-cli/tests/covgap_commands_vendor.rs index b837f058..5785b504 100644 --- a/crates/socket-patch-cli/tests/covgap_commands_vendor.rs +++ b/crates/socket-patch-cli/tests/covgap_commands_vendor.rs @@ -177,6 +177,18 @@ fn run_cli(cwd: &Path, args: &[&str], extra_env: &[(&str, &str)]) -> (i32, Strin cmd.env_remove(key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd.env("SOCKET_TELEMETRY_DISABLED", "1"); for (k, v) in extra_env { cmd.env(k, v); diff --git a/crates/socket-patch-cli/tests/e2e_nuget_dotnet_build.rs b/crates/socket-patch-cli/tests/e2e_nuget_dotnet_build.rs index 79701d3a..a86b4ddd 100644 --- a/crates/socket-patch-cli/tests/e2e_nuget_dotnet_build.rs +++ b/crates/socket-patch-cli/tests/e2e_nuget_dotnet_build.rs @@ -91,6 +91,27 @@ fn pinned_version() -> Option { .filter(|v| !v.is_empty()) } +/// Serializes every `dotnet` process this binary spawns. The .NET 9 PAL +/// races when two processes create the same named mutex while the machine's +/// shared-memory root does not exist yet (a fresh CI runner has no +/// `/tmp/.dotnet`): both first-run `dotnet restore`s take NuGet's +/// `NuGet-Migrations` mutex, and the loser dies with `System.IO.IOException: +/// ... 'NuGet-Migrations' ... mkdir("/tmp/.dotnet/shm/session", +/// AllUsers_ReadWriteExecute) == -1; errno == EEXIST` before restoring +/// anything. The hosted and vendored tests run in parallel, so without this +/// the SDK 9 leg failed whichever test lost. (Reproduced in +/// `mcr.microsoft.com/dotnet/sdk:9.0`: two concurrent first-run CLI +/// commands with fresh HOMEs and `/tmp/.dotnet` wiped before each of 60 +/// rounds lost up to 9 of the 120 processes; none with the root pre-created +/// or the commands run one at a time.) Only the SDK phases +/// serialize — the socket-patch runs between them stay parallel. +static DOTNET_SPAWN: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +fn dotnet_output(cmd: &mut Command) -> std::io::Result { + let _one_at_a_time = DOTNET_SPAWN.lock().unwrap_or_else(|p| p.into_inner()); + cmd.output() +} + /// The SDK under test: its `dotnet` muxer, `--version`, and major. struct Dotnet { bin: PathBuf, @@ -127,12 +148,13 @@ impl Dotnet { ) .unwrap(); } - let out = Command::new(&bin) - .arg("--version") - .current_dir(sb.root()) - .env("DOTNET_CLI_TELEMETRY_OPTOUT", "1") - .env("DOTNET_NOLOGO", "1") - .output(); + let out = dotnet_output( + Command::new(&bin) + .arg("--version") + .current_dir(sb.root()) + .env("DOTNET_CLI_TELEMETRY_OPTOUT", "1") + .env("DOTNET_NOLOGO", "1"), + ); let version = match out { Ok(o) if o.status.success() => String::from_utf8_lossy(&o.stdout).trim().to_string(), _ => { @@ -198,7 +220,7 @@ impl Dotnet { // A dotnet-install.sh SDK dir: pin the muxer's own root. cmd.env("DOTNET_ROOT", self.bin.parent().unwrap()); } - cmd.output().expect("spawn dotnet restore") + dotnet_output(&mut cmd).expect("spawn dotnet restore") } fn restore_ok(&self, sb: &Sandbox, cwd: &Path, store: &Path, extra: &[&str], what: &str) { diff --git a/crates/socket-patch-cli/tests/e2e_redirect_yarn_berry_build.rs b/crates/socket-patch-cli/tests/e2e_redirect_yarn_berry_build.rs index 29378e31..b73bf72e 100644 --- a/crates/socket-patch-cli/tests/e2e_redirect_yarn_berry_build.rs +++ b/crates/socket-patch-cli/tests/e2e_redirect_yarn_berry_build.rs @@ -99,7 +99,7 @@ fn has_corepack_pm(pm: &str) -> bool { }; // Isolated too: this probe is what actually downloads the package manager // the first time, and corepack stores it under `COREPACK_HOME`. - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.args([pm, "--version"]) .current_dir(probe.path()) .env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"); @@ -143,12 +143,13 @@ fn scrub_socket_env(cmd: &mut Command) { } fn corepack(cwd: &Path, pm: &str, args: &[&str], extra_env: &[(&str, &str)]) -> Output { - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.arg(pm).args(args).current_dir(cwd); // Scrub FIRST (it removes YARN_* / SOCKET_* from the inherited env), then // set the hermetic flags so they survive. scrub_socket_env(&mut cmd); cache_env::isolate(&mut cmd); + yarn_berry_common::pin_berry_ci_defaults(&mut cmd, pm); cmd.env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0") // Hermetic: no global mirror/cache. Without this, yarn's persistent // `~/.yarn/berry` global cache serves a previously-fetched archive @@ -158,7 +159,7 @@ fn corepack(cwd: &Path, pm: &str, args: &[&str], extra_env: &[(&str, &str)]) -> for (k, v) in extra_env { cmd.env(k, v); } - cmd.output().expect("failed to run corepack") + yarn_berry_common::berry_spawn_output(&mut cmd).expect("failed to run corepack") } fn run_socket(cwd: &Path, args: &[&str]) -> (i32, String, String) { @@ -237,7 +238,7 @@ fn bootstrap_berry_checksum(tmp: &Path, patched_tgz: &Path) -> Option { if !out.status.success() { skip!( "SKIP e2e_redirect_yarn_berry_build: bootstrap yarn install failed:\n{}", - String::from_utf8_lossy(&out.stderr) + yarn_berry_common::yarn_output(&out) ); return None; } @@ -331,10 +332,17 @@ async fn berry_hosted_project( skip!( "SKIP e2e_redirect_yarn_berry_build ({tag}): fixture `yarn install` failed \ (registry unreachable?):\n{}", - String::from_utf8_lossy(&install.stderr) + yarn_berry_common::yarn_output(&install) ); return None; } + // Windows line endings (yarn writes CRLF there): see yarn_berry_common. + yarn_berry_common::adopt_yarn_line_endings( + &proj, + yarn_berry(), + &format!("redirect-{tag}"), + &["package.json", "yarn.lock"], + ); let installed_dir = proj.join("node_modules").join(DEP); let orig = std::fs::read(installed_dir.join("index.js")).expect("installed index.js"); let registry_lock = std::fs::read(proj.join("yarn.lock")).expect("registry yarn.lock"); diff --git a/crates/socket-patch-cli/tests/e2e_vendor_yarn_berry_build.rs b/crates/socket-patch-cli/tests/e2e_vendor_yarn_berry_build.rs index e2ac1318..f847a1e4 100644 --- a/crates/socket-patch-cli/tests/e2e_vendor_yarn_berry_build.rs +++ b/crates/socket-patch-cli/tests/e2e_vendor_yarn_berry_build.rs @@ -93,7 +93,7 @@ fn binary() -> PathBuf { fn has_corepack_pm(pm: &str) -> bool { // Isolated too: this probe is what actually downloads the package manager // the first time, and corepack stores it under `COREPACK_HOME`. - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.args([pm, "--version"]) .env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"); cache_env::isolate(&mut cmd); @@ -105,17 +105,18 @@ fn has_corepack_pm(pm: &str) -> bool { } fn corepack(cwd: &Path, pm: &str, args: &[&str], extra_env: &[(&str, &str)]) -> Output { - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.arg(pm).args(args).current_dir(cwd); // Scrub FIRST (it removes YARN_* / SOCKET_* from the inherited env), then // seed the hermetic flags so they survive (Command: last env call wins). scrub_socket_env(&mut cmd); cache_env::isolate(&mut cmd); + yarn_berry_common::pin_berry_ci_defaults(&mut cmd, pm); cmd.env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"); for (k, v) in extra_env { cmd.env(k, v); } - cmd.output().expect("failed to run corepack") + yarn_berry_common::berry_spawn_output(&mut cmd).expect("failed to run corepack") } /// Remove ambient `SOCKET_*` and `YARN_*` vars (so a developer's settings @@ -329,10 +330,17 @@ async fn run_berry_capstone(driver: VendorDriver) { skip!( "SKIP e2e_vendor_yarn_berry_build: fixture `yarn install` failed (registry \ unreachable?):\n{}", - String::from_utf8_lossy(&install.stderr) + yarn_berry_common::yarn_output(&install) ); return; } + // Windows line endings (yarn writes CRLF there): see yarn_berry_common. + yarn_berry_common::adopt_yarn_line_endings( + &proj, + yarn_berry(), + &format!("vendor-{driver:?}"), + &["package.json", "yarn.lock"], + ); let installed_index = proj.join("node_modules").join(DEP).join("index.js"); let orig = std::fs::read(&installed_index).expect("installed index.js"); diff --git a/crates/socket-patch-cli/tests/e2e_vex_build/deno.rs b/crates/socket-patch-cli/tests/e2e_vex_build/deno.rs index 20c55f7e..e40287c2 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_build/deno.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_build/deno.rs @@ -16,10 +16,11 @@ //! `manifest_not_found`) and makes ZERO patch-API requests, with and //! without `--no-verify`; //! 3. VENDORED: `scan --mode vendored --vendor-source build` likewise -//! commits no wiring Deno would consume, and `vex` again has nothing to -//! attest; -//! 4. AGENT (manifest) mode, unchanged: a staged manifest + blob → -//! `apply --vex` patches the installed file in place and attests it with +//! commits no wiring Deno would consume and (vendored mode being +//! manifest-free) leaves no manifest or ledger record, so `vex` again has +//! nothing to attest; +//! 4. AGENT (manifest) mode, unchanged: a staged manifest + blob attests +//! nothing until applied; `apply --vex` patches the installed file in place and attests it with //! the plain (no `(redirected)` / `(vendored)`) provenance; `deno run` //! now prints `PATCHED` (Deno consumes the patched bytes); the standalone //! `vex` with the manifest attests the same; with the manifest deleted it @@ -448,27 +449,18 @@ fn deno_hosted_and_vendored_never_attest_manifest_mode_unchanged() { "deno {v}: vendored scan edited committed files" ); assert_eq!(std::fs::read(&installed).unwrap(), pristine); - // The vendored scan's download phase left a manifest for the patch it - // could not wire. Nothing applied it, so even WITH that manifest the - // pristine installed tree attests nothing … - assert!(project.join(".socket/manifest.json").is_file()); - let out = run_vex( - &binary(), - &project, - &VexRun { - product: Some(PRODUCT.to_string()), - ..VexRun::offline() - } - .env("DENO_DIR", &deno_dir), - ); - assert_eq!( - out.code, - Some(1), - "deno {v}: unapplied manifest patch: {out}" - ); - assert_absent(out.doc.as_ref(), PURL); - // … and without it there is nothing to attest at all. - strip_manifest(&project); + // Vendored mode is manifest-free: its download phase is detached (the + // vendor ledger alone carries the records), and the refused vendor step + // recorded nothing in that ledger either — so the failed run left no + // record anywhere for `vex` to attest from. + assert_eq!(env["download"]["detached"], true, "deno {v}: {env:#}"); + let ledger = project.join(".socket/vendor/state.json"); + if let Ok(state) = std::fs::read_to_string(&ledger) { + assert!( + !state.contains(NAME), + "deno {v}: a refused vendor step recorded {NAME} in the ledger:\n{state}" + ); + } assert_nothing_to_attest(&project, &deno_dir, &uri, &format!("deno {v} vendored")); let _ = std::fs::remove_dir_all(project.join(".socket")); assert_eq!(deno.run_main(&project, &deno_dir), "PRISTINE"); @@ -498,6 +490,23 @@ fn deno_hosted_and_vendored_never_attest_manifest_mode_unchanged() { .unwrap(), ) .unwrap(); + // Nothing applied the staged manifest yet, so even WITH it the pristine + // installed tree attests nothing. + let unapplied = run_vex( + &binary(), + &project, + &VexRun { + product: Some(PRODUCT.to_string()), + ..VexRun::offline() + } + .env("DENO_DIR", &deno_dir), + ); + assert_eq!( + unapplied.code, + Some(1), + "deno {v}: unapplied manifest patch: {unapplied}" + ); + assert_absent(unapplied.doc.as_ref(), PURL); let quiet = PatchApi::empty(); let apply = VexRun { proxy_url: Some(quiet.uri()), diff --git a/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs b/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs index 52c7e632..095e3bc1 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_build/hatch.rs @@ -345,8 +345,16 @@ fn flow(flavor: Flavor, mode: Mode) { out_text(&out) ); assert_eq!(git_sha256(&bytes), git_sha256(&patched), "{what}"); + // Both sides canonical: a macOS framework CPython (python.org / + // actions/setup-python, Homebrew) realpaths `sys.prefix`, so `six.__file__` + // reads `/private/var/…` while `hatch env find` echoes the `/var/…` + // spelling of the same temp dir. + let (module_real, env_real) = ( + module.canonicalize().unwrap(), + env_dir.canonicalize().unwrap(), + ); assert!( - module.starts_with(&env_dir), + module_real.starts_with(&env_real), "{what}: {module:?} outside {env_dir:?}" ); record( diff --git a/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs b/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs index e477710f..7549b0fe 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs @@ -9,8 +9,9 @@ //! written, the same-run VEX attests from the lock's sha256 pin); vendored //! = `scan --vendor --vendor-source build --vex` over the pristine //! install (the patched wheel is committed under -//! `.socket/vendor/pypi//`, the lock is rewired to it, manifest + -//! ledger are written); +//! `.socket/vendor/pypi//`, the lock is rewired to it, and only the +//! ledger is written — vendored mode is manifest-free, so the ledger +//! entry is detached and embeds the patch record); //! 3. a FRESH checkout of only the committable files (`pyproject.toml`, //! `poetry.lock`, `.socket/`) is installed by the real `poetry install` //! and the imported `six` is proven to be the PATCHED bytes (vendored @@ -924,8 +925,25 @@ fn poetry_vendored_fresh_install_then_manifestless_vex() { lock.contains(&wheel_sha), "the lock pins the committed wheel:\n{lock}" ); - assert!(project.join(".socket/manifest.json").is_file()); - assert!(project.join(".socket/vendor/state.json").is_file()); + // Vendored mode is manifest-free (v5.0, CLI_CONTRACT `scan --vendor`): + // the ledger entry is detached and embeds the patch record, and + // `.socket/manifest.json` is never written. + assert!( + !project.join(".socket/manifest.json").exists(), + "vendored scan wrote a manifest: {env}" + ); + let ledger: Value = serde_json::from_slice( + &std::fs::read(project.join(".socket/vendor/state.json")).expect("vendor ledger written"), + ) + .unwrap(); + let entry = ledger["entries"] + .as_object() + .into_iter() + .flat_map(|m| m.values()) + .find(|e| e["uuid"] == VENDORED_UUID) + .unwrap_or_else(|| panic!("no ledger entry for {VENDORED_UUID}: {ledger:#}")); + assert_eq!(entry["detached"], true, "{ledger:#}"); + assert_eq!(entry["record"]["uuid"], VENDORED_UUID, "{ledger:#}"); let doc: Value = serde_json::from_slice(&std::fs::read(&embedded).unwrap()).unwrap(); assert_attested( &doc, diff --git a/crates/socket-patch-cli/tests/e2e_vex_lockfile/yarn_berry.rs b/crates/socket-patch-cli/tests/e2e_vex_lockfile/yarn_berry.rs index 5cab1abb..918335ba 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_lockfile/yarn_berry.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_lockfile/yarn_berry.rs @@ -18,6 +18,7 @@ //! | scoped | `@scope/pkg` hosted `__archiveUrl` + vendored `@scope/pkg-.tgz` | both attest `pkg:npm/%40scope/pkg@v` (events: `@scope`) from the API record; offline `record_unavailable`, zero requests; pristine install `not_applied` | //! | npm alias | `"lp@npm:left-pad@1.3.0"` installed at `node_modules/lp` | the REAL package is attested after hashing the alias dir; tampered `hash_mismatch`; stale pristine `not_applied` (regression: the alias dir used to be "not installed", so the lock pin attested it) | //! | multi-descriptor + CRLF + BOM | one block for `^1.0.0, ^1.3.0` | attests from the `10c0` pin | +//! | vendored CRLF + BOM | lock AND root `package.json` CRLF + BOM (yarn's Windows output) | the committed artifact attests; offline `record_unavailable`; `resolutions` reverted: nothing discovered | //! | two uuids for one package | two live blocks wiring one purl to different patches | never attested (`wiring_conflict`) | //! | user `yarn patch` on top | `patch:` entry wrapping the hosted locator | the hosted base entry still attests when the user patch leaves the Socket file intact; a user patch rewriting it is `hash_mismatch` | //! | foreign / look-alike hosts, uuid-shaped token only | `__archiveUrl` not on the patch host, or whose LAST uuid is the grant token of another patch | nothing discovered (exit 2), zero requests | @@ -660,6 +661,72 @@ fn multi_descriptor_key_crlf_and_bom() { attested(&out, LP, Marker::Redirected, "multi-descriptor CRLF BOM"); } +/// The VENDORED pair on the Windows shapes: yarn berry writes a new +/// `yarn.lock` AND the root `package.json` it first pretty-prints with +/// `os.EOL` (CRLF), and editors add a BOM. The CRLF + BOM lock entry and the +/// CRLF + BOM `resolutions` mapping are both read, so the committed artifact +/// attests (offline without a ledger: `record_unavailable`); with the +/// mapping reverted the `file:` entry is orphaned and nothing is wired. +#[test] +fn vendored_crlf_and_bom_lock_and_manifest() { + let windows = |text: &str| format!("\u{feff}{}", text.replace('\n', "\r\n")); + let manifest = |resolutions: Option| { + let mut doc = serde_json::json!({ + "name": "app", + "version": "1.0.0", + "dependencies": { "left-pad": "1.3.0" } + }); + if let Some(r) = resolutions { + doc["resolutions"] = r; + } + windows(&format!( + "{}\n", + serde_json::to_string_pretty(&doc).unwrap() + )) + }; + let tmp = tempfile::tempdir().unwrap(); + let cwd = tmp.path(); + let rel = vendored_rel(UUID, "left-pad-1.3.0.tgz"); + write_tgz(cwd, &rel, PATCHED); + put( + cwd, + ".yarnrc.yml", + b"nodeLinker: node-modules\r\nenableGlobalCache: false\r\n", + ); + put( + cwd, + "package.json", + manifest(Some( + serde_json::json!({ "left-pad": format!("file:./{rel}") }), + )) + .as_bytes(), + ); + put( + cwd, + "yarn.lock", + windows(&format!( + "{}{}{}", + header(), + workspace_block("app", ".", &[("left-pad", "npm:1.3.0")]), + vendored_block("left-pad", "1.3.0", &rel, ROOT_LOCATOR) + )) + .as_bytes(), + ); + let api = api_for(LP); + let out = run_vex(&binary(), cwd, &VexRun::online(&api)); + attested(&out, LP, Marker::Vendored, "vendored CRLF BOM"); + let before = api.request_count(); + let out = run_vex(&binary(), cwd, &VexRun::offline()); + omitted(&out, LP, "record_unavailable", "vendored CRLF BOM offline"); + assert_eq!(api.request_count(), before, "offline: zero requests"); + + // The `resolutions` mapping reverted (still CRLF + BOM): the lock's + // `file:` entry is orphaned — yarn would not install it. + put(cwd, "package.json", manifest(None).as_bytes()); + let out = run_vex(&binary(), cwd, &VexRun::online(&api)); + nothing_discovered(&out, "vendored CRLF BOM, resolutions reverted"); +} + /// Two LIVE blocks wiring the same package version to two different /// patches: which one yarn installs is not the lock's to say, so neither is /// attested. diff --git a/crates/socket-patch-cli/tests/e2e_yarn4_pnpm_linker_build.rs b/crates/socket-patch-cli/tests/e2e_yarn4_pnpm_linker_build.rs index 2062951b..fa165f10 100644 --- a/crates/socket-patch-cli/tests/e2e_yarn4_pnpm_linker_build.rs +++ b/crates/socket-patch-cli/tests/e2e_yarn4_pnpm_linker_build.rs @@ -95,7 +95,7 @@ fn has_corepack_pm(pm: &str) -> bool { let Ok(probe) = tempfile::tempdir() else { return false; }; - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.args([pm, "--version"]) .current_dir(probe.path()) .env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"); @@ -136,17 +136,18 @@ fn scrub_socket_env(cmd: &mut Command) { } fn corepack(cwd: &Path, pm: &str, args: &[&str], extra_env: &[(&str, &str)]) -> Output { - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.arg(pm).args(args).current_dir(cwd); // Scrub FIRST, then the hermetic flags so they survive (last env wins). scrub_socket_env(&mut cmd); cache_env::isolate(&mut cmd); + yarn_berry_common::pin_berry_ci_defaults(&mut cmd, pm); cmd.env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0") .env("YARN_ENABLE_GLOBAL_CACHE", "false"); for (k, v) in extra_env { cmd.env(k, v); } - cmd.output().expect("failed to run corepack") + yarn_berry_common::berry_spawn_output(&mut cmd).expect("failed to run corepack") } fn run_socket(cwd: &Path, args: &[&str]) -> (i32, String, String) { @@ -308,7 +309,7 @@ fn bootstrap_berry_checksum(tmp: &Path, patched_tgz: &Path) -> Option { if !out.status.success() { skip!( "SKIP e2e_yarn4_pnpm_linker_build: bootstrap yarn install failed:\n{}", - String::from_utf8_lossy(&out.stderr) + yarn_berry_common::yarn_output(&out) ); return None; } @@ -347,10 +348,17 @@ fn install_pnpm_fixture(tag: &str, tmp: &Path, proj: &Path) -> Option> { skip!( "SKIP e2e_yarn4_pnpm_linker_build ({tag}): fixture `yarn install` failed \ (registry unreachable?):\n{}", - String::from_utf8_lossy(&install.stderr) + yarn_berry_common::yarn_output(&install) ); return None; } + // Windows line endings (yarn writes CRLF there): see yarn_berry_common. + yarn_berry_common::adopt_yarn_line_endings( + proj, + yarn_berry(), + &format!("pnpm-linker-{tag}"), + &["package.json", "yarn.lock"], + ); assert_pnpm_store_layout(proj, tag); // Read THROUGH the symlink — the same path discovery crawls. let orig = std::fs::read(proj.join("node_modules").join(DEP).join("index.js")) diff --git a/crates/socket-patch-cli/tests/e2e_yarn4_workspaces_build.rs b/crates/socket-patch-cli/tests/e2e_yarn4_workspaces_build.rs index 786c1f21..f87be21e 100644 --- a/crates/socket-patch-cli/tests/e2e_yarn4_workspaces_build.rs +++ b/crates/socket-patch-cli/tests/e2e_yarn4_workspaces_build.rs @@ -95,7 +95,7 @@ fn has_corepack_pm(pm: &str) -> bool { let Ok(probe) = tempfile::tempdir() else { return false; }; - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.args([pm, "--version"]) .current_dir(probe.path()) .env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"); @@ -134,17 +134,18 @@ fn scrub_socket_env(cmd: &mut Command) { } fn corepack(cwd: &Path, pm: &str, args: &[&str], extra_env: &[(&str, &str)]) -> Output { - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.arg(pm).args(args).current_dir(cwd); // Scrub FIRST, then the hermetic flags so they survive (last env wins). scrub_socket_env(&mut cmd); cache_env::isolate(&mut cmd); + yarn_berry_common::pin_berry_ci_defaults(&mut cmd, pm); cmd.env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0") .env("YARN_ENABLE_GLOBAL_CACHE", "false"); for (k, v) in extra_env { cmd.env(k, v); } - cmd.output().expect("failed to run corepack") + yarn_berry_common::berry_spawn_output(&mut cmd).expect("failed to run corepack") } fn run_socket(cwd: &Path, args: &[&str]) -> (i32, String, String) { @@ -306,7 +307,7 @@ fn bootstrap_berry_checksum(tmp: &Path, patched_tgz: &Path) -> Option { if !out.status.success() { skip!( "SKIP e2e_yarn4_workspaces_build: bootstrap yarn install failed:\n{}", - String::from_utf8_lossy(&out.stderr) + yarn_berry_common::yarn_output(&out) ); return None; } @@ -338,10 +339,17 @@ fn install_workspace_fixture(tag: &str, tmp: &Path, proj: &Path) -> Option bool { let Ok(probe) = tempfile::tempdir() else { return false; }; - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.args([pm, "--version"]) .current_dir(probe.path()) .env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"); @@ -127,18 +127,19 @@ fn scrub_socket_env(cmd: &mut Command) { } fn corepack(cwd: &Path, pm: &str, args: &[&str], extra_env: &[(&str, &str)]) -> Output { - let mut cmd = Command::new("corepack"); + let mut cmd = yarn_berry_common::corepack_command(); cmd.arg(pm).args(args).current_dir(cwd); // Scrub FIRST (it removes YARN_* / SOCKET_* from the inherited env), then // set the hermetic flags so they survive (Command: last env call wins). scrub_socket_env(&mut cmd); cache_env::isolate(&mut cmd); + yarn_berry_common::pin_berry_ci_defaults(&mut cmd, pm); cmd.env("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0") .env("YARN_ENABLE_GLOBAL_CACHE", "false"); for (k, v) in extra_env { cmd.env(k, v); } - cmd.output().expect("failed to run corepack") + yarn_berry_common::berry_spawn_output(&mut cmd).expect("failed to run corepack") } fn run_socket(cwd: &Path, args: &[&str]) -> (i32, String, String) { @@ -280,10 +281,18 @@ async fn refusal_case(tag: &str, yarn_pm: &str, compression_zero: bool, expected skip!( "SKIP e2e_yarn_legacy_cachekey_refusal_build ({tag}): fixture `yarn install` \ failed (registry unreachable?):\n{}", - String::from_utf8_lossy(&install.stderr) + yarn_berry_common::yarn_output(&install) ); return; } + // Windows line endings (yarn 2/3 write CRLF there too): a CRLF legacy + // lock must still be refused for its cacheKey, never for its endings. + yarn_berry_common::adopt_yarn_line_endings( + &proj, + yarn_pm, + &format!("legacy-{tag}"), + &["package.json", "yarn.lock"], + ); // 2. The cacheKey pin — the empirical fact the refusal contract below is // conditioned on. Fails FIRST, with a self-describing message, if a diff --git a/crates/socket-patch-cli/tests/in_process_redirect.rs b/crates/socket-patch-cli/tests/in_process_redirect.rs index 6da34950..7338c7d4 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect.rs @@ -680,6 +680,13 @@ async fn mock_reference_with_berry(server: &MockServer) { /// Write a project whose only lockfile is a yarn-berry `yarn.lock` resolving /// `@npm:` (spike B3 shape, cacheKey 10c0). fn write_berry_project(root: &Path) { + write_berry_project_spelled(root, str::to_string); +} + +/// [`write_berry_project`] with the lock's bytes passed through `spell` — +/// the Windows shapes (yarn writes a new lockfile with CRLF there; a +/// `core.autocrlf` checkout does the same on any OS; editors add a BOM). +fn write_berry_project_spelled(root: &Path, spell: impl Fn(&str) -> String) { std::fs::write( root.join("package.json"), format!( @@ -696,7 +703,7 @@ fn write_berry_project(root: &Path) { .unwrap(); std::fs::write( root.join("yarn.lock"), - format!( + spell(&format!( "# This file is generated by running \"yarn install\" inside your project.\n\ # Manual changes might be lost - proceed with caution!\n\n\ __metadata:\n version: 8\n cacheKey: 10c0\n\n\ @@ -707,7 +714,7 @@ fn write_berry_project(root: &Path) { resolution: \"consumer@workspace:.\"\n dependencies:\n \ {NAME}: \"npm:^{VERSION}\"\n languageName: unknown\n linkType: soft\n", "3".repeat(128) - ), + )), ) .unwrap(); } @@ -767,6 +774,139 @@ async fn scan_redirect_rewrites_yarn_berry_lock() { ); } +/// The berry leg on the Windows lock shapes: yarn berry writes a NEW +/// `yarn.lock` with `os.EOL` (CRLF on Windows), a `core.autocrlf` checkout +/// produces the same on any OS, and editors add a BOM. The hosted chain +/// must redirect the dep (never `redirected: 0` with a line-ending +/// refusal), keep every line CRLF and the BOM, record the lock's on-disk +/// CRLF fragments in the ledger, stay a no-op on re-run, and `rollback` +/// must restore the pristine lock byte-for-byte. +#[tokio::test] +#[serial] +async fn scan_redirect_rewrites_crlf_and_bom_yarn_berry_locks_and_rollback_restores_them() { + let server = MockServer::start().await; + mock_discovery(&server).await; + mock_reference_with_berry(&server).await; + mock_view(&server).await; + let encoded = socket_patch_core::utils::uri::encode_uri_component(HOSTED_URL); + + for (label, bom) in [("crlf", ""), ("bom+crlf", "\u{feff}")] { + let tmp = tempfile::tempdir().unwrap(); + write_berry_project_spelled(tmp.path(), |t| format!("{bom}{}", t.replace('\n', "\r\n"))); + let lock_path = tmp.path().join("yarn.lock"); + let pristine = std::fs::read(&lock_path).unwrap(); + + let env = run_redirect_subprocess(tmp.path(), &server.uri()); + assert_eq!(env["redirect"]["redirected"], 1, "{label}: {env:#}"); + assert!( + warning_codes(&env).is_empty(), + "{label}: no line-ending refusal (or any other warning): {env:#}" + ); + let lock = std::fs::read_to_string(&lock_path).unwrap(); + assert!( + lock.contains(&format!("::__archiveUrl={encoded}\"\r\n")) + && lock.contains(&format!(" checksum: {BERRY_CHECKSUM}\r\n")), + "{label}: the entry is redirected in CRLF: {lock:?}" + ); + assert_eq!( + lock.matches('\n').count(), + lock.matches("\r\n").count(), + "{label}: every line keeps CRLF" + ); + assert_eq!(lock.starts_with('\u{feff}'), !bom.is_empty(), "{label}"); + + let ledger = read_ledger(tmp.path()); + let edit = ledger["edits"] + .as_array() + .unwrap() + .iter() + .find(|e| e["kind"] == "redirect_yarn_berry_entry") + .unwrap_or_else(|| panic!("{label}: a berry ledger edit: {ledger:#}")); + for side in ["original", "new"] { + let fragment = edit[side].as_str().unwrap(); + assert!( + fragment.contains("\r\n") && !fragment.replace("\r\n", "").contains('\n'), + "{label}: the ledger's {side} is the on-disk CRLF fragment: {fragment:?}" + ); + assert!( + String::from_utf8_lossy(if side == "original" { + &pristine + } else { + lock.as_bytes() + }) + .contains(fragment), + "{label}: {side} is a verbatim slice of the file" + ); + } + + // Re-run: in sync, byte-stable, no new ledger edit. + let env = run_redirect_subprocess(tmp.path(), &server.uri()); + assert_eq!(env["redirect"]["redirected"], 1, "{label}: {env:#}"); + assert_eq!( + std::fs::read_to_string(&lock_path).unwrap(), + lock, + "{label}" + ); + assert_eq!( + read_ledger(tmp.path())["edits"].as_array().unwrap().len(), + ledger["edits"].as_array().unwrap().len(), + "{label}: a re-run appends nothing" + ); + + let (code, env) = rollback_json(tmp.path()); + assert_eq!(code, Some(0), "{label}: rollback: {env:#}"); + assert_eq!( + std::fs::read(&lock_path).unwrap(), + pristine, + "{label}: rollback restores the pristine CRLF lock byte-for-byte" + ); + assert!( + !tmp.path() + .join(".socket/vendor/redirect-state.json") + .exists(), + "{label}: the emptied ledger is removed" + ); + } +} + +/// A berry lock whose line endings are MIXED (CRLF and LF, or a bare CR) +/// cannot be kept in one style — and yarn itself rejects it under +/// `--immutable` — so the hosted run refuses it untouched with a code that +/// names the line endings and the `yarn install` remedy, redirecting +/// nothing and writing no ledger. +#[tokio::test] +#[serial] +async fn scan_redirect_refuses_a_mixed_line_ending_yarn_berry_lock() { + let server = MockServer::start().await; + mock_discovery(&server).await; + mock_reference_with_berry(&server).await; + mock_view(&server).await; + + let tmp = tempfile::tempdir().unwrap(); + write_berry_project_spelled(tmp.path(), |t| { + t.replace('\n', "\r\n") + .replacen("proceed with caution!\r\n", "proceed with caution!\n", 1) + }); + let lock_path = tmp.path().join("yarn.lock"); + let before = std::fs::read(&lock_path).unwrap(); + + let env = run_redirect_subprocess(tmp.path(), &server.uri()); + assert_eq!(env["redirect"]["redirected"], 0, "{env:#}"); + assert!( + warning_codes(&env).contains(&"redirect_yarn_berry_mixed_line_endings".to_string()), + "{env:#}" + ); + let detail = redirect_warning_detail(&env, "redirect_yarn_berry_mixed_line_endings"); + assert!(detail.contains("yarn install"), "remedy named: {detail}"); + assert_eq!(std::fs::read(&lock_path).unwrap(), before, "untouched"); + assert!( + !tmp.path() + .join(".socket/vendor/redirect-state.json") + .exists(), + "no ledger for a refused rewrite" + ); +} + /// Classic (v1) yarn.lock with CRLF line endings (Windows `core.autocrlf` /// checkout): the full hosted chain must repoint the TARGET entry — not /// whichever entry sorts first — and keep every untouched line CRLF @@ -1393,6 +1533,18 @@ fn scrubbed_cli() -> std::process::Command { cmd.env_remove(&key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd } diff --git a/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs b/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs index 63b767d6..06990734 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs @@ -763,6 +763,18 @@ fn scrubbed_cli() -> std::process::Command { cmd.env_remove(&key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd } diff --git a/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs b/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs index 8053eeab..c8dc1fff 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs @@ -382,6 +382,18 @@ async fn scan_output(root: &Path, server: &MockServer, extra: &[&str]) -> std::p cmd.env_remove(key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd.env("SOCKET_TELEMETRY_DISABLED", "1") .args(["scan", "--mode", "hosted", "--yes", "--cwd"]) .arg(root) diff --git a/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs b/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs index 55688250..dcb3f171 100644 --- a/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs +++ b/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs @@ -131,6 +131,18 @@ fn scrubbed_cli() -> std::process::Command { cmd.env_remove(&key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd } diff --git a/crates/socket-patch-cli/tests/in_process_rollback_vendored.rs b/crates/socket-patch-cli/tests/in_process_rollback_vendored.rs index 1e95a855..10e14d1d 100644 --- a/crates/socket-patch-cli/tests/in_process_rollback_vendored.rs +++ b/crates/socket-patch-cli/tests/in_process_rollback_vendored.rs @@ -229,6 +229,18 @@ fn run_cli(cwd: &Path, args: &[&str]) -> (i32, String, String) { cmd.env_remove(key); } } + // In-process tests in this binary `std::env::set_var` these via + // `apply_env_toggles`; one set by a parallel test between the scan + // above and the spawn would be inherited, so remove them + // unconditionally (see in_process_vendor.rs `run_cli`). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd.env("SOCKET_TELEMETRY_DISABLED", "1"); let out = cmd.output().expect("spawn socket-patch binary"); ( diff --git a/crates/socket-patch-cli/tests/in_process_vendor.rs b/crates/socket-patch-cli/tests/in_process_vendor.rs index 3baaa836..54ddc426 100644 --- a/crates/socket-patch-cli/tests/in_process_vendor.rs +++ b/crates/socket-patch-cli/tests/in_process_vendor.rs @@ -230,6 +230,21 @@ fn run_cli(cwd: &Path, args: &[&str], extra_env: &[(&str, &str)]) -> (i32, Strin cmd.env_remove(key); } } + // The in-process tests in this binary run `apply_env_toggles`, which + // `std::env::set_var`s these on the shared test process. A toggle set + // by a parallel test after the scan above but before the spawn would + // be inherited, so remove them unconditionally: `Command` applies the + // removals to the environment captured at spawn time. Seen on + // test-release: a hosted scan here inherited SOCKET_OFFLINE=1 and + // refused to run ("cannot run with --offline/SOCKET_OFFLINE"). + for key in [ + "SOCKET_OFFLINE", + "SOCKET_DEBUG", + "SOCKET_API_URL", + "SOCKET_PROXY_URL", + ] { + cmd.env_remove(key); + } cmd.env("SOCKET_TELEMETRY_DISABLED", "1"); for (k, v) in extra_env { cmd.env(k, v); @@ -823,6 +838,593 @@ async fn revendor_new_uuid_carries_original_forward_yarn_berry() { assert!(!root.join(".socket/vendor").exists(), "vendor tree pruned"); } +// ───────────────────────────────────────────────────────────────────── +// 8e. yarn berry on the Windows file shapes: CRLF, BOM, mixed endings +// ───────────────────────────────────────────────────────────────────── + +/// The berry root manifest the way yarn pretty-prints it (LF form). +const BERRY_WIN_PKG: &str = r#"{ + "name": "berry-win", + "version": "1.0.0", + "private": true, + "dependencies": { + "left-pad": "1.3.0" + } +} +"#; + +/// The matching yarn 4 lock (LF form). +fn berry_win_lock() -> String { + format!( + "# This file is generated by running \"yarn install\" inside your project.\n\ + # Manual changes might be lost - proceed with caution!\n\n\ + __metadata:\n version: 8\n cacheKey: 10c0\n\n\ + \"berry-win@workspace:.\":\n version: 0.0.0-use.local\n \ + resolution: \"berry-win@workspace:.\"\n dependencies:\n \ + left-pad: \"npm:1.3.0\"\n languageName: unknown\n linkType: soft\n\n\ + \"left-pad@npm:1.3.0\":\n version: 1.3.0\n \ + resolution: \"left-pad@npm:1.3.0\"\n checksum: 10c0/{}\n \ + languageName: node\n linkType: hard\n", + "3".repeat(128) + ) +} + +/// `text` in the Windows shape yarn berry writes (`os.EOL` = CRLF for a +/// file it creates or first pretty-prints), optionally BOM'd by an editor. +fn windows_shape(text: &str, bom: bool) -> String { + format!( + "{}{}", + if bom { "\u{feff}" } else { "" }, + text.replace('\n', "\r\n") + ) +} + +/// A berry project (root manifest + lock as given, `.yarnrc.yml`, the +/// installed copy) plus the offline manifest + blob `vendor` reads. +fn stage_berry_project(root: &Path, pkg: &str, lock: &str) { + let installed = root.join("node_modules/left-pad"); + std::fs::create_dir_all(&installed).unwrap(); + std::fs::write( + installed.join("package.json"), + br#"{"name":"left-pad","version":"1.3.0"}"#, + ) + .unwrap(); + std::fs::write(installed.join("index.js"), ORIG_INDEX).unwrap(); + std::fs::write(root.join("package.json"), pkg).unwrap(); + std::fs::write(root.join("yarn.lock"), lock).unwrap(); + std::fs::write( + root.join(".yarnrc.yml"), + "nodeLinker: node-modules\r\nenableGlobalCache: false\r\n", + ) + .unwrap(); + let before_hash = compute_git_sha256_from_bytes(ORIG_INDEX); + let after_hash = compute_git_sha256_from_bytes(PATCHED_INDEX); + let manifest = json!({ "patches": { PURL: patch_record(&before_hash, &after_hash) } }); + std::fs::create_dir_all(root.join(".socket/blobs")).unwrap(); + std::fs::write( + root.join(".socket/manifest.json"), + serde_json::to_vec_pretty(&manifest).unwrap(), + ) + .unwrap(); + std::fs::write(root.join(".socket/blobs").join(&after_hash), PATCHED_INDEX).unwrap(); +} + +/// yarn berry writes BOTH files CRLF on Windows (a new lock and the +/// manifest it first pretty-prints get `os.EOL`). Vendoring must keep each +/// file's layout — CRLF, BOM, trailing newline — so the wired pair is what +/// yarn itself would have written; the re-run is byte-stable; and +/// `vendor --revert` restores both files byte-for-byte (the Windows CI +/// failure this pins: the revert wrote package.json back LF). +#[tokio::test] +async fn berry_crlf_and_bom_vendor_and_revert_round_trip_byte_exact() { + for bom in [false, true] { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + let (pkg, lock) = ( + windows_shape(BERRY_WIN_PKG, bom), + windows_shape(&berry_win_lock(), bom), + ); + stage_berry_project(root, &pkg, &lock); + + let (code, env) = vendor_cli(root, &[]); + assert_eq!(code, 0, "bom={bom}: vendor: {env:#}"); + assert_eq!(env["summary"]["applied"], 1, "bom={bom}: {env:#}"); + for rel in ["package.json", "yarn.lock"] { + let text = std::fs::read_to_string(root.join(rel)).unwrap(); + assert_eq!( + text.matches('\n').count(), + text.matches("\r\n").count(), + "bom={bom}: {rel} keeps CRLF: {text:?}" + ); + assert_eq!(text.starts_with('\u{feff}'), bom, "bom={bom}: {rel} BOM"); + assert!( + text.contains(".socket/vendor/npm/"), + "bom={bom}: {rel} wired" + ); + } + let wired: Vec> = ["package.json", "yarn.lock"] + .iter() + .map(|rel| std::fs::read(root.join(rel)).unwrap()) + .collect(); + + let (code, env) = vendor_cli(root, &[]); + assert_eq!(code, 0, "bom={bom}: re-vendor: {env:#}"); + for (rel, bytes) in ["package.json", "yarn.lock"].iter().zip(&wired) { + assert_eq!( + &std::fs::read(root.join(rel)).unwrap(), + bytes, + "bom={bom}: re-vendor leaves {rel} byte-identical" + ); + } + + let (code, env) = vendor_cli(root, &["--revert"]); + assert_eq!(code, 0, "bom={bom}: revert: {env:#}"); + assert_eq!( + std::fs::read_to_string(root.join("package.json")).unwrap(), + pkg, + "bom={bom}: package.json restored byte-for-byte" + ); + assert_eq!( + std::fs::read_to_string(root.join("yarn.lock")).unwrap(), + lock, + "bom={bom}: yarn.lock restored byte-for-byte" + ); + assert!(!root.join(".socket/vendor").exists(), "bom={bom}: pruned"); + } +} + +/// A berry lock or root manifest whose line endings MIX CRLF and LF has no +/// single style to keep (and yarn itself rejects such a lock under +/// `--immutable`): the package fails with the dedicated code before any +/// write, both files byte-identical and no vendor tree created. +#[tokio::test] +async fn berry_mixed_line_endings_fail_closed_with_code() { + let crlf_lock = windows_shape(&berry_win_lock(), false); + let crlf_pkg = windows_shape(BERRY_WIN_PKG, false); + let mixed = |crlf: &str| crlf.replacen("\r\n", "\n", 1); + for (label, pkg, lock) in [ + ("mixed lock", crlf_pkg.clone(), mixed(&crlf_lock)), + ("mixed package.json", mixed(&crlf_pkg), crlf_lock.clone()), + ] { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + stage_berry_project(root, &pkg, &lock); + let (code, env) = vendor_cli(root, &[]); + assert_eq!(code, 1, "{label}: {env:#}"); + let failed = find_event(&env, "failed", Some("vendor_yarn_berry_mixed_line_endings")); + assert!( + failed.to_string().contains("yarn install"), + "{label}: the remedy is named: {failed}" + ); + assert_eq!( + std::fs::read_to_string(root.join("package.json")).unwrap(), + pkg, + "{label}" + ); + assert_eq!( + std::fs::read_to_string(root.join("yarn.lock")).unwrap(), + lock, + "{label}" + ); + assert!(!root.join(".socket/vendor").exists(), "{label}"); + } +} + +/// Mount the hosted-mode API (batch discovery, per-package search, a granted +/// reference carrying the yarn-berry-zip checksum, the patch view) for the +/// berry takeover legs. Returns the hosted tarball URL. +async fn mount_berry_hosted_api(server: &wiremock::MockServer) -> String { + use wiremock::matchers::{method, path, path_regex}; + use wiremock::{Mock, ResponseTemplate}; + let org = "test-org"; + let hosted_url = format!( + "{}/patch/npm/left-pad/1.3.0/44444444-4444-4444-8444-444444444444/{UUID}/left-pad-1.3.0.tgz", + server.uri() + ); + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{org}/patches/batch"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "packages": [{ "purl": PURL, "patches": [{ + "uuid": UUID, "purl": PURL, "tier": "free", "cveIds": [], "ghsaIds": [], + "severity": "high", "title": "berry takeover fixture" + }]}], + "canAccessPaidPatches": false, + }))) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{org}/patches/by-package/.+$" + ))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "patches": [{ + "uuid": UUID, "purl": PURL, "publishedAt": "2026-01-01T00:00:00Z", + "description": "x", "license": "MIT", "tier": "free", "vulnerabilities": {} + }], + "canAccessPaidPatches": false, + }))) + .mount(server) + .await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{org}/patches/package"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "results": { UUID: { + "status": "granted", "url": hosted_url, "purl": PURL, + "artifacts": [ + { "kind": "tarball", "url": hosted_url, + "integrity": { "sha512": "sha512-unused-by-berry==" } }, + { "kind": "yarn-berry-zip", "url": hosted_url, + "integrity": { "yarnBerry10c0": format!("10c0/{}", "7".repeat(128)) } } + ], + "registryOverride": null + }} + }))) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{org}/patches/view/{UUID}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "uuid": UUID, "purl": PURL, "publishedAt": "2026-01-01T00:00:00Z", + "files": { "package/index.js": { + "beforeHash": compute_git_sha256_from_bytes(ORIG_INDEX), + "afterHash": compute_git_sha256_from_bytes(PATCHED_INDEX), + }}, + "vulnerabilities": {}, + "description": "x", "license": "MIT", "tier": "free" + }))) + .mount(server) + .await; + hosted_url +} + +/// `scan --mode hosted --json --yes` through the binary. +fn hosted_scan_cli(root: &Path, api_url: &str) -> (i32, Value) { + let (code, stdout, stderr) = run_cli( + root, + &[ + "scan", + "--mode", + "hosted", + "--json", + "--yes", + "--cwd", + root.to_str().unwrap(), + "--api-url", + api_url, + "--org", + "test-org", + "--api-token", + "fake-token", + ], + &[], + ); + let env: Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|e| { + panic!( + "scan --mode hosted --json must emit JSON: {e}\nstdout:\n{stdout}\nstderr:\n{stderr}" + ) + }); + (code, env) +} + +/// Mode takeovers on the Windows shapes, both directions, hermetic. Hosted → +/// vendored: `vendor` reverts the CRLF hosted redirect (the ledger's CRLF +/// fragments) before wiring, and `vendor --revert` then lands on the +/// pristine CRLF + BOM pair. Vendored → hosted: `scan --mode hosted` +/// reverts the vendored pair first (package.json back byte-exact, BOM and +/// CRLF included), redirects the CRLF lock, and `rollback` restores the +/// pristine lock. +#[tokio::test] +async fn berry_crlf_takeovers_round_trip_both_directions() { + let server = wiremock::MockServer::start().await; + let hosted_url = mount_berry_hosted_api(&server).await; + let encoded = socket_patch_core::utils::uri::encode_uri_component(&hosted_url); + let (pkg, lock) = ( + windows_shape(BERRY_WIN_PKG, true), + windows_shape(&berry_win_lock(), false), + ); + let assert_crlf = |root: &Path, ctx: &str| { + for rel in ["package.json", "yarn.lock"] { + let text = std::fs::read_to_string(root.join(rel)).unwrap(); + assert_eq!( + text.matches('\n').count(), + text.matches("\r\n").count(), + "{ctx}: {rel} keeps CRLF" + ); + } + }; + + // ── hosted → vendored ── + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + stage_berry_project(root, &pkg, &lock); + let (code, env) = hosted_scan_cli(root, &server.uri()); + assert_eq!(code, 0, "hosted scan: {env:#}"); + assert_eq!(env["redirect"]["redirected"], 1, "{env:#}"); + let hosted_lock = std::fs::read_to_string(root.join("yarn.lock")).unwrap(); + assert!(hosted_lock.contains(&encoded), "{hosted_lock:?}"); + assert_crlf(root, "hosted"); + + let (code, env) = vendor_cli(root, &[]); + assert_eq!(code, 0, "vendor over the hosted redirect: {env:#}"); + assert_eq!(env["summary"]["applied"], 1, "{env:#}"); + assert!( + env.to_string() + .contains("vendor_takeover_reverted_redirect"), + "the takeover is surfaced: {env:#}" + ); + assert!( + !root.join(".socket/vendor/redirect-state.json").exists(), + "the superseded redirect ledger is dropped" + ); + let vendored_lock = std::fs::read_to_string(root.join("yarn.lock")).unwrap(); + assert!( + !vendored_lock.contains("__archiveUrl") && vendored_lock.contains(".socket/vendor/npm/"), + "fully vendored: {vendored_lock:?}" + ); + assert_crlf(root, "hosted→vendored"); + let (code, env) = vendor_cli(root, &["--revert"]); + assert_eq!(code, 0, "revert: {env:#}"); + assert_eq!( + std::fs::read_to_string(root.join("yarn.lock")).unwrap(), + lock + ); + assert_eq!( + std::fs::read_to_string(root.join("package.json")).unwrap(), + pkg + ); + + // ── vendored → hosted ── + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + stage_berry_project(root, &pkg, &lock); + let (code, env) = vendor_cli(root, &[]); + assert_eq!(code, 0, "vendor: {env:#}"); + let (code, env) = hosted_scan_cli(root, &server.uri()); + assert_eq!(code, 0, "hosted scan over the vendored pair: {env:#}"); + assert_eq!(env["redirect"]["redirected"], 1, "{env:#}"); + assert!( + env.to_string() + .contains("redirect_takeover_reverted_vendored"), + "the takeover is surfaced: {env:#}" + ); + assert_eq!( + std::fs::read_to_string(root.join("package.json")).unwrap(), + pkg, + "the vendored resolutions entry is reverted byte-exactly (BOM + CRLF kept)" + ); + let hosted_lock = std::fs::read_to_string(root.join("yarn.lock")).unwrap(); + assert!( + hosted_lock.contains(&encoded) && !hosted_lock.contains(".socket/vendor/"), + "fully hosted: {hosted_lock:?}" + ); + assert_crlf(root, "vendored→hosted"); + let (code, stdout, stderr) = run_cli( + root, + &[ + "rollback", + "--json", + "--yes", + "--offline", + "--cwd", + root.to_str().unwrap(), + ], + &[], + ); + assert_eq!(code, 0, "rollback: {stdout}\n{stderr}"); + assert_eq!( + std::fs::read_to_string(root.join("yarn.lock")).unwrap(), + lock, + "rollback restores the pristine CRLF lock" + ); + assert_eq!( + std::fs::read_to_string(root.join("package.json")).unwrap(), + pkg + ); +} + +/// `scan --mode hosted --json --yes ` through the binary. +fn hosted_scan_cli_with(root: &Path, api_url: &str, extra: &[&str]) -> (i32, Value) { + let mut args = vec![ + "scan", + "--mode", + "hosted", + "--json", + "--yes", + "--cwd", + root.to_str().unwrap(), + "--api-url", + api_url, + "--org", + "test-org", + "--api-token", + "fake-token", + ]; + args.extend_from_slice(extra); + let (code, stdout, stderr) = run_cli(root, &args, &[]); + let env: Value = serde_json::from_str(stdout.trim()).unwrap_or_else(|e| { + panic!( + "scan --mode hosted --json must emit JSON: {e}\nstdout:\n{stdout}\nstderr:\n{stderr}" + ) + }); + (code, env) +} + +/// The mode wiring a takeover would touch: the berry pair, `.yarnrc.yml`, +/// and everything under `.socket/vendor/` (vendor ledger, artifact, marker, +/// redirect ledger), as `(relative path, bytes)`. +fn berry_wiring_snapshot(root: &Path) -> std::collections::BTreeMap> { + fn walk(root: &Path, dir: &Path, out: &mut std::collections::BTreeMap>) { + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + walk(root, &path, out); + } else { + let rel = path + .strip_prefix(root) + .unwrap() + .to_string_lossy() + .into_owned(); + out.insert(rel, std::fs::read(&path).unwrap()); + } + } + } + let mut out = std::collections::BTreeMap::new(); + for rel in ["package.json", "yarn.lock", ".yarnrc.yml"] { + out.insert(rel.to_string(), std::fs::read(root.join(rel)).unwrap()); + } + walk(root, &root.join(".socket/vendor"), &mut out); + out +} + +/// Mode takeovers must refuse a berry project the NEW mode would refuse +/// BEFORE reverting the OLD mode's wiring. The reverts keep line endings as +/// they are (a mixed lock stays mixed), while the forward hosted rewriter +/// and vendored backend refuse a mixed file (and an unsupported +/// `.yarnrc.yml` compressionLevel) — so reverting first left the package +/// unpatched in BOTH modes: `scan --mode hosted` reported +/// `redirect_takeover_reverted_vendored` ("now fully hosted") then +/// `redirected: 0`; `vendor` reported `vendor_takeover_reverted_redirect` +/// then failed. Each leg (wet and --dry-run) asserts the old mode's wiring +/// stays byte-identical, the refusal carries the new mode's code, and no +/// takeover is announced. +#[tokio::test] +async fn berry_takeovers_refuse_before_reverting_the_old_mode() { + let server = wiremock::MockServer::start().await; + mount_berry_hosted_api(&server).await; + let (pkg, lock) = ( + windows_shape(BERRY_WIN_PKG, true), + windows_shape(&berry_win_lock(), false), + ); + // Each breakage lands AFTER the old mode is wired. `mix`: an editor + // saves one header line of `rel` with LF. `compression`: the project + // opts into a compressionLevel neither mode can reproduce. + type Break = fn(&Path, &str); + let mix: Break = |root, rel| { + let text = std::fs::read_to_string(root.join(rel)).unwrap(); + std::fs::write(root.join(rel), text.replacen("\r\n", "\n", 1)).unwrap(); + }; + let compression: Break = |root, _| { + std::fs::write( + root.join(".yarnrc.yml"), + "nodeLinker: node-modules\r\nenableGlobalCache: false\r\ncompressionLevel: 9\r\n", + ) + .unwrap(); + }; + + // ── vendored → hosted ── + for (label, breakage, rel, code) in [ + ( + "mixed lock", + mix, + "yarn.lock", + "redirect_yarn_berry_mixed_line_endings", + ), + ( + "compressionLevel", + compression, + "", + "redirect_yarn_berry_cache_unsupported", + ), + ] { + for dry in [true, false] { + let ctx = format!("vendored→hosted {label} dry={dry}"); + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + stage_berry_project(root, &pkg, &lock); + let (exit, env) = vendor_cli(root, &[]); + assert_eq!(exit, 0, "{ctx}: vendor: {env:#}"); + breakage(root, rel); + let before = berry_wiring_snapshot(root); + let extra: &[&str] = if dry { &["--dry-run"] } else { &[] }; + let (_, env) = hosted_scan_cli_with(root, &server.uri(), extra); + let text = env.to_string(); + assert!(text.contains(code), "{ctx}: refused with {code}: {env:#}"); + for announced in [ + "redirect_takeover_reverted_vendored", + "redirect_would_revert_vendored", + ] { + assert!( + !text.contains(announced), + "{ctx}: no takeover ({announced}): {env:#}" + ); + } + assert_eq!(env["redirect"]["redirected"], 0, "{ctx}: {env:#}"); + let skipped = env["redirect"]["skipped"] + .as_array() + .cloned() + .unwrap_or_default(); + assert!( + skipped + .iter() + .any(|s| s["purl"] == PURL && s["reason"] == code), + "{ctx}: the purl is skipped with the refusal's code: {env:#}" + ); + assert_eq!( + berry_wiring_snapshot(root), + before, + "{ctx}: the vendored wiring, ledger and artifact stay byte-identical" + ); + } + } + + // ── hosted → vendored ── + for (label, breakage, rel, code) in [ + ( + "mixed lock", + mix, + "yarn.lock", + "vendor_yarn_berry_mixed_line_endings", + ), + ( + "mixed package.json", + mix, + "package.json", + "vendor_yarn_berry_mixed_line_endings", + ), + ( + "compressionLevel", + compression, + "", + "vendor_yarn_berry_cache_unsupported", + ), + ] { + for dry in [true, false] { + let ctx = format!("hosted→vendored {label} dry={dry}"); + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + stage_berry_project(root, &pkg, &lock); + let (exit, env) = hosted_scan_cli_with(root, &server.uri(), &[]); + assert_eq!(exit, 0, "{ctx}: hosted scan: {env:#}"); + assert_eq!(env["redirect"]["redirected"], 1, "{ctx}: {env:#}"); + breakage(root, rel); + let before = berry_wiring_snapshot(root); + let extra: &[&str] = if dry { &["--dry-run"] } else { &[] }; + let (exit, env) = vendor_cli(root, extra); + assert_eq!(exit, 1, "{ctx}: the refusal fails the run: {env:#}"); + let failed = find_event(&env, "failed", Some(code)); + assert_eq!(failed["purl"], PURL, "{ctx}: {failed}"); + let text = env.to_string(); + for announced in [ + "vendor_takeover_reverted_redirect", + "vendor_would_revert_redirect", + ] { + assert!( + !text.contains(announced), + "{ctx}: no takeover ({announced}): {env:#}" + ); + } + assert_eq!( + berry_wiring_snapshot(root), + before, + "{ctx}: the hosted lock edits and redirect ledger stay byte-identical" + ); + } + } +} + // ───────────────────────────────────────────────────────────────────── // 9. offline with no local source // ───────────────────────────────────────────────────────────────────── diff --git a/crates/socket-patch-cli/tests/maven_build_common/mod.rs b/crates/socket-patch-cli/tests/maven_build_common/mod.rs index e69d79f0..c583e8b3 100644 --- a/crates/socket-patch-cli/tests/maven_build_common/mod.rs +++ b/crates/socket-patch-cli/tests/maven_build_common/mod.rs @@ -17,7 +17,9 @@ //! Every Maven run is hermetic: a per-test local repository //! (`-Dmaven.repo.local`), a per-test user `settings.xml` (`-s`, so the //! developer's `~/.m2/settings.xml` mirrors/proxies never apply), batch -//! mode, and `MAVEN_ARGS` / `MAVEN_OPTS` / `MAVEN_CONFIG` scrubbed. The +//! mode, `MAVEN_ARGS` / `MAVEN_OPTS` / `MAVEN_CONFIG` scrubbed, and the CI +//! markers Maven 4 sniffs ([`CI_DETECTOR_ENV`]) removed, so a leg logs +//! exactly what a developer's terminal run logs. The //! dependency plugin is pinned so every Maven line resolves with the same //! plugin (the default bound version differs per Maven release). @@ -32,6 +34,43 @@ pub const MVN_ENV: &str = "SOCKET_PATCH_MAVEN_E2E_MVN"; pub const VERSION_ENV: &str = "SOCKET_PATCH_MAVEN_E2E_VERSION"; pub const REQUIRED_ENV: &str = "SOCKET_PATCH_MAVEN_E2E_REQUIRED"; +/// What Maven 4's `CIDetector`s key on (4.0.0-rc-6 `cisupport`: generic +/// `CI`, GitHub `GITHUB_ACTIONS`, CircleCI, Jenkins `WORKSPACE`, TeamCity, +/// Travis). When one is set, Maven 4 swaps in the `QuietMavenTransferListener` +/// even under `-B` — no "Downloading from …" lines and, crucially, no +/// "Checksum validation failed" warning for a rejected `checksumPolicy=fail` +/// download, which is the evidence the vendored TAMPER probe asserts. On a +/// GitHub runner every Maven 4 leg would otherwise log differently from the +/// same run on a laptop. Maven 3 reads none of these, so scrubbing them is a +/// no-op there. (`--force-interactive` also disables the detection, but it +/// flips the run interactive and Maven 3 rejects the flag.) +pub const CI_DETECTOR_ENV: &[&str] = &[ + "CI", + "GITHUB_ACTIONS", + "CIRCLECI", + "WORKSPACE", + "TEAMCITY_VERSION", + "TRAVIS", +]; + +/// `mvn` with the ambient Maven configuration and CI markers scrubbed. +fn mvn_command(program: &OsString) -> Command { + let mut cmd = Command::new(program); + for key in [ + "MAVEN_ARGS", + "MAVEN_OPTS", + "MAVEN_CONFIG", + "M2_HOME", + "MAVEN_REPO_LOCAL", + ] + .into_iter() + .chain(CI_DETECTOR_ENV.iter().copied()) + { + cmd.env_remove(key); + } + cmd +} + /// Pinned so 3.6 → 4.x all run the same goal implementation (3.6.1 still /// supports Maven 3.2.5+, so the oldest line in the matrix can load it). pub const DEPENDENCY_PLUGIN: &str = "org.apache.maven.plugins:maven-dependency-plugin:3.6.1"; @@ -83,12 +122,7 @@ impl Mvn { let program: OsString = std::env::var_os(MVN_ENV) .filter(|v| !v.is_empty()) .unwrap_or_else(|| if cfg!(windows) { "mvn.cmd" } else { "mvn" }.into()); - let out = match Command::new(&program) - .args(["-v", "-B"]) - .env_remove("MAVEN_ARGS") - .env_remove("MAVEN_OPTS") - .output() - { + let out = match mvn_command(&program).args(["-v", "-B"]).output() { Ok(out) => out, Err(e) => { skip( @@ -157,8 +191,8 @@ impl Mvn { /// `mvn -B ` in `cwd` against the local repository `m2`, with the /// user settings file `settings`. pub fn run(&self, cwd: &Path, m2: &Path, settings: &Path, args: &[&str]) -> Output { - let mut cmd = Command::new(&self.program); - cmd.current_dir(cwd) + mvn_command(&self.program) + .current_dir(cwd) .arg("-B") .arg("-s") .arg(settings) @@ -166,12 +200,8 @@ impl Mvn { .arg("-Dstyle.color=never") .arg("-Dmaven.test.skip=true") .args(args) - .env_remove("MAVEN_ARGS") - .env_remove("MAVEN_OPTS") - .env_remove("MAVEN_CONFIG") - .env_remove("M2_HOME") - .env_remove("MAVEN_REPO_LOCAL"); - cmd.output().expect("spawn mvn") + .output() + .expect("spawn mvn") } /// Resolve the project's dependencies into `/` (the diff --git a/crates/socket-patch-cli/tests/mode_migration_npm.rs b/crates/socket-patch-cli/tests/mode_migration_npm.rs index ba20fb58..ecb3e6bc 100644 --- a/crates/socket-patch-cli/tests/mode_migration_npm.rs +++ b/crates/socket-patch-cli/tests/mode_migration_npm.rs @@ -464,6 +464,16 @@ fn stage_yarn_fixture(tag: &str, pm: &str, berry: bool) -> Option { ); return None; } + if berry { + // Windows line endings (yarn writes CRLF there; `EOL_ENV=crlf` + // reproduces it here): the takeovers must round-trip CRLF files. + yarn_berry_common::adopt_yarn_line_endings( + &proj, + pm, + &format!("mode-migration-{tag}"), + &["package.json", "yarn.lock"], + ); + } let orig = std::fs::read(proj.join("node_modules").join(DEP).join("index.js")) .expect("installed index.js"); assert!( diff --git a/crates/socket-patch-cli/tests/redirect_npm_allow_remote.rs b/crates/socket-patch-cli/tests/redirect_npm_allow_remote.rs index b77547f4..4697de93 100644 --- a/crates/socket-patch-cli/tests/redirect_npm_allow_remote.rs +++ b/crates/socket-patch-cli/tests/redirect_npm_allow_remote.rs @@ -714,9 +714,15 @@ async fn outer_npm_config_layers_are_respected() { &[("npm_config_allow_remote", "none")], ); assert_eq!(code, 0, "{stderr}"); + // The warning names the variable as the OS reports it. Windows env names + // are case-insensitive, and `scan_hosted_env`'s blanking of + // `NPM_CONFIG_ALLOW_REMOTE` makes the child see that spelling there, so + // match the name case-insensitively. assert!( stderr.contains(&format!("Warning ({CODE}): ")) - && stderr.contains("npm_config_allow_remote=none") + && stderr + .to_ascii_lowercase() + .contains("npm_config_allow_remote=none") && stderr.contains("would not take effect"), "{stderr}" ); diff --git a/crates/socket-patch-cli/tests/yarn_berry_common/mod.rs b/crates/socket-patch-cli/tests/yarn_berry_common/mod.rs index 91ed43f7..b9f5e7cf 100644 --- a/crates/socket-patch-cli/tests/yarn_berry_common/mod.rs +++ b/crates/socket-patch-cli/tests/yarn_berry_common/mod.rs @@ -29,6 +29,20 @@ //! unreachable" soft-skip into a failure, for a CI leg that provisioned //! corepack on purpose. //! +//! # Line endings ([`EOL_ENV`]) +//! +//! yarn berry writes a file it CREATES with the OS line ending (`os.EOL`) +//! and keeps an existing file's majority ending on every later write +//! (`normalizeLineEndings` in yarnpkg-fslib's `FakeFS.ts`, used by +//! `Project.persistLockfile` and `Workspace.persistManifest`). On Windows +//! every fixture here therefore runs on CRLF files: the first `yarn install` +//! writes a CRLF `yarn.lock` and re-renders the compact fixture +//! `package.json` pretty — with CRLF. [`EOL_ENV`]`=crlf` reproduces that on +//! macOS / Linux: [`adopt_yarn_line_endings`] re-spells the files the +//! fixture install wrote CRLF, and every later yarn run keeps them CRLF. +//! Either way it prints one `BERRY-EOL||||yarn=|flow=` +//! line per file: the ending yarn itself wrote, and the one the flow runs on. +//! //! # The manifest-less VEX matrix ([`run_manifestless_vex_matrix`]) //! //! A hosted (`scan --mode hosted`) or vendored (`vendor`, `get --mode @@ -78,6 +92,70 @@ pub fn yarn_e2e_required() -> bool { std::env::var(REQUIRED_ENV).is_ok_and(|v| v == "1") } +/// `=crlf`: run the real-yarn flows on CRLF files on every OS, the way +/// yarn berry writes them on Windows (module docs, "Line endings"). +pub const EOL_ENV: &str = "SOCKET_PATCH_YARN_BERRY_EOL"; + +/// Whether the flows run on CRLF files: always on Windows (yarn writes them +/// that way there), elsewhere when [`EOL_ENV`] is `crlf`. +pub fn windows_line_endings() -> bool { + cfg!(windows) || std::env::var(EOL_ENV).is_ok_and(|v| v.eq_ignore_ascii_case("crlf")) +} + +/// The line-ending style of `bytes`, for the `BERRY-EOL` report. +fn eol_style(bytes: &[u8]) -> &'static str { + let crlf = bytes.windows(2).filter(|w| w == b"\r\n").count(); + let lf = bytes.iter().filter(|&&b| b == b'\n').count() - crlf; + match (crlf, lf) { + (0, 0) => "none", + (_, 0) => "crlf", + (0, _) => "lf", + _ => "mixed", + } +} + +/// Call right after a fixture's first `yarn install`: under +/// [`windows_line_endings`], re-spell each of `files` (relative to `dir`) +/// CRLF, as yarn itself writes them on Windows, and report each file as a +/// `BERRY-EOL||||yarn=|flow=` line (the +/// ending yarn wrote, the one the flow runs on). yarn keeps the majority +/// ending on every later write, +/// so the rest of the flow (socket-patch's rewrites, the fresh-checkout +/// installs, the VEX matrix) runs on CRLF files. On Windows the files are +/// CRLF already and the re-spelling is a no-op. Returns whether it +/// converted anything. +pub fn adopt_yarn_line_endings(dir: &Path, yarn_spec: &str, flow: &str, files: &[&str]) -> bool { + let crlf = windows_line_endings(); + let mut converted = false; + for rel in files { + let path = dir.join(rel); + let bytes = std::fs::read(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display())); + let written = eol_style(&bytes); + let mut flow_style = written; + if crlf { + let text = String::from_utf8(bytes).expect("yarn writes UTF-8"); + let respelled = text.replace("\r\n", "\n").replace('\n', "\r\n"); + if respelled != text { + std::fs::write(&path, &respelled).unwrap(); + converted = true; + } + flow_style = eol_style(respelled.as_bytes()); + } + println!("BERRY-EOL|{yarn_spec}|{flow}|{rel}|yarn={written}|flow={flow_style}"); + } + let _ = std::io::stdout().flush(); + converted +} + +/// [`eol_style`] classifies each file by its line breaks. +#[test] +fn eol_style_classifies_line_breaks() { + assert_eq!(eol_style(b"{}"), "none"); + assert_eq!(eol_style(b"a\nb\n"), "lf"); + assert_eq!(eol_style(b"a\r\nb\r\n"), "crlf"); + assert_eq!(eol_style(b"a\r\nb\n"), "mixed"); +} + /// The corepack spec (`yarn@`) of the yarn 4 release under test. /// /// Panics on a non-4.x [`VERSION_ENV`]: these suites prove the SUPPORTED @@ -109,6 +187,141 @@ pub fn yarn_berry() -> &'static str { .as_str() } +/// A `Command` for `corepack`. On Windows Node installs corepack as the +/// `corepack.cmd` batch shim, and `Command::new("corepack")` resolves only +/// `corepack.exe`, so every berry suite's availability probe reported +/// "`corepack yarn@4.12.0` unavailable" on the windows-latest yarn-berry leg +/// although corepack was on PATH. +pub fn corepack_command() -> std::process::Command { + std::process::Command::new(if cfg!(windows) { + "corepack.cmd" + } else { + "corepack" + }) +} + +/// Serializes yarn berry spawns on Windows. The suites' parallel tests share +/// one yarn cache folder (`cache_env::isolate` points `YARN_CACHE_FOLDER` at +/// a single per-run root), and two yarn processes fetching the same package +/// both write `.zip-.tmp` then rename it over the cache zip. On +/// Windows the loser's rename fails with `EPERM` while the winner holds the +/// file (seen on the windows-latest yarn-berry 4.12.0 leg: +/// `EPERM: operation not permitted, rename '...left-pad-npm-1.3.0-....zip-....tmp'`). +/// Unix rename-over is atomic, so other platforms stay parallel. Only the +/// yarn processes serialize; the socket-patch runs between them do not. +static BERRY_SPAWN: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +/// Run a yarn/corepack `Command`, one at a time on Windows (see +/// [`BERRY_SPAWN`]). +pub fn berry_spawn_output(cmd: &mut std::process::Command) -> std::io::Result { + let _one_at_a_time = + cfg!(windows).then(|| BERRY_SPAWN.lock().unwrap_or_else(|p| p.into_inner())); + cmd.output() +} + +/// Both output streams of a finished yarn run, for a failure message. yarn +/// berry reports its errors (YN0028, YN0018, …) on stdout and usually writes +/// nothing to stderr, so a stderr-only message hides the reason. +pub fn yarn_output(out: &Output) -> String { + format!( + "stdout:\n{}\nstderr:\n{}", + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) +} + +/// Pin the yarn berry defaults that depend on whether yarn thinks it is +/// running under CI. Apply it after the `YARN_*` scrub and `cache_env::isolate`, +/// and before the call site's own env. +/// +/// yarn 3+ turns `enableImmutableInstalls` on by default when it detects CI +/// (ci-info: `CI`, `GITHUB_ACTIONS`, …). Under that default, the plain +/// `yarn install` that creates each fixture's lockfile fails with YN0028 ("The +/// lockfile would have been created by this install, which is explicitly +/// forbidden"), exit 1 and nothing on stderr. That broke every hosted, +/// vendored, pnpm-linker, workspaces and yarn 3 refusal fixture on the +/// ubuntu/macOS yarn-berry legs. yarn 2 keeps the default off, which is why +/// its refusal legs stayed green. The pin: +/// +/// * forces `CI=true`, so a developer's local run gets the same defaults as +/// the CI leg. Without the second pin, every suite fails locally too, +/// instead of only on a runner; +/// * sets `YARN_ENABLE_IMMUTABLE_INSTALLS=false`, so a plain `install` may +/// write the lock. The fresh-checkout installs are unaffected because they +/// pass `--immutable` explicitly, and yarn's flag outranks the setting; +/// * sets `YARN_ENABLE_HARDENED_MODE=false` for yarn 4 (`yarn_spec`'s +/// major). yarn 4 turns hardened mode on when it detects a GitHub Actions +/// run for a public pull request, and then re-resolves every lock entry +/// against the registry. The fresh-checkout installs point the registry at +/// an unreachable address on purpose (they must install from the committed +/// lock and the hosted tarball alone), so hardened mode failed them with +/// ECONNREFUSED 127.0.0.1:1 on PR runs only. Hardened mode guards against +/// untrusted lockfiles; these suites exercise socket-patch's own rewrites, +/// and `--immutable --check-cache` still verifies every checksum. yarn 2 +/// and 3 predate the setting and refuse every command while it is set +/// ("Usage Error: Unrecognized or legacy configuration settings found: +/// enableHardenedMode"), so for them it is removed instead. +pub fn pin_berry_ci_defaults<'c>( + cmd: &'c mut std::process::Command, + yarn_spec: &str, +) -> &'c mut std::process::Command { + cmd.env("CI", "true") + .env("YARN_ENABLE_IMMUTABLE_INSTALLS", "false"); + if yarn_major(yarn_spec).is_none_or(|major| major >= 4) { + cmd.env("YARN_ENABLE_HARDENED_MODE", "false") + } else { + cmd.env_remove("YARN_ENABLE_HARDENED_MODE") + } +} + +/// The major version of a corepack yarn spec (`yarn@3.8.7` → 3). +fn yarn_major(yarn_spec: &str) -> Option { + yarn_spec + .strip_prefix("yarn@") + .unwrap_or(yarn_spec) + .split('.') + .next()? + .parse() + .ok() +} + +/// [`pin_berry_ci_defaults`] wins over an earlier value for every variable +/// (`get_envs` reports the last value set for each key), and only yarn 4+ +/// gets the hardened-mode pin — yarn 2/3 reject the unknown setting. +#[test] +fn pin_berry_ci_defaults_sets_ci_and_disables_implicit_immutable() { + for (spec, hardened) in [ + ("yarn@4.12.0", Some(Some("false".into()))), + ("yarn@4.0.2", Some(Some("false".into()))), + ("yarn@3.8.7", Some(None)), + ("yarn@2.4.3", Some(None)), + ] { + let mut cmd = corepack_command(); + cmd.env("YARN_ENABLE_IMMUTABLE_INSTALLS", "true") + .env("YARN_ENABLE_HARDENED_MODE", "true"); + pin_berry_ci_defaults(&mut cmd, spec); + let envs: std::collections::HashMap<_, _> = cmd + .get_envs() + .map(|(k, v)| (k.to_os_string(), v.map(|v| v.to_os_string()))) + .collect(); + assert_eq!( + envs.get(std::ffi::OsStr::new("CI")), + Some(&Some("true".into())), + "{spec}" + ); + assert_eq!( + envs.get(std::ffi::OsStr::new("YARN_ENABLE_IMMUTABLE_INSTALLS")), + Some(&Some("false".into())), + "{spec}" + ); + assert_eq!( + envs.get(std::ffi::OsStr::new("YARN_ENABLE_HARDENED_MODE")), + hardened.as_ref(), + "{spec}: removed (None) for yarn 2/3, pinned off for yarn 4" + ); + } +} + /// The cache-zip checksum a real yarn wrote into `lock` (the first entry /// `checksum:`), normalized to the prefixed `10c0/` the patch API's /// `yarnBerry10c0` carries. yarn 4.0.x writes the BARE hex under cacheKey @@ -140,7 +353,7 @@ pub fn expected_checksum_line(yarn_lock: &str, checksum_10c0: &str) -> String { } } -/// Run `f` on a fresh OS thread and return its result (re-raising a panic)./// Run `f` on a fresh OS thread and return its result (re-raising a panic). +/// Run `f` on a fresh OS thread and return its result (re-raising a panic). /// /// [`PatchApi`] owns its own tokio runtime; creating, blocking on or /// dropping one from inside a `#[tokio::test]` body panics ("Cannot start diff --git a/crates/socket-patch-core/src/package_json/detect.rs b/crates/socket-patch-core/src/package_json/detect.rs index 2aa5fb65..74f9f679 100644 --- a/crates/socket-patch-core/src/package_json/detect.rs +++ b/crates/socket-patch-core/src/package_json/detect.rs @@ -1,4 +1,4 @@ -use crate::vendor::common::{detect_indent, serialize_json}; +use crate::vendor::common::JsonLayout; /// Package manager type for selecting the correct command prefix. #[derive(Debug, Clone, Copy, PartialEq)] @@ -318,17 +318,20 @@ fn remove_package_json_object(package_json: &mut serde_json::Value) -> ScriptRem } } -/// Re-serialize a package.json, keeping the indent unit the file already uses. +/// Re-serialize a package.json in the layout the file already uses. /// -/// serde's `to_string_pretty` is hard-wired to 2 spaces, so a 4-space or -/// tab-indented manifest came back reformatted top to bottom — turning a -/// two-key edit into a whole-file diff. The vendor backends already respect the -/// project's formatting when they rewrite package.json / lockfiles; reuse the -/// same helpers so `setup` touches only the lines it means to. +/// serde's `to_string_pretty` is hard-wired to 2 spaces and bare `\n`, so a +/// 4-space or tab-indented manifest came back reformatted top to bottom, and +/// a Windows one (yarn berry's persistManifest pretty-prints with `os.EOL`) +/// flipped every CRLF to LF and lost its BOM — turning a two-key edit into a +/// whole-file diff that `setup --remove` could never undo byte-exactly. The +/// vendor backends render through [`JsonLayout`] (BOM, indent, line ending, +/// trailing-newline shape); reuse it so `setup` touches only the lines it +/// means to. fn serialize_preserving_indent(value: &serde_json::Value, original: &str) -> String { - let indent = detect_indent(strip_bom(original)); - match serialize_json(value, &indent) { - // Always valid UTF-8: serde_json emits escaped ASCII/UTF-8 only. + match JsonLayout::of(original).render(value) { + // Always valid UTF-8: the original was a &str and serde_json emits + // escaped ASCII/UTF-8 only. Ok(bytes) => String::from_utf8_lossy(&bytes).into_owned(), // Serializing a `Value` cannot fail; fall back to the 2-space form. Err(_) => serde_json::to_string_pretty(value).unwrap_or_default() + "\n", diff --git a/crates/socket-patch-core/src/package_json/update.rs b/crates/socket-patch-core/src/package_json/update.rs index 0c419d9e..140f251c 100644 --- a/crates/socket-patch-core/src/package_json/update.rs +++ b/crates/socket-patch-core/src/package_json/update.rs @@ -422,7 +422,11 @@ mod tests { result.error ); let content = fs::read_to_string(&pkg).await.unwrap(); - let parsed: serde_json::Value = serde_json::from_str(&content).unwrap(); + // The BOM survives the rewrite (the editor that added it keeps it). + let body = content + .strip_prefix('\u{feff}') + .expect("setup must keep the manifest's BOM"); + let parsed: serde_json::Value = serde_json::from_str(body).unwrap(); assert!(parsed["scripts"]["postinstall"].is_string()); assert!(parsed["scripts"]["dependencies"].is_string()); assert_eq!(parsed["scripts"]["build"], "tsc"); @@ -678,10 +682,81 @@ mod tests { ); let content = fs::read_to_string(&pkg).await.unwrap(); assert!(!content.contains("socket-patch")); - let parsed: serde_json::Value = serde_json::from_str(&content).unwrap(); + let body = content + .strip_prefix('\u{feff}') + .expect("setup --remove must keep the manifest's BOM"); + let parsed: serde_json::Value = serde_json::from_str(body).unwrap(); assert_eq!(parsed["scripts"]["build"], "tsc"); } + /// A Windows yarn-berry manifest (persistManifest pretty-prints with + /// `os.EOL`, so CRLF; an editor may add a BOM; some tools drop the final + /// newline) must keep its layout through `setup`, and `setup --remove` + /// must land byte-identical on the pre-setup file. serde's serializer + /// emits bare `\n` and no BOM, which used to flip every line to LF — a + /// whole-file diff yarn then keeps (it follows the majority ending). + #[tokio::test] + async fn test_setup_then_remove_round_trips_crlf_bom_and_final_newline_shape() { + let lf = "{\n \"name\": \"x\",\n \"version\": \"1.0.0\",\n \"scripts\": {\n \"build\": \"tsc\"\n },\n \"packageManager\": \"yarn@4.12.0\"\n}"; + let crlf = lf.replace('\n', "\r\n"); + let cases = [ + ("lf", format!("{lf}\n")), + ("lf-no-final-newline", lf.to_string()), + ("crlf", format!("{crlf}\r\n")), + ("bom-crlf", format!("\u{feff}{crlf}\r\n")), + ("crlf-no-final-newline", crlf.clone()), + ("bom-lf", format!("\u{feff}{lf}\n")), + ("crlf-two-final-newlines", format!("{crlf}\r\n\r\n")), + ]; + for (label, original) in cases { + let dir = tempfile::tempdir().unwrap(); + let pkg = dir.path().join("package.json"); + fs::write(&pkg, &original).await.unwrap(); + + let up = update_package_json(&pkg, false, PackageManager::Npm).await; + assert_eq!(up.status, UpdateStatus::Updated, "{label}: {:?}", up.error); + let wired = fs::read_to_string(&pkg).await.unwrap(); + assert_eq!( + wired.starts_with('\u{feff}'), + original.starts_with('\u{feff}'), + "{label}: BOM presence must survive setup:\n{wired:?}" + ); + if original.contains("\r\n") { + assert!( + !wired.replace("\r\n", "").contains('\n'), + "{label}: setup left a bare LF in a CRLF manifest:\n{wired:?}" + ); + } else { + assert!( + !wired.contains('\r'), + "{label}: setup added CR to an LF manifest" + ); + } + let trailer = |t: &str| t.len() - t.trim_end_matches(['\r', '\n']).len(); + assert_eq!( + trailer(&wired), + trailer(&original), + "{label}: trailing-newline shape must survive setup:\n{wired:?}" + ); + let parsed: serde_json::Value = + serde_json::from_str(wired.trim_start_matches('\u{feff}')).unwrap(); + assert!(parsed["scripts"]["postinstall"].is_string(), "{label}"); + + let down = remove_package_json(&pkg, false).await; + assert_eq!( + down.status, + RemoveStatus::Removed, + "{label}: {:?}", + down.error + ); + assert_eq!( + fs::read_to_string(&pkg).await.unwrap(), + original, + "{label}: setup --remove must restore the pre-setup bytes" + ); + } + } + /// mkfifo(2) directly rather than shelling out to the `mkfifo` binary — /// same helper as the find.rs FIFO tests: fork/exec flakes under heavy /// parallel load and the syscall needs no process at all. diff --git a/crates/socket-patch-core/src/patch/redirect/mod.rs b/crates/socket-patch-core/src/patch/redirect/mod.rs index bb535b86..4a4eebc6 100644 --- a/crates/socket-patch-core/src/patch/redirect/mod.rs +++ b/crates/socket-patch-core/src/patch/redirect/mod.rs @@ -24,6 +24,7 @@ use serde_json::{json, Value}; use crate::crawlers::composer_crawler::normalize_version; use crate::utils::digest::is_hex64_lower; +use crate::utils::line_endings::{to_lf, LineEndings}; use crate::vendor::yarn_berry_lock::yarnrc_compression_level; mod bun_binary; @@ -2252,18 +2253,37 @@ fn rewrite_yarn_classic( // `::__archiveUrl=` binding, and `checksum:` becomes // our precomputed `integrity.yarnBerry10c0`. The descriptor KEY + package.json // are untouched (the `name@npm:^range` descriptor still satisfies, so -// `--immutable` passes). Byte-for-byte twin of the TS `rewriteYarnBerry`. +// `--immutable` passes). Byte-for-byte twin of the TS `rewriteYarnBerry` on +// LF locks; the CRLF / BOM round trip below has no TS counterpart yet. /// Only cacheKey `10c0` (yarn 4, compressionLevel 0 default) has a checksum we /// can reproduce offline; matches the vendored backend's `SUPPORTED_CACHE_KEY`. const YARN_BERRY_SUPPORTED_CACHE_KEY: &str = "10c0"; +/// Whether ledger edits of `kind` are yarn.lock blocks — the fragments the +/// yarn rewriters record in the lock's ON-DISK line endings, which the +/// reverts (the per-purl takeover and the whole-ledger replay) may respell +/// in the live lock's ending when a `core.autocrlf` checkout changed it +/// ([`crate::utils::line_endings::fragments_in_eol_of`]). +pub(crate) fn yarn_lock_fragment_kind(kind: &str) -> bool { + matches!( + kind, + "redirect_yarn_berry_entry" | "redirect_yarn_classic_entry" + ) +} + /// A yarn.lock is berry (v2+) when it carries the `__metadata:` header block; /// anything else is a classic v1 lock. Shared by both yarn rewriters and /// lockfile discovery (`vex::discover::yarn`) so the grammar split cannot -/// drift. +/// drift. A leading BOM is encoding, not key text (yarn's YAML parser drops +/// it), so a header-less lock opening with `\u{feff}__metadata:` is berry +/// too. pub(crate) fn is_berry_lock(content: &str) -> bool { - content.lines().any(|line| line.starts_with("__metadata:")) + content + .strip_prefix('\u{feff}') + .unwrap_or(content) + .lines() + .any(|line| line.starts_with("__metadata:")) } /// The `cacheKey:` value from the `__metadata` block (berry writes it unquoted: @@ -2282,6 +2302,72 @@ fn berry_cache_key(content: &str) -> Option { None } +/// The project-level refusals of the yarn berry hosted rewriter — the gates +/// that hold for every dep of the lock, whatever the overrides: a MIXED +/// line-ending lock, an unsupported `cacheKey`, and a `.yarnrc.yml` +/// `compressionLevel` other than 0. `Ok` for a lock that is not berry (the +/// classic rewriter owns those). +/// +/// Exposed so the vendored→hosted mode takeover (`scan`/`get --mode hosted` +/// over a vendored berry purl) can refuse BEFORE it reverts the vendored +/// wiring: the vendored revert never refuses on line endings (it keeps a +/// mixed lock mixed), so without this preflight the takeover stripped the +/// live vendored patch and then this rewriter refused the lock, leaving the +/// package unpatched in both modes — the bun twin is +/// [`preflight_bun_hosted`]. +/// +/// Line endings: yarn berry writes a NEW lockfile with the OS line ending +/// (`os.EOL`: CRLF on Windows) and keeps an existing file's majority ending +/// on every later write (`normalizeLineEndings` in yarnpkg-fslib +/// `FakeFS.ts`, called by `Project.persistLockfile`); a `core.autocrlf` +/// checkout turns an LF lock into CRLF on any OS. A uniform CRLF lock is +/// supported (rewritten LF-normalized and re-expanded). A MIXED lock has no +/// single style to restore, and yarn cannot keep one either: `--immutable` +/// compares the file with its own majority-normalized re-render and fails +/// (YN0028), while a plain install rewrites every minority line — so it is +/// refused untouched, `yarn install` normalizes it first. +pub fn preflight_yarn_berry_hosted(lock: &str, yarnrc: Option<&str>) -> Result<(), RewriteWarning> { + if !is_berry_lock(lock) { + return Ok(()); + } + let body = lock.strip_prefix('\u{feff}').unwrap_or(lock); + if LineEndings::of(body) == LineEndings::Mixed { + return Err(RewriteWarning { + code: "redirect_yarn_berry_mixed_line_endings".into(), + detail: "yarn.lock mixes CRLF and LF line endings (or holds a bare carriage \ + return), so no single line ending can be kept, and yarn itself \ + rejects it under `--immutable` (YN0028) — run `yarn install` once to \ + normalize the lock, then re-run; leaving it untouched" + .into(), + }); + } + // Refuse any lock whose cache checksum we can't reproduce + // offline. A guessed `checksum:` bricks installs (YN0018). + let key = berry_cache_key(&to_lf(body)); + if key.as_deref() != Some(YARN_BERRY_SUPPORTED_CACHE_KEY) { + return Err(RewriteWarning { + code: "redirect_yarn_berry_cache_unsupported".into(), + detail: format!( + "yarn.lock cacheKey is `{}`; only `{YARN_BERRY_SUPPORTED_CACHE_KEY}` \ + (yarn 4, compressionLevel 0 default) has an offline-reproducible cache checksum", + key.as_deref().unwrap_or("(missing)") + ), + }); + } + if let Some(level) = yarnrc.and_then(yarnrc_compression_level) { + if level != "0" { + return Err(RewriteWarning { + code: "redirect_yarn_berry_cache_unsupported".into(), + detail: format!( + ".yarnrc.yml sets `compressionLevel: {level}`, which changes berry's \ + cache checksums; only compressionLevel 0 (the yarn 4 default) is supported" + ), + }); + } + } + Ok(()) +} + fn rewrite_yarn_berry( files: &BTreeMap, overrides: &[DepOverride], @@ -2293,56 +2379,34 @@ fn rewrite_yarn_berry( if npm.is_empty() || !files.contains_key("yarn.lock") { return; } - let content = &files["yarn.lock"]; + let raw = &files["yarn.lock"]; // The classic rewriter handles a v1 lock; berry stays out of its way. - if !is_berry_lock(content) { + if !is_berry_lock(raw) { return; } - // Whole-file gates. A CRLF lock collapses the `\n\n` block grammar (a - // `\r\n\r\n` file contains no `\n\n`), so `berry_cache_key` used to come - // back None and the refusal below misdiagnosed a perfectly good - // `cacheKey: 10c0` lock as "cacheKey is `(missing)`" — sending Windows - // users chasing yarn cache config instead of line endings. Same - // fail-closed outcome, honest diagnosis. - if content.contains("\r\n") { - result.warnings.push(RewriteWarning { - code: "redirect_yarn_berry_crlf_unsupported".into(), - detail: "yarn.lock has CRLF (Windows) line endings; the redirect's \ - byte-surgical rewrite only supports LF — normalize the \ - file to LF line endings and re-run" - .into(), - }); - return; - } - // Refuse any lock whose cache checksum we can't reproduce - // offline. A guessed `checksum:` bricks installs (YN0018). - let key = berry_cache_key(content); - if key.as_deref() != Some(YARN_BERRY_SUPPORTED_CACHE_KEY) { - result.warnings.push(RewriteWarning { - code: "redirect_yarn_berry_cache_unsupported".into(), - detail: format!( - "yarn.lock cacheKey is `{}`; only `{YARN_BERRY_SUPPORTED_CACHE_KEY}` \ - (yarn 4, compressionLevel 0 default) has an offline-reproducible cache checksum", - key.as_deref().unwrap_or("(missing)") - ), - }); + // Line endings (see [`preflight_yarn_berry_hosted`] for when yarn writes + // CRLF): a CRLF lock is rewritten LF-normalized (the `\n\n` block + // grammar never splits a `\r\n\r\n` file) and re-expanded, so every + // untouched byte round-trips and the ledger records the lock's on-disk + // CRLF fragments. A leading BOM rides outside the blocks; a mixed lock + // is refused by the preflight. + let (bom, body) = match raw.strip_prefix('\u{feff}') { + Some(rest) => ("\u{feff}", rest), + None => ("", raw.as_str()), + }; + // Project-level gates (line endings, cacheKey, compressionLevel), shared + // with the vendored→hosted takeover preflight so a takeover never + // reverts vendored wiring this rewriter then refuses. + if let Err(warning) = + preflight_yarn_berry_hosted(raw, files.get(".yarnrc.yml").map(String::as_str)) + { + result.warnings.push(warning); return; } - if let Some(rc) = files.get(".yarnrc.yml") { - if let Some(level) = yarnrc_compression_level(rc) { - if level != "0" { - result.warnings.push(RewriteWarning { - code: "redirect_yarn_berry_cache_unsupported".into(), - detail: format!( - ".yarnrc.yml sets `compressionLevel: {level}`, which changes berry's \ - cache checksums; only compressionLevel 0 (the yarn 4 default) is supported" - ), - }); - return; - } - } - } + let eol = LineEndings::of(body); + let normalized = to_lf(body); + let content: &str = &normalized; let mut blocks: Vec = content.split("\n\n").map(String::from).collect(); let resolution_re = @@ -2546,13 +2610,16 @@ fn rewrite_yarn_berry( } matched_any = true; if rewritten != *block { + // The ledger records the lock's on-disk bytes (CRLF lines for + // a CRLF lock), so every revert's byte-exact `replacen` + // matches what the file really holds. result.edits.push(FileEdit { path: "yarn.lock".into(), kind: "redirect_yarn_berry_entry".into(), action: "rewritten".into(), key: Some(format!("{fname}@{}", dep.version)), - original: Some(Value::String(block.clone())), - new: Some(Value::String(rewritten.clone())), + original: Some(Value::String(eol.restore(block).into_owned())), + new: Some(Value::String(eol.restore(&rewritten).into_owned())), }); *block = rewritten; changed = true; @@ -2566,7 +2633,10 @@ fn rewrite_yarn_berry( } } if changed { - result.files.insert("yarn.lock".into(), blocks.join("\n\n")); + let out = blocks.join("\n\n"); + result + .files + .insert("yarn.lock".into(), format!("{bom}{}", eol.restore(&out))); } } @@ -11940,34 +12010,252 @@ packages: ); } - /// A CRLF berry lock must be diagnosed as a line-ending problem, not as - /// `cacheKey is \`(missing)\`` — the lock's cacheKey IS 10c0; only the - /// `\n\n` block grammar fails on `\r\n\r\n`. Fail-closed either way. + /// A berry lock the way real yarn 4 writes it — header, `__metadata`, + /// a decoy entry, the target, the root workspace — for the line-ending + /// cells below. + fn berry_lock_two_entries() -> String { + format!( + "# This file is generated by running \"yarn install\" inside your project.\n\ + # Manual changes might be lost - proceed with caution!\n\n\ + __metadata:\n version: 8\n cacheKey: 10c0\n\n\ + \"aaa-decoy@npm:^1.0.0\":\n version: 1.0.0\n \ + resolution: \"aaa-decoy@npm:1.0.0\"\n checksum: 10c0/{}\n \ + languageName: node\n linkType: hard\n\n\ + \"app@workspace:.\":\n version: 0.0.0-use.local\n \ + resolution: \"app@workspace:.\"\n dependencies:\n \ + aaa-decoy: \"npm:^1.0.0\"\n left-pad: \"npm:^1.3.0\"\n \ + languageName: unknown\n linkType: soft\n\n\ + \"left-pad@npm:^1.3.0\":\n version: 1.3.0\n \ + resolution: \"left-pad@npm:1.3.0\"\n checksum: 10c0/{}\n \ + languageName: node\n linkType: hard\n", + "1".repeat(128), + "3".repeat(128) + ) + } + + /// A CRLF berry lock — what yarn itself writes on Windows (a new + /// lockfile gets `os.EOL`) and what a `core.autocrlf` checkout hands + /// any OS — is rewritten in place: exactly the target entry changes, + /// every line keeps its CRLF, a leading BOM survives, and the ledger + /// records the ON-DISK (CRLF) fragments, so the revert's byte-exact + /// `replacen(new, original)` restores the input and a re-run is a no-op. + #[test] + fn berry_crlf_and_bom_locks_round_trip_byte_exact() { + let checksum = format!("10c0/{}", "7".repeat(128)); + let url = "http://p.test/patch/npm/left-pad/1.3.0/t/u/left-pad-1.3.0.tgz"; + let ovr = berry_override("left-pad", "1.3.0", url, &checksum); + let lf = berry_lock_two_entries(); + let mut lf_files = BTreeMap::new(); + lf_files.insert("yarn.lock".to_string(), lf.clone()); + let mut lf_result = RewriteResult::default(); + rewrite_yarn_berry(&lf_files, std::slice::from_ref(&ovr), &mut lf_result); + let lf_out = lf_result.files["yarn.lock"].clone(); + + for (label, bom, crlf) in [ + ("crlf", "", true), + ("bom+lf", "\u{feff}", false), + ("bom+crlf", "\u{feff}", true), + ] { + let respell = |s: &str| { + let s = if crlf { + s.replace('\n', "\r\n") + } else { + s.to_string() + }; + format!("{bom}{s}") + }; + let input = respell(&lf); + let mut files = BTreeMap::new(); + files.insert("yarn.lock".to_string(), input.clone()); + let mut r = RewriteResult::default(); + rewrite_yarn_berry(&files, std::slice::from_ref(&ovr), &mut r); + assert!(r.warnings.is_empty(), "{label}: {:?}", r.warnings); + let out = &r.files["yarn.lock"]; + assert_eq!( + *out, + respell(&lf_out), + "{label}: the LF rewrite, in the input's own line ending" + ); + if crlf { + assert_eq!( + out.matches('\n').count(), + out.matches("\r\n").count(), + "{label}: every line keeps CRLF" + ); + } + assert_eq!( + out.starts_with('\u{feff}'), + !bom.is_empty(), + "{label}: BOM kept" + ); + assert!( + out.contains(&crate::utils::uri::encode_uri_component(url)), + "{label}: {out}" + ); + + // One edit; its fragments are the on-disk bytes of the entry. + assert_eq!(r.edits.len(), 1, "{label}"); + let edit = &r.edits[0]; + let (orig, new) = ( + edit.original.as_ref().and_then(Value::as_str).unwrap(), + edit.new.as_ref().and_then(Value::as_str).unwrap(), + ); + assert_eq!( + (orig, new), + ( + respell( + lf_result.edits[0] + .original + .as_ref() + .unwrap() + .as_str() + .unwrap() + ) + .trim_start_matches('\u{feff}'), + respell(lf_result.edits[0].new.as_ref().unwrap().as_str().unwrap()) + .trim_start_matches('\u{feff}'), + ), + "{label}: fragments in the lock's on-disk form" + ); + assert!(input.contains(orig) && out.contains(new), "{label}"); + assert!( + orig.contains("\"left-pad@npm:^1.3.0\":") && !orig.contains("aaa-decoy@npm"), + "{label}: the fragment is the target entry alone: {orig:?}" + ); + assert_eq!( + out.replacen(new, orig, 1), + input, + "{label}: the ledger revert restores the input byte-exactly" + ); + + // Re-run over the rewritten lock: nothing to do. + let mut files = BTreeMap::new(); + files.insert("yarn.lock".to_string(), out.clone()); + let mut again = RewriteResult::default(); + rewrite_yarn_berry(&files, std::slice::from_ref(&ovr), &mut again); + assert!( + again.files.is_empty() && again.edits.is_empty() && again.warnings.is_empty(), + "{label}: re-run must be a no-op: {:?}", + again.warnings + ); + } + } + + /// A lock mixing CRLF and LF (or holding a bare CR) has no single line + /// ending to keep — and yarn itself rejects it under `--immutable` + /// (YN0028) — so it is refused untouched with a code that names the + /// line endings, never rewritten half-and-half. + #[test] + fn berry_mixed_line_endings_are_refused_untouched() { + let checksum = format!("10c0/{}", "7".repeat(128)); + let ovr = berry_override("left-pad", "1.3.0", "http://p.test/lp.tgz", &checksum); + let crlf = berry_lock_two_entries().replace('\n', "\r\n"); + let first_crlf = crlf.find("\r\n").unwrap(); + let mixed_lf = format!("{}\n{}", &crlf[..first_crlf], &crlf[first_crlf + 2..]); + let bare_cr = crlf.replacen("proceed with caution!", "proceed\rwith caution!", 1); + for (label, lock) in [("crlf+lf", mixed_lf), ("bare cr", bare_cr)] { + let mut files = BTreeMap::new(); + files.insert("yarn.lock".to_string(), lock); + let mut r = RewriteResult::default(); + rewrite_yarn_berry(&files, std::slice::from_ref(&ovr), &mut r); + assert!(r.files.is_empty() && r.edits.is_empty(), "{label}"); + assert_eq!( + warning_codes(&r), + vec!["redirect_yarn_berry_mixed_line_endings"], + "{label}" + ); + let detail = &r.warnings[0].detail; + assert!( + detail.contains("CRLF") && detail.contains("yarn install"), + "{label}: detail names the cause and the remedy: {detail}" + ); + } + } + + /// The public takeover preflight is the rewriter's own project gate: + /// the same codes for a mixed lock, an unsupported cacheKey (CRLF lock + /// included) and a non-zero compressionLevel; `Ok` for a supported LF / + /// CRLF / BOM'd berry lock and for any classic lock. + #[test] + fn berry_hosted_preflight_mirrors_the_rewriter_gates() { + let lf = berry_lock_two_entries(); + let crlf = lf.replace('\n', "\r\n"); + for ok in [ + lf.clone(), + crlf.clone(), + format!("\u{feff}{crlf}"), + classic_lock_two_entries().replacen("\n", "\r\n", 1), + ] { + assert_eq!( + preflight_yarn_berry_hosted(&ok, None).map_err(|w| w.code), + Ok(()), + "{ok:?}" + ); + } + let code = |lock: &str, rc: Option<&str>| { + preflight_yarn_berry_hosted(lock, rc).map_err(|w| w.code) + }; + assert_eq!( + code(&crlf.replacen("\r\n", "\n", 1), None), + Err("redirect_yarn_berry_mixed_line_endings".to_string()) + ); + assert_eq!( + code(&crlf.replace("cacheKey: 10c0", "cacheKey: 8"), None), + Err("redirect_yarn_berry_cache_unsupported".to_string()) + ); + assert_eq!( + code(&crlf, Some("compressionLevel: 9\r\n")), + Err("redirect_yarn_berry_cache_unsupported".to_string()) + ); + assert_eq!(code(&crlf, Some("compressionLevel: 0\n")), Ok(())); + } + + /// The whole-file gates read the NORMALIZED lock: a CRLF lock at an + /// unsupported cacheKey is refused naming THAT key — never + /// "`(missing)`", which is what the `\n\n` grammar made of a CRLF + /// `__metadata` block before line endings were handled. #[test] - fn berry_crlf_lock_diagnosed_as_crlf_not_missing_cache_key() { + fn berry_crlf_lock_cache_key_gate_names_the_real_key() { let checksum = format!("10c0/{}", "7".repeat(128)); let ovr = berry_override("left-pad", "1.3.0", "http://p.test/lp.tgz", &checksum); let mut files = BTreeMap::new(); files.insert( "yarn.lock".to_string(), - berry_lock("10c0").replace('\n', "\r\n"), + berry_lock("8").replace('\n', "\r\n"), ); let mut r = RewriteResult::default(); rewrite_yarn_berry(&files, std::slice::from_ref(&ovr), &mut r); - assert!(r.files.is_empty(), "CRLF lock must not be rewritten"); + assert!(r.files.is_empty()); assert_eq!( warning_codes(&r), - vec!["redirect_yarn_berry_crlf_unsupported"], - "the refusal must name CRLF, not the cache key: {:?}", - r.warnings + vec!["redirect_yarn_berry_cache_unsupported"] ); assert!( - r.warnings[0].detail.contains("CRLF"), - "detail must name the line endings: {}", + r.warnings[0].detail.contains("`8`"), + "the detail names the lock's cacheKey: {}", r.warnings[0].detail ); } + /// A BOM directly in front of `__metadata:` (a header-less lock saved + /// by a Windows editor) is still berry: the classic rewriter stays out, + /// the berry one rewrites it and keeps the BOM. + #[test] + fn berry_bom_before_metadata_is_still_berry() { + let checksum = format!("10c0/{}", "7".repeat(128)); + let ovr = berry_override("left-pad", "1.3.0", "http://p.test/lp.tgz", &checksum); + let lock = berry_lock("10c0").replace("# header\n\n", "\u{feff}"); + assert!(lock.starts_with("\u{feff}__metadata:"), "{lock:?}"); + assert!(is_berry_lock(&lock)); + let mut files = BTreeMap::new(); + files.insert("yarn.lock".to_string(), lock); + let r = rewrite_registry_redirect(&files, std::slice::from_ref(&ovr)); + assert!(r.warnings.is_empty(), "{:?}", r.warnings); + let out = &r.files["yarn.lock"]; + assert!(out.starts_with("\u{feff}__metadata:"), "{out:?}"); + assert!(out.contains("::__archiveUrl="), "{out}"); + } + /// CRLF locks preserve their newline style through hosted rewriting. #[test] fn pnpm_crlf_lock_is_rewritten_without_changing_line_endings() { diff --git a/crates/socket-patch-core/src/patch/redirect/replay.rs b/crates/socket-patch-core/src/patch/redirect/replay.rs index eaba41a8..9c603b16 100644 --- a/crates/socket-patch-core/src/patch/redirect/replay.rs +++ b/crates/socket-patch-core/src/patch/redirect/replay.rs @@ -487,6 +487,22 @@ pub async fn revert_remaining_redirect_edits( } continue; } + // yarn lock fragments are whole blocks recorded in the + // lock's on-disk line endings, and a `core.autocrlf` + // checkout on another OS re-spells the lock (never the + // committed ledger): when neither fragment matches + // verbatim, try both in the live file's uniform ending. + let respelled = (super::yarn_lock_fragment_kind(&edit.kind) + && !content.contains(new) + && !content.contains(original)) + .then(|| { + crate::utils::line_endings::fragments_in_eol_of(&content, original, new) + }) + .flatten(); + let (original, new) = match &respelled { + Some((original, new)) => (original.as_str(), new.as_str()), + None => (original, new), + }; // `new` before `original`: original may be a substring // of new (Cargo.toml insert, maven version suffix). if content.contains(new) { @@ -925,6 +941,106 @@ mod tests { } } + /// yarn lock edits replay across a `core.autocrlf` checkout switch: the + /// ledger's fragments are the lock's on-disk bytes at redirect time (the + /// berry and classic rewriters record CRLF blocks for a CRLF lock), the + /// live lock may since be in the OTHER uniform ending — the revert lands + /// on the original block in the live file's ending. A mixed live file + /// proves nothing and refuses; a non-yarn kind keeps the verbatim-only + /// contract. + #[tokio::test] + async fn yarn_edits_revert_across_a_checkout_line_ending_switch() { + let head = "# yarn\n\n__metadata:\n version: 8\n cacheKey: 10c0\n\n"; + let original = "\"left-pad@npm:^1.3.0\":\n version: 1.3.0\n \ + resolution: \"left-pad@npm:1.3.0\"\n checksum: 10c0/aaaa\n \ + languageName: node\n linkType: hard"; + let new = "\"left-pad@npm:^1.3.0\":\n version: 1.3.0\n \ + resolution: \"left-pad@npm:1.3.0::__archiveUrl=http%3A%2F%2Fp.test%2Flp.tgz\"\n \ + checksum: 10c0/bbbb\n languageName: node\n linkType: hard"; + let spell = |text: &str, crlf: bool| { + if crlf { + text.replace('\n', "\r\n") + } else { + text.to_string() + } + }; + for kind in ["redirect_yarn_berry_entry", "redirect_yarn_classic_entry"] { + for (recorded_crlf, live_crlf) in + [(true, true), (true, false), (false, true), (false, false)] + { + let label = format!("{kind} recorded_crlf={recorded_crlf} live_crlf={live_crlf}"); + let dir = TempDir::new().unwrap(); + write( + dir.path(), + "yarn.lock", + &spell(&format!("{head}{new}\n"), live_crlf), + ) + .await; + let mut state = state_with( + vec![edit( + "yarn.lock", + kind, + "rewritten", + Some(&spell(original, recorded_crlf)), + Some(&spell(new, recorded_crlf)), + )], + &["pkg:npm/left-pad@1.3.0"], + ); + let out = revert_remaining_redirect_edits(dir.path(), &mut state, false).await; + assert!(out.fully_reverted(), "{label}: {:?}", out.refusals); + assert_eq!( + read(dir.path(), "yarn.lock").await, + spell(&format!("{head}{original}\n"), live_crlf), + "{label}" + ); + assert!(state.edits.is_empty(), "{label}"); + } + } + + // A mixed live lock: refused whole, byte-untouched, edit kept. + let dir = TempDir::new().unwrap(); + let mixed = format!("{}{}\n", spell(head, true), new); + write(dir.path(), "yarn.lock", &mixed).await; + let mut state = state_with( + vec![edit( + "yarn.lock", + "redirect_yarn_berry_entry", + "rewritten", + Some(&spell(original, true)), + Some(&spell(new, true)), + )], + &["pkg:npm/left-pad@1.3.0"], + ); + let out = revert_remaining_redirect_edits(dir.path(), &mut state, false).await; + assert!(!out.fully_reverted(), "a mixed lock must refuse"); + assert_eq!(read(dir.path(), "yarn.lock").await, mixed); + assert_eq!(state.edits.len(), 1); + + // A non-yarn line-oriented kind keeps the verbatim-only contract. + let dir = TempDir::new().unwrap(); + write( + dir.path(), + "requirements.txt", + "a==1\r\nleft-pad @ https://patch.example/x.whl\r\n", + ) + .await; + let mut state = state_with( + vec![edit( + "requirements.txt", + "redirect_requirements_line", + "rewritten", + Some("a==1\nleft-pad==1.3.0"), + Some("a==1\nleft-pad @ https://patch.example/x.whl"), + )], + &["pkg:pypi/left-pad@1.3.0"], + ); + let out = revert_remaining_redirect_edits(dir.path(), &mut state, false).await; + assert!( + !out.fully_reverted(), + "only yarn blocks are respelled across line endings" + ); + } + // ---------- ReplaceFragment ---------- #[tokio::test] diff --git a/crates/socket-patch-core/src/patch/redirect/takeover.rs b/crates/socket-patch-core/src/patch/redirect/takeover.rs index 7ea38176..a26862cb 100644 --- a/crates/socket-patch-core/src/patch/redirect/takeover.rs +++ b/crates/socket-patch-core/src/patch/redirect/takeover.rs @@ -783,6 +783,19 @@ pub async fn revert_npm_redirect_purl( edit.path )); }; + // A yarn block recorded on a CRLF checkout, replayed on an LF + // one (or the reverse — `core.autocrlf` re-spells the lock on + // every OS switch, never the committed ledger): when neither + // fragment matches verbatim, try both in the file's ending. + let respelled = (super::yarn_lock_fragment_kind(&edit.kind) + && !content.contains(new) + && !content.contains(orig)) + .then(|| crate::utils::line_endings::fragments_in_eol_of(&content, orig, new)) + .flatten(); + let (orig, new) = match &respelled { + Some((orig, new)) => (orig.as_str(), new.as_str()), + None => (orig, new), + }; if content.contains(new) { staged.insert(edit.path.clone(), Some(content.replacen(new, orig, 1))); out.reverted_files.push(edit.path.clone()); @@ -1698,6 +1711,92 @@ mod tests { assert!(state.edits.is_empty(), "edits dropped"); } + /// A CRLF (and BOM'd) berry lock — yarn's own output on Windows — + /// round-trips through the takeover byte-exactly: the rewriter records + /// the CRLF fragments, the revert replays them. Across checkouts too: a + /// ledger written on a CRLF checkout reverts an LF checkout of the same + /// commit and vice versa (`core.autocrlf` re-spells the lock, never the + /// committed ledger), landing on the pristine lock in the LIVE file's + /// endings. A lock whose endings are mixed proves nothing: it refuses + /// as drift, byte-untouched, ledger intact. + #[tokio::test] + async fn npm_berry_crlf_lock_round_trips_across_checkouts() { + let respell = |text: &str, crlf: bool| { + let lf = text.replace("\r\n", "\n"); + if crlf { + lf.replace('\n', "\r\n") + } else { + lf + } + }; + for (label, bom, recorded_crlf, live_crlf) in [ + ("crlf", "", true, true), + ("bom+crlf", "\u{feff}", true, true), + ("recorded crlf, reverted lf", "", true, false), + ("recorded lf, reverted crlf", "\u{feff}", false, true), + ] { + let pristine = format!("{bom}{}", respell(&berry_pristine(), recorded_crlf)); + let (tmp, mut state) = npm_redirected_fixture("yarn.lock", &pristine).await; + let root = tmp.path(); + let edit = state + .edits + .iter() + .find(|e| e.kind == "redirect_yarn_berry_entry") + .expect("berry edit"); + let recorded = edit.original.as_ref().and_then(Value::as_str).unwrap(); + assert_eq!( + recorded.contains("\r\n"), + recorded_crlf, + "{label}: the ledger records the on-disk endings: {recorded:?}" + ); + let wired = tokio::fs::read_to_string(root.join("yarn.lock")) + .await + .unwrap(); + tokio::fs::write(root.join("yarn.lock"), respell(&wired, live_crlf)) + .await + .unwrap(); + + revert_redirect_purl(root, &mut state, NPM_PURL, false) + .await + .unwrap_or_else(|e| panic!("{label}: revert must succeed: {e}")); + assert_eq!( + tokio::fs::read_to_string(root.join("yarn.lock")) + .await + .unwrap(), + format!("{bom}{}", respell(&berry_pristine(), live_crlf)), + "{label}: the pristine lock, in the live file's endings" + ); + assert!(state.edits.is_empty(), "{label}: edits dropped"); + } + + // Mixed live endings: neither the verbatim nor a respelled fragment + // is provable — refuse, touch nothing, keep the ledger. + let pristine = respell(&berry_pristine(), true); + let (tmp, mut state) = npm_redirected_fixture("yarn.lock", &pristine).await; + let root = tmp.path(); + let wired = tokio::fs::read_to_string(root.join("yarn.lock")) + .await + .unwrap(); + let mixed = wired.replacen(" languageName: node\r\n", " languageName: node\n", 1); + assert_ne!(mixed, wired, "the fixture edit must hit"); + tokio::fs::write(root.join("yarn.lock"), &mixed) + .await + .unwrap(); + let edits_before = state.edits.len(); + let err = revert_redirect_purl(root, &mut state, NPM_PURL, false) + .await + .expect_err("a mixed lock must refuse"); + assert!(err.contains("drifted"), "{err}"); + assert_eq!( + tokio::fs::read_to_string(root.join("yarn.lock")) + .await + .unwrap(), + mixed, + "byte-untouched" + ); + assert_eq!(state.edits.len(), edits_before, "ledger intact"); + } + #[tokio::test] async fn npm_package_lock_v2_round_trips_both_trees() { let (tmp, mut state) = diff --git a/crates/socket-patch-core/src/utils/line_endings.rs b/crates/socket-patch-core/src/utils/line_endings.rs new file mode 100644 index 00000000..b3d155f0 --- /dev/null +++ b/crates/socket-patch-core/src/utils/line_endings.rs @@ -0,0 +1,180 @@ +//! A text file's line-ending style, for the writers that must hand a file +//! back in the style they found it. +//! +//! The round trip the yarn-berry writers use (the pattern the other +//! CRLF-preserving rewriters in this crate follow too): classify the file, +//! refuse [`LineEndings::Mixed`] (there is no single style to restore), +//! operate on the LF-normalized text ([`to_lf`]), and re-expand whatever is +//! written or recorded with [`LineEndings::restore`]. + +use std::borrow::Cow; + +/// How a text file ends its lines. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum LineEndings { + /// No line break at all: an empty file, or one unterminated line. + None, + /// Every line break is `\n`. + Lf, + /// Every line break is `\r\n`. + Crlf, + /// `\r\n` and bare `\n` both occur, or a `\r` stands outside a `\r\n` + /// pair. There is no single style to restore, so an LF-normalize and + /// re-expand round trip would rewrite lines nobody asked to touch. + Mixed, +} + +impl LineEndings { + /// Classify `text`. + pub(crate) fn of(text: &str) -> Self { + let bytes = text.as_bytes(); + let (mut crlf, mut lf) = (false, false); + for (i, &b) in bytes.iter().enumerate() { + match b { + b'\n' if i > 0 && bytes[i - 1] == b'\r' => crlf = true, + b'\n' => lf = true, + b'\r' if bytes.get(i + 1) != Some(&b'\n') => return Self::Mixed, + _ => {} + } + } + match (crlf, lf) { + (false, false) => Self::None, + (false, true) => Self::Lf, + (true, false) => Self::Crlf, + (true, true) => Self::Mixed, + } + } + + /// `lf` — text built or edited in LF form — spelled in this style: + /// every `\n` becomes `\r\n` for a CRLF file; any other style is + /// returned as is. + pub(crate) fn restore(self, lf: &str) -> Cow<'_, str> { + match self { + Self::Crlf => Cow::Owned(lf.replace('\n', "\r\n")), + _ => Cow::Borrowed(lf), + } + } +} + +/// `text` with every `\r\n` folded to `\n` (borrowed when there is none). +pub(crate) fn to_lf(text: &str) -> Cow<'_, str> { + if text.contains("\r\n") { + Cow::Owned(text.replace("\r\n", "\n")) + } else { + Cow::Borrowed(text) + } +} + +/// The line terminator yarn berry itself would write into a file holding +/// `text`, minus its operating-system fallback: `\r\n` when CRLF breaks +/// strictly outnumber LF ones, `\n` otherwise (a tie, a file with no break +/// at all, bare `\r`s). Mirrors `getEndOfLine` in yarnpkg-fslib's +/// `FakeFS.ts`, which falls back to `os.EOL` where this falls back to LF — +/// a re-serialization must not depend on the OS it runs on. +pub(crate) fn majority_terminator(text: &str) -> &'static str { + let crlf = text.matches("\r\n").count(); + let lf = text.matches('\n').count() - crlf; + if crlf > lf { + "\r\n" + } else { + "\n" + } +} + +/// A recorded `(original, new)` fragment pair of a line-oriented text edit, +/// respelled in `content`'s line endings when the two disagree wholesale. +/// +/// The yarn rewriters record their fragments in the lock's on-disk form +/// (CRLF lines for a CRLF lock), and every revert matches them +/// byte-exactly. But the committed ledger keeps those JSON-escaped +/// fragments verbatim while a `core.autocrlf` checkout re-spells the lock +/// itself: Git for Windows' default install converts LF to CRLF on +/// checkout, and a macOS or Linux clone of the same commit is LF. When the +/// live file is uniformly one style and the fragments uniformly the other, +/// they record the same edit; anything else (a mixed file, fragments that +/// disagree with each other) proves nothing and stays a drift refusal. +pub(crate) fn fragments_in_eol_of( + content: &str, + original: &str, + new: &str, +) -> Option<(String, String)> { + let fragments = match (LineEndings::of(original), LineEndings::of(new)) { + (a, b) if a == b => a, + (LineEndings::None, style) | (style, LineEndings::None) => style, + _ => return None, + }; + match (LineEndings::of(content), fragments) { + (LineEndings::Crlf, LineEndings::Lf) => { + Some((original.replace('\n', "\r\n"), new.replace('\n', "\r\n"))) + } + (LineEndings::Lf, LineEndings::Crlf) => { + Some((original.replace("\r\n", "\n"), new.replace("\r\n", "\n"))) + } + _ => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn classifies_every_style() { + assert_eq!(LineEndings::of(""), LineEndings::None); + assert_eq!(LineEndings::of("one line"), LineEndings::None); + assert_eq!(LineEndings::of("a\nb\n"), LineEndings::Lf); + assert_eq!(LineEndings::of("a\r\nb\r\n"), LineEndings::Crlf); + assert_eq!(LineEndings::of("a\r\nb"), LineEndings::Crlf); + assert_eq!(LineEndings::of("a\r\nb\n"), LineEndings::Mixed); + assert_eq!(LineEndings::of("a\nb\r\n"), LineEndings::Mixed); + // A bare CR is never a style we can restore, alone or beside CRLF. + assert_eq!(LineEndings::of("a\rb"), LineEndings::Mixed); + assert_eq!(LineEndings::of("a\r\nb\rc\r\n"), LineEndings::Mixed); + assert_eq!(LineEndings::of("trailing\r"), LineEndings::Mixed); + // A BOM is not a line ending. + assert_eq!(LineEndings::of("\u{feff}a\r\n"), LineEndings::Crlf); + } + + #[test] + fn normalize_and_restore_round_trip_a_uniform_file() { + for text in ["a\nb\n\nc", "a\r\nb\r\n\r\nc", "no break", ""] { + let style = LineEndings::of(text); + assert_eq!(style.restore(&to_lf(text)), text, "{text:?}"); + } + assert!(matches!(to_lf("a\nb"), Cow::Borrowed(_))); + assert!(matches!(LineEndings::Lf.restore("a\nb"), Cow::Borrowed(_))); + } + + #[test] + fn majority_terminator_follows_yarn_but_never_the_os() { + assert_eq!(majority_terminator("a\r\nb\r\nc\n"), "\r\n"); + assert_eq!(majority_terminator("a\r\nb\nc\n"), "\n"); + assert_eq!(majority_terminator("a\r\nb\n"), "\n", "a tie is LF"); + assert_eq!(majority_terminator("{}"), "\n", "no break: LF, not os.EOL"); + } + + #[test] + fn fragments_are_respelled_only_across_uniform_styles() { + let (orig, new) = ("k:\n a: 1\n", "k:\n a: 2\n"); + assert_eq!( + fragments_in_eol_of("x\r\nk:\r\n a: 2\r\n", orig, new), + Some(("k:\r\n a: 1\r\n".into(), "k:\r\n a: 2\r\n".into())) + ); + let (orig, new) = ("k:\r\n a: 1", "k:\r\n a: 2"); + assert_eq!( + fragments_in_eol_of("x\nk:\n a: 2\n", orig, new), + Some(("k:\n a: 1".into(), "k:\n a: 2".into())) + ); + // Same style, a mixed file, disagreeing or line-free fragments: + // nothing to respell. + assert_eq!(fragments_in_eol_of("x\n", "a\nb", "a\nc"), None); + assert_eq!(fragments_in_eol_of("x\r\ny\n", "a\nb", "a\nc"), None); + assert_eq!(fragments_in_eol_of("x\r\n", "a\nb", "a\r\nc"), None); + assert_eq!(fragments_in_eol_of("x\r\n", "ab", "ac"), None); + // One single-line side takes the other side's style. + assert_eq!( + fragments_in_eol_of("x\r\n", "ab", "a\nc"), + Some(("ab".into(), "a\r\nc".into())) + ); + } +} diff --git a/crates/socket-patch-core/src/utils/mod.rs b/crates/socket-patch-core/src/utils/mod.rs index ad349708..840cdbf2 100644 --- a/crates/socket-patch-core/src/utils/mod.rs +++ b/crates/socket-patch-core/src/utils/mod.rs @@ -3,6 +3,7 @@ pub mod env_compat; pub mod fs; pub mod notice; pub(crate) mod http; +pub(crate) mod line_endings; pub mod pdm_lock; pub mod pipenv; pub mod poetry_lock; diff --git a/crates/socket-patch-core/src/vendor/common.rs b/crates/socket-patch-core/src/vendor/common.rs index 78e41987..b8602e6f 100644 --- a/crates/socket-patch-core/src/vendor/common.rs +++ b/crates/socket-patch-core/src/vendor/common.rs @@ -163,6 +163,76 @@ pub(crate) fn serialize_json(value: &Value, indent: &str) -> std::io::Result serde_json::Result { + serde_json::from_slice(bytes.strip_prefix(b"\xef\xbb\xbf").unwrap_or(bytes)) +} + +/// The byte layout a re-serialized JSON manifest keeps from the text it +/// replaces, so a vendor edit and its revert change nothing but the edited +/// keys: the leading UTF-8 BOM, the indent unit ([`detect_indent`]), the +/// line terminator, and whatever trails the closing brace (the +/// trailing-newline shape, verbatim). +/// +/// The terminator is CRLF for a CRLF file — yarn berry writes a +/// `package.json` it creates or first pretty-prints with `os.EOL`, so every +/// Windows project carries one — LF for an LF or single-line file, and, for +/// a file mixing both, the majority terminator yarn itself would rewrite it +/// with ([`majority_terminator`]; the forward vendor paths refuse such a +/// file before this runs, so only a revert reaches that arm). +pub(crate) struct JsonLayout { + bom: bool, + indent: String, + eol: &'static str, + trailer: String, +} + +impl JsonLayout { + /// The layout of `text` (a manifest's current contents). + pub(crate) fn of(text: &str) -> Self { + use crate::utils::line_endings::{majority_terminator, LineEndings}; + let (bom, body) = match text.strip_prefix('\u{feff}') { + Some(rest) => (true, rest), + None => (false, text), + }; + let content = body.trim_end_matches([' ', '\t', '\r', '\n']); + let eol = match LineEndings::of(body) { + LineEndings::Crlf => "\r\n", + LineEndings::Mixed => majority_terminator(body), + LineEndings::Lf | LineEndings::None => "\n", + }; + Self { + bom, + indent: detect_indent(body), + eol, + trailer: body[content.len()..].to_string(), + } + } + + /// `value` pretty-printed ([`serialize_json`]) in this layout. + pub(crate) fn render(&self, value: &Value) -> std::io::Result> { + let mut pretty = serialize_json(value, &self.indent)?; + // serialize_json's own trailing newline gives way to the trailer. + pretty.pop(); + let pretty = String::from_utf8(pretty).map_err(std::io::Error::other)?; + let mut out = String::with_capacity(pretty.len() + self.trailer.len() + 3); + if self.bom { + out.push('\u{feff}'); + } + // serde_json escapes every newline INSIDE a string value, so each + // `\n` it emits is a line break of the layout. + if self.eol == "\n" { + out.push_str(&pretty); + } else { + out.push_str(&pretty.replace('\n', self.eol)); + } + out.push_str(&self.trailer); + Ok(out.into_bytes()) + } +} + /// Serialize `(name, bytes, unix mode)` entries — in the given order — into /// a deterministic zip: a fixed DOS timestamp (1980-01-01 00:00:00) and a /// fixed deflate level, so rebuilding the same content always yields @@ -780,6 +850,70 @@ mod tests { use crate::hash::git_sha256::compute_git_sha256_from_bytes; + /// A manifest re-rendered in its own layout is byte-identical to itself + /// whenever it is in the canonical pretty shape (what yarn / npm write): + /// indent, CRLF vs LF, BOM and the trailing-newline shape all carry. + #[test] + fn json_layout_round_trips_canonical_manifests_in_every_shape() { + let lf = "{\n \"name\": \"a\",\n \"dependencies\": {\n \"x\": \"1\"\n }\n}\n"; + let shapes = [ + lf.to_string(), + lf.replace('\n', "\r\n"), + format!("\u{feff}{lf}"), + format!("\u{feff}{}", lf.replace('\n', "\r\n")), + lf.trim_end().to_string(), + lf.trim_end().replace('\n', "\r\n"), + format!("{lf}\n"), + lf.replace(" ", "\t"), + lf.replace(" ", " ").replace('\n', "\r\n"), + ]; + for text in shapes { + let value = parse_json_manifest(text.as_bytes()).unwrap(); + let out = JsonLayout::of(&text).render(&value).unwrap(); + assert_eq!(String::from_utf8(out).unwrap(), text, "{text:?}"); + } + } + + /// An edit lands in the file's layout; a single-line file becomes a + /// pretty LF one (no line ending to inherit, and none from the OS); a + /// mixed file takes the ending yarn would rewrite it with (majority, + /// ties LF); a newline inside a string value stays escaped. + #[test] + fn json_layout_renders_edits_in_the_file_layout() { + let render = |text: &str, value: serde_json::Value| { + String::from_utf8(JsonLayout::of(text).render(&value).unwrap()).unwrap() + }; + let value = serde_json::json!({ "a": "x\ny", "b": 1 }); + assert_eq!( + render("\u{feff}{\r\n \"a\": 0\r\n}\r\n", value.clone()), + "\u{feff}{\r\n \"a\": \"x\\ny\",\r\n \"b\": 1\r\n}\r\n" + ); + assert_eq!( + render("{\"a\":0}", value.clone()), + "{\n \"a\": \"x\\ny\",\n \"b\": 1\n}" + ); + assert_eq!( + render("{\r\n \"a\": 0,\r\n \"c\": 2\n}\r\n", value.clone()), + "{\r\n \"a\": \"x\\ny\",\r\n \"b\": 1\r\n}\r\n", + "majority CRLF" + ); + assert_eq!( + render("{\r\n \"a\": 0\n}", value), + "{\n \"a\": \"x\\ny\",\n \"b\": 1\n}", + "a tie is LF" + ); + } + + #[test] + fn parse_json_manifest_reads_past_one_bom_only() { + assert_eq!( + parse_json_manifest(b"\xef\xbb\xbf{\"a\":1}").unwrap(), + serde_json::json!({ "a": 1 }) + ); + assert!(parse_json_manifest(b"\xef\xbb\xbf\xef\xbb\xbf{}").is_err()); + assert!(parse_json_manifest(b"{} \xef\xbb\xbf").is_err()); + } + /// `[project] dependencies` and every optional-dependencies extra, by /// PEP 508 name; groups, tool tables, non-array extras and non-string /// members are not PEP 621 declarations. diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs b/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs index 98eb1e11..90c2047a 100644 --- a/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs +++ b/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs @@ -930,6 +930,34 @@ async fn yarn_berry_registry_resolutions_inventory_with_checksums() { assert!(!entries.iter().any(|e| e.name == "fixture"), "{entries:?}"); } +/// yarn berry writes a CRLF `yarn.lock` on Windows (a new lockfile gets +/// `os.EOL`), and editors add a BOM: the Windows spellings — header-less +/// too — inventory exactly like the LF lock, with no stray `\r` riding into +/// a checksum pin. +#[tokio::test] +async fn yarn_berry_crlf_and_bom_locks_inventory_like_their_lf_twin() { + let tmp = tempfile::tempdir().unwrap(); + write(tmp.path(), "yarn.lock", YARN_BERRY).await; + let (_, lf) = inventory_npm_lock(tmp.path()).await.unwrap().unwrap(); + let headerless = YARN_BERRY.trim_start_matches(|c| c != '_'); + for lock in [ + YARN_BERRY.replace('\n', "\r\n"), + format!("\u{feff}{}", YARN_BERRY.replace('\n', "\r\n")), + format!("\u{feff}{}", headerless.replace('\n', "\r\n")), + ] { + let tmp = tempfile::tempdir().unwrap(); + write(tmp.path(), "yarn.lock", &lock).await; + let (flavor, entries) = inventory_npm_lock(tmp.path()).await.unwrap().unwrap(); + assert_eq!(flavor, NpmLockFlavor::YarnBerry, "{lock:?}"); + assert_eq!(sorted_pairs(&entries), sorted_pairs(&lf), "{lock:?}"); + assert_eq!( + entry(&entries, "left-pad").integrity, + LockIntegrity::BerryChecksum("10c0/deadbeefcafe==".into()), + "no stray \\r in the pin: {lock:?}" + ); + } +} + // ── bun ─────────────────────────────────────────────────────────────── const BUN_LOCK: &str = r#"{ diff --git a/crates/socket-patch-core/src/vendor/mod.rs b/crates/socket-patch-core/src/vendor/mod.rs index c0b3aed8..d331a849 100644 --- a/crates/socket-patch-core/src/vendor/mod.rs +++ b/crates/socket-patch-core/src/vendor/mod.rs @@ -105,10 +105,13 @@ pub use state::{ carry_forward_wiring, load_state, lookup_entry, save_state, VendorEntry, VendorState, VENDOR_STATE_REL, }; +// The hosted→vendored takeover refuses a berry project the backend would +// refuse BEFORE it reverts the hosted redirect. pub use verify::{ artifact_is_file_shaped, check_vendored_artifact, compute_dir_inventory, file_sha256_hex, ArtifactHealth, }; +pub use yarn_berry_lock::yarn_berry_vendor_preflight; use std::collections::{HashMap, HashSet}; use std::path::Path; diff --git a/crates/socket-patch-core/src/vendor/npm_flavor.rs b/crates/socket-patch-core/src/vendor/npm_flavor.rs index e2c5a8c2..41176de7 100644 --- a/crates/socket-patch-core/src/vendor/npm_flavor.rs +++ b/crates/socket-patch-core/src/vendor/npm_flavor.rs @@ -284,7 +284,13 @@ async fn read_lock(project_root: &Path, name: &str) -> Result Result { let text = read_lock(project_root, "yarn.lock").await?; - let head: Vec<&str> = text.lines().take(YARN_SNIFF_HEAD_LINES).collect(); + // CRLF lines split like LF ones; a leading BOM is not key text. + let head: Vec<&str> = text + .strip_prefix('\u{feff}') + .unwrap_or(&text) + .lines() + .take(YARN_SNIFF_HEAD_LINES) + .collect(); // Berry wins the check (it must never be mistaken for classic). The // node-modules linker keeps packages on disk for staging, and berry's // cache-zip checksum is reproducible from our tarball (berry_zip), so the @@ -709,6 +715,25 @@ mod tests { touch(tmp.path(), "yarn.lock", "garbage: true\n").await; let (code, _) = detect_npm_lock_flavor(tmp.path()).await.unwrap_err(); assert_eq!(code, "vendor_lockfile_version_unsupported"); + + // Windows spellings sniff the same: CRLF lines, and a BOM right in + // front of a header-less `__metadata:` / the v1 comment. + for (lock, want) in [ + (YARN_BERRY.replace('\n', "\r\n"), NpmLockFlavor::YarnBerry), + ( + format!("\u{feff}{}", YARN_BERRY.trim_start_matches(|c| c != '_')), + NpmLockFlavor::YarnBerry, + ), + ( + format!("\u{feff}{}", YARN_V1.replace('\n', "\r\n")), + NpmLockFlavor::YarnClassic, + ), + ] { + let tmp = tempfile::tempdir().unwrap(); + touch(tmp.path(), "yarn.lock", &lock).await; + let (flavor, _) = detect_npm_lock_flavor(tmp.path()).await.unwrap(); + assert_eq!(flavor, want, "{lock:?}"); + } } #[tokio::test] diff --git a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs index 0bc89071..04eb8fa9 100644 --- a/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_berry_lock.rs @@ -27,6 +27,17 @@ //! the package.json edit is unwound when the lock write fails — a resolutions //! entry without its lock counterpart would make a plain `yarn install` //! re-resolve and rewrite the lock underneath the user. +//! +//! Line endings: yarn writes a file it creates with `os.EOL` — so on Windows +//! BOTH files are CRLF (a fresh `yarn.lock`, and a `package.json` it first +//! pretty-prints) — and keeps each file's majority ending on every later +//! write; a `core.autocrlf` checkout makes them CRLF on any OS. The lock +//! entry is spliced in the file's own terminator and `package.json` is +//! re-serialized in its own layout ([`JsonLayout`]: BOM, indent, terminator, +//! trailing newline), so vendor + revert round-trip both byte-exactly. A +//! file mixing CRLF and LF is refused before any write +//! (`vendor_yarn_berry_mixed_line_endings`, see +//! [`refuse_mixed_line_endings`]); a revert never refuses on line endings. use std::path::Path; @@ -39,11 +50,12 @@ use crate::patch::apply::{normalize_file_path, PatchSources}; use crate::utils::fs::{ atomic_write_bytes_preserving_mode, read_regular_to_bytes, read_regular_to_string, }; +use crate::utils::line_endings::LineEndings; use crate::utils::socket_dir::remove_tree_and_prune; use crate::utils::uri::encode_uri_component; use super::berry_zip::berry_cache_checksum_10c0; -use super::common::{already_patched_result, detect_eol, detect_indent, refused, serialize_json}; +use super::common::{already_patched_result, parse_json_manifest, refused, JsonLayout}; use super::npm_common::{ done_failure_unstage, guard_coordinates, guard_revert_uuid_dir, stage_patch_pack, tgz_rel_leaf, }; @@ -52,7 +64,7 @@ use super::state::{ write_marker_or_warn, VendorArtifact, VendorEntry, VendorMarker, WiringAction, WiringRecord, }; use super::yarn_classic_lock::{ - body_field_line, lines_to_json, pattern_real_name, read_yarn_lock, replace_block, + block_eol, body_field_line, lines_to_json, pattern_real_name, read_yarn_lock, replace_block, revert_recorded_block, scan_blocks, split_berry_key_patterns, split_pattern, LockBlock, }; use super::{RevertOpts, RevertOutcome, VendorOutcome, VendorWarning}; @@ -103,52 +115,19 @@ pub async fn vendor_yarn_berry( Ok(t) => t, Err(outcome) => return *outcome, }; + // A uniformly CRLF lock (what yarn writes on Windows) is spliced in its + // own line ending below; only a mixed one is refused. + if let Some(outcome) = refuse_mixed_line_endings(YARN_LOCK, &lock_text) { + return outcome; + } let blocks = scan_blocks(&lock_text); - let Some(meta) = berry_metadata(&blocks) else { - return refused( - "vendor_lockfile_version_unsupported", - "yarn.lock has no `__metadata:` entry — not a yarn berry lockfile".to_string(), - ); - }; - let cache_key = berry_field(&meta.lines, "cacheKey").unwrap_or(""); - if cache_key != SUPPORTED_CACHE_KEY { - // The checksum is sha512 of the cache archive, whose bytes depend on - // the cache format version + compression; only 10c0 (stored entries) - // is reproducible offline. Emitting a guess would brick installs - // with YN0018, so refuse. - return refused( - "vendor_yarn_berry_cache_unsupported", - format!( - "yarn.lock cacheKey is `{cache_key}`; only `{SUPPORTED_CACHE_KEY}` (yarn 4 \ - with compressionLevel 0, the default) has an offline-reproducible cache \ - checksum — remove custom compression settings and re-run `yarn install`" - ), - ); + if let Some(outcome) = refuse_unsupported_cache(&blocks) { + return outcome; } // ── 3. .yarnrc.yml knobs that change the checksum (spike B4) ───────── - match read_regular_to_string(&project_root.join(YARNRC)).await { - Ok(rc) => { - if let Some(level) = yarnrc_compression_level(&rc) { - if level != "0" { - return refused( - "vendor_yarn_berry_cache_unsupported", - format!( - "{YARNRC} sets `compressionLevel: {level}`, which changes berry's \ - cache checksums; only compressionLevel 0 (the yarn 4 default) is \ - supported" - ), - ); - } - } - } - Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} - Err(e) => { - return refused( - "vendor_yarn_berry_cache_unsupported", - format!("cannot read {YARNRC} to verify the cache configuration: {e}"), - ); - } + if let Some(outcome) = refuse_unsupported_compression(project_root).await { + return outcome; } // ── 4. Root workspace name (the lock key/resolution embed it) ──────── @@ -172,7 +151,13 @@ pub async fn vendor_yarn_berry( ); } }; - let pkg: Value = match serde_json::from_slice(&pkg_bytes) { + // Its layout (BOM, indent, line ending, trailing newline) carries into + // the rewritten bytes; a mixed-ending file has none to carry. + let pkg_text = String::from_utf8_lossy(&pkg_bytes); + if let Some(outcome) = refuse_mixed_line_endings(PACKAGE_JSON, &pkg_text) { + return outcome; + } + let pkg: Value = match parse_json_manifest(&pkg_bytes) { Ok(v) => v, Err(e) => { return refused( @@ -419,8 +404,7 @@ pub async fn vendor_yarn_berry( }; res_obj.insert(name.to_string(), Value::String(spec.clone())); } - let pkg_indent = detect_indent(&String::from_utf8_lossy(&pkg_bytes)); - let new_pkg_bytes = match serialize_json(&new_pkg, &pkg_indent) { + let new_pkg_bytes = match JsonLayout::of(&pkg_text).render(&new_pkg) { Ok(b) => b, Err(e) => { return done_failure_unstage( @@ -433,7 +417,12 @@ pub async fn vendor_yarn_berry( .await } }; - let new_lock_text = replace_block(&lock_text, target, &new_lines, detect_eol(&lock_text)); + let new_lock_text = replace_block( + &lock_text, + target, + &new_lines, + block_eol(&lock_text, target), + ); if let Err(e) = commit_pair( project_root, &new_pkg_bytes, @@ -632,7 +621,7 @@ pub async fn revert_yarn_berry_opts( let pkg_path = project_root.join(PACKAGE_JSON); match read_regular_to_bytes(&pkg_path).await { Ok(bytes) => { - let mut pkg: Value = match serde_json::from_slice(&bytes) { + let mut pkg: Value = match parse_json_manifest(&bytes) { Ok(v) => v, // Fail-closed: rewriting a manifest we cannot parse // risks destroying it. @@ -653,8 +642,10 @@ pub async fn revert_yarn_berry_opts( ); } if changed { - let indent = detect_indent(&String::from_utf8_lossy(&bytes)); - match serialize_json(&pkg, &indent) { + // The file's current layout — the one vendoring kept — + // so the restore lands byte-identical on the pre-vendor + // bytes (CRLF, BOM and trailing newline included). + match JsonLayout::of(&String::from_utf8_lossy(&bytes)).render(&pkg) { Ok(out) => { if let Err(e) = atomic_write_bytes_preserving_mode(&pkg_path, &out).await @@ -829,6 +820,123 @@ fn revert_resolution_record( // ───────────────────────────── vendor internals ───────────────────────────── +/// The pre-write refusal for a berry file (`yarn.lock` / `package.json`) +/// whose line endings mix CRLF and LF, or that holds a bare CR. +/// +/// yarn berry keeps ONE line ending per file: a new file gets `os.EOL` +/// (CRLF on Windows) and every later write re-renders the whole file in its +/// majority ending (`normalizeLineEndings` in yarnpkg-fslib `FakeFS.ts`, +/// used by `Project.persistLockfile` and `Workspace.persistManifest`). A +/// uniformly CRLF or LF file is therefore spliced (yarn.lock) or +/// re-serialized (package.json) in its own ending; a mixed one has no +/// ending to keep — and `yarn install --immutable` already rejects a mixed +/// lock (YN0028), because the re-render differs from the file. `yarn +/// install` normalizes both files, after which vendoring proceeds. +fn refuse_mixed_line_endings(file: &str, text: &str) -> Option { + (LineEndings::of(text) == LineEndings::Mixed).then(|| { + refused( + "vendor_yarn_berry_mixed_line_endings", + format!( + "{file} mixes CRLF and LF line endings (or holds a bare carriage return), so \ + no single line ending can be kept — yarn rewrites the file with one ending \ + on its next install and rejects a lockfile like this under `--immutable` \ + (YN0028); run `yarn install` once to normalize it, then re-run" + ), + ) + }) +} + +/// The `__metadata` / `cacheKey` gate: the checksum is sha512 of the cache +/// archive, whose bytes depend on the cache format version + compression; +/// only 10c0 (stored entries) is reproducible offline. Emitting a guess would +/// brick installs with YN0018, so refuse. +fn refuse_unsupported_cache(blocks: &[LockBlock]) -> Option { + let Some(meta) = berry_metadata(blocks) else { + return Some(refused( + "vendor_lockfile_version_unsupported", + "yarn.lock has no `__metadata:` entry — not a yarn berry lockfile".to_string(), + )); + }; + let cache_key = berry_field(&meta.lines, "cacheKey").unwrap_or(""); + (cache_key != SUPPORTED_CACHE_KEY).then(|| { + refused( + "vendor_yarn_berry_cache_unsupported", + format!( + "yarn.lock cacheKey is `{cache_key}`; only `{SUPPORTED_CACHE_KEY}` (yarn 4 \ + with compressionLevel 0, the default) has an offline-reproducible cache \ + checksum — remove custom compression settings and re-run `yarn install`" + ), + ) + }) +} + +/// The `.yarnrc.yml` `compressionLevel` gate (spike B4): any level but 0 +/// changes berry's cache checksums. +async fn refuse_unsupported_compression(project_root: &Path) -> Option { + match read_regular_to_string(&project_root.join(YARNRC)).await { + Ok(rc) => yarnrc_compression_level(&rc) + .filter(|level| *level != "0") + .map(|level| { + refused( + "vendor_yarn_berry_cache_unsupported", + format!( + "{YARNRC} sets `compressionLevel: {level}`, which changes berry's \ + cache checksums; only compressionLevel 0 (the yarn 4 default) is \ + supported" + ), + ) + }), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => Some(refused( + "vendor_yarn_berry_cache_unsupported", + format!("cannot read {YARNRC} to verify the cache configuration: {e}"), + )), + } +} + +/// The project-level refusals [`vendor_yarn_berry`] raises before any +/// write, whatever the purl: mixed line endings in yarn.lock or +/// package.json, an unsupported `cacheKey`, a non-zero `.yarnrc.yml` +/// `compressionLevel`. `None` unless the project's npm flavor is yarn berry +/// (the probe `vendor_npm_any` routes on) and every gate passes. +/// +/// For the hosted→vendored mode takeover (`vendor`, `scan`/`get --mode +/// vendored` over a hosted-redirected purl): the takeover reverts the +/// hosted lock edits and drops the redirect-ledger record BEFORE this +/// backend runs, and a hosted revert keeps a mixed lock mixed — so without +/// this preflight a refusal here landed after the hosted redirect was gone, +/// leaving the package unpatched in both modes. Returns `(code, detail)`, +/// exactly the refusal the backend would raise. +pub async fn yarn_berry_vendor_preflight(project_root: &Path) -> Option<(&'static str, String)> { + use super::npm_flavor::{detect_npm_lock_flavor, NpmLockFlavor}; + if !matches!( + detect_npm_lock_flavor(project_root).await, + Ok((NpmLockFlavor::YarnBerry, _)) + ) { + return None; + } + let into_pair = |outcome: VendorOutcome| match outcome { + VendorOutcome::Refused { code, detail } => Some((code, detail)), + _ => None, + }; + // An unreadable file is left to the backend's own refusal. + let lock_text = read_yarn_lock(project_root).await.ok()?; + if let Some(outcome) = refuse_mixed_line_endings(YARN_LOCK, &lock_text) { + return into_pair(outcome); + } + if let Some(outcome) = refuse_unsupported_cache(&scan_blocks(&lock_text)) { + return into_pair(outcome); + } + if let Some(outcome) = refuse_unsupported_compression(project_root).await { + return into_pair(outcome); + } + let pkg_bytes = read_regular_to_bytes(&project_root.join(PACKAGE_JSON)) + .await + .ok()?; + refuse_mixed_line_endings(PACKAGE_JSON, &String::from_utf8_lossy(&pkg_bytes)) + .and_then(into_pair) +} + /// Commit the pair in contract order — package.json first, yarn.lock second /// — unwinding package.json to its original bytes when the lock write fails /// (a resolutions entry without its lock counterpart would let a plain @@ -1067,7 +1175,12 @@ fn root_workspace_name(blocks: &[LockBlock]) -> Option { /// enough: yarn writes the knob as a top-level scalar (spike B4), and any /// value we cannot positively read as `0` makes the caller refuse. Shared /// with the hosted-redirect rewriter, whose cache-checksum gate is identical. +/// CRLF lines split like LF ones (`str::lines`), and a leading BOM is +/// skipped the way yarn's YAML parser skips it — otherwise a knob on the +/// first line of a BOM'd file would read as unset (the offline-reproducible +/// default) while yarn applies it and every install fails YN0018. pub(crate) fn yarnrc_compression_level(rc: &str) -> Option<&str> { + let rc = rc.strip_prefix('\u{feff}').unwrap_or(rc); rc.lines().find_map(|line| { let rest = line.strip_prefix("compressionLevel:")?; Some(rest.trim().trim_matches(['\'', '"'])) @@ -3321,6 +3434,254 @@ __metadata: ); } + /// `text` with every line break respelled CRLF (idempotent). + fn crlf(text: &str) -> String { + text.replace("\r\n", "\n").replace('\n', "\r\n") + } + + /// The Windows shape — yarn writes a new `yarn.lock` and the + /// `package.json` it first pretty-prints with `os.EOL` (CRLF) — plus a + /// BOM and a missing trailing newline: vendoring keeps each file's + /// layout (the spike's LF oracle bytes in that layout), the ledger + /// records terminator-free lines, the re-run is in sync, and revert + /// lands byte-exactly on the pre-vendor files. + #[tokio::test] + async fn crlf_bom_and_newline_shapes_vendor_and_revert_byte_exact() { + for (label, pkg_shape, lock_shape) in [ + ( + "crlf", + crlf as fn(&str) -> String, + crlf as fn(&str) -> String, + ), + ( + "bom+crlf", + (|t: &str| format!("\u{feff}{}", crlf(t))) as fn(&str) -> String, + (|t: &str| format!("\u{feff}{}", crlf(t))) as fn(&str) -> String, + ), + ( + "bom+lf pkg, lf lock", + (|t: &str| format!("\u{feff}{t}")) as fn(&str) -> String, + (|t: &str| t.to_string()) as fn(&str) -> String, + ), + ( + "crlf pkg without final newline, crlf lock", + (|t: &str| crlf(t.strip_suffix('\n').unwrap())) as fn(&str) -> String, + crlf as fn(&str) -> String, + ), + ] { + let pkg_before = pkg_shape(B3_BEFORE_PKG); + let lock_before = lock_shape(B3_BEFORE_LOCK); + let fx = fixture_with(&pkg_before, &lock_before).await; + let (result, entry, warnings) = expect_done(fx.vendor(false).await); + assert!(result.success, "{label}: {:?}", result.error); + assert!(warnings.is_empty(), "{label}: {warnings:?}"); + let entry = entry.expect("a ledger entry"); + + let (hash6, checksum) = fx.packed_berry_facts().await; + assert_eq!( + tokio::fs::read_to_string(fx.pkg_path()).await.unwrap(), + pkg_shape(B3_AFTER_PKG), + "{label}: package.json keeps its layout" + ); + assert_eq!( + tokio::fs::read_to_string(fx.lock_path()).await.unwrap(), + lock_shape(&spike_after_lock(&hash6, &checksum)), + "{label}: yarn.lock keeps its line endings and BOM" + ); + // Ledger lines carry no terminators, whatever the file's ending. + let lock_rec = entry + .wiring + .iter() + .find(|w| w.kind == KIND_LOCK_ENTRY) + .unwrap(); + for v in [&lock_rec.original, &lock_rec.new] { + let text = serde_json::to_string(v).unwrap(); + assert!(!text.contains("\\r"), "{label}: {text}"); + } + assert_eq!( + lock_rec.original.as_ref().unwrap()[0], + json!("\"left-pad@npm:1.3.0\":"), + "{label}: the BOM never leaks into a recorded key" + ); + + let (pkg_wired, lock_wired) = ( + tokio::fs::read(fx.pkg_path()).await.unwrap(), + tokio::fs::read(fx.lock_path()).await.unwrap(), + ); + let (result, again, _) = expect_done(fx.vendor(false).await); + assert!(result.success, "{label}: {:?}", result.error); + assert!(again.is_none(), "{label}: the re-run is in sync"); + assert_eq!(tokio::fs::read(fx.pkg_path()).await.unwrap(), pkg_wired); + assert_eq!(tokio::fs::read(fx.lock_path()).await.unwrap(), lock_wired); + + let outcome = revert_yarn_berry(&entry, fx.root(), false).await; + assert!(outcome.success, "{label}: {:?}", outcome.error); + assert!( + outcome.warnings.is_empty(), + "{label}: {:?}", + outcome.warnings + ); + assert_eq!( + tokio::fs::read(fx.pkg_path()).await.unwrap(), + pkg_before.as_bytes(), + "{label}: package.json restored byte-exactly" + ); + assert_eq!( + tokio::fs::read(fx.lock_path()).await.unwrap(), + lock_before.as_bytes(), + "{label}: yarn.lock restored byte-exactly" + ); + assert!(!fx.tgz_path().exists(), "{label}: artifact removed"); + } + } + + /// A lock or manifest mixing CRLF and LF (or holding a bare CR) has no + /// single line ending to keep: vendoring refuses before any write, with + /// a code and a detail naming the file and the `yarn install` remedy. + #[tokio::test] + async fn mixed_line_endings_refuse_before_any_write() { + let half = |t: &str| { + let c = crlf(t); + let at = c.rfind("\r\n").unwrap(); + format!("{}\n{}", &c[..at], &c[at + 2..]) + }; + for (file, pkg, lock) in [ + (YARN_LOCK, B3_BEFORE_PKG.to_string(), half(B3_BEFORE_LOCK)), + ( + YARN_LOCK, + B3_BEFORE_PKG.to_string(), + B3_BEFORE_LOCK.replacen("proceed with", "proceed\rwith", 1), + ), + (PACKAGE_JSON, half(B3_BEFORE_PKG), crlf(B3_BEFORE_LOCK)), + ] { + let fx = fixture_with(&pkg, &lock).await; + let detail = expect_refused( + fx.vendor(false).await, + "vendor_yarn_berry_mixed_line_endings", + ); + assert!( + detail.starts_with(file) && detail.contains("yarn install"), + "{file}: {detail}" + ); + fx.assert_untouched().await; + } + } + + /// The takeover preflight raises exactly the project-level refusal the + /// backend raises (same code, same detail) — the hosted→vendored + /// takeover relies on it to refuse BEFORE reverting the hosted redirect + /// — and stays silent on a supported pair (CRLF included) and on a + /// project whose npm flavor is not yarn berry. + #[tokio::test] + async fn takeover_preflight_raises_the_backends_project_refusals() { + let half = |t: &str| { + let c = crlf(t); + let at = c.rfind("\r\n").unwrap(); + format!("{}\n{}", &c[..at], &c[at + 2..]) + }; + for (label, pkg, lock, yarnrc) in [ + ( + "mixed lock", + crlf(B3_BEFORE_PKG), + half(B3_BEFORE_LOCK), + None, + ), + ( + "mixed package.json", + half(B3_BEFORE_PKG), + crlf(B3_BEFORE_LOCK), + None, + ), + ( + "compressionLevel", + crlf(B3_BEFORE_PKG), + crlf(B3_BEFORE_LOCK), + Some("compressionLevel: 9\r\n"), + ), + ] { + let fx = fixture_with(&pkg, &lock).await; + if let Some(rc) = yarnrc { + tokio::fs::write(fx.root().join(YARNRC), rc).await.unwrap(); + } + let (code, detail) = yarn_berry_vendor_preflight(fx.root()) + .await + .unwrap_or_else(|| panic!("{label}: the preflight must refuse")); + let backend = expect_refused(fx.vendor(false).await, code); + assert_eq!(detail, backend, "{label}: the backend's own detail"); + fx.assert_untouched().await; + } + for (label, pkg, lock) in [ + ("lf", B3_BEFORE_PKG.to_string(), B3_BEFORE_LOCK.to_string()), + ("crlf", crlf(B3_BEFORE_PKG), crlf(B3_BEFORE_LOCK)), + ( + "classic", + B3_BEFORE_PKG.to_string(), + "# yarn lockfile v1\n\n\n\"left-pad@1.3.0\":\n version \"1.3.0\"\n".to_string(), + ), + ] { + let fx = fixture_with(&pkg, &lock).await; + assert_eq!( + yarn_berry_vendor_preflight(fx.root()).await, + None, + "{label}: nothing to refuse" + ); + } + } + + /// Revert never refuses on line endings. A lock mixed AFTER vendoring + /// (an editor saving one line LF into a CRLF lock) restores the entry in + /// the terminator of the block it replaces, every other byte kept; a + /// BOM added to package.json since vendoring stays. + #[tokio::test] + async fn revert_keeps_foreign_line_endings_and_a_later_bom() { + let fx = fixture_with(&crlf(B3_BEFORE_PKG), &crlf(B3_BEFORE_LOCK)).await; + let (_, entry, _) = expect_done(fx.vendor(false).await); + let entry = entry.unwrap(); + let wired = tokio::fs::read_to_string(fx.lock_path()).await.unwrap(); + let mixed = wired.replacen("proceed with caution!\r\n", "proceed with caution!\n", 1); + assert_ne!(mixed, wired, "the fixture edit must hit"); + tokio::fs::write(fx.lock_path(), &mixed).await.unwrap(); + let pkg = tokio::fs::read_to_string(fx.pkg_path()).await.unwrap(); + tokio::fs::write(fx.pkg_path(), format!("\u{feff}{pkg}")) + .await + .unwrap(); + + let outcome = revert_yarn_berry(&entry, fx.root(), false).await; + assert!(outcome.success, "{:?}", outcome.error); + assert_eq!( + tokio::fs::read_to_string(fx.lock_path()).await.unwrap(), + crlf(B3_BEFORE_LOCK).replacen( + "proceed with caution!\r\n", + "proceed with caution!\n", + 1 + ), + "the entry comes back CRLF; the foreign LF line stays LF" + ); + assert_eq!( + tokio::fs::read_to_string(fx.pkg_path()).await.unwrap(), + format!("\u{feff}{}", crlf(B3_BEFORE_PKG)), + "the resolutions entry is gone; the BOM added since stays" + ); + } + + /// A `.yarnrc.yml` saved with a BOM (and CRLF) still has its first-line + /// `compressionLevel` knob read — yarn applies it, so it must refuse. + #[test] + fn yarnrc_compression_level_reads_past_a_bom_and_crlf() { + assert_eq!( + yarnrc_compression_level("\u{feff}compressionLevel: mixed\r\nnodeLinker: pnp\r\n"), + Some("mixed") + ); + assert_eq!( + yarnrc_compression_level("nodeLinker: pnp\r\ncompressionLevel: 0\r\n"), + Some("0") + ); + assert_eq!( + yarnrc_compression_level("\u{feff}nodeLinker: pnp\r\n"), + None + ); + } + /// yarn 4.0.x spells `10c0` checksums bare, 4.1+ prefixed: a written /// entry follows the lock (an `--immutable` install rejects a respelled /// checksum with YN0028). A lock with no checksum keeps the prefix. diff --git a/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs b/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs index 207632e0..c003c56c 100644 --- a/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs +++ b/crates/socket-patch-core/src/vendor/yarn_classic_lock.rs @@ -546,7 +546,10 @@ pub(super) fn revert_recorded_block( )); return false; }; - replace_block(text, block, &original, detect_eol(text)) + // The recorded lines carry no terminators: the restored block takes + // the one the live block is written in, so a lock whose endings + // were mixed since vendoring keeps every other line as it is. + replace_block(text, block, &original, block_eol(text, block)) }; *text = edit; true @@ -727,20 +730,29 @@ pub(crate) struct LockBlock { } /// Scan a lockfile into blocks, CRLF-aware. Comments, blank lines, and -/// anything else outside blocks are left to the splicer untouched. +/// anything else outside blocks are left to the splicer untouched. A +/// leading UTF-8 BOM is encoding, not text (yarn's parsers drop it): it is +/// stripped from the first line and kept OUT of that line's span, so a +/// header-less lock still yields its first key and a splice keeps the BOM. pub(crate) fn scan_blocks(text: &str) -> Vec { // (start, end-incl-terminator, content-without-terminator, terminated) let mut lines: Vec<(usize, usize, &str, bool)> = Vec::new(); let mut pos = 0; for seg in text.split_inclusive('\n') { - let start = pos; + let mut start = pos; pos += seg.len(); let terminated = seg.ends_with('\n'); let mut content = seg; if terminated { content = &content[..content.len() - 1]; } - let content = content.strip_suffix('\r').unwrap_or(content); + let mut content = content.strip_suffix('\r').unwrap_or(content); + if start == 0 { + if let Some(rest) = content.strip_prefix('\u{feff}') { + start = '\u{feff}'.len_utf8(); + content = rest; + } + } lines.push((start, pos, content, terminated)); } let mut blocks = Vec::new(); @@ -775,6 +787,20 @@ fn is_body_line(s: &str) -> bool { s.starts_with(' ') || s.starts_with('\t') } +/// The line terminator `block` is written in: its first line's (`\r\n` or +/// `\n`), else — a block that is one unterminated last line — the file's +/// dominant one ([`detect_eol`]). For a uniformly-ended lock this is the +/// file's own terminator; in a lock whose endings were mixed after the +/// fact it keeps a restored block in the style of the block it replaces. +pub(super) fn block_eol(text: &str, block: &LockBlock) -> &'static str { + let span = &text[block.start..block.end]; + match span.find('\n') { + Some(i) if span[..i].ends_with('\r') => "\r\n", + Some(_) => "\n", + None => detect_eol(text), + } +} + /// Splice `new_lines` over `block`'s byte range, preserving every byte /// outside it. pub(super) fn replace_block( @@ -2161,6 +2187,33 @@ left-pad@^1.3.0: assert!(classic_field(&blocks[1].lines, "resolved").is_none()); } + /// A leading BOM is not key text: a header-less lock still yields its + /// first key (without the BOM), and a splice of that first block keeps + /// the BOM in place. Each block reports its own terminator, so a + /// restore into a lock mixed after the fact keeps the block's style. + #[test] + fn scan_blocks_skip_a_bom_and_report_each_block_terminator() { + let text = "\u{feff}__metadata:\r\n version: 8\r\n\r\n\"a@npm:1\":\n version: 1\n"; + let blocks = scan_blocks(text); + let keys: Vec<&str> = blocks.iter().map(|b| b.key.as_str()).collect(); + assert_eq!(keys, vec!["__metadata", "\"a@npm:1\""]); + assert_eq!(blocks[0].lines[0], "__metadata:"); + for b in &blocks { + assert_eq!( + replace_block(text, b, &b.lines, block_eol(text, b)), + text, + "{}", + b.key + ); + } + assert_eq!(block_eol(text, &blocks[0]), "\r\n"); + assert_eq!(block_eol(text, &blocks[1]), "\n"); + // An unterminated one-line block falls back to the file's ending. + let last = "x:\r\n v: 1\r\n\r\ny:"; + let blocks = scan_blocks(last); + assert_eq!(block_eol(last, &blocks[1]), "\r\n"); + } + /// A second canonical uuid, distinct from [`UUID`], for re-vendor tests. const UUID_B: &str = "0a1b2c3d-4e5f-4a6b-8c7d-9e0f1a2b3c4d"; diff --git a/docs/ecosystems.md b/docs/ecosystems.md index c5437b8a..8f8dae9e 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -78,7 +78,11 @@ The backticked slug in each row is the value `-e`/`--ecosystems` accepts (e.g. - **yarn berry** — the redirect edits the `yarn.lock` entry only (cacheKey `10c0` / yarn 4), and `.yarnrc.yml`'s `compressionLevel` must stay 0. The node-modules linker is e2e-covered; PnP is untested for hosted — the lock rewrite fires, but PnP's - `.yarn/cache` resolution isn't exercised. + `.yarn/cache` resolution isn't exercised. CRLF locks — what yarn writes on Windows, + and what a `core.autocrlf` checkout produces anywhere — are rewritten in their own + line ending (a BOM is kept); a lock mixing CRLF and LF is refused + (`redirect_yarn_berry_mixed_line_endings`) until `yarn install` normalizes it. See + [yarn berry compatibility](testing/yarn-berry-compatibility.md). - **yarn `npm:` aliases (classic & berry)** — a lock entry that consumes the patched package only through an alias descriptor (`"safe-pad@npm:left-pad@^1.3.0"`) is left untouched, with a `redirect_yarn_classic_alias_skipped` / diff --git a/docs/testing/yarn-berry-compatibility.md b/docs/testing/yarn-berry-compatibility.md new file mode 100644 index 00000000..4ea50420 --- /dev/null +++ b/docs/testing/yarn-berry-compatibility.md @@ -0,0 +1,109 @@ +# yarn berry compatibility + +socket-patch supports yarn berry at cacheKey `10c0` — yarn 4 with the default +`compressionLevel: 0`, the one cache-zip checksum recipe it can reproduce +offline — in both modes: hosted (`scan --mode hosted` rewrites the lock entry +to the hosted `::__archiveUrl=`) and vendored (`vendor` wires the root +`package.json` `resolutions` plus the lock's `file:` entry). yarn 2 and 3 +(cacheKeys `7` / `8`) are refused by both modes. The node-modules and pnpm +linkers are covered end to end; Plug'n'Play keeps packages inside +`.yarn/cache` zips, so `vendor` refuses it (`vendor_yarn_berry_unsupported`) +and so does `apply` (`yarn_pnp_unsupported`), while standalone `vex` still +attests a hosted lock's `checksum:` pin. + +## Test matrix + +The `yarn-berry-e2e` job in `.github/workflows/ci.yml` runs +`scripts/yarn-berry-vex-matrix.sh` with `SOCKET_PATCH_YARN_E2E_REQUIRED=1` +(a toolchain or registry problem fails instead of skipping): + +| OS | yarn | +| --- | --- | +| ubuntu-latest | 4.0.2 (bare-hex checksums), 4.1.0 (first `10c0/` spelling), 4.6.0, 4.12.0, 4.18.0 | +| macos-latest | 4.12.0 | +| windows-latest | 4.12.0 | + +Each release drives four real-yarn suites — `e2e_redirect_yarn_berry_build`, +`e2e_vendor_yarn_berry_build`, `e2e_yarn4_pnpm_linker_build` and +`e2e_yarn4_workspaces_build` — each ending in the manifest-less VEX matrix of +`tests/yarn_berry_common`, plus `e2e_yarn_legacy_cachekey_refusal_build` +against yarn 2.4.3 and 3.8.7. `mode_migration_npm` covers both mode +takeovers against yarn 4.12.0. Hermetic twins of every shape (no toolchain) +live in `in_process_redirect`, `in_process_vendor`, `e2e_vex_lockfile` and the +core unit tests. + +## Line endings + +yarn berry keeps one line ending per file, chosen by the same function for +the lockfile and every manifest. At tag `@yarnpkg/cli/4.12.0` (the same code +ships at 4.0.0 and 4.18.0; the published 3.8.7, 4.0.2, 4.6.0, 4.9.2 and +4.18.0 bundles carry it verbatim, and 2.4.3 applies the same rule through +`changeFilePromise`): + +- `packages/yarnpkg-fslib/sources/FakeFS.ts` (lines 799–812): + `getEndOfLine(content)` returns `os.EOL` when `content` has no line break — + the file is new — and otherwise `\r\n` only when CRLF breaks strictly + outnumber LF ones (a tie is LF); `normalizeLineEndings(original, next)` + respells every break of `next` that way. +- `packages/yarnpkg-core/sources/Project.ts`: `persistLockfile` (lines + 2014–2033) writes the generated lock through `normalizeLineEndings` against + the current file; the `--immutable` check (lines 1789–1858) fails with + YN0028 whenever `normalizeLineEndings(initialLockfile, generateLockfile())` + differs from the file — so a uniformly CRLF lock passes, while a lock with + mixed endings (or a BOM, which the re-render never writes) always fails. +- `packages/yarnpkg-core/sources/Workspace.ts` (lines 217–229): + `persistManifest` writes `JSON.stringify(data, null, indent) + "\n"` through + `changeFilePromise(…, {automaticNewlines: true})`, the same rule, after every + install (`Project.ts` line 1881, `--immutable` included). + `Manifest.loadFromText` strips a BOM when reading (`Manifest.ts` lines + 135–146 and 984–990); the rewrite never writes one back. +- `packages/yarnpkg-parsers/sources/syml.ts`: `parseSyml` reads the lock with + js-yaml, which accepts CRLF. + +So on Windows every yarn berry project starts CRLF: the first `yarn install` +writes a CRLF `yarn.lock`, and a `package.json` yarn pretty-prints for the +first time (any compact one) comes back CRLF. On macOS and Linux both are LF. +An existing file keeps its majority ending on every OS. Git adds its own +path to CRLF: `core.autocrlf=true` (the Git for Windows installer's default) +checks LF-committed text out as CRLF, and so does `core.autocrlf=true` or a +`text eol=crlf` attribute on macOS and Linux; a lock committed with CRLF stays +CRLF in every checkout. + +What socket-patch does with those files: + +| | hosted (`yarn.lock`) | vendored (`yarn.lock` + root `package.json`) | +| --- | --- | --- | +| uniformly LF or CRLF | rewritten in the file's own ending; ledger fragments recorded as on disk | lock entry spliced in the file's ending; `package.json` re-serialized in its own layout (BOM, indent, ending, trailing newline) | +| leading BOM | kept | kept, both files | +| mixed CRLF / LF, or a bare CR | refused untouched: `redirect_yarn_berry_mixed_line_endings` | refused before any write: `vendor_yarn_berry_mixed_line_endings` | +| revert (`rollback`, `remove`, takeovers) | byte-exact; a ledger recorded before a uniform LF ↔ CRLF checkout flip is replayed respelled; a mixed lock refuses as drift | byte-exact; a lock mixed after vendoring gets the restored entry in the terminator of the entry it replaces | +| mode takeover into this mode | the berry gates (line endings, `cacheKey`, `compressionLevel`) run BEFORE the vendored wiring is reverted; a refused purl stays vendored, byte-identical | the backend's project gates (both files' line endings, `cacheKey`, `compressionLevel`) run BEFORE the hosted redirect is reverted; a refused purl stays hosted, byte-identical | + +`setup` / `setup --remove` write `package.json` in the same layout-keeping +way (BOM, indent, ending, trailing newline), so the pair round-trips +byte-exactly on a CRLF manifest too. + +Every reader — manifest-less `vex`, the lockfile inventory, the npm flavor +sniff, `repair` — splits CRLF lines like LF ones and skips a leading BOM. +The shared hosted golden fixtures stay LF: their TypeScript twin in the +depscan backend has no CRLF path yet. + +## Running the suites locally + +```bash +SOCKET_PATCH_YARN_E2E_REQUIRED=1 SOCKET_PATCH_YARN_BERRY_VERSION=4.12.0 \ +COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \ +cargo test -p socket-patch-cli \ + --test e2e_redirect_yarn_berry_build --test e2e_vendor_yarn_berry_build \ + --test e2e_yarn4_pnpm_linker_build --test e2e_yarn4_workspaces_build \ + --test e2e_yarn_legacy_cachekey_refusal_build -- --nocapture +``` + +`SOCKET_PATCH_YARN_BERRY_EOL=crlf` runs the same flows on CRLF files on macOS +or Linux: right after each fixture's first `yarn install`, the files yarn +wrote are respelled CRLF — what yarn itself writes on Windows — and yarn keeps +them CRLF on every later write. Each fixture file prints one +`BERRY-EOL||||yarn=|flow=` line: the ending +yarn wrote (`lf` on macOS and Linux, `crlf` on Windows) and the one the flow +ran on. The `mode_migration_npm` berry legs honor the variable too (run them +with `CI` unset: they do not pin yarn's CI defaults).