Support hosted Pipenv patches and safe vendoring

.github/workflows/pipenv-compatibility.yml

@@ -0,0 +1,100 @@
+name: Pipenv compatibility
+
+# Drives the real CLI and real Pipenv releases through hosted, vendored,
+# agent and out-of-tree agent mode on Linux and macOS (the `test` job already
+# runs the in-process hosted Pipenv CLI tests on all three platforms,
+# Windows included). The pre-2018 releases need Docker and stay in the local
+# matrix (`docs/testing/pipenv-compatibility.md`). Needs network (PyPI + the
+# public patch service); no Socket token.
+
+on:
+  pull_request:
+    paths:
+      - 'crates/socket-patch-core/src/patch/redirect/pipenv.rs'
+      - 'crates/socket-patch-core/src/vendor/pypi_pipenv.rs'
+      - 'crates/socket-patch-core/src/vendor/pypi.rs'
+      - 'crates/socket-patch-core/src/vendor/lock_inventory.rs'
+      - 'crates/socket-patch-core/src/crawlers/python_crawler.rs'
+      - 'crates/socket-patch-core/src/utils/pipenv.rs'
+      - 'crates/socket-patch-cli/src/commands/scan/hosted.rs'
+      - 'crates/socket-patch-cli/src/commands/vendor.rs'
+      - 'crates/socket-patch-cli/src/commands/rollback.rs'
+      - 'crates/socket-patch-cli/src/commands/vex.rs'
+      - 'crates/socket-patch-core/src/patch/redirect/replay.rs'
+      - 'scripts/backtest-pipenv.py'
+      - '.github/workflows/pipenv-compatibility.yml'
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  matrix:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # Every 2018+ major, plus the shapes that exercise the relock and
+          # line-ending paths on the newest and the last Python-3.8 release.
+          - os: ubuntu-latest
+            versions: 2018.11.26 2020.11.15 2021.11.23 2022.12.19 2023.12.1 2024.4.1 2025.1.3 2026.8.0
+            shapes: direct
+          - os: ubuntu-latest
+            versions: 2022.12.19 2026.8.0
+            shapes: crlf marker-excluded extras category
+          - os: macos-latest
+            versions: 2018.11.26 2022.12.19 2023.12.1 2026.8.0
+            shapes: direct
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - name: Install Rust
+        run: rustup show
+      - name: Cache cargo
+        uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          key: pipenv-compat
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.8.0
+      - name: Install the Pythons the Pipenv releases run on
+        run: uv python install 3.8 3.12
+      - name: Build the CLI
+        run: |
+          cargo build --locked -p socket-patch-cli
+          mkdir -p "$RUNNER_TEMP/bin"
+          cp target/debug/socket-patch "$RUNNER_TEMP/bin/socket-patch"
+      - name: Run the Pipenv matrix
+        env:
+          SOCKET_NO_CONFIG: '1'
+          SOCKET_NO_UPDATE_CHECK: '1'
+          # The runners carry whatever patch release uv ships; the harness
+          # only needs one 3.8 and one 3.12.
+          BACKTEST_PY38: '3.8'
+          BACKTEST_PY312: '3.12'
+          PIPENV_VERSIONS: ${{ matrix.versions }}
+          PIPENV_SHAPES: ${{ matrix.shapes }}
+        run: |
+          # shellcheck disable=SC2086
+          python3 scripts/backtest-pipenv.py \
+            --socket-patch "$RUNNER_TEMP/bin/socket-patch" \
+            --socket-patch-revision "$GITHUB_SHA" \
+            --output "$RUNNER_TEMP/pipenv-compat" \
+            --versions $PIPENV_VERSIONS \
+            --shapes $PIPENV_SHAPES \
+            --modes hosted vendored agent agent-oot \
+            --jobs 4
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        if: always()
+        with:
+          name: pipenv-compat-${{ matrix.os }}-${{ strategy.job-index }}
+          path: |
+            ${{ runner.temp }}/pipenv-compat/summary.json
+            ${{ runner.temp }}/pipenv-compat/summary.md
+          if-no-files-found: warn
+          retention-days: 7

CHANGELOG.md

@@ -51,6 +51,43 @@ into the new version's section — see docs/releasing.md.
 
 ### Added
 
+- **Pipenv projects can use hosted patches, and vendored patches keep every
+  category.** `scan --mode hosted` rewrites every `Pipfile.lock` category
+  (`default`, `develop`, Pipenv 2022+ named categories) that pins the patched
+  release to the hosted wheel — `file` references for Pipenv 2018 and later,
+  `path` for 7–11 (probed once with `pipenv --version`; `SOCKET_PIPENV_MAJOR`
+  pins it), pipfile-spec < 6 refused — preserving markers, extras, unrelated
+  entries, the Pipfile and its content hash, with per-entry rollback
+  (`redirect_pipenv_entry`). Vendored mode keeps custom categories and extras,
+  uses `path` for wheels with extras (Pipenv 2022's file-URL bug) and refuses
+  installers older than 2018 (`pypi_pipenv_installer_unsupported`). A stale
+  Pipfile.lock only vetoes the sibling Python rewriters on a real pin/source
+  conflict (`redirect_pipenv_refused`); anything else is
+  `redirect_pipenv_skipped`. Measured across the last stable release of all
+  18 published Pipenv majors — see `docs/testing/pipenv-compatibility.md` and
+  `scripts/backtest-pipenv.py`.
+- **`Pipfile.lock` is inventoried.** Lock-only Pipenv checkouts (a fresh
+  clone with nothing installed) now discover their pins in every mode —
+  hosted redirects them, vendored fetches the pristine wheel by one of the
+  lock's recorded digests (`LockIntegrity::Sha256AnyOf`, resolved through
+  PyPI's JSON API and verified against the same digest) and agent/scan list
+  them as lockfile-only packages. Previously they discovered nothing and
+  exited 0. Socket's own references stay discoverable, so a re-scan of an
+  already-redirected or already-vendored lock-only checkout re-confirms it;
+  a lock that resolves only from private indexes is never looked up on
+  pypi.org.
+- **Pipenv's out-of-tree virtualenv is discovered.** Agent mode (bare `scan`,
+  `rollback`, `vex`) now finds `$WORKON_HOME/<dir>-<hash>[-<python>]` (the
+  `.venv` file pointer, `PIPENV_CUSTOM_VENV_NAME` and `PIPENV_PIPFILE`
+  included) exactly as Pipenv 7 through 2026 place it, instead of falling
+  through to the global interpreter's site-packages.
+- **Pipenv stale-install guard.** Pipenv never reinstalls a release that is
+  already present, so a hosted or vendored rewrite over a warm venv leaves the
+  upstream bytes installed; `redirect_pypi_stale_install` /
+  `pypi_pipenv_stale_install` now say so, naming the site-packages dir and
+  the verified remedy (`pipenv run pip uninstall -y <pkg> && pipenv sync`, or
+  a clean `pipenv --rm && pipenv sync`), and the stale purl is excluded from
+  the same-run `--vex`.
 - **Poetry projects take hosted patches, and vendored patches now cover every
   `poetry.lock` generation.** `scan --mode hosted` rewrites `poetry.lock` to a
   `[package.source] type = "url"` pointing at the Socket-hosted, SHA-256-pinned
@@ -200,6 +237,23 @@ into the new version's section — see docs/releasing.md.
 
 ### Fixed
 
+- **Rollback after a Pipenv relock no longer refuses forever.** `pipenv lock`
+  (and `update`, and `install <other>` before 2024) regenerates a redirected
+  or vendored entry to registry shape on every Pipenv major; that is now the
+  desired end state — the hosted edit retires and the vendored record is
+  dropped (`vendor_lock_entry_relocked`) — instead of a permanent drift
+  refusal that held every pypi revert and kept the orphaned wheel dir. A
+  foreign `file`/`path` reference is still drift.
+- **Same-run `--vex` attests lock-only pypi redirects.** The confirmed purl is
+  unqualified while the ledger records the API's artifact-qualified purl;
+  both sides now match on the qualifier-stripped purl, so a lock-only Pipenv
+  (or uv) checkout no longer exits 1 `no_applicable_patches` after
+  redirecting its lock.
+- **The Pipenv installer probe runs only when a patch targets the lock**, warns
+  only when the lock was actually rewritten, resolves `pipenv` on absolute
+  `PATH` entries only (a relative entry would have executed a `pipenv` planted
+  in the scanned repository), finds `.bat`/`.cmd` shims on Windows, and takes
+  only the token after `version` (never a stray `Python 3.12` banner).
 - Hosted Python redirects now warn when installed files still contain upstream
   or modified bytes and omit those packages from same-run VEX. The read-only
   probe covers Poetry virtualenvs, repeats on re-scans, and uses persisted patch

README.md

@@ -254,6 +254,51 @@ Mode support varies by ecosystem — e.g. Go can't do hosted, Rush monorepos can
 vendored. See the full **[mode × ecosystem matrix](docs/ecosystems.md#mode--ecosystem-matrix)**
 for details and per-ecosystem caveats.
 
+### Pipenv compatibility
+
+Hosted mode rewrites every `Pipfile.lock` category that pins the patched
+release (`default`, `develop`, and Pipenv 2022+ named categories) and
+preserves the Pipfile, its content hash, markers, extras, and unrelated lock
+entries, so `pipenv install --deploy`, `pipenv sync` and `pipenv verify` keep
+passing. The reference shape follows the installing Pipenv: releases 7–11
+need `path` references, 2018 and later use `file` references, and lock
+formats before `pipfile-spec: 6` (Pipenv 0–6) are refused without changing
+the lock. Socket Patch probes `pipenv --version` once per run (only when a
+patch targets the lock); `SOCKET_PIPENV_MAJOR=<major>` pins the answer for
+machines without pipenv on PATH. Hosted references carry both the `#sha256=`
+URL fragment (verified by Pipenv 2023+) and a `hashes` entry (verified by
+2018–2022; Pipenv 11 verifies either), so a tampered lock fails to install
+on every supported release.
+
+Vendored mode requires Pipenv 2018 or later. Wheels with extras use `path`
+references to avoid Pipenv 2022's local-file URL parsing bug. Pipenv 2023+
+does not enforce hashes on local wheels; commit the wheel and run
+`socket-patch vex --product <purl>` (a Pipfile names no project, so pass the
+product purl explicitly).
+
+Fresh checkouts work in every mode: a clone with only `Pipfile` +
+`Pipfile.lock` is discovered from the lock (hosted redirects it, vendored
+fetches the pristine wheel by one of the lock's recorded digests), and agent
+mode finds Pipenv's default out-of-tree virtualenv under `WORKON_HOME`
+without `pipenv run`.
+
+Pipenv never reinstalls a release that is already present: `pipenv install`,
+`pipenv install --deploy` and `pipenv sync` all exit 0 and keep the installed
+bytes, on every Pipenv major. A hosted or vendored rewrite therefore
+protects fresh installs, and Socket Patch warns
+(`redirect_pypi_stale_install` / `pypi_pipenv_stale_install`) when a venv
+still holds the upstream release, with the verified remedy:
+`pipenv run pip uninstall -y <pkg> && pipenv sync` (or `pipenv --rm &&
+pipenv sync`). Do not use `pipenv uninstall <pkg>` for this — it rewrites the
+Pipfile and re-locks the patch away. `pipenv lock` / `pipenv update`
+regenerate the entry to its registry reference (a silent unpatch): re-run
+Socket Patch afterwards; `rollback` retires the stale record cleanly.
+
+`scripts/backtest-pipenv.py` drives the real CLI and the last stable release
+of every published Pipenv major through hosted, vendored, agent and
+out-of-tree agent mode, and `docs/testing/pipenv-compatibility.md` holds the
+measured boundaries and results.
+
 ## Common tasks
 
 ### Patch everything that can be patched