diff --git a/.agents/skills/build-openshell-mxc-windows/SKILL.md b/.agents/skills/build-openshell-mxc-windows/SKILL.md index 5f0d617b08..a38763e070 100644 --- a/.agents/skills/build-openshell-mxc-windows/SKILL.md +++ b/.agents/skills/build-openshell-mxc-windows/SKILL.md @@ -30,7 +30,7 @@ The Windows build lane is implemented by these tracked files: | `tasks/windows.toml` | Mise task entry points for `windows:*` commands. | | `tasks/rust.toml`, `tasks/test.toml`, and `tasks/markdown.toml` | Windows routing for compiler-bearing checks, explicit Unix-only test skips, and Markdown dependency setup. | | `tasks/scripts/windows-msvc.ps1` | PowerShell wrapper that enters the Visual Studio developer environment and invokes Cargo. | -| `.github/workflows/windows-msvc.yml` | Manually dispatched GitHub Actions jobs with architecture-specific Rust caches for x64 and future ARM64 Windows validation. | +| `.github/workflows/windows-msvc.yml` | PR/merge-queue lint and test plus main/manual cache seeding and dependent binary builds on native x64 and ARM64 runners. | | `architecture/windows-msvc-build.md` | Design notes and validation contract. | | `.agents/skills/build-openshell-mxc-windows/` | This skill and companion reference material. | @@ -115,7 +115,7 @@ The lane targets a Windows host with Visual Studio Build Tools and rustup. | Visual C++ ARM64 tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.ARM64 -property installationPath` | Required for native ARM64 check, build, and tests and for x64-to-ARM64 check/build. Tests always require a native runner. | | Visual C++ ARM64 Spectre-mitigated libraries | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Runtimes.ARM64.Spectre -property installationPath` | Required by `regorus` through `msvc_spectre_libs`; the build fails when the selected MSVC toolset lacks `lib\spectre\arm64`. | | Visual C++ Clang tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.Llvm.Clang -property installationPath` | Provides host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for ARM64 crypto dependencies such as `ring` and `aws-lc-sys`. On ARM64, the wrapper uses `VC\Tools\Llvm\Arm64\bin`. | -| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja. The x64-to-ARM64 path adds Ninja to `PATH` for native dependencies but keeps bundled Z3 on CMake's Visual Studio ARM64 generator with native MSVC `cl.exe`. | +| Visual C++ CMake tools | `vswhere -latest -products * -requires Microsoft.VisualStudio.Component.VC.CMake.Project -property installationPath` | Provides CMake and Ninja for native dependencies. The x64-to-ARM64 path adds Ninja to `PATH`; Z3 uses an architecture-specific prebuilt release. | | Windows SDK | `where.exe rc.exe` from a Developer PowerShell | Install an SDK containing target libraries and ARM64 tools. | | Rust via rustup | `rustc --version` | Add each target being validated: `x86_64-pc-windows-msvc` and/or `aarch64-pc-windows-msvc`. The wrapper also adds the selected target. | | mise | `mise --version` | Used as a task runner only. | @@ -135,8 +135,9 @@ from this skill. | `CARGO_TARGET_DIR` | `target` under repo root | Override Cargo output location. Use a short absolute path when x64-to-ARM64 builds approach Windows path-length limits. | | `Z3_LIBRARY_PATH_OVERRIDE` | unset | Directory containing an x64 system `libz3.lib`; not valid for ARM64. | | `Z3_SYS_Z3_HEADER` | unset | Full `z3.h` path required with a system Z3 library. | -| `Z3_SYS_BUNDLED_DIR_OVERRIDE` | pinned source cached under `CARGO_TARGET_DIR` when explicit, otherwise `%LOCALAPPDATA%\OpenShell\cache\z3` | Use an existing Z3 source tree containing `src/api/z3.h`; otherwise the wrapper fetches the pinned revision through Git and sets this automatically. | -| `RUSTC_WRAPPER` | cleared by wrapper | The wrapper clears inherited values because `--skip-tools` does not provision `sccache`. | +| `Z3_SYS_Z3_VERSION` | `4.16.0` | Pinned official prebuilt Z3 release selected by the wrapper. | +| `READ_ONLY_GITHUB_TOKEN` | unset | Optional token for the Z3 release lookup; GitHub Actions supplies `github.token`. | +| `RUSTC_WRAPPER` | inherited | The wrapper resolves an available command to an absolute path. If it is unavailable, the wrapper warns and continues without compiler caching. | Legacy fork variables such as `OPENSHELL_UPSTREAM`, `OPENSHELL_MXC_FORK_DIR`, and `OPENSHELL_MXC_FORK_BRANCH` are no longer part @@ -189,17 +190,19 @@ order: 6. Focused unsupported-driver contract tests. 7. Artifact reporting. -The GitHub Actions jobs use architecture-specific `Swatinem/rust-cache` -entries for the Cargo registry and dependency target artifacts. Failed runs -also save their usable dependency artifacts. The workflow remains manually -dispatched until cache-hit runtimes justify restoring automatic triggers. +The GitHub Actions jobs layer architecture-specific `Swatinem/rust-cache` +entries for Cargo registry and dependency target artifacts with sccache's GHA +backend for cacheable Rust compiler outputs. Failed runs also save their usable +dependency artifacts. Pull-request mirrors and merge queues run Clippy for the +Windows-supported workspace and e2e crates plus Rust tests. Pushes to `main` and +manual dispatches run the same lint and test commands in a cache-seed job, +followed by a dependent release-binary build job. The seed and PR jobs use the +same cache namespaces. The binaries are not uploaded or published. The ARM64 check/build steps in this x64-host contract are cross-builds. The wrapper discovers and adds host-native LLVM and Ninja to `PATH`, requires the ARM64 compiler and Spectre-mitigated libraries, lets ARM64 crypto crates select -`clang-cl`, and keeps bundled Z3 on native MSVC `cl.exe` with CMake's Visual -Studio ARM64 generator. Z3 does not use Ninja because `z3-sys 0.10.9` passes -the MSBuild-only `-m` argument. +`clang-cl`, and downloads the official prebuilt ARM64 Z3 static library. On ARM64 hosts, validate the native ARM64 check, build, and test path. The wrapper rejects test targets that do not match the host architecture, so x64 @@ -209,7 +212,8 @@ commands above on an ARM64 host. The repository-wide `mise run pre-commit` task is also supported on Windows. Its Rust check, Clippy, and test dependencies enter the same MSVC environment -for the native host target and clear inherited `RUSTC_WRAPPER`. Linux glibc +for the native host target and use an inherited compiler wrapper when it is +available. Linux glibc installer tests and Linux service/RPM packaging-asset tests skip explicitly; the Linux build-environment shell-helper test also skips; cross-platform checks continue to run. The blocking Windows Clippy pass excludes unsupported @@ -261,9 +265,9 @@ The focused contract tasks for either native architecture run: windows_builtin_compute_drivers_report_unsupported ``` -These tests are also included in the full x64 workspace test run; the focused -task intentionally re-runs them so unsupported Windows behavior is visible in -the CI report. +These tests are also included in the full x64 workspace test run. The focused +task is available for local diagnosis; GitHub Actions does not re-run it after +the full suite. ## Test Accounting Guidance @@ -293,19 +297,13 @@ Useful log files: | `test-x86_64-pc-windows-msvc-unsupported-*.log` | Focused unsupported-driver contract output. | | `test-aarch64-pc-windows-msvc-unsupported-*.log` | Focused native ARM64 contract output. | -The first bundled-Z3 check or test can spend several minutes in CMake/MSBuild -without much console output because Cargo output is redirected to the log. Look -for native `MSBuild.exe` workers before treating the process as stalled. The -wrapper fetches the pinned Z3 source through Git before Cargo starts. It caches -under an explicitly configured `CARGO_TARGET_DIR`, or under the current user's -local application data directory when Cargo uses its default target tree. -Concurrent commands publish the validated source through an atomic directory -rename, so x64 and ARM64 validation can share the cache safely. The wrapper does -not rely on the rate-limited GitHub Contents API used by `z3-sys`. A failed -fetch reports the partial checkout path for diagnosis. The artifact report -computes SHA256 through .NET directly and does not rely on the -`Get-FileHash` module being available inside the mise-launched Windows -PowerShell process. +The first check downloads the pinned official Z3 archive for the target +architecture through `z3-sys`. GitHub Actions authenticates the lookup with its +read-only workflow token; local users can set `READ_ONLY_GITHUB_TOKEN` if an +unauthenticated lookup is rate-limited. Cargo stores the extracted library in +its target tree, so the Windows target cache reuses it. The artifact report +computes SHA256 through .NET directly and does not rely on the `Get-FileHash` +module being available inside the mise-launched Windows PowerShell process. ## Common Fix Patterns diff --git a/.agents/skills/build-openshell-mxc-windows/reference.md b/.agents/skills/build-openshell-mxc-windows/reference.md index 16b9a48586..f570c04472 100644 --- a/.agents/skills/build-openshell-mxc-windows/reference.md +++ b/.agents/skills/build-openshell-mxc-windows/reference.md @@ -9,7 +9,7 @@ maintaining the existing build-only Windows MSVC lane. |---|---| | `tasks/windows.toml` | Mise task definitions for `windows:*`. | | `tasks/scripts/windows-msvc.ps1` | Visual Studio environment discovery, rustup target setup, Cargo invocation, logs, artifact report. | -| `.github/workflows/windows-msvc.yml` | Manual GitHub Actions x64 job and disabled ARM64 scaffold, each with an architecture-specific Rust dependency cache. | +| `.github/workflows/windows-msvc.yml` | PR/merge-queue lint and test plus main/manual cache seeding and dependent binary builds on native x64 and ARM64 runners. | | `architecture/windows-msvc-build.md` | Human-readable design contract. | ## Commands @@ -46,8 +46,9 @@ if ($arch -eq [System.Runtime.InteropServices.Architecture]::Arm64) { The native test tasks reject a target that does not match the host architecture. Do not report x64 compatibility-under-emulation coverage from an ARM64 run. -The wrapper adds missing rustup targets and clears inherited -`RUSTC_WRAPPER`. It does not install Visual Studio, Rust, Docker, Kubernetes, +The wrapper adds missing rustup targets and preserves an inherited +`RUSTC_WRAPPER` when the command is available. Otherwise, it warns and clears +the setting. It does not install Visual Studio, Rust, Docker, Kubernetes, Podman, WSL, Hyper-V, or VM tooling. On Windows, `mise run pre-commit` routes `rust:check`, `rust:lint`, and @@ -67,11 +68,10 @@ file. For ARM64, verify the Visual Studio instance contains the ARM64 MSVC tools, ARM64 Spectre-mitigated libraries, Clang tools, CMake tools, and a Windows SDK. Clang supplies host-native `libclang.dll` for `bindgen` and `clang-cl.exe` for -ARM64 crypto dependencies such as `ring` and `aws-lc-sys`. Native ARM64 uses -the normal bundled-Z3 CMake path. An x64-to-ARM64 check/build discovers and -adds host-native Ninja to `PATH`, while the crypto crates select `clang-cl`. -Bundled Z3 uses CMake's Visual Studio ARM64 generator with native MSVC `cl.exe` -because `z3-sys 0.10.9` passes the MSBuild-only `-m` argument. Use a short +ARM64 crypto dependencies such as `ring` and `aws-lc-sys`. Native and +x64-to-ARM64 builds use the official prebuilt Z3 4.16.0 static library for the +target architecture. An x64-to-ARM64 check/build discovers and adds host-native +Ninja to `PATH`, while the crypto crates select `clang-cl`. Use a short `CARGO_TARGET_DIR` if Windows path-length limits are reached. ## Unsupported Driver Rules diff --git a/.github/workflows/windows-msvc.yml b/.github/workflows/windows-msvc.yml index 3ad1cc68f0..4ecff4d42c 100644 --- a/.github/workflows/windows-msvc.yml +++ b/.github/workflows/windows-msvc.yml @@ -1,50 +1,199 @@ -name: Windows MSVC (build-only) +name: Windows MSVC + on: + merge_group: + types: [checks_requested] + push: + branches: + - main + - "pull-request/[0-9]+" workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +env: + CARGO_INCREMENTAL: "0" + jobs: - x64: - runs-on: windows-2025 + pr_metadata: + name: Resolve PR metadata + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + outputs: + should_run: ${{ github.ref == 'refs/heads/main' || steps.gate.outputs.should_run == 'true' }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: jdx/mise-action@c2a87611a18de5b3828c5652fe268e992400cb5c # v4.3.0 + + - id: gate + if: github.ref != 'refs/heads/main' + uses: ./.github/actions/pr-gate + + pr-check: + name: PR lint and test (${{ matrix.arch }}) + needs: pr_metadata + if: >- + needs.pr_metadata.outputs.should_run == 'true' && + github.ref != 'refs/heads/main' && + github.event_name != 'workflow_dispatch' + strategy: + fail-fast: false + matrix: + include: + - arch: x64 + runner: windows-2025 + rust_target: x86_64-pc-windows-msvc + - arch: arm64 + runner: windows-11-arm + rust_target: aarch64-pc-windows-msvc + runs-on: ${{ matrix.runner }} + env: + READ_ONLY_GITHUB_TOKEN: ${{ github.token }} + SCCACHE_GHA_ENABLED: "true" + SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}-prebuilt-z3-v2 + SCCACHE_CLIENT_SIDE: "1" + SCCACHE_BASEDIRS: ${{ github.workspace }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: jdx/mise-action@e6a8b3978addb5a52f2b4cd9d91eafa7f0ab959d # v4.2.0 + with: + install: false + experimental: true + - name: Configure GHA sccache backend + uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10 + - uses: dtolnay/rust-toolchain@4be7066ada62dd38de10e7b70166bc74ed198c30 # stable + with: + toolchain: "1.95.0" + targets: ${{ matrix.rust_target }} + components: clippy + - name: Install cargo-nextest + run: mise install --locked github:nextest-rs/nextest + - name: Cache Rust target and registry + uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + env: + # Keep rust-cache's cargo metadata calls independent of the compiler + # wrapper; build steps still inherit sccache from the job. + RUSTC_WRAPPER: "" + with: + # Keep the prebuilt-Z3 experiment isolated from earlier target caches. + shared-key: windows-msvc-${{ matrix.arch }}-prebuilt-z3-v1 + cache-targets: "true" + cache-on-failure: "true" + cache-bin: "false" + - name: Lint Windows-supported crates + run: mise run --skip-tools windows:lint:${{ matrix.arch }} + - name: Test + run: mise run --skip-tools windows:test:${{ matrix.arch }} + - name: sccache stats + if: always() + run: sccache --show-stats + + cache-seed: + name: Seed cache (${{ matrix.arch }}) + needs: pr_metadata + if: >- + needs.pr_metadata.outputs.should_run == 'true' && + (github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch') + strategy: + fail-fast: false + matrix: + include: + - arch: x64 + runner: windows-2025 + rust_target: x86_64-pc-windows-msvc + - arch: arm64 + runner: windows-11-arm + rust_target: aarch64-pc-windows-msvc + runs-on: ${{ matrix.runner }} + env: + READ_ONLY_GITHUB_TOKEN: ${{ github.token }} + SCCACHE_GHA_ENABLED: "true" + SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}-prebuilt-z3-v2 + SCCACHE_CLIENT_SIDE: "1" + SCCACHE_BASEDIRS: ${{ github.workspace }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: jdx/mise-action@e6a8b3978addb5a52f2b4cd9d91eafa7f0ab959d # v4.2.0 with: install: false experimental: true - - uses: dtolnay/rust-toolchain@master + - name: Configure GHA sccache backend + uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10 + - uses: dtolnay/rust-toolchain@4be7066ada62dd38de10e7b70166bc74ed198c30 # stable with: toolchain: "1.95.0" - targets: x86_64-pc-windows-msvc + targets: ${{ matrix.rust_target }} + components: clippy + - name: Install cargo-nextest + run: mise install --locked github:nextest-rs/nextest - name: Cache Rust target and registry uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + env: + RUSTC_WRAPPER: "" with: - shared-key: windows-msvc-x64 + shared-key: windows-msvc-${{ matrix.arch }}-prebuilt-z3-v1 cache-targets: "true" cache-on-failure: "true" cache-bin: "false" - - run: mise run --skip-tools windows:check:x64 - - run: mise run --skip-tools windows:build:x64 - - run: mise run --skip-tools windows:test:x64 - - run: mise run --skip-tools windows:test:unsupported:x64 - arm64: - # TODO: provision a windows-arm64 self-hosted runner - runs-on: [self-hosted, windows-arm64] - if: false # flip to true once the runner is online + - name: Lint Windows-supported crates + # Keep the seed workload identical to the pull-request workload so its + # target and compiler caches are reusable by subsequent pull requests. + run: mise run --skip-tools windows:lint:${{ matrix.arch }} + - name: Test + run: mise run --skip-tools windows:test:${{ matrix.arch }} + - name: sccache stats + if: always() + run: sccache --show-stats + + build: + name: Build binaries (${{ matrix.arch }}) + needs: cache-seed + strategy: + fail-fast: false + matrix: + include: + - arch: x64 + runner: windows-2025 + rust_target: x86_64-pc-windows-msvc + - arch: arm64 + runner: windows-11-arm + rust_target: aarch64-pc-windows-msvc + runs-on: ${{ matrix.runner }} + env: + READ_ONLY_GITHUB_TOKEN: ${{ github.token }} + SCCACHE_GHA_ENABLED: "true" + SCCACHE_GHA_VERSION: windows-msvc-${{ matrix.arch }}-prebuilt-z3-v2 + SCCACHE_CLIENT_SIDE: "1" + SCCACHE_BASEDIRS: ${{ github.workspace }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: jdx/mise-action@c2a87611a18de5b3828c5652fe268e992400cb5c # v4.3.0 + - uses: jdx/mise-action@e6a8b3978addb5a52f2b4cd9d91eafa7f0ab959d # v4.2.0 with: install: false experimental: true - - uses: dtolnay/rust-toolchain@master + - name: Configure GHA sccache backend + uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10 + - uses: dtolnay/rust-toolchain@4be7066ada62dd38de10e7b70166bc74ed198c30 # stable with: toolchain: "1.95.0" - targets: aarch64-pc-windows-msvc + targets: ${{ matrix.rust_target }} - name: Cache Rust target and registry uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + env: + RUSTC_WRAPPER: "" with: - shared-key: windows-msvc-arm64 + shared-key: windows-msvc-${{ matrix.arch }}-prebuilt-z3-v1 cache-targets: "true" cache-on-failure: "true" cache-bin: "false" - - run: mise run --skip-tools windows:check:arm64 - - run: mise run --skip-tools windows:build:arm64 + - name: Build release binaries + run: mise run --skip-tools windows:build:${{ matrix.arch }} + - name: sccache stats + if: always() + run: sccache --show-stats diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ab1fa2e7a5..91c2be4b63 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -338,13 +338,21 @@ compiles Z3 from source during the Rust build and requires CMake 3.16+: cargo build -p openshell-prover --features bundled-z3 ``` -For x86-64 Windows MSVC builds, use one of these Z3 paths: - +For x86-64 and ARM64 Windows MSVC builds, use one of these Z3 paths: + +- Prebuilt Z3 (the default for `windows:*` tasks): `z3-sys` downloads the + pinned Z3 4.16.0 GitHub release for the target architecture on the first + build. Cargo reuses the extracted archive from its target directory. Windows + CI authenticates the GitHub API request with `READ_ONLY_GITHUB_TOKEN` and + preserves the archive in the architecture-specific Cargo target cache. For + cold local builds, you may set `READ_ONLY_GITHUB_TOKEN` to avoid anonymous + GitHub API rate limits. - System Z3: point `Z3_LIBRARY_PATH_OVERRIDE` at the directory containing the - 64-bit MSVC Z3 library and `Z3_SYS_Z3_HEADER` at the full path to `z3.h`. + target-compatible MSVC Z3 library and `Z3_SYS_Z3_HEADER` at the full path to `z3.h`. The `windows:*` tasks use this path automatically when `Z3_LIBRARY_PATH_OVERRIDE` is set. -- Bundled Z3: pass `--features bundled-z3` so `z3-sys` builds Z3 from source. +- Bundled Z3: for direct Cargo builds, pass `--features bundled-z3` so `z3-sys` + builds Z3 from source. `openshell-prover` itself has no `bindgen`/`libclang` dependency, so building just this crate does not require `LIBCLANG_PATH`: @@ -357,7 +365,7 @@ cargo build -p openshell-prover --target x86_64-pc-windows-msvc --features bundl To build the full set of Windows binaries, including `openshell-gateway.exe` and `openshell.exe`, use the `windows:build:x64` mise task instead of a -single-crate `cargo build`. It builds Z3 from source (bundled) by default. A +single-crate `cargo build`. It downloads the pinned prebuilt Z3 release by default. A full build also compiles crates that use `bindgen` (e.g. the MXC driver on Windows), so it requires `libclang.dll`; if LLVM is not on the default search path, set `LIBCLANG_PATH` to the directory containing `libclang.dll`: @@ -367,7 +375,7 @@ $env:LIBCLANG_PATH='C:\Program Files\Microsoft Visual Studio\2022\\VC\T mise run --skip-tools windows:build:x64 ``` -To use a local x64 Z3 release instead of the bundled build, set +To use a local x64 Z3 release instead of the prebuilt download, set `Z3_LIBRARY_PATH_OVERRIDE` and `Z3_SYS_Z3_HEADER` before running the task: ```powershell diff --git a/Cargo.lock b/Cargo.lock index edc940ea68..599c59f353 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -880,6 +880,15 @@ dependencies = [ "either", ] +[[package]] +name = "bzip2" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" +dependencies = [ + "libbz2-rs-sys", +] + [[package]] name = "capctl" version = "0.2.4" @@ -1135,6 +1144,12 @@ dependencies = [ "unicode-xid", ] +[[package]] +name = "constant_time_eq" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" + [[package]] name = "core-foundation" version = "0.10.1" @@ -1467,6 +1482,12 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b" +[[package]] +name = "deflate64" +version = "0.1.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" + [[package]] name = "delegate" version = "0.13.5" @@ -1609,6 +1630,7 @@ dependencies = [ "const-oid 0.10.2", "crypto-common 0.2.2", "ctutils", + "zeroize", ] [[package]] @@ -1916,6 +1938,7 @@ checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" dependencies = [ "crc32fast", "miniz_oxide", + "zlib-rs", ] [[package]] @@ -2492,6 +2515,7 @@ dependencies = [ "tokio", "tokio-rustls", "tower-service", + "webpki-roots", ] [[package]] @@ -3195,6 +3219,12 @@ version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" +[[package]] +name = "libbz2-rs-sys" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c" + [[package]] name = "libc" version = "0.2.189" @@ -3298,6 +3328,15 @@ version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" +[[package]] +name = "lzma-rust2" +version = "0.16.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca93e534d1142d1d0dcca6d25fe302508a5dfb40b302802904577725ea0b695b" +dependencies = [ + "sha2 0.11.0", +] + [[package]] name = "matchers" version = "0.2.0" @@ -5084,6 +5123,12 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" +[[package]] +name = "ppmd-rust" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "efca4c95a19a79d1c98f791f10aebd5c1363b473244630bb7dbde1dc98455a24" + [[package]] name = "ppv-lite86" version = "0.2.21" @@ -5659,6 +5704,7 @@ dependencies = [ "wasm-bindgen", "wasm-bindgen-futures", "web-sys", + "webpki-roots", ] [[package]] @@ -7150,6 +7196,7 @@ checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" dependencies = [ "deranged", "itoa", + "js-sys", "num-conv", "powerfmt", "serde_core", @@ -7654,6 +7701,12 @@ dependencies = [ "thiserror 2.0.18", ] +[[package]] +name = "typed-path" +version = "0.12.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e" + [[package]] name = "typenum" version = "1.20.1" @@ -8717,7 +8770,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c18b0a91a13522d21b3414847667de2b2056a721a3edcb5b6ee6858352d58db4" dependencies = [ "pkg-config", + "reqwest 0.12.28", + "serde_json", "z3-src", + "zip", ] [[package]] @@ -8814,12 +8870,57 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "zip" +version = "8.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b" +dependencies = [ + "aes", + "bzip2", + "constant_time_eq", + "crc32fast", + "deflate64", + "flate2", + "getrandom 0.4.2", + "hmac 0.13.0", + "indexmap", + "lzma-rust2", + "memchr", + "pbkdf2", + "ppmd-rust", + "sha1 0.11.0", + "time", + "typed-path", + "zeroize", + "zopfli", + "zstd", +] + +[[package]] +name = "zlib-rs" +version = "0.6.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b142a20ec14a91d5bc708c1dc21b080c550113d8aa77afa29635673a65dd02c5" + [[package]] name = "zmij" version = "1.0.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" +[[package]] +name = "zopfli" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249" +dependencies = [ + "bumpalo", + "crc32fast", + "log", + "simd-adler32", +] + [[package]] name = "zstd" version = "0.13.3" diff --git a/architecture/windows-msvc-build.md b/architecture/windows-msvc-build.md index 6b0d675a9c..274aae2560 100644 --- a/architecture/windows-msvc-build.md +++ b/architecture/windows-msvc-build.md @@ -54,11 +54,18 @@ creating misleading Windows driver artifacts. ## Mise Lane -The GitHub Actions workflow is manually dispatched. Each architecture restores -and saves a dedicated Rust cache containing the Cargo registry and dependency -build artifacts, including artifacts from failed runs. Keep the workflow manual -until cache-hit runtimes demonstrate that it is suitable for pull requests and -merges to `main`. +The GitHub Actions workflow runs Clippy for the Windows-supported workspace and +e2e crates plus Rust tests for pull-request mirror branches and merge queues. On +pushes to `main`, a cache-seed job runs the same lint and test commands before a +dependent job builds the release binaries. Manual dispatches exercise the same +seed-then-build path. The binaries remain CI validation artifacts and are not +uploaded or published. + +Each job restores and saves a dedicated Rust cache containing the Cargo +registry and dependency build artifacts, including artifacts from failed runs. +The seed job and pull-request job use the same Cargo target and sccache +namespaces. The release build waits for the seed job, then restores its newly +warmed cache rather than compiling concurrently from a cold cache. Windows validation is exposed through `tasks/windows.toml`: @@ -66,10 +73,12 @@ Windows validation is exposed through `tasks/windows.toml`: |---|---| | `windows:check:x64` | Check the x64 MSVC gateway/CLI build graph. | | `windows:check:arm64` | Check the ARM64 MSVC gateway/CLI build graph. | +| `windows:lint:x64` | Run Clippy over the Windows-supported workspace for x64 MSVC. | +| `windows:lint:arm64` | Run Clippy over the Windows-supported workspace for ARM64 MSVC. | | `windows:build:x64` | Build release x64 `openshell-gateway.exe` and `openshell.exe`. | | `windows:build:arm64` | Build release ARM64 `openshell-gateway.exe` and `openshell.exe`. | -| `windows:test:x64` | Run native x64 workspace tests, including MXC mapper and lifecycle tests, while excluding unsupported Windows packages as top-level test targets. | -| `windows:test:arm64` | Run native ARM64 workspace tests with the same package exclusions. | +| `windows:test:x64` | Run native x64 workspace tests with the nextest CI profile and server test support, while excluding unsupported Windows packages as top-level test targets. | +| `windows:test:arm64` | Run the same suite natively on ARM64. | | `windows:test:unsupported:x64` | Run focused gateway-composition tests for unsupported driver contracts. | | `windows:test:unsupported:arm64` | Run the same focused contracts natively on ARM64. | | `windows:ci` | Run check, build, test, unsupported-contract tests, and artifact reporting. | @@ -77,8 +86,10 @@ Windows validation is exposed through `tasks/windows.toml`: The Windows tasks call `tasks/scripts/windows-msvc.ps1`. The wrapper discovers Visual Studio's `VsDevCmd.bat` with `vswhere` or by enumerating installed release directories, validates the requested compiler and ARM64 Spectre -libraries, adds rustup MSVC targets, clears inherited `RUSTC_WRAPPER`, and -keeps build artifacts under the normal Cargo target tree. +libraries, adds rustup MSVC targets, preserves an inherited `RUSTC_WRAPPER` +when the command is available, and keeps build artifacts under the normal +Cargo target tree. If the wrapper command is unavailable, it warns and clears +the setting so local builds continue without compiler caching. On Windows, the generic `rust:check`, `rust:lint`, and `test:rust` tasks call the same wrapper with the host-native MSVC target. The wrapper preserves the Unix Cargo commands on Linux and macOS, excludes unsupported Windows runtime @@ -90,20 +101,23 @@ packaging-asset tests; its cross-platform Python, Markdown, license, and documentation checks still run. Test tasks require the Rust target architecture to match the Windows host, so an ARM64 test result is native coverage rather than x64 emulation coverage. -By default it enables bundled Z3 for reproducible Windows builds. When +By default it enables the `z3-sys` prebuilt-release feature and pins Z3 4.16.0. +On a clean target directory, `z3-sys` downloads the official static library for +the selected Windows architecture instead of compiling Z3 through +CMake/MSBuild. GitHub Actions supplies its read-only workflow token for the +release lookup, and the Cargo target cache preserves the extracted library for +subsequent runs. When `Z3_LIBRARY_PATH_OVERRIDE` points at a directory containing `libz3.lib`, the wrapper uses that system Z3 instead and requires `Z3_SYS_Z3_HEADER` to point at -the full path to `z3.h`. For bundled builds, the wrapper fetches the Z3 source -revision pinned by `z3-sys` through Git and sets -`Z3_SYS_BUNDLED_DIR_OVERRIDE`. When `CARGO_TARGET_DIR` is explicit, the wrapper -uses it for the source cache. Otherwise, it caches under the current user's -local application data directory, outside the checkout. Publishing uses an -atomic directory rename so concurrent x64 and ARM64 commands can share the -cache safely. This keeps downloaded sources outside the checkout by default and -avoids the unauthenticated GitHub API lookup in the `z3-sys` build script, which -can fail with HTTP 403 when a shared runner or developer network exhausts its -API rate limit. An explicitly set `Z3_SYS_BUNDLED_DIR_OVERRIDE` remains -supported and must contain `src/api/z3.h`. +the full path to `z3.h`. Local clean builds use the unauthenticated GitHub API +unless `READ_ONLY_GITHUB_TOKEN` is set. + +GitHub Actions layers the Cargo target cache with sccache's GitHub Actions +backend. The target cache lets Cargo skip intact dependency builds; sccache +recovers cacheable Rust compiler outputs when source changes invalidate part of +that target tree. CI enables client-side mode and normalizes the checkout root +for stable compiler cache keys. The target-cache action runs its metadata step +with `RUSTC_WRAPPER` cleared so cache maintenance does not depend on sccache. The lane uses `mise run --skip-tools windows:*` because Windows Rust comes from rustup and linking comes from Visual Studio Build Tools. Mise orchestrates the @@ -114,45 +128,47 @@ Spectre-mitigated libraries, host-native Clang tools, CMake tools, and an ARM64-capable Windows SDK. Clang provides `libclang.dll` for `bindgen` and `clang-cl.exe` for ARM64 crypto dependencies. During x64-to-ARM64 check/build, the wrapper discovers and adds the Visual Studio-bundled Ninja to `PATH` for -native dependencies. It lets `cmake-rs` select the Visual Studio ARM64 -generator with native MSVC `cl.exe` for bundled Z3 so the Z3 build does not -inherit the crypto crates' compiler requirement. Z3 stays on the Visual Studio -generator because `z3-sys` emits an MSBuild-only `-m` argument that Ninja -rejects. Artifact hashing uses .NET SHA256 directly because module autoloading -in the mise-launched Windows PowerShell process is not guaranteed. +native dependencies. Z3 uses the official prebuilt ARM64 static library, so it +does not inherit compiler settings from those native dependencies. Artifact +hashing uses .NET SHA256 directly because module autoloading in the +mise-launched Windows PowerShell process is not guaranteed. The wrapper defaults Cargo compilation to four jobs. Set `OPENSHELL_WINDOWS_BUILD_JOBS` to a positive integer to override that limit. A host-local mutex serializes wrapper-owned Cargo commands so concurrent -pre-commit tasks do not multiply the process count while bundled Z3 compiles. +pre-commit tasks do not multiply the compiler process count. The wrapper does not set `CL` or `_CL_`: those variables are also consumed by `clang-cl`, where MSVC's `/MP` option can be interpreted as an input file and break ARM64 crypto dependency builds. ## CI Shape -The x64 GitHub Actions job runs on `windows-2025` and executes: +The x64 GitHub Actions jobs run on `windows-2025`; native ARM64 jobs run on +`windows-11-arm`. Pull-request mirrors and merge queues execute the matching +architecture-specific tasks: + +```powershell +mise run --skip-tools windows:lint: +mise run --skip-tools windows:test: +``` + +Pushes to `main` and manual dispatches first seed the shared caches with those +same lint and test commands. After the seed succeeds, a separate job executes: ```powershell -mise run --skip-tools windows:check:x64 -mise run --skip-tools windows:build:x64 -mise run --skip-tools windows:test:x64 -mise run --skip-tools windows:test:unsupported:x64 +mise run --skip-tools windows:build: ``` -The cache is partitioned by architecture so incompatible x64 and ARM64 target -artifacts cannot collide. It does not cache Cargo-installed binaries, which -also keeps the disabled self-hosted ARM64 scaffold from modifying persistent -runner tooling. +The server test-support suite includes the unsupported-driver contract test, so +CI does not run the focused test task a second time. The focused task remains +available for local diagnosis. -The local aggregate `windows:ci` task cross-builds ARM64 on an x64 host. The -GitHub x64 job currently runs only the x64 tasks, and native ARM64 tests remain -exclusive to an ARM64 runner. +The hosted workflow uses architecture-specific cache namespaces and does not +cache Cargo-installed binaries. -The ARM64 job is scaffolded but disabled until a Windows ARM64 runner is -available. Once enabled, it should run check, release build, native workspace -tests, and the focused unsupported-driver contracts for -`aarch64-pc-windows-msvc`. +The local aggregate `windows:ci` task can still cross-build ARM64 on an x64 +host. Hosted tests use architecture-matched runners, so ARM64 test results are +native rather than emulated coverage. ## Validation Contract diff --git a/crates/openshell-conformance/src/plan.rs b/crates/openshell-conformance/src/plan.rs index ad955ea5f6..955c4b7765 100644 --- a/crates/openshell-conformance/src/plan.rs +++ b/crates/openshell-conformance/src/plan.rs @@ -124,9 +124,19 @@ fn validate_command(label: &str, command: &Path, timeout_secs: u64) -> Result<() mod tests { use super::*; + #[cfg(not(windows))] + const RESTART_GATEWAY_COMMAND: &str = "/usr/local/libexec/restart-gateway"; + #[cfg(windows)] + const RESTART_GATEWAY_COMMAND: &str = r"C:\usr\local\libexec\restart-gateway"; + + #[cfg(not(windows))] + const DIAGNOSTICS_COMMAND: &str = "/usr/local/libexec/diagnostics"; + #[cfg(windows)] + const DIAGNOSTICS_COMMAND: &str = r"C:\usr\local\libexec\diagnostics"; + #[test] fn parses_a_smoke_and_continuity_plan() { - let plan = ConformancePlan::parse( + let plan = ConformancePlan::parse(&format!( r#" version = 1 @@ -139,10 +149,10 @@ mod tests { [[runs.actions]] name = "gateway-upgrade" - command = "/usr/local/libexec/restart-gateway" + command = '{RESTART_GATEWAY_COMMAND}' timeout_secs = 120 "#, - ) + )) .expect("valid plan"); assert_eq!(plan.runs.len(), 2); @@ -151,18 +161,18 @@ mod tests { #[test] fn parses_plan_diagnostics() { - let plan = ConformancePlan::parse( + let plan = ConformancePlan::parse(&format!( r#" version = 1 [diagnostics] - command = "/usr/local/libexec/diagnostics" + command = '{DIAGNOSTICS_COMMAND}' timeout_secs = 60 [[runs]] scenario = "smoke" "#, - ) + )) .expect("valid plan with diagnostics"); assert_eq!( diff --git a/crates/openshell-driver-mxc/src/driver.rs b/crates/openshell-driver-mxc/src/driver.rs index 1ded7130c4..1f248d2d18 100644 --- a/crates/openshell-driver-mxc/src/driver.rs +++ b/crates/openshell-driver-mxc/src/driver.rs @@ -294,7 +294,7 @@ impl MxcComputeBackend { } } - fn validate_sandbox_fields(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { + fn validate_sandbox_fields(sandbox: &DriverSandbox) -> Result<(), tonic::Status> { if let Some(spec) = &sandbox.spec { if effective_driver_gpu_count(driver_gpu_requirements( spec.resource_requirements.as_ref(), @@ -335,7 +335,7 @@ impl MxcComputeBackend { } pub fn validate_sandbox_create(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { - self.validate_sandbox_fields(sandbox)?; + Self::validate_sandbox_fields(sandbox)?; let policy = sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref()); self.map_sandbox_policy(&sandbox.id, policy)?; Ok(()) @@ -356,7 +356,7 @@ impl MxcComputeBackend { pub async fn create_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { let sandbox_id = sandbox.id.clone(); - self.validate_sandbox_fields(sandbox)?; + Self::validate_sandbox_fields(sandbox)?; let sandbox_config = sandbox_config(sandbox)?; // Policy translation is deterministic and side-effect free. Do it before diff --git a/crates/openshell-driver-mxc/src/grpc.rs b/crates/openshell-driver-mxc/src/grpc.rs index dfe8f66b29..9a166057ef 100644 --- a/crates/openshell-driver-mxc/src/grpc.rs +++ b/crates/openshell-driver-mxc/src/grpc.rs @@ -202,6 +202,20 @@ mod tests { assert!(error.message().contains("does not support restarting")); } + #[tokio::test] + async fn sandbox_authentication_is_not_supported() { + let service = + ComputeDriverService::new(MxcComputeBackend::new(MxcComputeConfig::default())); + + let error = service + .authenticate_sandbox(Request::new(AuthenticateSandboxRequest::default())) + .await + .expect_err("MXC must not advertise sandbox credential authentication"); + + assert_eq!(error.code(), tonic::Code::Unimplemented); + assert!(error.message().contains("does not authenticate")); + } + #[tokio::test] async fn workspace_lifecycle_is_an_idempotent_no_op() { let service = diff --git a/crates/openshell-prover/Cargo.toml b/crates/openshell-prover/Cargo.toml index ee815f3a3f..b620280e46 100644 --- a/crates/openshell-prover/Cargo.toml +++ b/crates/openshell-prover/Cargo.toml @@ -12,6 +12,7 @@ repository.workspace = true [features] bundled-z3 = ["z3/bundled"] +prebuilt-z3 = ["z3/gh-release"] [dependencies] z3 = { workspace = true } diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index 898eef334d..3ceef6ba39 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -119,6 +119,7 @@ default = ["telemetry"] ## that contains no telemetry endpoint, HTTP client, or emission code. telemetry = ["openshell-core/telemetry"] bundled-z3 = ["openshell-prover/bundled-z3"] +prebuilt-z3 = ["openshell-prover/prebuilt-z3"] test-support = [] [dev-dependencies] diff --git a/crates/openshell-server/src/config_file.rs b/crates/openshell-server/src/config_file.rs index 65709748b1..ef6f9a3f37 100644 --- a/crates/openshell-server/src/config_file.rs +++ b/crates/openshell-server/src/config_file.rs @@ -332,7 +332,7 @@ pub enum ConfigFileError { Parse { path: PathBuf, #[source] - source: toml::de::Error, + source: Box, }, #[error( "unsupported gateway config version {version}; this build only supports version {SCHEMA_VERSION}" @@ -387,7 +387,7 @@ pub fn load(path: &Path) -> Result { } let file: ConfigFile = toml::from_str(&contents).map_err(|source| ConfigFileError::Parse { path: path.to_path_buf(), - source, + source: Box::new(source), })?; if let Some(version) = file.openshell.version diff --git a/deny.toml b/deny.toml index c0b4db8c1b..adbc5bd52b 100644 --- a/deny.toml +++ b/deny.toml @@ -38,6 +38,9 @@ allow = [ "BSD-2-Clause", "BSD-3-Clause", "BSL-1.0", + # Permissive, non-copyleft license pulled in by the prebuilt Z3 archive + # path (z3-sys -> zip -> bzip2); compatible with this Apache-2.0 project. + "bzip2-1.0.6", "ISC", "Zlib", "0BSD", diff --git a/mise.lock b/mise.lock index 597f54fecb..522084b559 100644 --- a/mise.lock +++ b/mise.lock @@ -134,6 +134,40 @@ checksum = "sha256:b8514ed7552e148b0a032114f745118dcb801791adafafeaf9935e4bfb0ed url = "https://github.com/mozilla/sccache/releases/download/v0.16.0/sccache-v0.16.0-x86_64-pc-windows-msvc.zip" url_api = "https://api.github.com/repos/mozilla/sccache/releases/assets/452060720" +[[tools."github:nextest-rs/nextest"]] +version = "cargo-nextest-0.9.143" +backend = "github:nextest-rs/nextest" + +[tools."github:nextest-rs/nextest"."platforms.linux-arm64"] +checksum = "sha256:2a64b3566a92508550a7ab29c3e8db25472ca37730ecb4d22100b6aa440c2a68" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-aarch64-unknown-linux-gnu.tar.gz" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501881925" +provenance = "github-attestations" + +[tools."github:nextest-rs/nextest"."platforms.linux-x64"] +checksum = "sha256:66786b9abe23920d022a182d1416b1bbc8130dd4872a9553d76985a1708dcd1e" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-unknown-linux-gnu.tar.gz" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501882467" +provenance = "github-attestations" + +[tools."github:nextest-rs/nextest"."platforms.macos-arm64"] +checksum = "sha256:4830d430411148d17602a75cc880bfb4dc8dac153dea59a48a2ef4cc93577f07" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-universal-apple-darwin.tar.gz" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501885637" +provenance = "github-attestations" + +[tools."github:nextest-rs/nextest"."platforms.windows-arm64"] +checksum = "sha256:58c1637ba2396e6c556aa0092f9aa4388695594b8ddda5a4b8b39212574678ce" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-aarch64-pc-windows-msvc.zip" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501886128" +provenance = "github-attestations" + +[tools."github:nextest-rs/nextest"."platforms.windows-x64"] +checksum = "sha256:c670ba18e8731fd2eff33a47af33a0fa53d1afa6d0678344e82dc6f8fc7344ac" +url = "https://github.com/nextest-rs/nextest/releases/download/cargo-nextest-0.9.143/cargo-nextest-0.9.143-x86_64-pc-windows-msvc.zip" +url_api = "https://api.github.com/repos/nextest-rs/nextest/releases/assets/501885754" +provenance = "github-attestations" + [[tools."github:rust-cross/cargo-zigbuild"]] version = "0.22.3" backend = "github:rust-cross/cargo-zigbuild" diff --git a/mise.toml b/mise.toml index 94cafcc738..89d4ef5727 100644 --- a/mise.toml +++ b/mise.toml @@ -45,6 +45,7 @@ k3d = { version = "5.8.3", os = ["macos"] } zig = "0.14.1" "github:rust-secure-code/cargo-auditable" = "0.7.5" "github:rust-cross/cargo-zigbuild" = "0.22.3" +"github:nextest-rs/nextest" = { version = "cargo-nextest-0.9.143", os = ["windows"] } "npm:markdownlint-cli2" = "0.22.0" [tools."github:mozilla/sccache"] diff --git a/tasks/scripts/windows-msvc.ps1 b/tasks/scripts/windows-msvc.ps1 index 0a0410f4ca..cab42fb719 100644 --- a/tasks/scripts/windows-msvc.ps1 +++ b/tasks/scripts/windows-msvc.ps1 @@ -32,24 +32,11 @@ if (-not (Test-Path $LogDir)) { } $LogDir = (Resolve-Path $LogDir).Path -$TargetDirWasConfigured = -not [string]::IsNullOrWhiteSpace($env:CARGO_TARGET_DIR) $TargetDir = $env:CARGO_TARGET_DIR -if (-not $TargetDirWasConfigured) { +if ([string]::IsNullOrWhiteSpace($TargetDir)) { $TargetDir = Join-Path $RepoRoot "target" } -$BundledZ3CacheRoot = $TargetDir -if (-not $TargetDirWasConfigured) { - $userCacheRoot = [Environment]::GetFolderPath([Environment+SpecialFolder]::LocalApplicationData) - if ([string]::IsNullOrWhiteSpace($userCacheRoot)) { - $userCacheRoot = $env:LOCALAPPDATA - } - if ([string]::IsNullOrWhiteSpace($userCacheRoot)) { - $userCacheRoot = [IO.Path]::GetTempPath() - } - $BundledZ3CacheRoot = Join-Path $userCacheRoot "OpenShell\cache\z3" -} - $BuildJobsValue = $env:OPENSHELL_WINDOWS_BUILD_JOBS if ([string]::IsNullOrWhiteSpace($BuildJobsValue)) { $BuildJobsValue = $env:CARGO_BUILD_JOBS @@ -66,14 +53,11 @@ $WindowsCargoMutex = [System.Threading.Mutex]::new($false, "Local\OpenShellWindo $UnsupportedDriverPackageExcludes = "--exclude openshell-driver-docker --exclude openshell-driver-kubernetes --exclude openshell-driver-kubernetes-secrets --exclude openshell-driver-podman --exclude openshell-driver-vault --exclude openshell-driver-vm --exclude openshell-sandbox --exclude openshell-supervisor-network --exclude openshell-supervisor-process --exclude openshell-vfio" $WindowsClippyPackageExcludes = $UnsupportedDriverPackageExcludes $WindowsClippyLintArgs = "-D warnings -A dead-code -A unused-imports -A clippy::unused-async" -$BundledZ3WorkspaceFeatures = "--features openshell-prover/bundled-z3" -$BundledZ3ServerFeatures = "--features openshell-server/bundled-z3,openshell-prover/bundled-z3" -$BundledZ3Repository = "https://github.com/Z3Prover/z3.git" -$BundledZ3SysVersion = "0.11.0" -# This is the matching Z3 4.16.0 source revision. Update both pins together. -$BundledZ3Revision = "ddb49568d3520e99799e364fb22f35fc67d887b1" -$Z3WorkspaceFeatures = $BundledZ3WorkspaceFeatures -$Z3ServerFeatures = $BundledZ3ServerFeatures +$PrebuiltZ3WorkspaceFeatures = "--features openshell-prover/prebuilt-z3" +$PrebuiltZ3ServerFeatures = "--features openshell-server/prebuilt-z3,openshell-prover/prebuilt-z3" +$PrebuiltZ3Version = "4.16.0" +$Z3WorkspaceFeatures = $PrebuiltZ3WorkspaceFeatures +$Z3ServerFeatures = $PrebuiltZ3ServerFeatures function Get-VsInstallRoots { $programFiles = @( @@ -367,109 +351,14 @@ function Resolve-Z3HeaderPath([string] $HeaderPath) { return (Resolve-Path $HeaderPath).Path } -function Assert-BundledZ3Source([string] $SourcePath, [string] $ExpectedRevision) { - if (-not (Test-Path $SourcePath -PathType Container)) { - throw "Bundled Z3 source directory does not exist: $SourcePath" - } - - $header = Join-Path $SourcePath "src\api\z3.h" - if (-not (Test-Path $header -PathType Leaf)) { - throw "Bundled Z3 source directory does not contain src\api\z3.h: $SourcePath" - } - - if (-not [string]::IsNullOrWhiteSpace($ExpectedRevision)) { - $actualRevision = (& git -C $SourcePath rev-parse HEAD 2>$null) - if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($actualRevision)) { - throw "Could not verify the bundled Z3 source revision at: $SourcePath" - } - if ($actualRevision.Trim() -ne $ExpectedRevision) { - throw "Bundled Z3 source revision mismatch at ${SourcePath}: expected $ExpectedRevision, found $($actualRevision.Trim())" - } - } - - return (Resolve-Path $SourcePath).Path -} - -function Resolve-BundledZ3Source { - if (-not [string]::IsNullOrWhiteSpace($env:Z3_SYS_BUNDLED_DIR_OVERRIDE)) { - return Assert-BundledZ3Source $env:Z3_SYS_BUNDLED_DIR_OVERRIDE "" - } - - $cargoLock = Get-Content (Join-Path $RepoRoot "Cargo.lock") -Raw - $packagePattern = '(?ms)^\[\[package\]\]\s+name = "z3-sys"\s+version = "([^"]+)"' - $packageMatches = [regex]::Matches($cargoLock, $packagePattern) - if ($packageMatches.Count -ne 1 -or $packageMatches[0].Groups[1].Value -ne $BundledZ3SysVersion) { - throw "Bundled Z3 source pin expects z3-sys $BundledZ3SysVersion. Update the version and revision pins for the z3-sys version in Cargo.lock." - } - - $revisionPrefix = $BundledZ3Revision.Substring(0, 12) - $sourcePath = Join-Path $BundledZ3CacheRoot "z3-source-$revisionPrefix" - if (Test-Path $sourcePath) { - return Assert-BundledZ3Source $sourcePath $BundledZ3Revision - } - - if (-not (Get-Command git.exe -ErrorAction SilentlyContinue)) { - throw "Bundled Z3 source preparation requires git.exe on PATH." - } - if (-not (Test-Path $BundledZ3CacheRoot -PathType Container)) { - New-Item -ItemType Directory -Force -Path $BundledZ3CacheRoot | Out-Null - } - - $stagingPath = "$sourcePath.partial-$([guid]::NewGuid().ToString('N'))" - Write-Host "==> Fetching bundled Z3 source" - Write-Host " repository: $BundledZ3Repository" - Write-Host " revision: $BundledZ3Revision" - Write-Host " cache: $sourcePath" - - & git init --quiet $stagingPath - if ($LASTEXITCODE -ne 0) { - throw "git init failed while preparing bundled Z3 source at: $stagingPath" - } - & git -C $stagingPath remote add origin $BundledZ3Repository - if ($LASTEXITCODE -ne 0) { - throw "git remote add failed while preparing bundled Z3 source at: $stagingPath" - } - & git -C $stagingPath fetch --quiet --depth 1 origin $BundledZ3Revision - if ($LASTEXITCODE -ne 0) { - throw "git fetch failed for bundled Z3 revision $BundledZ3Revision. Partial source remains at: $stagingPath" - } - & git -C $stagingPath checkout --quiet --detach FETCH_HEAD - if ($LASTEXITCODE -ne 0) { - throw "git checkout failed for bundled Z3 revision $BundledZ3Revision. Partial source remains at: $stagingPath" - } - - Assert-BundledZ3Source $stagingPath $BundledZ3Revision | Out-Null - try { - # Directory.Move is an atomic rename on the same volume and, unlike - # Move-Item, fails when the destination already exists. A concurrent - # x64/ARM64 invocation can therefore win publication without the loser - # nesting its staging directory inside the shared cache. - [IO.Directory]::Move($stagingPath, $sourcePath) - } catch { - if (-not (Test-Path $sourcePath -PathType Container)) { - throw - } - Write-Host "==> Reusing bundled Z3 source published by another process" - } finally { - if (Test-Path $stagingPath -PathType Container) { - try { - Remove-Item -LiteralPath $stagingPath -Recurse -Force - } catch { - Write-Warning "Could not remove redundant bundled Z3 staging directory: $stagingPath" - } - } - } - return Assert-BundledZ3Source $sourcePath $BundledZ3Revision -} - function Configure-Z3 { if ([string]::IsNullOrWhiteSpace($env:Z3_LIBRARY_PATH_OVERRIDE)) { - Write-Host "==> Z3: bundled" - $env:Z3_SYS_BUNDLED_DIR_OVERRIDE = Resolve-BundledZ3Source - Write-Host " Z3_SYS_BUNDLED_DIR_OVERRIDE=$env:Z3_SYS_BUNDLED_DIR_OVERRIDE" + Write-Host "==> Z3: prebuilt release" + $env:Z3_SYS_Z3_VERSION = $PrebuiltZ3Version + Write-Host " Z3_SYS_Z3_VERSION=$env:Z3_SYS_Z3_VERSION" return [pscustomobject]@{ - WorkspaceFeatures = $BundledZ3WorkspaceFeatures - ServerFeatures = $BundledZ3ServerFeatures + WorkspaceFeatures = $PrebuiltZ3WorkspaceFeatures + ServerFeatures = $PrebuiltZ3ServerFeatures } } @@ -516,15 +405,24 @@ function Invoke-VsCargo { $targetArch = Get-VsTargetArch $RustTarget $hostArch = Get-HostArch $logPath = Join-Path $LogDir $LogName + $rustcWrapper = "" + if (-not [string]::IsNullOrWhiteSpace($env:RUSTC_WRAPPER)) { + $wrapperCommand = Get-Command $env:RUSTC_WRAPPER -ErrorAction SilentlyContinue + if ($wrapperCommand) { + $rustcWrapper = $wrapperCommand.Source + } else { + Write-Warning "RUSTC_WRAPPER '$env:RUSTC_WRAPPER' was not found; continuing without it." + } + } $environmentSetup = @( "set `"CARGO_TARGET_DIR=$TargetDir`"", "set `"CARGO_BUILD_JOBS=$WindowsBuildJobs`"", "set `"CARGO_INCREMENTAL=0`"", - "set `"RUSTC_WRAPPER=`"" + "set `"RUSTC_WRAPPER=$rustcWrapper`"" ) if ($hostArch -eq "amd64" -and $RustTarget -eq "aarch64-pc-windows-msvc") { - # Let cmake-rs select MSVC cl.exe for bundled Z3. AWS-LC selects - # clang-cl inside its own ARM64 build script. + # Native ARM64 dependencies select their own compilers. Clear inherited + # overrides so AWS-LC can select clang-cl inside its build script. $environmentSetup += @( "set `"CC=`"", "set `"CXX=`"", @@ -605,6 +503,10 @@ function Invoke-Lint([string] $RustTarget) { -RustTarget $RustTarget ` -CargoArgs "cargo clippy --manifest-path e2e/rust/Cargo.toml --all-targets --no-deps --target $RustTarget -- $WindowsClippyLintArgs" ` -LogName "lint-$RustTarget-e2e.log" + Invoke-VsCargo ` + -RustTarget $RustTarget ` + -CargoArgs "cargo check --manifest-path examples/governance-interceptor/Cargo.toml --all-targets --target $RustTarget" ` + -LogName "lint-$RustTarget-governance-interceptor.log" } function Invoke-Build([string] $RustTarget) { @@ -618,7 +520,7 @@ function Invoke-Test([string] $RustTarget) { Assert-NativeTestTarget $RustTarget Invoke-VsCargo ` -RustTarget $RustTarget ` - -CargoArgs "cargo test --workspace $UnsupportedDriverPackageExcludes --target $RustTarget --no-fail-fast $Z3WorkspaceFeatures" ` + -CargoArgs "cargo nextest run --profile ci --workspace $UnsupportedDriverPackageExcludes --target $RustTarget $Z3WorkspaceFeatures" ` -LogName "test-$RustTarget.log" } @@ -626,12 +528,8 @@ function Invoke-PreCommitTest([string] $RustTarget) { Assert-NativeTestTarget $RustTarget Invoke-VsCargo ` -RustTarget $RustTarget ` - -CargoArgs "cargo test --workspace --exclude openshell-server $UnsupportedDriverPackageExcludes --target $RustTarget --no-fail-fast $Z3WorkspaceFeatures" ` - -LogName "test-$RustTarget-precommit-workspace.log" - Invoke-VsCargo ` - -RustTarget $RustTarget ` - -CargoArgs "cargo test -p openshell-server --features test-support --target $RustTarget --no-fail-fast $Z3ServerFeatures" ` - -LogName "test-$RustTarget-precommit-server.log" + -CargoArgs "cargo nextest run --profile ci --workspace $UnsupportedDriverPackageExcludes --target $RustTarget --features openshell-server/test-support $Z3ServerFeatures" ` + -LogName "test-$RustTarget-precommit.log" } function Invoke-UnsupportedContractTests([string] $RustTarget) { @@ -750,7 +648,7 @@ switch ($Action) { foreach ($rustTarget in $targets) { Invoke-Build $rustTarget } - Invoke-Test "x86_64-pc-windows-msvc" + Invoke-PreCommitTest "x86_64-pc-windows-msvc" Invoke-UnsupportedContractTests "x86_64-pc-windows-msvc" Show-Artifacts $targets } diff --git a/tasks/windows.toml b/tasks/windows.toml index 59c2a8f2c2..708ba2c248 100644 --- a/tasks/windows.toml +++ b/tasks/windows.toml @@ -1,8 +1,9 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Windows MSVC build-only tasks. These intentionally live outside the default -# Linux/macOS `ci` task so the established Linux build path remains unchanged. +# Windows MSVC validation and build tasks. These intentionally live outside +# the Linux/macOS task definitions so each hosted workflow has an explicit +# platform-native entry point. ["windows:check"] description = "Check Windows x64 and ARM64 MSVC targets" @@ -19,6 +20,16 @@ description = "Check Windows ARM64 MSVC target" run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 check aarch64-pc-windows-msvc" +["windows:lint:x64"] +description = "Lint Windows x64 MSVC workspace targets with Clippy" +run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" +run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 lint x86_64-pc-windows-msvc" + +["windows:lint:arm64"] +description = "Lint Windows ARM64 MSVC workspace targets with Clippy" +run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" +run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 lint aarch64-pc-windows-msvc" + ["windows:build"] description = "Build Windows x64 and ARM64 release binaries" run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" @@ -35,14 +46,14 @@ run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 build aarch64-pc-windows-msvc" ["windows:test:x64"] -description = "Run Windows x64 MSVC workspace tests" +description = "Run Windows x64 MSVC workspace tests with the CI nextest profile and server test support" run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" -run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test x86_64-pc-windows-msvc" +run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-precommit x86_64-pc-windows-msvc" ["windows:test:arm64"] -description = "Run Windows ARM64 MSVC workspace tests on a native ARM64 host" +description = "Run Windows ARM64 MSVC workspace tests with the CI nextest profile and server test support on a native ARM64 host" run = "echo 'windows:* tasks require a Windows MSVC host' && exit 1" -run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test aarch64-pc-windows-msvc" +run_windows = "powershell -NoProfile -ExecutionPolicy Bypass -File tasks/scripts/windows-msvc.ps1 test-precommit aarch64-pc-windows-msvc" ["windows:test:unsupported:x64"] description = "Run focused Windows unsupported compute-driver contract tests"