Skip to content

Latest commit

 

History

History
359 lines (323 loc) · 22 KB

repository-workflows.md

File metadata and controls

359 lines (323 loc) · 22 KB

Repository checks and publication boundaries

Read this reference when changing hooks, workflows, caches, runners or publication configuration. CI explains the pull-request checks: the single required check ci-ok, the tiers, labels, the lane table in scripts/ci/lanes.py, lane runners, running a lane locally and adding one. Agents load $change-maple-ci. Commands and paths are relative to the monorepo root.

Pre-commit hook

./setup-hooks.sh installs .githooks/pre-commit. It asks scripts/ci/plan.py hook --staged which components to check: a lane's own paths select the component in its hook field, without CI's fan-out to consumers, and an unclassified path selects every component. Each selected component's .githooks/pre-commit runs inside its Nix flake: the root .#ci shell for repository checks, Research, Auth, services/updates/ and scripts/ci/agent-updates.sh; the component flakes for apps/maple-agent/, sdk/, proxy/ and services/opensecret?submodules=1. Without Nix the same commands run from PATH with a warning.

The hook runs the fast part of the selected lanes: format, lint, type checks and unit tests. The Agent runs its agent-lint and agent-test recipes; Research Rust runs formatting and tests, without Clippy; repository changes run actionlint on staged workflows and the scripts/ci tests. Integration suites, cargo-deny, nix flake check and packaging stay in CI. MAPLE_HOOK_FULL=1 runs slower variants where they exist, MAPLE_HOOK_ONLY=a,b restricts the components, MAPLE_HOOK_NO_NIX=1 runs from PATH even with Nix installed, and MAPLE_HOOK_SKIP=1 or git commit --no-verify skips the hook. scripts/ci/test_plan.py keeps the hook's components and its branches in step with the lane table.

Caches

  • Pull requests restore Actions caches, and no pull-request write reaches a cache master reads. The exceptions are the EIF jobs. opensecret-eif.yml's eif job (fork PRs that edit PCR approvals, and manual runs outside master) has no FlakeHub credentials, so its Nix store goes to the run's branch-scoped Actions cache, which master and other pull requests cannot read. Same-repository PR EIF builds (ci.yml's os-eif and opensecret-eif.yml's eif-trusted) push to the FlakeHub Nix cache by design, and fall back to the branch-scoped Actions cache when FlakeHub does not initialize (cache policy).
  • Lane caches (Swatinem/rust-cache, prefix ci-v2) are keyed by lane and toolchain pins (the workspace's flake.lock, and OpenSecret's rust-toolchain.toml), which must match exactly, then by rust-cache's lockfile hash, which may not: a version bump restores its exact cache, and any other lockfile change the newest cache of the same lane and toolchain. The restore key also hashes the runner's rustc versions (the Windows lanes' rustup pin among them) and CARGO*/RUST* variables, so a runner image with a new preinstalled Rust starts cold once, until master saves again. rust-cache's save keeps every build of a crate still in the graph, so a master run that will save a new cache first deletes the lane's target directory: each saved cache holds one dependency graph. agent-supply-chain and agent-provider, which only resolve the Agent's dependency graph, share agent-cargo-home: the Cargo registry and git sources (zed and cua among them) without a target directory. Master pushes and the nightly run save them, into the store of the lane's own runner vendor; a runner from the other vendor (a failover) only restores. Heavy Rust caches therefore live in Blacksmith's store. GitHub's holds what GitHub-hosted jobs need: the lane caches of agent-full-windows (saved only by the nightly run) and agent-windows-updater, the agent-package-windows and agent-package-windows-arm64 packaging caches (below), the Agent's ARM64 release build, the signed Research Windows build's Bun binary, the release, the Nix installer and that EIF check. test_workflow_policy.py lists these writers.
  • The signed master builds write the keys release.yml restores by prefix (<os>-sccache-* for macOS and Linux, xcode-cache-*, release-onnxruntime-ios-*, <os>-bun-*, <os>-gradle-*). Research Windows builds use no compiler cache. No pull-request job writes these prefixes; the Research platform lanes only restore them. The iOS ONNX key text is identical in every workflow that uses it. A release job reads only the store of its own runner vendor, so release.yml's build-web (GitHub-hosted) starts with a cold Bun cache; Bun installs in seconds.
  • agent-desktop-build.yml's unsigned builders save the exact key ci-v1-agent-release-<os>-<graph hash> on master pushes, with no fallback keys. scripts/ci/cargo_graph_key.py hashes the Agent's flake.lock and its Cargo.lock without the versions of workspace and path crates, which Cargo rebuilds anyway: a release version bump keeps the key, and a change to any third-party package or dependency edge, or to the toolchain (for macOS also scripts/ci/apple-toolchain.json), makes the next build cold. Pushes build only Prod; the nightly Dev and dispatched builds restore the same key without saving (the profile changes only Agent workspace crates, which Cargo rebuilds). No ci.yml lane reads it. The Windows ARM64 build never runs on a push, so it saves its own ci-v1-agent-arm64-release-<graph hash> from whichever run misses. The Agent packaging lanes build thin LTO into their own ci-v1-agent-package-<os>-* caches (ci-v1-agent-package-ARM64-Windows-* for the native ARM64 lane), which only the nightly master run saves. No packaging family's key or restore prefix matches another cache's keys, so a target never restores another target's objects. The Agent's signing and publish jobs restore nothing.

Concurrency

Workflow Group Cancelled
ci.yml per PR, or per merge-queue head or SHA superseded PR runs only
frontend-tests, rust-tests, web-build, desktop-build, android-build, proxy-rust none never
codeql.yml per SHA never
agent-desktop-build.yml per run never
mobile-build.yml, ios-dev-testflight.yml one queue each (queue: max) never
opensecret-eif.yml, pages-preview-build.yml, auth-pages-ci.yml per PR or SHA PR runs only
nightly.yml one group never
Publishers their own serial groups never

Contracts

scripts/ci/test_contracts.py runs in the repo lane and imports the code on the other side of each contract. A failure means the workflow change is wrong:

  1. release.yml and the publishers (pages-publish, proxy-publish, updates-publish, sdk-publish-npm, sdk-publish-rust, auth-pages-publish, opensecret-eif-release, opensecret-release-statement-dev) change only with owner review. Exactly one workflow is named Release, and every workflow_run listener names its producer and checks the head repository, conclusion and event.
  2. agent-desktop-build.yml's run number is the Agent build number, baked into every package and the update manifests its publish job writes, so never rename that file; its Windows signing job sets MAPLE_PACKAGE_BUILD_NUMBER to it.
  3. The release preflight (.agents/skills/release-maple/scripts/ci_evidence.py) needs the evidence jobs of frontend-tests, rust-tests, web-build, desktop-build, android-build and mobile-build on master pushes, and a CodeQL that is not red: the newest finished scan of the release commit or an ancestor succeeded (a scan still running does not block). These run on every relevant master push and are never cancelled.
  4. The signed master builds write the release cache keys above (the Agent's is cargo_graph_key.py's hash of its lockfiles, plus the Xcode pin on macOS, with no fallback), and ios-dev-testflight.yml's run number is the Maple Dev build number: never rename that file.
  5. Every OpenSecret EIF builder that uses FlakeHub (ci.yml's os-eif, opensecret-eif.yml's eif-trusted and the release build) runs on the release builder's runner class, and only the listed jobs receive id-token: write or use FlakeHub. The unprivileged fork job stays on ubuntu-24.04-arm. An EIF approval mismatch fails only where that environment's approval files are edited (or a manual run outside master); on master it is a notice.
  6. Each protected environment has a fixed set of consumer jobs; windows-signing runs only on GitHub-hosted runners.
  7. Docs, guides and READMEs name only workflows that exist. Docs select no lane, so this runs with the repo lane and nightly; run just lane repo after editing docs.

The Pages and Auth publishers also trust their producers (pages-preview-build.yml, pages-production-build.yml, auth-pages-ci.yml, auth-pages-build.yml) by file, artifact name and gates; test_pages_workflows.py and test_pages_auth_workflows.py pin them.

scripts/ci/test_workflow_policy.py holds the rules every workflow keeps: no pull_request_target, and a fixed set of workflows that run on pull requests; actions pinned to commits; checkouts without persisted credentials; read-only workflow permissions, with write scopes granted to the jobs that need them; secrets only inside their environment's jobs; a pinned, verified Nix install; a timeout on every job; ci-ok needing every other ci.yml job; and declared runner labels.

Signed builds and publication

Master pushes start signed Research builds (desktop-build, android-build, mobile-build, ios-dev-testflight) when app-change-detection.yml and scripts/ci/change_detection.py find Research inputs for that platform; the scripts only ci.yml's unsigned lanes run select none. Only mobile-build and ios-dev-testflight can also be dispatched, which always builds; desktop and Android release evidence comes only from master pushes. The selector routes proxy runtime and Rust SDK build inputs to the native builds, because the Research shell builds both from the tree. Research builds the in-tree TypeScript SDK, so its source selects the same builds as a frontend change; Auth's pin does not. Agent Prod desktop packages build on master pushes that change Agent inputs (plan.py only agent-desktop), Agent Dev packages nightly, and both on every manual dispatch; a dispatch with release=true builds Prod and publishes it as the next update. Signing jobs keep their protected environments; approving one trusts its runner with that environment's secrets. To rehearse the signed Research release scripts outside CI, run scripts/ci/signed-release-rehearsal.sh <desktop|ios|android|all> in the root Nix shell that builds the platform (.#apple, .#android, and .#desktop-linux on Linux or .#ci on macOS); its usage text lists the signing variables, and nothing is uploaded.

Routine development never authorizes releases, signing, store submission, deployment, or live-service changes. Never push to master as a validation step: app inputs start production-shaped signed builds and can upload iOS artifacts to TestFlight. Creating a GitHub Release starts release builds and downstream publication. Use $release-maple only for explicitly requested release work, and report the tag and commit before publishing.

For Pages, read the deployment guide. Credential-bearing publication runs from trusted master; publishing flags, protected environments and Cloudflare build controls are operator configuration, not consequences of merging source.

Agent Windows updater checks

The agent-windows-updater lane (see scripts/ci/lanes.py for what selects it) runs the updater's native filesystem, Authenticode-rejection, transaction and recovery tests and the bootstrap integration test on GitHub's Windows image. Like every windows matrix lane it has no signing credential or protected environment. A pass qualifies no official signature, signed installed upgrade, visible GUI or paired remote upgrade.

OpenSecret checks

ci.yml runs os-rust, os-flake-check (the component flake's checks), os-helpers (the parent helper images), deps-opensecret (when its manifest, lockfile or deny.toml changes), os-pcr (the root pcr-compatibility check for PCR tooling and approval edits) and os-sdk-e2e, which tests both SDKs against services/opensecret/ from the same checkout. The os-eif lane builds dev and prod EIF candidates on same-repository PRs to master that change measured inputs (or carry the ci:eif label) without editing PCR approvals, on the release builder's blacksmith-16vcpu-ubuntu-2404-arm class, and reports their PCRs. opensecret-eif.yml compares measurements on PRs that edit approved PCR JSON, on relevant master changes and on manual runs; that approval status is not part of ci-ok. A mismatch fails a manual run outside master, and a PR only in an environment whose approval files it edits. On master it is pending approval: the job passes with a notice and the built PCRs in its summary. Build and comparison errors always fail. These checks never change approvals, sign, release, or authorize deployment. Master and same-repository PR EIF builds receive OIDC for FlakeHub caching; fork PRs and non-master manual runs never do. Verify cache changes on fresh hosted runners, not a warm local Nix store. Details: CI approval checks and the cache policy.

Nix runtime

Jobs install Determinate Nix 3.24.2 through an immutable DeterminateSystems/determinate-nix-action commit with source-tag: v3.24.2 (.github/actions/setup-nix, or inline in the sacred workflows), then run scripts/ci/verify-nix-runtime.sh. Keep the revision and tag in step when upgrading. No GitHub token reaches the installer; FlakeHub authentication is limited to jobs with id-token: write.

The pinned actionlint carries scripts/ci/patches/actionlint-stream-stdin.patch, so analyzer input streams after the child starts; remove it when upstream includes this. The macOS shell caps Bash heredoc pipes at 512 bytes, and larger bodies use a temporary file.

Runners and toolchains

CI runners assigns the ci.yml lanes. The signed and release workflows use:

Work Runner
Research macOS, Linux, Android and iOS builds, which compile and sign (master and release); rust-tests.yml; the Agent's unsigned master builds Blacksmith (blacksmith-6vcpu-macos-27, blacksmith-8vcpu-ubuntu-2404, blacksmith-8vcpu-windows-2025)
Release build-windows maple-windows-2025-16core
Signed Research Windows master build; Agent Windows signing windows-2025
Agent Windows ARM64 build windows-11-arm
Artifact verifiers, TestFlight submission, the Agent's signing, attestation and publishing, CodeQL, web and frontend evidence Other GitHub-hosted runners
Proxy release builds and their PR rehearsal (proxy-rust.yml) GitHub-hosted Linux and Windows, Blacksmith macOS
EIF builders that use FlakeHub blacksmith-16vcpu-ubuntu-2404-arm

The Agent's macOS signing, both TestFlight submissions, the macOS release verifiers and the Metal Toolchain export need the pinned Xcode, so they run on GitHub's xcode-27 label, a public-preview image without an SLA. If GitHub renames or retires it, move them to the label that carries Xcode 27.1, such as macos-27 once that image ships it. Jobs that launch or inspect artifacts without selecting Xcode stay on macos-26, an older macOS than the builds link against.

The Agent's signing and attestation, windows-signing jobs and every artifact verifier stay GitHub-hosted, so provenance never names a self-hosted runner (Blacksmith counts as self-hosted). The Agent's unsigned master Windows build compiles on Blacksmith because its fat-LTO link runs on one thread, so 16 cores are no faster. Blacksmith's Windows workspace inherits a foreign write ACE that the updater's storage validation rejects, so the updater tests stay on GitHub's image. Jobs that compile and sign install the pinned Artifact Signing module before compiling, so a failed module download fails early; the module's own NuGet packages and the Azure login still come after the compile. The installer retries the PowerShell Gallery and its CDN and accepts only the pinned SHA-256. Rust, MSVC, Perl and the linker are pinned by .github/actions/setup-agent-windows for the Agent.

Research desktop and iOS builds select Xcode 27.1 (27A9275) from scripts/ci/apple-toolchain.json through scripts/ci/select-xcode.py; the root Nix shell exports the same version for its Apple recipes. Do not rely on a runner's default Xcode. The iOS ONNX pins verify artifacts against that exact build, so an Xcode upgrade means rebuilding and reviewing those hashes and validating desktop and iOS packaging. The Agent's signed macOS build compiles gpui's Metal shaders: Blacksmith's macOS image has no Metal Toolchain and cannot download one, so a GitHub xcode-27 job exports it as a one-day artifact that the build imports, and the build's release cache key hashes the Xcode pin. A newer CI image does not raise the shipped deployment targets: Research supports macOS 13.4 and iOS 16.0.

Research Linux packaging builds AppImage and DEB with Tauri, then the RPM from the normalized DEB payload. AppImage staging bundles libglvnd's GL/GLES dispatch libraries, leaving GPU vendor drivers on the host. The existing string patcher rewrites libepoxy's Nix fallback directory to $ORIGIN and literal Nix library filenames (including GStreamer's libsoup loader) to bundled sonames before closure repair. The patcher retains its build interpreter until these rewrites finish, then receives the distributable interpreter. Portability verification rejects those paths in ELF runtime constants (.rodata), separately from DT_NEEDED and RPATH checks; obsolete dynamic strings and source locations are not runtime dependencies. Plugin search directories and patterns are outside this literal-library check. scripts/ci/sccache-stats.sh reports compiler-cache statistics with the build's own cache configuration. The packaged executable runs on the host's glibc, so it links against zig's glibc LINUX_GLIBC_FLOOR (scripts/ci/_common.sh; 2.39, Ubuntu 24.04) through cargo-zigbuild rather than the Nix shell's glibc, and scripts/ci/elf_glibc_floor.py rejects a packaged maple that needs newer symbol versions. zig does not apply the Nix compiler wrapper's hardening, so LINUX_C_HARDENING_FLAGS gives C dependencies the same flags and the same check rejects a maple that imports no fortified glibc function. The AppImage has its own, higher floor. It bundles Nix-built WebKitGTK, GTK and GStreamer, which need the pinned nixpkgs' glibc and cannot be relinked, while the host still supplies libc. LINUX_APPIMAGE_GLIBC_FLOOR declares the newest glibc any bundled file needs (2.44). AppRun checks the host's glibc first and, below that floor, explains the requirement and points to the DEB and RPM instead of failing in the dynamic loader. Package verification scans every ELF file in the AppImage and fails when the bundle needs more or less than the declared floor, so a nixpkgs bump that moves it must change the constant in the same change. Raising it drops the distributions below it for AppImage users. The AppImage also bundles what its libraries find only at runtime, which the DT_NEEDED closure cannot see: glib-networking's TLS module, without which every webview HTTPS request fails, and GTK3's compiled GSettings schemas, without which a file dialog aborts the process. AppRun sets GIO_MODULE_DIR to the bundled module. The dev shell sets its own, so only package verification can catch a gap: scripts/ci/appimage_gio_runtime.py loads the extracted AppImage's libraries and fails unless they have a TLS backend and the file chooser's schema and template. GPU-facing libraries go the other way: like glibc, libgbm and libdrm come from the host, because they must match its GPU drivers and kernel. A bundled libgbm searches only its build's driver directory (/run/opengl-driver from nixpkgs), so WebKit's GPU renderer fails and the window stays blank. linuxdeploy already excludes both, and the runtime-closure check rejects an AppImage that bundles them. The RPM builder preserves explicit bundle.linux.rpm.depends from the Tauri configuration. WebKitGTK (webkit2gtk4.1) is required on Fedora, including KDE; package verification rejects RPMs missing that requirement. Install a downloaded RPM through Discover or sudo dnf install ./Maple-<version>-1.x86_64.rpm so the package manager resolves dependencies, rather than using rpm --nodeps.

For a side-by-side hosted Dev installation, use nix run --no-update-lock-file .#research-dev-desktop, not an ordinary Research PR package. The Linux Dev command stages its own DEB payload and normalizes it into the separately named maple-research-dev RPM under target/research-dev. It uses the fixed Dev native identity and callback scheme, disables updates, and verifies the final package contains only Dev-owned executable, launcher, icon, and runtime paths. See the Research Dev package guide.

SDK consumption

Follow the consumer version policy. Client apps build the in-tree SDK; Auth pins a published version and upgrades alone. SDK publication serves external users and Auth, not in-repo clients.

Maintaining guidance

Re-check prescriptive language against source, tests, and executed workflows. Correct material workflow, ownership, or validation changes alongside the implementation. Keep unrelated drift separately scoped and avoid duplicated policy. Public guides describe implementation, supported setup, tests, and CI; credential administration and private operational procedures belong with their owners outside this repository.