From fe70d3bb8502c3cd1d4c9059c5f8957bfaed8e06 Mon Sep 17 00:00:00 2001 From: Jane Chu <7559015+janechu@users.noreply.github.com> Date: Wed, 9 Sep 2026 21:19:00 -0700 Subject: [PATCH] fix: pass governed inputs to CD release job Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e26f1001-5364-4bfa-a4e9-45f472307ff2 --- .github/workflows/README.md | 6 +-- azure-pipelines-cd.yml | 100 ++++++++++++++++++++++-------------- 2 files changed, 65 insertions(+), 41 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 45e7710da60..e46f744dd5c 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -13,14 +13,14 @@ All CI workflows that run against pull requests are configured to skip draft PRs ## Continuous Deployment -Nightly publishing is split into two coordinated jobs so that npm credentials never leave the Azure environment. GitHub Releases are the source of truth, and `deployed/` git marker tags track which releases have already been published. +Nightly publishing is coordinated across GitHub Actions and Azure Pipelines so that npm credentials never leave the Azure environment. GitHub Releases are the source of truth, and `deployed/` git marker tags track which releases have already been published. - **`cd-github-releases.yml`** (GitHub Actions) runs nightly via cron (`0 8 * * *` UTC, ~12am PST) and on `workflow_dispatch`. It does **not** bump versions or push source changes to `main` — version bumps land on `main` through ordinary human-authored pull requests (for example, by running `npm run bump` locally and opening a PR). The cron is scheduled ~1 hour before the Azure CD pipeline (09:00 UTC) so any GitHub releases this job creates are picked up by that same night's publish run. The workflow has two jobs: 1. **`detect`** — checks out `main` with `fetch-depth: 0` and runs [`build/scripts/create-github-releases.mjs --check-only`](../../build/scripts/create-github-releases.mjs). The script walks the workspaces tree (no `npm ci` required), computes `${name}_v${version}` for every non-private workspace, and emits `hasMissingReleases=true` if any of those git tags do not yet exist. 2. **`release`** runs only when missing releases exist. Installs Node, the Rust toolchain (for `cargo package`), and the npm workspace dependencies, builds the repo, then runs the script in default mode. For every missing release the script: packs the npm tarball into `publish_artifacts_npm/`, packs any paired Rust crates into `publish_artifacts_crates/`, and creates the GitHub release with all assets attached via `gh release create --target `. `@microsoft/fast-build` is a bundled release: it uses one npm package, one tag, and one GitHub release containing both `microsoft-fast-build` and `microsoft-fast-convert` crate assets. The `gh` CLI creates the git tag atomically with the release, so "tag exists" and "release exists" are always the same fact — a failed release is safely retried on the next workflow run, with no orphan tag stranded behind. The script errors if a paired crate's version does not match the npm package's version — but this is purely a safety net: the [`postbump` hook in `beachball.config.js`](../../beachball.config.js) rewrites each crate's `Cargo.toml` (and the matching entry in `Cargo.lock`) automatically whenever `npm run bump` bumps the paired npm package, so they stay in sync from the same commit. - **`azure-pipelines-cd.yml`** (Azure Pipelines) runs every night at **1am PST (`0 9 * * *` UTC)** with `always: true` so it still runs on no-op nights (it is checking external GitHub state, not repo commits). It is split into two stages so the heavy publish work is skipped on no-op nights: 1. **`Check`** — runs [`build/scripts/download-github-releases.mjs --check-only`](../../build/scripts/download-github-releases.mjs). The script walks the current publishable workspaces, keeps only workspaces whose current `${name}_v${version}` release tag exists, filters out tags that already have a `deployed/` counterpart, and emits Azure Pipelines output variables for the overall deployment decision, npm dist-tag, and each package-specific release tag. No network calls to GitHub, npm, or crates.io are needed. - 2. **`Package`** — depends on `Check` and runs only when `needsDeployment == 'true'`. Conditional `DownloadGitHubRelease@0` tasks download undeployed release assets through the `fast` GitHub service connection, a shell step sorts them into `publish_artifacts_npm/` (`.tgz`) and `publish_artifacts_crates/` (`.crate`), configures npm to publish companion packages with the detected dist-tag, then `FAST.Release.PipelineTemplate.yml@fastPipelines` performs the actual `npm publish` / `cargo publish`. On success, the pipeline pushes a `deployed/` git marker tag for each release that was just published. The next nightly run will see those markers and skip the corresponding releases. + 2. **`Package`** — depends on `Check` and runs only when `needsDeployment == 'true'`. Its `PrepareReleaseInputs` job downloads undeployed release assets through the `fast` GitHub service connection, sorts them into `publish_artifacts_npm/` (`.tgz`) and `publish_artifacts_crates/` (`.crate`), configures the detected npm dist-tag, and publishes those files as a governed pipeline artifact. The checkout-free `Deploy` release job consumes that artifact through `templateContext.inputs`, then `FAST.Release.PipelineTemplate.yml@fastPipelines` performs the actual `npm publish` / `cargo publish`. After publishing succeeds, the ordinary `MarkDeployed` job checks out the repository and pushes a `deployed/` marker for each release. The next nightly run sees those markers and skips the corresponding releases. Both scripts are thin Node.js wrappers around existing CLI tools and repository metadata — no extra npm dependencies and no custom GitHub API client. Idempotency is enforced entirely through git tags (`${name}_v${version}` on the GitHub side, `deployed/${name}_v${version}` on the Azure side), so neither side needs to talk to npm.org or crates.io to decide whether work is required. @@ -35,7 +35,7 @@ When adding a new non-private workspace that should publish through CD: 1. Ensure the workspace is included in the root `package.json` `workspaces` list and has a `name` and `version`. 2. If the package has paired crate assets, place each crate at `crates//Cargo.toml`. By default, `` is the npm package name with the leading `@` removed and `/` replaced by `-`. `@microsoft/fast-build` is the special bundled release and pairs with both `crates/microsoft-fast-build/Cargo.toml` and `crates/microsoft-fast-convert/Cargo.toml`. 3. Add package-specific output variables to the `Package` stage in `azure-pipelines-cd.yml`. The output prefix is generated from the npm package name by converting `@microsoft/` to camel case. For example, `@microsoft/fast-foo` emits `fastFooNeedsDeployment` and `fastFooReleaseTag`. -4. Add a conditional `DownloadGitHubRelease@0` task for the package using the `fast` GitHub service connection, `defaultVersionType: 'specificTag'`, and the package's `$(ReleaseTag)` variable. +4. Add a conditional `DownloadGitHubRelease@0` task to `PrepareReleaseInputs` using the `fast` GitHub service connection, `defaultVersionType: 'specificTag'`, and the package's `$(ReleaseTag)` variable. Release jobs cannot check out source or directly acquire undeclared dependencies, so all downloaded assets must remain part of the governed `fast-release-inputs` pipeline artifact. 5. Confirm the artifact sorting step still covers the package assets. Packages should attach `.tgz` assets, and paired crates should also attach `.crate` assets. Example Azure additions for `@microsoft/fast-foo`: diff --git a/azure-pipelines-cd.yml b/azure-pipelines-cd.yml index 034e41c1fca..0975ad375ec 100644 --- a/azure-pipelines-cd.yml +++ b/azure-pipelines-cd.yml @@ -92,24 +92,15 @@ extends: fastTestHarnessNeedsDeployment: $[ stageDependencies.Check.CheckVersion.outputs['deploymentCheck.fastTestHarnessNeedsDeployment'] ] fastTestHarnessReleaseTag: $[ stageDependencies.Check.CheckVersion.outputs['deploymentCheck.fastTestHarnessReleaseTag'] ] jobs: - - job: Deploy + - job: PrepareReleaseInputs + displayName: Prepare governed release inputs templateContext: - type: releaseJob - isProduction: true + outputs: + - output: pipelineArtifact + targetPath: $(Build.ArtifactStagingDirectory)/fast-release-inputs + artifactName: fast-release-inputs steps: - - checkout: self - persistCredentials: "true" - fetchTags: true - - - script: | - git config --global user.email fastsvc@microsoft.com - git config --global user.name "Microsoft FAST Builds" - displayName: "Configure git for tag push" - - - task: UseNode@1 - inputs: - version: "22.x" - displayName: "Install Node.js" + - checkout: none - task: DownloadGitHubRelease@0 displayName: "Download @microsoft/fast-build release assets" @@ -153,26 +144,27 @@ extends: - script: | set -euo pipefail - rm -rf publish_artifacts_npm publish_artifacts_crates publish_artifacts_meta - mkdir -p publish_artifacts_meta + ROOT="$(Build.ArtifactStagingDirectory)/fast-release-inputs" + rm -rf "$ROOT" + mkdir -p "$ROOT/publish_artifacts_meta" if find "$(System.ArtifactsDirectory)" -name "*.tgz" -print -quit | grep -q .; then - mkdir -p publish_artifacts_npm - find "$(System.ArtifactsDirectory)" -name "*.tgz" -exec cp {} publish_artifacts_npm/ \; + mkdir -p "$ROOT/publish_artifacts_npm" + find "$(System.ArtifactsDirectory)" -name "*.tgz" -exec cp {} "$ROOT/publish_artifacts_npm/" \; fi if find "$(System.ArtifactsDirectory)" -name "*.crate" -print -quit | grep -q .; then - mkdir -p publish_artifacts_crates - find "$(System.ArtifactsDirectory)" -name "*.crate" -exec cp {} publish_artifacts_crates/ \; + mkdir -p "$ROOT/publish_artifacts_crates" + find "$(System.ArtifactsDirectory)" -name "*.crate" -exec cp {} "$ROOT/publish_artifacts_crates/" \; fi - printf '%s\n' "$(undeployedTags)" | tr ',' '\n' | sed '/^$/d' > publish_artifacts_meta/undeployed-tags.txt + printf '%s\n' "$(undeployedTags)" | tr ',' '\n' | sed '/^$/d' > "$ROOT/publish_artifacts_meta/undeployed-tags.txt" has_assets=false - if [ -d publish_artifacts_npm ] && find publish_artifacts_npm -maxdepth 1 -type f -print -quit | grep -q .; then + if [ -d "$ROOT/publish_artifacts_npm" ] && find "$ROOT/publish_artifacts_npm" -maxdepth 1 -type f -print -quit | grep -q .; then has_assets=true fi - if [ -d publish_artifacts_crates ] && find publish_artifacts_crates -maxdepth 1 -type f -print -quit | grep -q .; then + if [ -d "$ROOT/publish_artifacts_crates" ] && find "$ROOT/publish_artifacts_crates" -maxdepth 1 -type f -print -quit | grep -q .; then has_assets=true fi if [ "$has_assets" != "true" ]; then @@ -181,49 +173,81 @@ extends: fi echo "npm artifacts:" - if [ -d publish_artifacts_npm ]; then - find publish_artifacts_npm -maxdepth 1 -type f -print | sort + if [ -d "$ROOT/publish_artifacts_npm" ]; then + find "$ROOT/publish_artifacts_npm" -maxdepth 1 -type f -print | sort else echo "none" fi echo "crate artifacts:" - if [ -d publish_artifacts_crates ]; then - find publish_artifacts_crates -maxdepth 1 -type f -print | sort + if [ -d "$ROOT/publish_artifacts_crates" ]; then + find "$ROOT/publish_artifacts_crates" -maxdepth 1 -type f -print | sort else echo "none" fi echo "deployment markers:" - cat publish_artifacts_meta/undeployed-tags.txt - displayName: "Separate release artifacts" + cat "$ROOT/publish_artifacts_meta/undeployed-tags.txt" + displayName: "Prepare release input artifact" - script: | set -euo pipefail + ROOT="$(Build.ArtifactStagingDirectory)/fast-release-inputs" if [ -z "$(npm_config_tag)" ]; then echo "npm_config_tag was not provided by deployment detection." exit 1 fi echo "Publishing npm artifacts with dist-tag: $(npm_config_tag)" - printf '\ntag=%s\n' "$(npm_config_tag)" >> .npmrc - printf '%s\n' "$(npm_config_tag)" > publish_artifacts_meta/npm-dist-tag.txt + printf '\ntag=%s\n' "$(npm_config_tag)" > "$ROOT/.npmrc" + printf '%s\n' "$(npm_config_tag)" > "$ROOT/publish_artifacts_meta/npm-dist-tag.txt" displayName: "Configure npm publish dist-tag" + - job: Deploy + displayName: Publish packages + dependsOn: PrepareReleaseInputs + templateContext: + type: releaseJob + isProduction: true + inputs: + - input: pipelineArtifact + artifactName: fast-release-inputs + targetPath: $(System.DefaultWorkingDirectory) + steps: + - checkout: none + + - task: UseNode@1 + inputs: + version: "22.x" + displayName: "Install Node.js" + - template: FAST.Release.PipelineTemplate.yml@fastPipelines # Template reference + - job: MarkDeployed + displayName: Mark releases as deployed + dependsOn: Deploy + condition: succeeded() + steps: + - checkout: self + persistCredentials: "true" + fetchTags: true + + - script: | + git config --global user.email fastsvc@microsoft.com + git config --global user.name "Microsoft FAST Builds" + displayName: "Configure git for tag push" + # Push a `deployed/` marker tag for each release that was just # published so that the next nightly Check stage sees it and skips - # the release. Reads the list of tags prepared before publishing. + # the release. Reads the list of tags detected before publishing. # Idempotent: a marker tag that already exists locally (i.e. fetched # from origin via `fetchTags: true`) is left alone instead of failing. - script: | set -euo pipefail - META_FILE=publish_artifacts_meta/undeployed-tags.txt - if [ ! -s "$META_FILE" ]; then + if [ -z "$(undeployedTags)" ]; then echo "No tags to mark as deployed." exit 0 fi - while IFS= read -r tag; do + printf '%s\n' "$(undeployedTags)" | tr ',' '\n' | sed '/^$/d' | while IFS= read -r tag; do [ -z "$tag" ] && continue DEPLOY_TAG="deployed/${tag}" if git rev-parse --verify --quiet "refs/tags/${DEPLOY_TAG}" >/dev/null; then @@ -236,5 +260,5 @@ extends: # if the pipeline is re-run from a later commit. git tag "${DEPLOY_TAG}" "refs/tags/${tag}" git push origin "${DEPLOY_TAG}" - done < "$META_FILE" + done displayName: "Mark releases as deployed"