fix(ci): make stable release publication retryable

.agents/skills/release-captain/SKILL.md

@@ -234,7 +234,7 @@ git worktree remove --force "$wt"
 | Transient infrastructure failure in a pre-release, including a registry push | After approval, `gh run rerun <run-id> --failed`. `Release` won't select the run until all of it passes. |
 | Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Fix it in a new `-pre.N` (Monday step 3). |
 | Code defect or `Cargo Deny` advisory | Rerunning won't help. Fix it in a new `-pre.N` (Monday step 3). |
-| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. |
+| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. If the run's artifacts expired or its jobs were skipped, dispatch `Release` again with `-f tag=vX.Y.Z`; it publishes only what is missing. |
 
 ## Related
 

.agents/skills/release-openshell/SKILL.md

@@ -32,7 +32,9 @@ passed but publication failed, rerun the failed jobs before promoting the candid
 Release creates `vX.Y.Z` at that exact commit, then rebuilds and publishes all
 stable artifacts without rerunning qualification. Embedded versions change, so
 stable artifacts are not byte-identical to the qualified prerelease. An existing
-stable tag may be retried only if it already points to the selected commit.
+stable tag may be retried only if it already points to the selected commit. A retry
+publishes only missing artifacts; published images, charts, release assets, npm,
+and PyPI versions are kept, while the snap is uploaded again.
 Stable images are saved as OCI archives during the build and pushed under the
 stable version tags during publication. Their SBOMs are verified before updating
 `latest`; the qualified prerelease's commit-SHA image tags remain unchanged.

.agents/skills/test-release-canary/SKILL.md

@@ -95,7 +95,7 @@ Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${h
 
 ## Testing artifacts from a specific SHA
 
-`Release Dev` publishes two chart versions for every dev build (see `.github/actions/release-helm-oci/action.yml:89-102`):
+`Release Dev` publishes two chart versions for every dev build (see the push steps in `.github/actions/release-helm-oci/action.yml`):
 
 - `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` — floating, overwritten on every main push.
 - `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev.<sha>` — immutable, `appVersion` set to the same SHA so it pulls the matching `gateway`, `sandbox`, and `supervisor` images.

.agents/skills/watch-github-actions/SKILL.md

@@ -190,7 +190,7 @@ qualification; stable publication is blocked, while published pre-release
 artifacts remain available. The Pre-release failure notification includes the
 run link. Both callers share `release-build.yml` and `release-publish.yml`.
 Qualification evidence remains in Actions; publishing a copy to OCI storage is best effort.
-Snap Store uploads happen only for stable releases after qualification passes.
+Pre-release uploads snaps to `latest/beta`; stable releases upload to `latest/stable`.
 
 For a release notes preview, dispatch `release.yml` with `notes-only=true` and
 `tag=<stable-tag>`. The generator resolves the previous reachable stable tag;
@@ -207,8 +207,8 @@ and qualification, and keeps the notes only in the `release-notes` artifact.
 This job is advisory: a notes failure emits a warning without failing the
 prerelease run or blocking promotion. Stable publication reuses the promoted
 prerelease's `release-notes` artifact, retagged, and generates notes only when
-that preview is unavailable. The `Select Qualified Prerelease` summary says
-which happened.
+that preview is unavailable. The Release workflow's `Release Notes` job summary
+says which happened, and Publish waits for that job.
 
 For `Codex Compatibility Review`, inspect its job summary and the
 `compatibility-review-run-<run-id>-attempt-<attempt>` artifact. Confirm the

.github/actions/release-helm-oci/action.yml

@@ -3,8 +3,8 @@
 
 name: Release Helm OCI
 description: >
-  Patch chart version/appVersion, refuse duplicate OCI versions on public
-  releases, package the gateway and workspace charts, and push them to GHCR OCI.
+  Patch chart version/appVersion, package the gateway and workspace charts, and
+  push them to GHCR OCI. Public releases never overwrite published versions.
 
 inputs:
   chart-version:
@@ -63,22 +63,38 @@ runs:
         echo "workspace_chart_dir=${WORKSPACE_CHART_DIR}" >> "$GITHUB_OUTPUT"
         echo "chart_version=${CHART_VERSION}" >> "$GITHUB_OUTPUT"
 
-    - name: Refuse duplicate chart version
+    # Public chart versions are never overwritten. A rerun keeps both published
+    # charts; a partial publication still fails for manual repair.
+    - name: Check for published chart version
+      id: published
       if: inputs.release-kind == 'public'
       env:
         CHART_VERSION: ${{ steps.prep.outputs.chart_version }}
       shell: bash
       run: |
         set -euo pipefail
+        found=0
         for chart in helm-chart openshell-workspace; do
           OCI_CHART="oci://ghcr.io/nvidia/openshell/${chart}"
-          if helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" >/dev/null 2>&1; then
-            echo "::error::Chart ${chart}:${CHART_VERSION} is already published. Use a new tag or delete the existing package first."
+          # Only "not found" means unpublished; any other lookup error fails
+          # rather than risk overwriting.
+          if lookup_error=$(helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" 2>&1 >/dev/null); then
+            found=$((found + 1))
+          elif [[ "${lookup_error}" != *"ghcr.io/nvidia/openshell/${chart}:${CHART_VERSION}: not found"* ]]; then
+            echo "::error::Could not check whether ${chart}:${CHART_VERSION} is published: ${lookup_error}"
             exit 1
           fi
         done
+        if [ "${found}" -eq 2 ]; then
+          echo "Charts ${CHART_VERSION} are already published; keeping them."
+          echo "published=true" >> "$GITHUB_OUTPUT"
+        elif [ "${found}" -ne 0 ]; then
+          echo "::error::Only one chart is published at ${CHART_VERSION}. Delete it or publish the other chart manually."
+          exit 1
+        fi
 
     - name: Package Helm chart
+      if: steps.published.outputs.published != 'true'
       env:
         GATEWAY_CHART_DIR: ${{ steps.prep.outputs.gateway_chart_dir }}
         WORKSPACE_CHART_DIR: ${{ steps.prep.outputs.workspace_chart_dir }}
@@ -91,6 +107,7 @@ runs:
         ls /tmp/helm-charts/*.tgz
 
     - name: Push Helm chart to GHCR OCI
+      if: steps.published.outputs.published != 'true'
       shell: bash
       run: |
         set -euo pipefail

.github/workflows/release-notes.yml

@@ -17,16 +17,27 @@ on:
         description: Keep notes failures from failing the calling workflow
         default: false
         type: boolean
+      preview-run-id:
+        description: Optional Pre-release run whose release-notes preview is reused instead of generating notes
+        default: ""
+        type: string
+      preview-tag:
+        description: Prerelease tag in the reused preview, replaced with release-tag
+        default: ""
+        type: string
     secrets:
       LLM_INFERENCE_API_KEY:
         description: NVIDIA Inference Hub API key
         required: true
 
 permissions:
+  actions: read
   contents: read
   pull-requests: read
 
 jobs:
+  # Callers depend on this job, so choose reuse or generation with step
+  # conditions; a skipped job would skip every job that depends on it.
   release-notes:
     name: Write Release Notes
     runs-on: ubuntu-latest
@@ -36,7 +47,30 @@ jobs:
       ANTHROPIC_BASE_URL: ${{ vars.ANTHROPIC_BASE_URL }}
       ANTHROPIC_MODEL: ${{ vars.ANTHROPIC_MODEL }}
     steps:
+      # Without a preview (missing or expired), the notes are generated instead.
+      - name: Reuse the prerelease notes preview
+        id: preview
+        if: ${{ inputs.preview-run-id != '' }}
+        shell: bash
+        env:
+          GH_TOKEN: ${{ github.token }}
+          RUN_ID: ${{ inputs.preview-run-id }}
+          PREVIEW_TAG: ${{ inputs.preview-tag }}
+          RELEASE_TAG: ${{ inputs.release-tag }}
+        run: |
+          set -euo pipefail
+          : "${PREVIEW_TAG:?preview-tag is required with preview-run-id}"
+          if gh run download "${RUN_ID}" -R "${GITHUB_REPOSITORY}" -n release-notes -D preview; then
+            sed "s/${PREVIEW_TAG//./\\.}\b/${RELEASE_TAG}/g" preview/release-notes.md > release-notes.md
+            rm -rf preview
+            echo "reused=true" >> "${GITHUB_OUTPUT}"
+            echo "Release notes are reused from the \`${PREVIEW_TAG}\` preview." >> "${GITHUB_STEP_SUMMARY}"
+          else
+            echo "Release notes will be regenerated; the \`${PREVIEW_TAG}\` preview is unavailable." >> "${GITHUB_STEP_SUMMARY}"
+          fi
+
       - name: Check out release source
+        if: ${{ steps.preview.outputs.reused != 'true' }}
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           fetch-depth: 0
@@ -45,6 +79,7 @@ jobs:
 
       - name: Prepare release notes
         id: comparison
+        if: ${{ steps.preview.outputs.reused != 'true' }}
         shell: bash
         env:
           RELEASE_TAG: ${{ inputs.release-tag }}
@@ -63,6 +98,7 @@ jobs:
           echo "url=${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/compare/${BASE_TAG}...${RELEASE_TAG}" >> "$GITHUB_OUTPUT"
 
       - name: Write notes with Claude
+        if: ${{ steps.preview.outputs.reused != 'true' }}
         uses: anthropics/claude-code-action@374f254f5efaf968abfe15d399fd90972e8138b9 # v1
         env:
           GH_TOKEN: ${{ github.token }}

.github/workflows/release-prerelease.yml

@@ -27,7 +27,7 @@
     concurrency:
       group: release-prerelease-tags
       cancel-in-progress: false
       queue: max
     outputs:
       tag: ${{ steps.tag.outputs.tag }}
       source-sha: ${{ steps.tag.outputs.source_sha }}
@@ -99,6 +99,7 @@
     name: Preview Release Notes
     needs: [resolve, build]
     permissions:
+      actions: read
       contents: read
       pull-requests: read
     uses: ./.github/workflows/release-notes.yml
@@ -114,7 +115,6 @@
     permissions:
       actions: read
       contents: write
-      pull-requests: read
       packages: write
       id-token: write
       attestations: write

.github/workflows/release-publish.yml

@@ -19,11 +19,6 @@ on:
         required: false
         default: false
         type: boolean
-      release-notes-reused:
-        description: Whether the calling run already uploaded release-notes from the promoted prerelease's preview
-        required: false
-        default: false
-        type: boolean
       source-sha:
         description: Source commit of the built artifacts in the calling run
         required: true
@@ -49,9 +44,6 @@ on:
         required: false
       GITLAB_CI_TRIGGER_URL:
         required: false
-      LLM_INFERENCE_API_KEY:
-        description: NVIDIA Inference Hub API key for stable release notes
-        required: false
 
 permissions:
   contents: read
@@ -264,11 +256,23 @@ jobs:
           set -euo pipefail
           for component in gateway sandbox supervisor; do
             image="ghcr.io/nvidia/openshell/${component}:${VERSION}"
-            # Archives are build artifacts from this run, not externally signed images.
-            nix shell --inputs-from . nixpkgs#skopeo -c \
-              skopeo --insecure-policy copy --all --preserve-digests \
-              --authfile "${HOME}/.docker/config.json" \
-              "oci-archive:artifacts/images/${component}.tar" "docker://${image}"
+            # Never replace a published stable image; a rerun keeps the original.
+            # Only a missing manifest means unpublished; any other lookup error
+            # fails rather than risk overwriting.
+            if lookup_error=$(nix shell --inputs-from . nixpkgs#skopeo -c \
+              skopeo inspect --raw --authfile "${HOME}/.docker/config.json" \
+              "docker://${image}" 2>&1 >/dev/null); then
+              echo "${image} is already published; keeping it."
+            elif [[ "${lookup_error}" != *"manifest unknown"* ]]; then
+              echo "::error::Could not check whether ${image} is published: ${lookup_error}"
+              exit 1
+            else
+              # Archives are build artifacts from this run, not externally signed images.
+              nix shell --inputs-from . nixpkgs#skopeo -c \
+                skopeo --insecure-policy copy --all --preserve-digests \
+                --authfile "${HOME}/.docker/config.json" \
+                "oci-archive:artifacts/images/${component}.tar" "docker://${image}"
+            fi
             tasks/scripts/verify-image-sbom.sh "${image}" --require-cargo
           done
 
@@ -326,28 +330,24 @@ jobs:
           persist-credentials: false
           ref: ${{ inputs.source-sha }}
 
-      - uses: ./.github/actions/release-helm-oci
+      # Charts come from the release source; the publish logic comes from this
+      # workflow so retrying an older tag uses the current duplicate handling.
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+          ref: ${{ github.workflow_sha }}
+          path: release-workflow
+          sparse-checkout: .github/actions/release-helm-oci
+
+      - uses: ./release-workflow/.github/actions/release-helm-oci
         with:
           chart-version: ${{ inputs.semver }}
           app-version: ${{ inputs.semver }}
           release-kind: public
 
-  release-notes:
-    name: Write Release Notes
-    if: ${{ !inputs.is-prerelease && !inputs.release-notes-reused }}
-    permissions:
-      contents: read
-      pull-requests: read
-    uses: ./.github/workflows/release-notes.yml
-    with:
-      release-tag: ${{ inputs.release-tag }}
-    secrets:
-      LLM_INFERENCE_API_KEY: ${{ secrets.LLM_INFERENCE_API_KEY }}
-
   release:
     name: Publish Release Assets
-    needs: [assemble, release-helm, release-notes]
-    if: ${{ !cancelled() && needs.assemble.result == 'success' && needs.release-helm.result == 'success' && (inputs.is-prerelease || inputs.release-notes-reused || needs.release-notes.result == 'success') }}
+    needs: [assemble, release-helm]
     runs-on: linux-amd64-cpu8
     timeout-minutes: 10
     permissions:
@@ -393,6 +393,7 @@ jobs:
               }
             }
 
+      # Stable callers upload release-notes in the same run before calling Publish.
       - name: Download release notes
         if: ${{ !inputs.is-prerelease }}
         uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -416,6 +417,8 @@ jobs:
           make_latest: ${{ steps.latest.outputs.make-latest == 'true' && 'true' || 'false' }}
           tag_name: ${{ env.RELEASE_TAG }}
           body_path: release/release-notes.md
+          # A rerun uploads only missing assets and keeps published ones.
+          overwrite_files: false
           files: |
             release/openshell-x86_64-unknown-linux-musl.tar.gz
             release/openshell-aarch64-unknown-linux-musl.tar.gz
@@ -591,3 +594,49 @@ jobs:
       publish: true
     secrets:
       FERN_TOKEN: ${{ secrets.FERN_TOKEN }}
+
+  # GitHub treats skipped jobs as passing, so a publisher that never ran would
+  # leave the run green. Fail when any publisher expected for this release kind
+  # did not succeed, so callers' failure notifications fire.
+  check-publication:
+    name: Check Publication
+    needs:
+      - assemble
+      - tag-ghcr-release
+      - release-helm
+      - release
+      - publish-snap
+      - publish-prerelease-snap
+      - publish-sdk-typescript
+      - tag-sdk-go
+      - trigger-wheel-publish
+      - publish-fern-docs
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 2
+    steps:
+      - name: Require every expected publisher to succeed
+        env:
+          NEEDS: ${{ toJSON(needs) }}
+          IS_PRERELEASE: ${{ inputs.is-prerelease }}
+          MAKE_LATEST: ${{ inputs.make-latest }}
+        run: |
+          set -euo pipefail
+          expected=(assemble tag-ghcr-release release-helm release)
+          if [[ "${IS_PRERELEASE}" == "true" ]]; then
+            expected+=(publish-prerelease-snap)
+          else
+            expected+=(publish-sdk-typescript tag-sdk-go trigger-wheel-publish publish-fern-docs)
+            if [[ "${MAKE_LATEST}" == "true" ]]; then
+              expected+=(publish-snap)
+            fi
+          fi
+          failed=0
+          for job in "${expected[@]}"; do
+            result=$(jq -r --arg job "${job}" '.[$job].result' <<<"${NEEDS}")
+            if [[ "${result}" != "success" ]]; then
+              echo "::error::${job} did not publish (result: ${result})"
+              failed=1
+            fi
+          done
+          exit "${failed}"