fix(build-push-docker-manifest): idempotent manifest + cosign reruns (RANE-4683)

[HashWrangler]

Summary

Hardens build-publish against manifest job flakes and non-idempotent reruns (RANE-4683).

Layer	Problem	Fix
Manifest create	Rerun calls imagetools create again → new index digest (digest drift)	Skip create when tag already points at the same platform digests; fail if tag exists with different digests
Cosign sign	Always signs the digest produced/inspected by create	Sign always in the action (safe with idempotent create); no in-action verify/skip-sign
Cosign verify	First-run flake: no signatures found right after sign; job rerun used to re-create the index	Isolated verify-manifest-signature job (needs: docker-manifest) with exponential backoff up to 10 minutes (job timeout-minutes: 12)
Manifest inspect	Tag visible before multi-arch manifest fully propagated (ECR lag)	Post-create inspect retries until platform digests match expected inputs

Parent epic: RANE-4695 private release improvements.

Design (final)

• Composite action owns create + sign + create-idempotency + ECR propagation guard.
• Reusable workflow owns post-sign verify as its own job, so a Sigstore propagation false-negative retries alone and never re-runs create/sign (the RANE-4683 digest-drift path).

Evidence

• v2.43.0 / SECHD-30708 — non-idempotent reruns
• chainlink + canary v2.990.5-beta.0 — cosign verify flake; rerun passed
• v2.55.0-rc.0 — verify false-negative → manifest job rerun → tag moved onto unsigned index 1070bfbf… while signed digest was ff93d395… (identical platform children)

Out of scope (separate PR)

• chainlink#22877 — Slack only posts when both core + ccip manifests succeed (RANE-4695 UX guard)

Test plan

• Changesets publish minor build-push-docker-manifest + patch reusable-docker-build-publish
• Confirm reusable-docker-build-publish @build-push-docker-manifest/v1 picks up release
• Tag build on chainlink/canary: first run green
• Re-run failed verify job only: digest unchanged, verify passes after backoff
• Simulate digest mismatch on existing tag → create job fails with explicit error

[github-actions]

👋 HashWrangler, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

[Copilot]

Pull request overview

This PR hardens the build-push-docker-manifest composite action to make build-publish reruns idempotent and reduce flakiness around Cosign verification/signature propagation.

Changes:

• Adds an idempotency guard for manifest creation by comparing expected vs existing platform digests and skipping imagetools create when they match.
• Skips cosign sign when an existing valid signature is already present, and adds retry logic to cosign verify to absorb Sigstore propagation lag.
• Adds a changeset to release these hardening updates as a minor version bump.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
actions/build-push-docker-manifest/action.yml	Adds digest comparison to avoid manifest digest drift on reruns; makes cosign signing/verification rerun-safe with skip + retry behavior.
.changeset/rane-4683-manifest-cosign-hardening.md	Declares a minor release for the manifest + cosign hardening changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[HashWrangler]

Pushed 9c15a99 — verification is now its own job, per @erikburt / @chainchad.

reusable-docker-build-publish: new verify-manifest-signature job (needs: docker-manifest) that owns the cosign verify + 5×10s Sigstore-propagation retry. The docker-manifest job now only creates + signs (and exposes a name-with-digest output the verify job consumes). So a "no signatures found" propagation false-negative retries alone — manifest create/sign is never re-run, which removes the index digest-drift path entirely.

build-push-docker-manifest: back to create + sign only. Since verify is decoupled and imagetools create is already idempotent (skips when the tag points at the same platform digests), I dropped the sign-skip verification shell — it only existed to survive verify-triggered reruns. Also dropped the jq guard per your note, Erik.

@chainchad re: "is it ever unsigned, or just a delay to verify" — concrete answer from today: it was signed, then a rerun stranded the tag. This exact flake blocked v2.55.0 GA this morning. On Jul 7 the RC index sha256:ff93d395… was signed and announced, but the in-run verify hit the propagation lag → the manifest job was rerun → imagetools create produced a new index sha256:1070bfbf… and moved the tag onto it, unsigned. release-final then failed cosign verify on the current tag digest. Both indexes wrap identical per-arch images (amd64 7a3b55ba…, arm64 a048eb51…) — pure digest drift. A standalone, independently-retryable verify job prevents this by construction. (GA unblock itself needs ProdSec to re-point/re-sign the release-ECR tag — separate SECHD ticket.)

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[HashWrangler]

Ready for re-review

All open review threads replied + resolved. Tip: 585dd63.

Final shape

1. build-push-docker-manifest — idempotent imagetools create + always cosign sign + ECR propagation inspect (no in-action verify / sign-skip / jq guard)
2. reusable-docker-build-publish — isolated verify-manifest-signature job with exponential backoff up to 10 min (timeout-minutes: 12)

Copilot today

• ✅ timeout mismatch (5m job vs 600s retry) — fixed in 585dd63
• ❌ jq preflight — won't fix (per Erik/Chad: assume available)

@erikburt @chainchad — please take another look when you can.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[chainchad]

There's a lot of complexity that we're adding to this action.

Since we're moving the verify step to its own job, what if we simplified this to make the manifest index idempotent by checking if the manifest index tag exists and then running successive steps only if it does not?

Example:

- name: Check if manifest already exists
  id: manifest-check
  shell: bash
  env:
    MANIFEST: ${{ steps.manifest-name.outputs.name }}:${{ inputs.docker-manifest-tag }}
  run: |
    if docker buildx imagetools inspect "${MANIFEST}" >/dev/null 2>&1; then
      echo "exists=true"  | tee -a "$GITHUB_OUTPUT"
    else
      echo "exists=false" | tee -a "$GITHUB_OUTPUT"
    fi

- name: Create and push Docker manifest
  if: steps.manifest-check.outputs.exists != 'true'
  ...