Skip to content

ci: make crates.io publish idempotent and retry index lag#735

Merged
lwshang merged 2 commits into
mainfrom
fix/publish-workflow-idempotent
Jul 5, 2026
Merged

ci: make crates.io publish idempotent and retry index lag#735
lwshang merged 2 commits into
mainfrom
fix/publish-workflow-idempotent

Conversation

@lwshang

@lwshang lwshang commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

What

Make the Publish to crates.io workflow idempotent and resilient to crates.io index lag.

Why — the 0.48.0 release half-failed

The publish job ran six cargo publish commands unguarded under set -e. During the 0.48.0 release (run):

  • ic-transport-types, ic-agent, ic-identity-hsm, ic-utils uploaded 0.48.0
  • cargo publish -p icx then failed: failed to select a version for the requirement ic-utils = "^0.48.0"ic-utils 0.48.0 had been uploaded ~60s earlier but hadn't yet appeared in the crates.io index (cargo's built-in wait timed out, then the resolve hard-failed). icx-cert never ran.

Two brittleness sources:

  1. Partial release can't be resumed. Re-dispatching aborts on the first command, since cargo publish -p ic-transport-types now errors with "crate version already uploaded" and set -e kills the step.
  2. Index lag. A dependent crate can fail to resolve a just-uploaded dependency for a minute or two.

Fix

Wrap each publish in a helper that:

  • treats "already uploaded/exists" as success → idempotent, so a re-dispatch finishes the crates that didn't make it without re-publishing the ones already live;
  • retries other failures up to 6× with a 30s backoff → rides out index lag between dependent crates.

Order (dependencies before dependents) is unchanged.

Rollout

The Identity team's blocker is already resolved — ic-agent 0.48.0 and ic-transport-types 0.48.0 are live. After merging this, re-dispatch the workflow: it will skip the four published crates and publish icx + icx-cert (both currently still at 0.47.3).

The publish job ran six `cargo publish` commands unguarded under `set -e`.
Two failure modes broke it:

  1. Partial release: if a later crate fails, the already-uploaded crates
     can't be re-published — a re-dispatch aborts immediately on the first
     `cargo publish` ("crate version already uploaded").
  2. Index lag: a dependent crate (e.g. icx -> ic-utils) can fail to
     resolve a just-uploaded dependency until the crates.io index catches
     up, which can take a minute or two.

Both hit the 0.48.0 release: ic-transport-types, ic-agent, ic-identity-hsm,
and ic-utils published, then `icx` failed to resolve `ic-utils = ^0.48.0`
because the index hadn't surfaced it yet.

Wrap each publish in a helper that treats "already uploaded/exists" as
success (idempotent re-runs) and retries other failures up to 6x with a
30s backoff (rides out index lag). Re-dispatching now finishes icx and
icx-cert without re-publishing the four crates already live.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lwshang lwshang requested a review from a team as a code owner July 5, 2026 02:10
@zeropath-ai

zeropath-ai Bot commented Jul 5, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to 72f7710.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► .github/workflows/publish.yml
    Refine cargo publish loop to retry on failures and handle already published crates; implement robust publish() with retries and index lag handling
► (No other files changed in diff)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the crates.io publish GitHub Actions workflow to be resilient to partial publishes and crates.io index lag by making publishing idempotent and adding retry/backoff behavior.

Changes:

  • Add a publish() helper that treats “already uploaded/exists” as success to allow safe re-dispatch after partial releases.
  • Retry failed publishes up to 6 times with a 30s backoff to tolerate crates.io index propagation delays.
  • Keep publish order dependency-first to reduce resolution failures between crates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/publish.yml Outdated
Don't sleep 30s after the last attempt fails — it only delayed the job's
failure. Gate the backoff on there being another attempt to come, and
include attempt count in the messages.

Addresses Copilot review feedback on #735.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lwshang lwshang enabled auto-merge (squash) July 5, 2026 02:20
@lwshang lwshang merged commit 86d485a into main Jul 5, 2026
21 of 25 checks passed
@lwshang lwshang deleted the fix/publish-workflow-idempotent branch July 5, 2026 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants