Skip to content

fix(bootstrap): an infra image build holds this machine's lock for its tag - #731

Merged
earakely-scale merged 1 commit into
mainfrom
edgararakelyan/infra-image-builds-take-the-machine-lock
Nov 17, 2026
Merged

earakely-scale merged 1 commit into
mainfrom
edgararakelyan/infra-image-builds-take-the-machine-lock

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

main's CI failed on the v0.9.1550 bump commit, so the release's main-ci check failed, no infra image was published and v0.9.1550 didn't reach PyPI. The failure was in integration-local-slow (8/12):

FAILED tst/integration/cli/run_bundle_built_envs_test.py::test_environments_and_a_universe_written_from_a_bundle_load_into_an_env_through_the_gateway
  Error: docker build of env-gateway failed (exit 1):
  ERROR: failed to build: failed to solve: image "docker.io/library/env-gateway:latest": already exists

When the gateway isn't published for the sources under test, as on the bump after #725 changed it, every bundle run that sets up its own default gateway builds env-gateway. Each test has a state root of its own, so the lock ensure_default_envs takes is per test. Two workers on one runner then built the same tag with the same Docker at once, and the second failed.

Builds of a host image already hold this machine's image-build: lock (host_images.py), and the integration tier shares that prefix across its workers (MACHINE_LOCKS in tst/integration/conftest.py). An infra image's build didn't take it.

What changes

bootstrap._build_and_put holds image-build:<tag> around its docker build, so two builds of one infra image tag take turns: two test workers, or two runs with state roots of their own on one machine.

Testing

  • New unit test: each infra kind (gateway, service-db, website browser) builds each of its images while holding image-build:<tag>. It fails on main (no lock held) and passes here; tst/unit/env/bootstrap_test.py: 52 passed.
  • The race needs two workers building at once on a runner whose Docker uses the containerd image store, so it isn't reproduced locally. integration-local-slow exercises the path whenever a PR changes the gateway.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Infra image builds now hold image-build:<tag> while Docker builds the image.

  • Infra image builds now take turns when they use the same tag.

Diagram

sequenceDiagram
    participant A as Worker A
    participant L as Shared tag lock
    participant D as Docker
    participant B as Worker B
    A->>L: Acquire image-build tag lock
    A->>D: Build image
    B->>L: Wait for the same lock
    D-->>A: Build finishes
    A->>L: Release lock
    L-->>B: Acquire lock
    B->>D: Build image
Loading

Reviews (1) · Last reviewed commit: "fix(bootstrap): an infra image build hol..." · Reviewed by Greptile

…s tag

Two test workers, each with a state root of its own, set up the gateway env at once on main's v0.9.1550 bump and
built env-gateway together; the second failed with "image ... already exists", main went red and the release didn't
publish. The build now holds image-build:<tag>, as a host image's build does, which the integration tier's workers
share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 17, 2026 03:40
@earakely-scale
earakely-scale merged commit 97d973a into main Nov 17, 2026
35 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/infra-image-builds-take-the-machine-lock branch November 17, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant