docker-container driver: builds connecting between container start and buildkitd readiness fail without retry (first-build bootstrap race)

[lyao-77]

Summary

The docker-container driver has a first-build bootstrap race: when a build connects to a freshly created (never-bootstrapped) builder after the buildkitd container reports Running but before buildkitd has bound its socket, the build fails immediately with no retry:

error: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory
ERROR: failed to build: listing workers for Build: failed to list workers: Unavailable: connection error: desc = "error reading server preface: EOF"

This is most visible when several builds target the same new builder concurrently (e.g. CI that creates a builder and fans out builds): the first build triggers bootstrap, and sibling builds that connect during the socket-bind window fail hard.

Likely the same underlying cause as #1570 (which reports the identical dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory symptom but without a root cause).

Root cause (verified against master, driver files unchanged since v0.35/v0.36)

The bootstrap path is protected against races, but the "already Running" fast path is not:

• driver/driver.go — Boot() skips Bootstrap() whenever Info() reports Running:

  if info.Status != Running {
      ...
      if err := d.Bootstrap(ctx, logger); err != nil { return nil, err }
  }
  c, err := d.Client(clientContext)

• driver/docker-container/driver.go — Client() → Dial() execs buildctl dial-stdio with no readiness check. dial-stdio succeeds (the exec starts) even while buildkitd is still binding its socket; the failure only surfaces on the first RPC as error reading server preface: EOF.
• driver/driver.go — Boot() only retries when the error matches ErrNotRunning:

  if errors.Is(err, ErrNotRunning{}) && try <= 2 { continue }
  return nil, err

  The docker-container driver never returns ErrNotRunning, so the connection error is fatal — no retry.

By contrast, the bootstrap path is race-tolerant: create() ignores the container name-conflict (cerrdefs.IsConflict) when two builds create the builder simultaneously, and wait() polls buildctl debug workers ~15× with backoff until buildkitd answers. Only the Running fast path in Boot() lacks that readiness wait.

Deterministic reproducer

The cpu-quota driver-opt throttles the container's CPU, which widens the buildkitd startup window and makes the race fire reliably.

#!/usr/bin/env bash
set -u
mkdir -p /tmp/bxrace && cd /tmp/bxrace
printf 'FROM scratch\nCOPY marker /marker\n' > Dockerfile
echo hi > marker

docker buildx rm -f racer 2>/dev/null
docker buildx create --name racer --driver docker-container --driver-opt cpu-quota=3000

# first build triggers bootstrap
docker buildx build --builder racer --quiet . >out.boot 2>&1 &

# wait until the buildkitd container is Running (but not necessarily ready)
ctr=buildx_buildkit_racer0
for _ in $(seq 1 6000); do
  [ "$(docker inspect -f '{{.State.Running}}' $ctr 2>/dev/null)" = true ] && break
done

# fan out sibling builds into the socket-bind window
for i in 1 2 3 4 5; do
  docker buildx build --builder racer --quiet . >out.$i 2>&1 &
  sleep 0.15
done
wait

grep -l 'server preface' out.* && echo RACE REPRODUCED

Possible fixes

1. When Info() reports Running, still verify readiness via the existing wait() loop before handing back the client — a cheap buildctl debug workers no-op once buildkitd answers.
2. Classify dial-stdio connection failures as ErrNotRunning so Boot()'s existing retry loop covers the window.

Happy to open a PR for (1) if that direction looks right.