Skip to content

ci: resolve Ubuntu's archive to Azure's mirror of it in the integration jobs - #813

Merged
earakely-scale merged 1 commit into
mainfrom
edgararakelyan/ci-ubuntu-archive-mirror
Nov 19, 2026
Merged

earakely-scale merged 1 commit into
mainfrom
edgararakelyan/ci-ubuntu-archive-mirror

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Why

On 11-19 Ubuntu's archive (archive.ubuntu.com, security.ubuntu.com, now behind Cloudflare) went unresponsive three times, for minutes to hours: index files took 20–40 s, then nothing answered at all.

Every image build and compose in the integration tiers runs apt against it: the ubuntu-24.04 base's Dockerfile, the composer's package resolution and the disk recipe's kernel stage. So shards hit the 20-minute test timeout, on main too: its v0.9.1583 run timed out in shards 1, 9 and 11. Meanwhile azure.archive.ubuntu.com, the mirror the runners' own apt uses, answered in tens of milliseconds.

What

A new step in integration-local, the slow shards and the gateway shards: .github/scripts/ubuntu_archive_dns.sh.

  • It runs dnsmasq on Docker's bridge address. dnsmasq answers archive.ubuntu.com and security.ubuntu.com with the Azure mirror's address and forwards every other name to the runner's resolver.
  • It points the runner's /etc/resolv.conf at that dnsmasq. Docker copies that resolver into containers, and BuildKit into each RUN, because it isn't a loopback address; user-defined networks forward to it through Docker's embedded DNS.
  • The step fails the job if the archive's names don't resolve to the mirror, or if other names don't resolve.

The mirror answers requests made to those hosts' names: Host: archive.ubuntu.com and Host: security.ubuntu.com both return 200 from its address. Images, Dockerfiles and src/ are unchanged. Outside CI, everything still uses Ubuntu's archive.

How it was tested

On an admin VM (Ubuntu 24.04, Docker 28), with the VM's resolver restored afterwards:

  • A BuildKit RUN resolved archive.ubuntu.com to the mirror, and apt-get update fetched 34 MB at 33.6 MB/s.
  • Containers on the default bridge and on a user-defined network resolved the archive's names to the mirror, and pypi.org through the forwarder.

This PR's CI ran it for real: the step succeeded on all 18 integration jobs (each fails if the archive's names don't resolve to the mirror), and the Local backends run was green in 14.7 minutes (run 44528375491).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

This PR appears safe to merge.

Summary

This PR redirects Ubuntu archive requests to Azure's mirror in the local integration jobs.

  • Local integration jobs use Azure’s mirror for Ubuntu archive lookups.

No actionable issues found.

Reviews (1) · Last reviewed commit: "ci: resolve Ubuntu's archive to Azure's ..." · Reviewed by Greptile

…on jobs

Ubuntu's archive stopped answering for minutes to hours at a time on 11-19 (index files taking 20-40 s, then no
answer at all), failing every image build and compose that runs apt in the integration tiers, on main too, while
azure.archive.ubuntu.com, the mirror the runners' own apt uses, stayed up. A step now runs dnsmasq on Docker's
bridge, answering archive.ubuntu.com and security.ubuntu.com with that mirror's address and forwarding every
other name to the runner's resolver, and points the runner's resolv.conf at it; Docker copies that into each
container, and BuildKit into each RUN, since it isn't a loopback address. The mirror answers those hosts' names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale marked this pull request as ready for review November 19, 2026 13:10
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 13:10
@earakely-scale
earakely-scale merged commit b0a1a03 into main Nov 19, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/ci-ubuntu-archive-mirror branch November 19, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant