Skip to content

ci: retry the network reads our own CI steps make - #662

Merged
bburda merged 3 commits into
mainfrom
ci/retry-rosdep-fetches
Sep 9, 2026
Merged

ci: retry the network reads our own CI steps make#662
bburda merged 3 commits into
mainfrom
ci/retry-rosdep-fetches

Conversation

@bburda

@bburda bburda commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Pull Request

Summary

Every CI job reads from the network before anything of ours is built, and rosdep update is the one of those reads this branch can wrap: it fetches the distribution index from raw.githubusercontent.com. A reset connection there fails the whole job with an error that has nothing to do with the change under test. Two jobs on two pull requests failed that way within an hour on the day this branch was opened, both on rosdep update with Connection reset by peer.

rosdep update now gets three attempts five seconds apart in every workflow step that calls it (ci.yml, quality.yml, opcua-plugin.yml). The ctcache install in the clang-tidy job gets the same loop: it is a git+https URL, so pip runs git clone in a subprocess to fetch it, and pip's own --retries only covers its HTTP session. A genuine failure still ends the job after the last attempt with its own error text; the loop only names the attempt.

The release-asset download in .github/actions/ros-apt-source gets a wider window. It already retried, but only for about 18 seconds, and a job on this branch died when GitHub answered 500 to all six attempts. That download is the one fetch in the action with nothing to fall back to: the version lookup above it can use the pinned value, but a 5xx on the .deb ends the step, and setup-ros then has no apt source, so the build stops at colcon: command not found. Measured against a server that answers 500 every time, the old flags make 6 attempts over 18 s and the new ones make 9 over 131 s.

One read stays unprotected, and it runs earlier than any of these. ros-tooling/setup-ros@v0.7 removes /etc/ros/rosdep/sources.list.d/20-default.list and then runs sudo rosdep init once, with no retry around it (src/setup-ros-ubuntu.ts:135-137 in the action). That call fetches 20-default.list from raw.githubusercontent.com, the same host that fails here, and it happens before the step this branch touches, so a reset there still ends the job. Wrapping it would mean not using the action, which is a larger change than this one.

apt-get update is left alone on purpose. apt has defaulted Acquire::Retries to 3 since 2.3.2. Measured on the three images this repository builds in, a fetch against a dead endpoint makes the same four attempts with the flag and without it:

image apt no flag -o Acquire::Retries=3 -o Acquire::Retries=0
ubuntu:jammy 2.4.14 4 attempts 4 attempts 1 attempt
ubuntu:noble 2.8.3 4 attempts 4 attempts 1 attempt
ubuntu:resolute 3.2.0 4 attempts 4 attempts 1 attempt

Issue

  • none: a CI hardening with no behaviour change in the repository's code

Type

  • Bug fix
  • New feature or tests
  • Breaking change
  • Documentation only

CI configuration only.


Testing

  • Every workflow file still loads as YAML.
  • The diff against main is the 11 rosdep update call sites, the ctcache install, and the curl flags plus their comment in the apt-source action. No other line is touched.
  • The loop was exercised the way the workflow runs it. Each step body was read out of the YAML and run under the shell GitHub Actions uses (bash --noprofile --norc -eo pipefail), with a stand-in rosdep and pip3 that fail a set number of times. set -e does not stop the step on a failed attempt: with zero, one or two failures the step ends at 0 and the following rosdep install still runs; with three failures the step ends at 1 and the last error is visible.

Checklist

  • Breaking changes are clearly described (and announced in docs / changelog if needed)
  • Tests were added or updated if needed
  • Docs were updated if behavior or public API changed

rosdep update reads the distribution index from raw.githubusercontent.com and
apt-get update reads the package indexes from the mirrors. Both run before
anything of ours is built, and a reset connection on either fails the job for
a reason that has nothing to do with the change under test. rosdep update now
gets three attempts, and apt-get update uses apt's own Acquire::Retries.
@bburda bburda self-assigned this Sep 7, 2026
Comment thread .github/workflows/ci.yml Outdated
@bburda
bburda marked this pull request as ready for review September 8, 2026 07:23
apt already retries a failed index fetch by itself. The default for
Acquire::Retries has been 3 since apt 2.3.2, and on the three images this
repository builds in - ubuntu:jammy (apt 2.4.14), ubuntu:noble (2.8.3)
and ubuntu:resolute (3.2.0) - a fetch against a dead endpoint makes four
attempts with the flag and four attempts without it, and one attempt with
Acquire::Retries=0. The calls go back to plain apt-get update.

The ctcache install in the clang-tidy job does need a loop. It is a
git+https URL, so pip runs git clone in a subprocess to fetch it, and
pip's own --retries only covers its HTTP session.
…outage

The release asset is the one fetch in this action with nothing to fall back
to. The version lookup above it can use the pinned value when the API is
unreachable, but a 5xx on the .deb ends the step, and the step failing takes
the whole job with it: setup-ros then has no apt source and the build stops
at "colcon: command not found".

Measured against a server that answers 500 every time: the old flags make 6
attempts over 18 seconds, the new ones make 9 over 131 seconds. A short
outage on the release host is now waited out instead of failing every job
that starts during it.
@bburda bburda changed the title ci: retry the network fetches every job starts with ci: retry the network reads our own CI steps make Sep 9, 2026
@bburda
bburda merged commit 226a02e into main Sep 9, 2026
21 checks passed
@bburda
bburda deleted the ci/retry-rosdep-fetches branch September 9, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants