ci: retry the network reads our own CI steps make - #662
Merged
Conversation
rosdep update reads the distribution index from raw.githubusercontent.com and apt-get update reads the package indexes from the mirrors. Both run before anything of ours is built, and a reset connection on either fails the job for a reason that has nothing to do with the change under test. rosdep update now gets three attempts, and apt-get update uses apt's own Acquire::Retries.
mfaferek93
reviewed
Sep 7, 2026
bburda
marked this pull request as ready for review
September 8, 2026 07:23
apt already retries a failed index fetch by itself. The default for Acquire::Retries has been 3 since apt 2.3.2, and on the three images this repository builds in - ubuntu:jammy (apt 2.4.14), ubuntu:noble (2.8.3) and ubuntu:resolute (3.2.0) - a fetch against a dead endpoint makes four attempts with the flag and four attempts without it, and one attempt with Acquire::Retries=0. The calls go back to plain apt-get update. The ctcache install in the clang-tidy job does need a loop. It is a git+https URL, so pip runs git clone in a subprocess to fetch it, and pip's own --retries only covers its HTTP session.
…outage The release asset is the one fetch in this action with nothing to fall back to. The version lookup above it can use the pinned value when the API is unreachable, but a 5xx on the .deb ends the step, and the step failing takes the whole job with it: setup-ros then has no apt source and the build stops at "colcon: command not found". Measured against a server that answers 500 every time: the old flags make 6 attempts over 18 seconds, the new ones make 9 over 131 seconds. A short outage on the release host is now waited out instead of failing every job that starts during it.
mfaferek93
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Every CI job reads from the network before anything of ours is built, and
rosdep updateis the one of those reads this branch can wrap: it fetches the distribution index from raw.githubusercontent.com. A reset connection there fails the whole job with an error that has nothing to do with the change under test. Two jobs on two pull requests failed that way within an hour on the day this branch was opened, both onrosdep updatewithConnection reset by peer.rosdep updatenow gets three attempts five seconds apart in every workflow step that calls it (ci.yml,quality.yml,opcua-plugin.yml). The ctcache install in the clang-tidy job gets the same loop: it is agit+httpsURL, so pip runsgit clonein a subprocess to fetch it, and pip's own--retriesonly covers its HTTP session. A genuine failure still ends the job after the last attempt with its own error text; the loop only names the attempt.The release-asset download in
.github/actions/ros-apt-sourcegets a wider window. It already retried, but only for about 18 seconds, and a job on this branch died when GitHub answered 500 to all six attempts. That download is the one fetch in the action with nothing to fall back to: the version lookup above it can use the pinned value, but a 5xx on the .deb ends the step, and setup-ros then has no apt source, so the build stops atcolcon: command not found. Measured against a server that answers 500 every time, the old flags make 6 attempts over 18 s and the new ones make 9 over 131 s.One read stays unprotected, and it runs earlier than any of these.
ros-tooling/setup-ros@v0.7removes/etc/ros/rosdep/sources.list.d/20-default.listand then runssudo rosdep initonce, with no retry around it (src/setup-ros-ubuntu.ts:135-137in the action). That call fetches20-default.listfrom raw.githubusercontent.com, the same host that fails here, and it happens before the step this branch touches, so a reset there still ends the job. Wrapping it would mean not using the action, which is a larger change than this one.apt-get updateis left alone on purpose. apt has defaultedAcquire::Retriesto 3 since 2.3.2. Measured on the three images this repository builds in, a fetch against a dead endpoint makes the same four attempts with the flag and without it:-o Acquire::Retries=3-o Acquire::Retries=0Issue
Type
CI configuration only.
Testing
mainis the 11rosdep updatecall sites, the ctcache install, and the curl flags plus their comment in the apt-source action. No other line is touched.bash --noprofile --norc -eo pipefail), with a stand-inrosdepandpip3that fail a set number of times.set -edoes not stop the step on a failed attempt: with zero, one or two failures the step ends at 0 and the followingrosdep installstill runs; with three failures the step ends at 1 and the last error is visible.Checklist