Skip to content

CI: cache the apt packages of the install steps - #104

Merged
compot-dev merged 4 commits into
masterfrom
ci-apt-timeout
Nov 18, 2026
Merged

compot-dev merged 4 commits into
masterfrom
ci-apt-timeout

Conversation

@compot-dev

@compot-dev compot-dev commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

The "Install cross-compiler" and "Install qemu-user" steps stop at 5 minutes (#71) and ran out of time seven times on 2026-11-18. With apt's output visible, the cause is the runners' Ubuntu mirror (us-east-2.ec2.archive.ubuntu.com) serving slowly but steadily: "Fetched 44.0 MB in 4min 34s (160 kB/s)", the install done at 5:07. So neither a longer limit (10 minutes timed out on aarch64 too) nor retries help.

Now the downloaded .deb files are kept with actions/cache (Dir::Cache::Archives in the workspace, .apt-archives):

  • the key is per job and per ISO week; a new week restores the last one, so apt fetches only packages that changed, and apt-get autoclean drops versions no longer in the archive;
  • the 5-minute limit from CI: no timeout on the apt-get steps, a stalled mirror holds a job #71 stays, apt drops a connection silent for 30 s and retries (Acquire::http::Timeout=30, Acquire::Retries=3), and update/install run without -qq so a slow step shows in the log.

Checked locally in ubuntu:24.04: the second install of qemu-user from the cache directory reports "Need to get 0 B/38.6 MB of archives".

CI only: the release files do not change, hence no-release.

@compot-dev compot-dev added the no-release Changes the build or the code without changing the binaries: no new version, no release label Nov 18, 2026
@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

PR Summary by Qodo

Extend CI apt install timeouts to 10 minutes

⚙️ Configuration changes 🕐 Less than 5 minutes

Grey Divider

AI Description

• Extend both apt install step timeouts from five to ten minutes to accommodate slow mirrors.
• Retain a timeout for stalled mirrors without changing release files.
Merge Risk 🟡 Medium

Blast radius 🟡 Medium 💥 2 capabilities

  • 2 affected capabilities within one bounded functional scope: CI build validation.
  • This capability is the end-to-end Linux matrix behavior that installs cross-compilers, builds the project, runs smoke/regression checks, and uploads binaries.
  • The changed install step directly gates that outcome across the ARM, aarch64, and mipsel matrix entries; these are variants of one bounded CI validation integration.
  • This is a distinct end-to-end CI operation producing and uploading the static mipsel artifact, separate from the Linux matrix's dynamically linked cross-build validation.
  • Both are part of the same bounded build-validation integration and share the workflow's CI purpose, while their job outputs and toolchain setup are independent.
Sources

.github/workflows/build.yml[81-86]

      # A stalled mirror fails the job instead of holding it (#71); the
      # install normally takes about 20 s, but a slow mirror has gone past
      # 5 min often enough to fail runs that would have finished.
      - name: Install cross-compiler
        if: matrix.platform.pkgs != ''
        timeout-minutes: 10
Behavioral complexity 🟢 Low 🧩 Straightforward

  • Before, both installation steps had a 5-minute timeout.
  • After, the cross-compiler and qemu-user installation steps have a 10-minute timeout.
  • The changed correctness condition is only the allowed duration before the existing failure boundary is enforced; no new branch, state transition, compatibility protocol, concurrency coordination, transformation, or invariant is introduced.
  • The two locations affect separate CI jobs but use the same simple timeout-rule change.
Sources

.github/workflows/build.yml[81-86]

      # A stalled mirror fails the job instead of holding it (#71); the
      # install normally takes about 20 s, but a slow mirror has gone past
      # 5 min often enough to fail runs that would have finished.
      - name: Install cross-compiler
        if: matrix.platform.pkgs != ''
        timeout-minutes: 10
Potential damage 🟡 Medium 🩹 Recoverable

  • A stalled package mirror can delay or fail CI validation, potentially blocking build/artifact confirmation until the timeout expires or a rerun succeeds.
  • The impact is confined to workflow execution and runner time.
  • Reverting the workflow restores the shorter timeout, but does not repair an in-progress runner; recovery requires timeout completion, package availability, or rerunning the affected job.
  • No persistent application data or external production effect is introduced.
  • Recoverable through package mirror recovery, timeout termination, or workflow rerun.
  • Restoring the YAML only changes future runs and does not compensate for runner time already consumed.
Sources

.github/workflows/build.yml[81-86]

      # A stalled mirror fails the job instead of holding it (#71); the
      # install normally takes about 20 s, but a slow mirror has gone past
      # 5 min often enough to fail runs that would have finished.
      - name: Install cross-compiler
        if: matrix.platform.pkgs != ''
        timeout-minutes: 10
Assessed commit 2386410 > Snapshot assessment: the current PR revision could not be verified.
Diagram

graph TD
  Linux["Linux matrix"] --> CrossApt["Cross-compiler install"] --> LinuxBuild["Linux builds"]
  Mipsel["Static mipsel job"] --> QemuApt["QEMU install"] --> StaticBuild["Static build"]
Loading
High-Level Assessment

Keep the step-level timeout increase: it addresses slow mirrors while preserving a limit for stalled installs. Adding retries or changing package installation would introduce complexity without addressing the observed runs that succeeded when rerun.

Files changed (1) +5 / -4

Other (1) +5 / -4
build.ymlAllow ten minutes for CI apt installs +5/-4

Allow ten minutes for CI apt installs

• Raises the Install cross-compiler and Install qemu-user step timeouts from five to ten minutes. Revises the cross-compiler comment to explain that slow mirrors can exceed five minutes.

.github/workflows/build.yml

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (1) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Install steps can run 10 minutes, not 5 📎 Requirement gap ⚙ Maintainability
Description
The Install cross-compiler step's timeout-minutes is raised from 5 to 10, so it no longer
matches the five-minute limit that rule 1 sets for stalled apt-get installs. The same change also
raises the Install qemu-user step from 5 to 10. The PR description states the intent: slow mirrors
ran past 5 minutes seven times, so the limit is deliberately relaxed. The rule's objective, ending a
stalled mirror without holding the job indefinitely, still holds at 10 minutes, so this reads as a
superseded rule rather than a defect. The ticket-derived rule should be updated or closed, or the
rationale recorded against issue #71, so later reviewers do not flag the new value.
Code

.github/workflows/build.yml[86]

+        timeout-minutes: 10
Evidence
Rule 1 requires timeout-minutes: 5 on both install steps. The diff sets timeout-minutes: 10 on
the cross-compiler step (line 86) and on the qemu-user step (line 196). The PR description
explicitly justifies the change, so the rule appears superseded by this PR rather than violated by
accident.

Limit both apt-get installation steps to five minutes
.github/workflows/build.yml[85-86]
.github/workflows/build.yml[195-196]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Rule 1 (ticket-derived from OpenIPC/vectis issue #71) requires `timeout-minutes: 5` on the Install cross-compiler and Install qemu-user steps. This PR raises both to 10 minutes on purpose, because slow apt mirrors on 2026-11-18 exceeded 5 minutes and failed otherwise-passing runs.
## Fix Focus Areas
- .github/workflows/build.yml[81-86]
- .github/workflows/build.yml[193-196]
## Recommended Fix
No workflow change is needed if the 10-minute value is intended. Update the compliance rule or the issue #71 record so it states the new limit and the reason, so future reviews do not flag the change. Alternatively, if the rule should stay at 5 minutes, revert both values, but only if the slow-mirror failures are otherwise handled.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@compot-dev compot-dev changed the title CI: give the apt install steps 10 minutes instead of 5 CI: retry a dead apt mirror connection instead of waiting out the step Nov 18, 2026
@compot-dev

Copy link
Copy Markdown
Collaborator Author

Qodo finding 1 (install steps can run 10 minutes, not 5): resolved in 9e586c5. The limit is back to 5 minutes as #71 sets it; instead apt now drops a dead mirror connection after 30 s and retries (Acquire::http::Timeout=30, Acquire::Retries=3), since the 10-minute limit timed out as well.

@compot-dev compot-dev changed the title CI: retry a dead apt mirror connection instead of waiting out the step CI: cache the apt packages of the install steps Nov 18, 2026
@compot-dev
compot-dev merged commit 423a4c3 into master Nov 18, 2026
27 of 28 checks passed
@compot-dev
compot-dev deleted the ci-apt-timeout branch November 18, 2026 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-release Changes the build or the code without changing the binaries: no new version, no release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant