Skip to content

s3: upload package files concurrently - #1616

Open
arges wants to merge 1 commit into
aptly-dev:masterfrom
arges:arges/concurrent-s3-publish
Open

s3: upload package files concurrently#1616
arges wants to merge 1 commit into
aptly-dev:masterfrom
arges:arges/concurrent-s3-publish

Conversation

@arges

@arges arges commented Aug 6, 2026

Copy link
Copy Markdown

Upload package files in bounded concurrent batches while keeping metadata generation and conflicting paths ordered. Default to four uploads and allow per-endpoint configuration.

Fixes #1615

Requirements

All new code should be covered with tests, documentation should be updated. CI should pass.

Also, to speed up things, if you could kindly "Allow edits and access to secrets by maintainers" in the
PR settings, as this allows us to rebase the PR on master, fix conflicts, run coverage and help with
implementing code and tests.

Description of the Change

This allows for s3 uploads to be done in parallel making uploads work much faster.

Checklist

  • allow Maintainers to edit PR (rebase, run coverage, help with tests, ...)
  • unit-test added (if change is algorithm)
  • functional test added/updated (if change is functional)
  • man page updated (if applicable)
  • bash completion updated (if applicable)
  • documentation updated
  • author name in AUTHORS

Upload package files in bounded concurrent batches while keeping metadata
generation and conflicting paths ordered. Default to four uploads and allow
per-endpoint configuration.

aptly-dev#1615

Signed-off-by: Chris J Arges <carges@cloudflare.com>
@arges
arges force-pushed the arges/concurrent-s3-publish branch from 90e1d0b to 2d44e00 Compare August 7, 2026 00:03
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.13580% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.04%. Comparing base (f59b0d2) to head (2d44e00).

Files with missing lines Patch % Lines
deb/publish.go 65.67% 12 Missing and 11 partials ⚠️
context/context.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1616      +/-   ##
==========================================
- Coverage   77.37%   77.04%   -0.34%     
==========================================
  Files         165      165              
  Lines       15747    15805      +58     
==========================================
- Hits        12185    12177       -8     
- Misses       2356     2417      +61     
- Partials     1206     1211       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3: publishing can be slow

1 participant