From 0bb8e6e51bae0747f2010f930e44f2fb88a228cb Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 11:33:30 +0000 Subject: [PATCH] docs: add extensive architecture and internals documentation Adds documentation/ with 14 chapters covering concepts, architecture, data model and DB key layout, package pool and published storage, core workflows, publishing internals, REST API and task system, CLI and context, configuration reference, signing, query language and dependency resolution, operations, development, and a glossary. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_014MapwAeQdsE1RTyqUizWwX --- README.rst | 3 + documentation/01-overview-and-concepts.md | 140 +++++++ documentation/02-architecture.md | 246 +++++++++++ documentation/03-data-model-and-database.md | 279 +++++++++++++ documentation/04-storage.md | 209 ++++++++++ documentation/05-workflows.md | 410 +++++++++++++++++++ documentation/06-publishing.md | 222 ++++++++++ documentation/07-rest-api-and-tasks.md | 334 +++++++++++++++ documentation/08-cli-and-context.md | 177 ++++++++ documentation/09-configuration.md | 210 ++++++++++ documentation/10-signing-and-verification.md | 143 +++++++ documentation/11-queries-and-dependencies.md | 189 +++++++++ documentation/12-operations.md | 163 ++++++++ documentation/13-development.md | 221 ++++++++++ documentation/14-glossary.md | 47 +++ documentation/README.md | 48 +++ 16 files changed, 3041 insertions(+) create mode 100644 documentation/01-overview-and-concepts.md create mode 100644 documentation/02-architecture.md create mode 100644 documentation/03-data-model-and-database.md create mode 100644 documentation/04-storage.md create mode 100644 documentation/05-workflows.md create mode 100644 documentation/06-publishing.md create mode 100644 documentation/07-rest-api-and-tasks.md create mode 100644 documentation/08-cli-and-context.md create mode 100644 documentation/09-configuration.md create mode 100644 documentation/10-signing-and-verification.md create mode 100644 documentation/11-queries-and-dependencies.md create mode 100644 documentation/12-operations.md create mode 100644 documentation/13-development.md create mode 100644 documentation/14-glossary.md create mode 100644 documentation/README.md diff --git a/README.rst b/README.rst index 90c911522..f401eae0c 100644 --- a/README.rst +++ b/README.rst @@ -21,6 +21,9 @@ Aptly is a swiss army knife for Debian repository management. Documentation is available at `http://www.aptly.info/ `_. For support please use open `issues `_ or `discussions `_. +In-depth documentation of the architecture, data model, workflows and internals lives in the +`documentation/ `_ directory. + Aptly features: * make mirrors of remote Debian/Ubuntu repositories, limiting by components/architectures diff --git a/documentation/01-overview-and-concepts.md b/documentation/01-overview-and-concepts.md new file mode 100644 index 000000000..0f26587c1 --- /dev/null +++ b/documentation/01-overview-and-concepts.md @@ -0,0 +1,140 @@ +# 1. Overview & Concepts + +## 1.1 What problem aptly solves + +`apt` clients consume *repositories*: a directory tree containing `dists//…` +index files (`Release`, `Packages`, `Sources`, `Contents`) and a `pool/` of package files. +Running one of those trees in production raises hard questions: + +* How do I get a **partial copy** of `archive.ubuntu.com` (only some components/architectures, + only some packages and their dependencies)? +* How do I make installation **repeatable** — i.e. freeze "what the archive looked like on + Tuesday" and ship exactly that to 500 machines? +* How do I **publish my own packages** as a proper, signed repository? +* How do I **promote** a tested set of packages from *staging* to *production*, atomically, + with the ability to roll back? + +aptly answers with a small set of composable objects and operations. Its stated goal (from the +CLI help in `cmd/cmd.go`) is "to establish repeatability and controlled changes in a +package-centric environment". + +## 1.2 The object model + +``` + (remote archive on the internet) + │ aptly mirror create / update + ▼ + ┌───────────────┐ snapshot ┌────────────┐ filter / merge / pull ┌────────────┐ + │ Mirror │ ──────────────► │ Snapshot │ ──────────────────────► │ Snapshot │ + │ (RemoteRepo) │ │ (immutable)│ │ (immutable)│ + └───────────────┘ └─────┬──────┘ └─────┬──────┘ + │ │ + ┌───────────────┐ snapshot │ aptly publish snapshot │ + │ Local repo │ ──────────────────────┘ │ │ + │ (LocalRepo) │◄── aptly repo add / include / import ▼ ▼ + └───────┬───────┘ ┌────────────────────────────────┐ + │ aptly publish repo (direct) │ Published repository │ + └──────────────────────────────────────────► │ (PublishedRepo) │ + │ = dists/ + pool/ on a storage │ + └────────────────────────────────┘ +``` + +| Object | Go type | Mutable? | What it holds | +|--------|---------|----------|---------------| +| **Package** | `deb.Package` | no | One `.deb`/`.udeb`/source package: name, version, architecture, dependencies, extra control fields, list of files with checksums | +| **Mirror** | `deb.RemoteRepo` | yes (via `update`) | Configuration of a remote archive (URL, distribution, components, architectures, filter) + the list of packages found at last update | +| **Local repository** | `deb.LocalRepo` | yes | A user-managed list of packages (added from files, or copied from other places) | +| **Snapshot** | `deb.Snapshot` | **no** | An immutable list of packages + provenance ("created from mirror X", "merge of A and B", …) | +| **Published repository** | `deb.PublishedRepo` | re-publishable | A binding *(storage, prefix, distribution)* → one source per *component* (snapshot or local repo), plus the generated files | +| **Package pool** | `aptly.PackagePool` | append/GC | De-duplicated storage of the actual package files | +| **Published storage** | `aptly.PublishedStorage` | | Where the published tree lives (directory, S3, GCS, Azure, Swift, JFrog) | + +Two crucial facts drive everything else: + +1. **Repositories, mirrors and snapshots do not contain packages — they contain *references* + to packages.** A reference is a short byte string (`"Pamd64 nginx 1.24.0-1 8f3a91c2"`); + the package record itself lives once in the database. Creating a snapshot of a + 100,000-package mirror therefore copies a sorted list of 100,000 short keys, not + 100,000 packages (see [chapter 3](03-data-model-and-database.md)). +2. **Package *files* are stored once**, in a pool addressed by SHA-256, no matter how many + mirrors/snapshots/publications use them (see [chapter 4](04-storage.md)). + +## 1.3 Life cycle of a package + +1. **Discovered.** Either parsed from a remote `Packages`/`Sources` index (mirror update) or + read from a `.deb`/`.dsc` on disk (`repo add`, `repo include`). +2. **Registered.** A `deb.Package` is written to the database under a key derived from + *(architecture, name, version, hash-of-files)*. Large/rarely used fields (file list, + dependency lists, extra control fields) are stored under separate "offload" keys. +3. **Downloaded/imported.** The package file(s) are downloaded (mirror) or copied (local repo) + into the pool at a path derived from the file's SHA-256. +4. **Referenced.** The package key is inserted into a `PackageRefList` belonging to a mirror, + local repo or snapshot. +5. **Published.** When a repository containing it is published, the file is hard-linked + (or symlinked/copied/uploaded) from the pool into `pool////…` on the + published storage, and a stanza describing it is written to `Packages`. +6. **Garbage collected.** After all references disappear (mirror dropped, snapshot deleted, + …) `aptly db cleanup` removes the package record and — if no other package references it — + the pool file. + +## 1.4 Typical workflows + +### A. Mirror upstream, freeze, publish + +```bash +aptly mirror create -architectures=amd64 -filter='nginx | curl' -filter-with-deps \ + bookworm-main http://deb.debian.org/debian bookworm main +aptly mirror update bookworm-main # download indexes + packages +aptly snapshot create bookworm-2026-09-30 from mirror bookworm-main +aptly publish snapshot -distribution=bookworm bookworm-2026-09-30 debian +# clients: deb http://host/debian bookworm main +``` + +### B. Publish your own packages + +```bash +aptly repo create -distribution=stable -component=main myrepo +aptly repo add myrepo ./build/*.deb # or: aptly repo include ./build/x.changes +aptly publish repo myrepo internal # publishes directly from the local repo +aptly repo add myrepo ./build/newer.deb +aptly publish update stable internal # re-publish (only changed state) +``` + +### C. Controlled upgrade (`pull`) and atomic switch + +```bash +# take one package (+dependencies) from a newer snapshot into the current one +aptly snapshot pull -no-remove prod-2026-09-01 bookworm-2026-09-30 prod-2026-10-01 nginx +aptly publish switch bookworm prod-2026-10-01 debian # atomically re-point the publication +# rollback: aptly publish switch bookworm prod-2026-09-01 debian +``` + +### D. Multi-component / multi-source publishing + +```bash +aptly publish snapshot -component=main,contrib -distribution=stable snap-main snap-contrib +``` + +Each component of a published repository is fed by its *own* source (snapshot or local repo). +All components of one publication must have the same source kind. + +### E. REST-driven automation (CI/CD) + +```bash +curl -F file=@app_1.0_amd64.deb http://aptly:8080/api/files/ci-upload +curl -X POST http://aptly:8080/api/repos/myrepo/file/ci-upload +curl -X PUT -H 'Content-Type: application/json' \ + -d '{"Signing":{"Passphrase":"…"}}' \ + http://aptly:8080/api/publish/internal/stable +``` + +## 1.5 What aptly deliberately does *not* do + +* It is **not a package build system** (it only ingests finished `.deb`/`.dsc`/`.changes`). +* It is **not a long-running daemon by default**: the CLI is a batch tool; `aptly api serve` + is the (optional) server mode. +* It does **not** support RPM/YUM repositories (the `aptly` Go package comment mentions + "doesn't depend directly on Debian or CentOS", but only Debian formats are implemented). +* It is **not multi-writer**: with the default LevelDB backend only one process may open the + database at a time (see [Operations](12-operations.md)). The `etcd` backend and the + server's `-no-lock` mode relax this. diff --git a/documentation/02-architecture.md b/documentation/02-architecture.md new file mode 100644 index 000000000..cc9435d4b --- /dev/null +++ b/documentation/02-architecture.md @@ -0,0 +1,246 @@ +# 2. Architecture + +## 2.1 System context + +```mermaid +flowchart LR + subgraph Operators + CLI["aptly CLI
(cmd/)"] + HTTPC["REST clients
curl, CI, dashboards"] + end + + subgraph aptly process + CMD["cmd: commander tree"] + API["api: gin router + handlers"] + CTX["context.AptlyContext
(wiring / lifecycle)"] + DEB["deb: domain model & algorithms"] + TASK["task: resource-aware
task queue"] + end + + subgraph Local state under rootDir + DB[("db/ LevelDB
(or etcd)")] + POOL[("pool/
package files")] + UP[("upload/")] + PUB[("public/")] + end + + UPSTREAM["Upstream Debian/Ubuntu
archives (HTTP/HTTPS/FTP)"] + STORES["Published storage:
S3 · GCS · Azure · Swift · JFrog · FS"] + GPG["GnuPG / internal PGP"] + APT["apt clients"] + + CLI --> CMD --> CTX + HTTPC --> API --> CTX + API --> TASK + CTX --> DEB + DEB --> DB + DEB --> POOL + DEB --> UPSTREAM + DEB --> STORES + DEB --> GPG + API --> UP + PUB --> APT + STORES --> APT +``` + +## 2.2 Layered view + +aptly is a single Go module (`github.com/aptly-dev/aptly`, `go 1.26`) producing a single +static binary. Its packages form clean layers; arrows below point from a package to what it +depends on. + +``` +┌───────────────────────────────────────────────────────────────────────────┐ +│ main.go (embeds VERSION and debian/aptly.conf, calls cmd.Run) │ +├───────────────────────────────────────────────────────────────────────────┤ +│ cmd/ CLI commands (commander) api/ REST handlers (gin) │ +├───────────────────────────────────────────────────────────────────────────┤ +│ context/ AptlyContext: config, DB, pool, storages, downloader, signer │ +│ task/ task list, resource locking, task output │ +├───────────────────────────────────────────────────────────────────────────┤ +│ deb/ domain: Package, PackageList, PackageRefList, RemoteRepo, │ +│ LocalRepo, Snapshot, PublishedRepo, collections, index files, │ +│ Changes, query engine, dependency & version logic │ +│ query/ parser for the package query language → deb.PackageQuery │ +├───────────────────────────────────────────────────────────────────────────┤ +│ aptly/ interfaces: PackagePool, PublishedStorage, Downloader, Progress, │ +│ ChecksumStorage, PublishedStorageProvider │ +├───────────────────────────────────────────────────────────────────────────┤ +│ Implementations of the interfaces │ +│ database/{goleveldb,etcddb} files/ (fs pool + fs publish) │ +│ s3/ gcs/ azure/ swift/ jfrog/ (publish; azure also pool) │ +│ http/ (downloader) pgp/ (signer/verifier) console/ (progress) │ +├───────────────────────────────────────────────────────────────────────────┤ +│ utils/ config, checksums, compression, logging, small helpers │ +│ systemd/activation socket activation for `api serve` │ +└───────────────────────────────────────────────────────────────────────────┘ +``` + +### Package responsibilities + +| Package | Key files | Responsibility | +|---------|-----------|----------------| +| `main` | `main.go` | Embeds `VERSION` and `debian/aptly.conf` (default config template); `os.Exit(cmd.Run(...))` | +| `cmd` | `cmd.go`, `run.go`, `mirror_*.go`, `repo_*.go`, `snapshot_*.go`, `publish_*.go`, `db_*.go`, `serve.go`, `api_serve.go`, `task_run.go`, `graph.go` … | One file per (sub)command; builds the `commander` tree; parses flags; calls into `deb`/`context` | +| `context` | `context.go` | `AptlyContext`: lazy singletons for config, DB, pool, published storages, downloader, progress, task list, GPG signer/verifier; signal handling; shutdown | +| `api` | `router.go`, `api.go`, `repos.go`, `mirror.go`, `snapshot.go`, `publish.go`, `files.go`, `packages.go`, `db.go`, `task.go`, `gpg.go`, `graph.go`, `s3.go`, `gcs.go`, `jfrog.go`, `storage.go`, `metrics.go`, `middleware.go`, `error.go` | HTTP handlers, request structs (with swagger annotations), sync/async task wrappers | +| `task` | `list.go`, `task.go`, `resources.go`, `output.go` | Background task queue with **resource-based mutual exclusion** | +| `deb` | see §2.4 | Everything Debian-specific and all repository logic | +| `query` | `lex.go`, `syntax.go`, `query.go` | Lexer and recursive-descent parser for query strings | +| `aptly` | `interfaces.go`, `conf.go`, `const.go`, `report.go`, `version.go` | Backend-neutral interfaces & globals (`Version`, `AptlyConf`, `EnableDebug`) | +| `database` | `database.go` + `goleveldb/`, `etcddb/` | KV `Storage` abstraction; two implementations | +| `files` | `package_pool.go`, `public.go` | Local filesystem pool + local filesystem published storage | +| `s3`, `gcs`, `azure`, `swift`, `jfrog` | `public.go`, plus `package_pool.go` in `azure` | Cloud publish targets; Azure can also host the pool | +| `http` | `download.go`, `grab.go`, `compression.go`, `temp.go`, `gcp_auth.go`, `fake.go` | Downloaders (default & grab), "try .xz/.bz2/.gz" logic, `ar+https` GCP Artifact Registry auth | +| `pgp` | `gnupg.go`, `internal.go`, `openpgp.go`, `gnupg_finder.go` | `Signer`/`Verifier` implemented via external `gpg` or pure Go | +| `console` | `progress.go` | Terminal `aptly.Progress` with bars and coloured output | +| `utils` | `config.go`, `checksum.go`, `compress.go`, `copyfile.go`, `logging.go`, `human.go`, `list.go`, `utils.go` | Shared helpers | +| `systemd/activation` | `listeners.go`, `files.go`, `packetconns.go` | Socket-activation helper used by `aptly api serve` | +| `docs` | `index.go`, `*.md`, generated `docs.go`/`swagger.*` | Swagger sources and embedded docs page | + +## 2.3 Dependency inversion: the `aptly` interfaces + +The `aptly` package (`aptly/interfaces.go`) is the seam that keeps the domain code +storage-agnostic. `deb` depends **only** on these interfaces: + +```go +type PackagePool interface { // where package *files* live + Verify(...) ; Import(...) ; LegacyPath(...) ; Size(...) ; Open(...) + FilepathList(...) ; Remove(...) +} +type LocalPackagePool interface { // optional capability: pool is on a local FS + Stat ; GenerateTempPath ; Link ; Symlink ; FullPath +} +type PublishedStorage interface { // where the *published tree* lives + MkDir ; PutFile ; RemoveDirs ; Remove ; LinkFromPool ; Filelist + RenameFile ; SymLink ; HardLink ; FileExists ; ReadLink +} +type PublishedStorageProvider interface { GetPublishedStorage(name string) (PublishedStorage, error) } +type Downloader interface { Download ; DownloadWithChecksum ; GetProgress ; GetLength } +type Progress interface { … bars + Printf/ColoredPrintf … } +type ChecksumStorage interface { Get(path) ; Update(path, *ChecksumInfo) } +``` + +Consequences: + +* Adding a new publish target means implementing `PublishedStorage` (see + [Development](13-development.md#add-a-published-storage-backend)). +* The Azure pool implements `PackagePool` but **not** `LocalPackagePool`, so hard-linking or + symlinking pool files into a published tree is impossible; `LinkFromPool` then must + copy/upload (`files.PublishedStorage.LinkFromPool` errors with *"cannot link … from + non-local pool"* unless `link_method: copy`). +* `Progress` lets the same algorithm run with a terminal progress bar (CLI) or a buffered + text log (`task.Output`, REST) — see [chapter 7](07-rest-api-and-tasks.md). +* `AptlyContext` itself implements `PublishedStorageProvider`, so `deb.PublishedRepo.Publish` + can lazily obtain the storage named in its `Storage` field (`""`, `filesystem:x`, + `s3:x`, `gcs:x`, `swift:x`, `azure:x`, `jfrog:x`). + +## 2.4 The `deb` package (the heart) + +| Concept | File(s) | Notes | +|---------|---------|-------| +| Parsing of RFC-822-like control data | `format.go` | `Stanza`, `ControlFileReader`; canonical field ordering when writing | +| `Package` | `package.go`, `package_files.go`, `package_deps.go`, `package_collection.go` | In-memory model with lazy "offloaded" fields; `PackageCollection` persists them | +| `PackageList` | `list.go` | In-memory set of packages, indexed for search, dependency verification, filter | +| `PackageRefList` | `reflist.go` | Sorted `[][]byte` of package keys; Diff/Merge/Subtract; msgpack-encoded | +| Mirror | `remote.go`, `ppa.go` | `RemoteRepo` + `RemoteRepoCollection`; index download & parsing; `ppa:` URL expansion | +| Local repo | `local.go`, `import.go`, `changes.go`, `uploaders.go` | `LocalRepo`; file collection/import; `.changes` handling; uploader ACL rules | +| Snapshot | `snapshot.go` | `Snapshot` + `SnapshotCollection` | +| Publication | `publish.go`, `index_files.go`, `contents.go` | `PublishedRepo` and `Publish()`; generation/signing/atomic rename of index files; contents index | +| Query engine | `query.go` | `PackageQuery` implementations (`AndQuery`, `OrQuery`, `NotQuery`, `FieldQuery`, `DependencyQuery`, `PkgQuery`, `MatchAllQuery`) | +| Version/deps | `version.go`, `package_deps.go` | Debian version comparison, dependency parsing | +| Factory | `collections.go` | `CollectionFactory` hands out one collection per type over a shared DB | +| Misc | `graph.go`, `find_dangling.go`, `checksum_collection.go`, `debian.go`, `deb.go` | Graphviz export, dangling-ref detection, checksum cache, `.deb`/`.dsc` readers | + +### Collections pattern + +Each persistent type has a `*Collection` (e.g. `SnapshotCollection`) taking a +`database.Storage`. A collection offers `Add`, `Update`, `Drop`, `ByName`, `ByUUID`, +`ForEach`, `Len`, and — importantly — `LoadComplete`. Objects are loaded **shallow** +(metadata only, cheap) and their `PackageRefList` is fetched from a separate key only when +`LoadComplete` is called. This keeps `aptly snapshot list` fast even with huge snapshots. + +`CollectionFactory` (`deb/collections.go`) is created once per command/request and caches +one instance of each collection, guarded by a mutex. `Flush()` drops them to reclaim memory. + +> **Lock order.** `api/api.go` documents the canonical acquisition order for collections to +> avoid deadlocks: `RemoteRepoCollection → LocalRepoCollection → SnapshotCollection → +> PublishedRepoCollection`. + +## 2.5 Runtime modes + +| Mode | Entry | Process model | DB handling | +|------|-------|---------------|-------------| +| **CLI batch** | `aptly …` | One process per command | Opens LevelDB (exclusive file lock) on first use; closes on exit. Long operations (mirror update) *close* the DB during downloading so other commands can run | +| **API server** | `aptly api serve -listen=:8080` | Long-lived HTTP server (gin); mutating requests run as tasks | DB stays open for the life of the server. With `-no-lock` the server closes the DB whenever there are zero in-flight requests | +| **Static file server** | `aptly serve -listen=:8080` | Long-lived; serves `rootDir/public` (and prints recommended `sources.list` lines) | Read-only lookup of published repos at start | +| **Task script** | `aptly task run -filename=cmds.txt` or stdin | Runs many aptly commands in a single process (one DB open) | Avoids repeated open/close cost | +| **Systemd-managed API** | `debian/aptly-api.service` | Runs `aptly api serve -config=/etc/aptly.conf … -listen=…` as user `aptly-api`. `aptly api serve` can also adopt a single listener fd passed by systemd socket activation (`systemd/activation`) | Same as API server | + +## 2.6 Concurrency model + +* **CLI**: essentially single-threaded orchestration; parallelism is confined to + * package downloading (`download_concurrency` goroutines in `cmd/mirror_update.go`), + * filesystem walking (`saracen/walker`), + * compression (`klauspost/pgzip`). +* **API**: gin serves each HTTP request in its own goroutine, but every operation that + mutates state is wrapped in a **task** (`task.List`). Tasks declare the *resources* they + need (e.g. the key of a local repo `"L"`); the list guarantees no two tasks with + intersecting resources run concurrently, and special keys + `__all__` / `__alllocalrepos__` express "everything"/"all local repos". See + [chapter 7](07-rest-api-and-tasks.md#73-the-task-system). +* **Database**: LevelDB is safe for concurrent goroutines inside one process. Cross-process + safety is provided by the LevelDB file lock ("resource temporarily unavailable" ⇒ retry with + jittered ~10 s sleeps up to `database_open_attempts`). +* **Mirror update guard**: a `RemoteRepo` records `Status` and `WorkerPID` in the DB; + `CheckLock()` refuses a second update when that PID is alive (override: `-force`). + +## 2.7 Cross-cutting concerns + +| Concern | Where | Behaviour | +|---------|-------|-----------| +| Configuration | `utils/config.go`, `context.config()` | YAML or JSON (with comments). Search order: `-config` flag, `~/.aptly.conf`, `/usr/local/etc/aptly.conf`, `/etc/aptly.conf`. If none found, a default (the embedded `debian/aptly.conf`) is written to `~/.aptly.conf` | +| Logging | `utils/logging.go` (zerolog) | `log_level`, `log_format: default|json`; JSON mode also makes gin log JSON (`api.JSONLogger`) | +| Error handling (CLI) | `context.Fatal` | Commands return `error`; `cmd.Run` converts to exit code 1 (2 for usage errors) via a panic/recover on `FatalError` | +| Error handling (API) | `api.AbortWithJSONError` | Sets JSON content type and aborts with status; gin's `ErrorLogger` middleware renders `{"error": "..."}` | +| Metrics | `api/metrics.go` | Prometheus counters/gauges/histograms (opt-in `enable_metrics_endpoint`) incl. per-repo package counts | +| Signals | `context.GoContextHandleSignals`, `cmd/api_serve.go` | First `^C` cancels the Go context (downloads stop, cleanup runs); second `^C` aborts immediately. API server waits for running tasks before shutdown | +| Reproducible builds | `deb/publish.go` | `SOURCE_DATE_EPOCH` env var overrides `Date:` in the top-level `Release` file | +| Debug tooling | `aptly/version.go` (`const EnableDebug`) | Compile-time constant (default `false`); when set to `true` at build time it adds `-cpuprofile`, `-memprofile`, `-memstats`/`-meminterval` flags and gin debug mode | + +## 2.8 Key design decisions and their rationale + +1. **References, not copies.** `PackageRefList` (sorted list of package keys) makes + snapshot/merge/diff operations linear merges over sorted byte slices and makes snapshots + nearly free. The price: deleting data needs a mark-and-sweep GC (`db cleanup`). +2. **Content-addressed pool.** Files are keyed by SHA-256, so identical files from different + mirrors/versions are stored once, and integrity can be re-verified cheaply. A MD5-keyed + *legacy* layout (aptly ≤ 1.0) is still read (unless `skip_legacy_pool`). +3. **Package key includes a hash of the file set** (`FilesHash`). Two "same name+version" + packages with different bytes (e.g. rebuilt without version bump) become *distinct keys*, + so a snapshot can never silently change content. +4. **Offloaded package fields.** The base `Package` record is small; `Files`, `Depends…` and + `Extra` are separate keys (`xF`, `xD`, `xE` prefixes). Listing, diffing and merging rarely + need them, so they are loaded lazily. +5. **Publishing writes to temp names and renames.** Index files are generated in a temp dir, + uploaded as `*.tmp` when re-publishing, then renamed — minimising the time clients could + see a half-written state (see [chapter 6](06-publishing.md)). +6. **One codebase, two front-ends.** CLI commands and REST handlers are separate thin + layers over the same `deb` operations; the REST layer adds tasks, JSON and locking. +7. **Interfaces at every I/O boundary** (DB, pool, published storage, downloader, signer, + progress) — unit tests inject fakes (`http/fake.go`, `files/mocks.go`, `console`), and new + back-ends are additive. +8. **Explicit DB open/close.** The CLI can release the LevelDB lock during long network + phases (`context.CloseDatabase()` / `ReOpenDatabase()`), which is why a mirror update + collects all download tasks first, closes the DB, downloads, reopens, then imports. + +## 2.9 Build & release artefacts + +* `go build` → `aptly` binary. `main.go` embeds `VERSION` (generated by `go generate` from + `make version`, based on `debian/changelog` + git commit for CI builds) and + `debian/aptly.conf`. +* Debian packaging (`debian/`): packages `aptly` (binary, man page, bash completion), + `aptly-api` (systemd unit + `/etc/aptly.conf` handling) and `aptly-dbg`. +* `completion.d/` — bash and zsh completions; `man/` + `_man/gen.go` — man page generation from + command help text. +* `docs/` + `make swagger` — OpenAPI generation via `swag` from handler annotations. diff --git a/documentation/03-data-model-and-database.md b/documentation/03-data-model-and-database.md new file mode 100644 index 000000000..74ae58457 --- /dev/null +++ b/documentation/03-data-model-and-database.md @@ -0,0 +1,279 @@ +# 3. Data Model & Database + +aptly's entire *metadata* state lives in an ordered key/value store. Package *files* do not +(see [chapter 4](04-storage.md)). This chapter describes the key space, value encodings, the +in-memory model built from them, and the storage back-ends. + +## 3.1 The KV abstraction (`database/database.go`) + +```go +type Storage interface { + Reader // Get(key) ([]byte, error) → database.ErrNotFound + Writer // Put(key, value) ; Delete(key) + PrefixReader // HasPrefix ; ProcessByPrefix ; KeysByPrefix ; FetchByPrefix + CreateBatch() Batch // buffered writes, applied by Write() + OpenTransaction() (Transaction, error) // Commit() / Discard() + CreateTemporary() (Storage, error) // scratch DB of the same type + Open() ; Close() ; CompactDB() ; Drop() +} +``` + +Everything else in aptly is expressed using only these operations, notably **prefix scans** +(all snapshots = every key starting with `S`). + +### Back-ends + +| Backend | Package | Selected by | Notes | +|---------|---------|-------------|-------| +| **LevelDB** (default) | `database/goleveldb` | `database_backend.type: leveldb` (or unset) | Embedded, single-process (exclusive file lock), bloom filter (10 bits), 256 cached open files. Path: `rootDir/db` or `database_backend.db_path`. `Put` skips writing if the value is unchanged. Temporary DBs are created in the OS temp dir with throttled compaction settings (`CompactionL0Trigger=32`, `WriteL0Pause=96`, `WriteL0Slowdown=64`) | +| **etcd** | `database/etcddb` | `database_backend.type: etcd`, `url: host:2379` | Network KV; allows several aptly processes to share state. Message limits raised to ~2 GiB, 30 s dial timeout. Temporary DBs are emulated by a UUID key prefix in the same cluster | + +`aptly db recover` (`cmd/db_recover.go`) runs `goleveldb.RecoverDB` to rebuild a corrupted +LevelDB. `system/leveldb2etcd.py` migrates LevelDB → etcd. + +## 3.2 Key space + +| First byte(s) | Meaning | Key format | Value | +|---------------|---------|-----------|-------| +| `P` | **Package** (core record) | `P ` e.g. `Pamd64 nginx 1.24.0-1 8f3a91c2` | `0xC1 0x01` marker + msgpack(`Package` public fields) | +| `xF` + package key | Package **files** (offloaded) | `xF` + full package key | msgpack(`PackageFiles`) | +| `xD` + package key | Package **dependencies** (offloaded) | `xD` + key | msgpack(`PackageDependencies`: Depends, PreDepends, Suggests, Recommends, Build-Depends…) | +| `xE` + package key | Package **extra** control fields (offloaded) | `xE` + key | msgpack(`Stanza`, i.e. `map[string]string`) | +| `xC` + package key | Cached **contents** (list of file paths in the `.deb`) | `xC` + key | msgpack(`[]string`); computed lazily on first publish with Contents enabled | +| `R` | **Remote repo** (mirror) | `R` | msgpack(`RemoteRepo`) — metadata only | +| `L` | **Local repo** | `L` | msgpack(`LocalRepo`) | +| `S` | **Snapshot** | `S` | msgpack(`Snapshot`) | +| `U` | **Published repo** | `U>>` (`StoragePrefix()` + `">>"` + `Distribution`) | msgpack(`PublishedRepo`) | +| `E` | **Package ref list** for a mirror/local repo/snapshot | `E` | msgpack(`PackageRefList{Refs [][]byte}`) | +| `E` (published) | Ref list *frozen at publish time* for a **local-repo** publication | `E` | msgpack(`PackageRefList`) | +| `C` | **Checksum cache** for pool files | `C` | msgpack(`utils.ChecksumInfo`) | + +Notes: + +* UUIDs are random v4 (`google/uuid`), assigned at creation; **names are only unique + indexes** enforced by the collections (renames just rewrite the metadata record). +* Because a ref list key is `E` and each mirror/local repo/snapshot has its own UUID, + the three never collide. Published local-repo ref lists append the component name to + the published repo's own UUID. +* The `P` scan is what makes "all packages in the DB" cheap: `PackageCollection.AllPackageRefs()` + is simply `KeysByPrefix("P")` — the very first byte deliberately excludes `xF/xD/xE/xC` + keys (they start with `x`). +* Published repos are **all loaded into memory** on first use (`PublishedRepoCollection.loadList`) + because they are few and lookups by *(storage, prefix, distribution)* / by source are + common. + +### Entity relationships + +```mermaid +erDiagram + PACKAGE ||--o{ PACKAGE_FILE : "has (xF)" + PACKAGE_FILE }o--|| POOL_FILE : "PoolPath (sha256 addressed)" + REMOTE_REPO ||--|| REFLIST : "E" + LOCAL_REPO ||--|| REFLIST : "E" + SNAPSHOT ||--|| REFLIST : "E" + REFLIST }o--o{ PACKAGE : "refs (P keys)" + SNAPSHOT }o--o{ SNAPSHOT : "SourceIDs (merge/pull/filter)" + SNAPSHOT }o--o| REMOTE_REPO : "SourceKind=repo" + SNAPSHOT }o--o| LOCAL_REPO : "SourceKind=local" + PUBLISHED_REPO }o--o{ SNAPSHOT : "Sources[component] (kind=snapshot)" + PUBLISHED_REPO }o--o{ LOCAL_REPO : "Sources[component] (kind=local)" + PUBLISHED_REPO ||--o{ REFLIST : "E (kind=local only)" +``` + +## 3.3 The Package record + +```go +type Package struct { + Name, Version, Architecture string + SourceArchitecture string // for source packages: real Architecture: field + Source string // binary → its source package name (+ optional version) + Provides []string + FilesHash uint64 // FNV-64a over the sorted file list + checksums + IsInstaller, IsSource, IsUdeb, V06Plus bool + // offloaded (lazy; not in the core record): + deps *PackageDependencies ; extra *Stanza ; files *PackageFiles ; contents []string + collection *PackageCollection +} +``` + +### Package key + +``` +P +``` + +* `V06Plus == true` (all packages created by aptly ≥ 0.6) includes `FilesHash`; older + packages use the *short key* `P `. +* `FilesHash = FNV-64a( for each file sorted by name: filename ‖ size ‖ MD5 ‖ SHA1 ‖ SHA256 )` + (`deb/package_files.go: PackageFiles.Hash`). Same name/version/arch but different bytes ⇒ + **different key**, so two mirrors can legitimately hold "different" `foo_1.0_amd64`. +* Keys sort lexicographically by architecture, then name, then version-as-string, then hash. + This ordering is **not** Debian version order; version comparisons use `CompareVersions`. + +### Where package data comes from + +| Source | Constructor | Notes | +|--------|-------------|-------| +| `Packages` stanza (mirror) | `NewPackageFromControlFile` | Pulls `Filename`, `Size`, `MD5sum`, `SHA1`, `SHA256`, `SHA512`; the `Filename` directory is remembered as `downloadPath` (for download only, not stored) | +| `Sources` stanza (mirror) | `NewSourcePackageFromControlFile` | Parses `Files`, `Checksums-Sha1/256/512`; `Architecture` is set to `source` | +| `.deb` file | `deb/deb.go` `GetControlFileFromDeb` → `NewPackageFromControlFile` | `ar` archive → `control.tar.{gz,xz,zst,…}` → `control` | +| `.udeb` | `NewUdebPackageFromControlFile` | Same but `IsUdeb` | +| `.dsc` | `deb/debian.go` `GetControlFileFromDsc` → source constructor | Also verifies referenced files exist next to it | +| Debian-installer images | `NewInstallerPackageFromControlFile` | Parses the installer `SHA256SUMS` file of a component/architecture and creates one synthetic package named `installer` (`IsInstaller`) whose files are the listed images; file sizes are obtained with `GetLength` (HTTP HEAD). Published under `dists///installer-/current/images` (`legacy-images` for `focal`) | + +### Offloading and lazy loading + +`Package.Files()`, `.Deps()`, `.Extra()` and `.Contents()` transparently load their data from +`xF/xD/xE/xC` keys through the `collection` back-pointer the first time they are used. The +publish loop explicitly drops (`pkg.files = nil` …) these after writing each stanza to keep +memory bounded on huge repositories. + +### Migration of old databases + +`PackageCollection.ByKey` detects the encoding by the `0xC1 0x01` marker: + +* Present → current format (msgpack of the split record). +* Absent → aptly < 0.4 "all fields in one struct" (`oldPackage`); it is decoded, converted and + **immediately rewritten** in the new format. + +Similar tolerant decoding exists in `RemoteRepo.Decode` (Go < 1.2 time encoding, codec +`TimeNotBuiltin`) and `PublishedRepo.Decode` (pre-0.6 single `SnapshotUUID`+`Component` fields +→ `Sources` map; default `SourceKind = snapshot`). + +## 3.4 PackageRefList — the unit of "repository content" + +```go +type PackageRefList struct { Refs [][]byte } // ALWAYS sorted lexicographically +``` + +Refs are package keys **without** any prefix other than the leading `P` (`p.Key("")`). +Because the slice is sorted, all set-like operations are linear two-pointer merges: + +| Operation | Method | Used by | +|-----------|--------|---------| +| Membership | `Has(p)` — binary search | conflict checks | +| Set difference | `Subtract(r)` | `db cleanup` (all keys − referenced keys) | +| Diff with details | `Diff(r, packageCollection)` | `snapshot diff` (CLI and `GET /api/snapshots/:name/diff/:with`); loads packages to classify entries as *added / removed / changed* | +| Merge | `Merge(r, overrideMatching, ignoreConflicting)` | `snapshot merge`, `db cleanup`, repo add | +| Latest per name | `FilterLatestRefs()` | `snapshot merge -latest` | +| Iterate | `ForEach` | everything | +| Build from list | `NewPackageRefListFromPackageList` | after filtering/pull | + +### Merge semantics (`deb/reflist.go: Merge`) + +`Merge(l, r, overrideMatching, ignoreConflicting)` walks both lists in key order: + +* identical refs → keep one; +* `overrideMatching=true` → when `(arch,name)` is equal on both sides, the **right** + package replaces the left one ("last wins"); +* `ignoreConflicting=false` → same `(arch,name,version)` but different file-hash: the right + one wins (conflict resolution); +* `ignoreConflicting=true, overrideMatching=false` → keep everything, a plain union. + +Callers: `snapshot merge` calls `Merge(right, overrideMatching, false)` with +`overrideMatching = !latest && !no-remove`, and afterwards calls `FilterLatestRefs()` for +`-latest` (`-latest` and `-no-remove` are mutually exclusive). `db cleanup` calls +`Merge(refs, false, true)` to build the union of all referenced packages. The API snapshot +create/pull handlers use `Merge(…, true, false)`. + +### Encoding + +`Encode()` → msgpack via `ugorji/go/codec` (`MsgpackHandle`). `Decode()` enables the codec's `ZeroCopy` +option, so decoded refs may alias the input buffer rather than copying it. + +## 3.5 PackageList — the in-memory working set + +`PackageList` (`deb/list.go`) is a **map** `key → *Package` plus optional indexes: + +* `packagesIndex` — packages sorted by name ascending, then version **descending** (newest first, + via `CompareVersions`), then architecture, for binary search (`PrepareIndex()`; required before + `Search/Filter`); +* `providesIndex` — virtual package name → providers. + +Two key functions exist: the *short key* (no file hash) prevents adding two packages of the +same name/version/arch — a `PackageConflictError` is returned (mirror update logs +"skipping package … duplicate in packages index"); the *full key* variant +(`NewPackageListWithDuplicates(true, …)`) allows duplicates. + +Typical life: `NewPackageListFromRefList(reflist, packageCollection, progress)` loads every +referenced `Package` from the DB (progress bar `BarGeneralBuildPackageList`), then callers +`PrepareIndex()`, `Filter`, `Search`, `VerifyDependencies`, `ForEachIndexed`, +`Append`, `Remove`… + +## 3.6 The persisted entities + +### RemoteRepo (`R…`) + +`UUID, Name, ArchiveRoot, Distribution, Components[], Architectures[], Meta (Release fields), +LastDownloadDate, ReleaseFiles (map path → ChecksumInfo), Filter, FilterWithDeps, +Status (idle/updating), WorkerPID, SkipComponentCheck, SkipArchitectureCheck, +DownloadSources, DownloadUdebs, DownloadInstaller, DownloadAppStream, AppStreamFiles`. +Ref list under `E`. *Flat* repositories (distribution ending in `/` or starting with +`.`) are detected in `NewRemoteRepo` and have no components. + +### LocalRepo (`L…`) + +`UUID, Name, Comment, DefaultDistribution, DefaultComponent, Uploaders (json ACL)`; ref list +`E`. + +### Snapshot (`S…`) + +`UUID, Name, CreatedAt, SourceKind ("repo"|"local"|"snapshot"), SourceIDs[], Description, +Origin, NotAutomatic, ButAutomaticUpgrades, AppStreamFiles`; ref list `E`. +Snapshots are immutable by convention: the only mutating operations are **rename** and +**description edit**. Provenance (`SourceKind`+`SourceIDs`) forms a DAG that +`aptly graph` (Graphviz) and `walkUpTree` (used to infer default distribution/component) +traverse. + +### PublishedRepo (`U…`) + +``` +UUID, Storage, Prefix, Distribution, +Origin, NotAutomatic, ButAutomaticUpgrades, Label, Suite, Codename, Version, +Architectures[], SourceKind ("snapshot"|"local"), +Sources: map[component]sourceUUID, +SkipContents, SkipBz2, AcquireByHash, SignedBy, MultiDist, +Revision {Sources: map[component]name} // staged, not-yet-applied source changes +``` + +* `(Storage, Prefix, Distribution)` must be unique (`CheckDuplicate`). +* For `SourceKind == "local"` the *ref list is copied* at publish time into + `E`; the publication therefore represents the local repo **as of + publish time** and only changes on `publish update`. +* `Revision` implements the "`publish source add/remove/update/…`" staging workflow: edit the + set of sources first, then apply them with `publish update` + (`PublishedRepo.Update` consumes/`DropRevision`s it). + +### Checksum cache (`C`) + +Pool files are validated against their DB-cached checksums rather than re-hashed on every +run. `ChecksumCollection` implements `aptly.ChecksumStorage`; `PackagePool.ensureChecksums` +fills it on demand. + +## 3.7 Transactions, batches, and consistency + +* **Package writes** use a transaction (`PackageCollection.Update` / + `UpdateInTransaction`) so the four keys of a package (`P`, `xF`, `xD`, `xE`) are atomic. + Mirror update opens **one transaction for the whole package set** in `FinalizeDownload` + (see [chapter 5](05-workflows.md#51-mirror-update)). +* **Entity + ref list** writes use a `Batch` (`collection.Update` → `Put(Key) ; Put(RefKey)`), + applied atomically by LevelDB's write batch. +* There is **no cross-entity transaction**: e.g. dropping a snapshot deletes `S…` and + `E…` in one batch, but the "is it published?" check is a prior read. In the API this is + protected by task resource locks, in the CLI by the process-exclusive DB lock. +* Referential integrity is **soft**: a snapshot references package keys that must exist. + `aptly db cleanup` never deletes a package referenced by any mirror, local repo, snapshot + or local-repo publication. The reverse problem — refs pointing to packages that no longer exist — + is repaired for **local repos** by `aptly db recover` (`deb/find_dangling.go`, + `FindDanglingReferences` → the refs are removed from the repo). + +## 3.8 Performance characteristics + +* A package record is small (core) plus a few KB of offloaded fields; a snapshot of N packages + costs on the order of N short keys, not N packages. +* Loading a ref list into a `PackageList` reads every referenced package from the DB (several + Gets per package) — this is the dominant cost of `snapshot merge/pull/filter` and `publish`, + and the reason those operations show progress bars. Ref-only operations (`Merge`, + `Subtract`, `Has`) never touch package records. +* Sizes vary with the mirror (long `Description` fields make the `xE` records the largest part + of the DB); measure your own mirrors with `du -sh $rootDir/db $rootDir/pool`. diff --git a/documentation/04-storage.md b/documentation/04-storage.md new file mode 100644 index 000000000..317a67522 --- /dev/null +++ b/documentation/04-storage.md @@ -0,0 +1,209 @@ +# 4. Package Pool & Published Storage + +aptly separates two storage concepts that are often conflated: + +| | **Package pool** | **Published storage** | +|-|------------------|-----------------------| +| Purpose | Authoritative, de-duplicated store of *every package file aptly knows about* | The tree that apt clients actually download (`dists/`, `pool/`) | +| Interface | `aptly.PackagePool` (+ optional `LocalPackagePool`) | `aptly.PublishedStorage` | +| Count | Exactly **one** per aptly instance | **Any number** (named endpoints) | +| Addressing | Content-addressed (SHA-256) | Human/apt layout (`pool////file.deb`) | +| Back-ends | Local filesystem, Azure Blob | Filesystem, S3, GCS, Azure Blob, Swift, JFrog Artifactory | +| Written by | mirror update, `repo add/include` | `publish snapshot/repo/update/switch` | +| Removed by | `db cleanup` | `publish drop`, `publish update/switch` cleanup | + +## 4.1 The local package pool (`files/package_pool.go`) + +### Layout + +For a file with SHA-256 `476e0cdac6bc757dd2b78bacc1325323b09c45ecb41d4562deec2a1c7c148405` +and name `my-package_1.2.3_all.deb`: + +``` +/47/6e/0cdac6bc757dd2b78bacc13253_my-package_1.2.3_all.deb + │ │ └── sha256[4:32] + "_" + basename + │ └────── sha256[2:4] + └───────── sha256[0:2] +``` + +(`buildPoolPath`: `hash[0:2]/hash[2:4]/hash[4:32] + "_" + filename`.) The pool root is +`rootDir/pool` unless `packagepool_storage.path` says otherwise. See also +[`files/README.md`](../files/README.md). + +**Legacy layout** (aptly ≤ 1.0): `//`. It is still *read* unless +`skip_legacy_pool: true` (the default for fresh configs); new files are never written that +way. Package records keep the `PoolPath` they were imported with, so mixed pools work. + +### Operations + +| Method | Behaviour | +|--------|-----------| +| `Import(src, basename, checksums, move, checksumStorage)` | Serialised by a mutex. Computes any missing checksums (MD5+SHA-256 mandatory; recomputes if size mismatched). If the target exists **with identical size**, it is trusted after `ensureChecksums`, and its path is returned (de-duplication); if size differs → error *"file … already exists"*. If a legacy path exists with same size, that path is returned. Otherwise it creates the directories and **tries `os.Link` (hard link) when source and pool are on the same device**, falling back to a copy. Finally it records checksums in the checksum cache and, if `move`, removes the source | +| `Verify(poolPath, basename, checksums, checksumStorage)` | Existence + size + checksum check. With no `poolPath` it *guesses* candidates from SHA-256 (modern) and MD5 (legacy). Fills back the full `ChecksumInfo` into the argument on success. Returns `(path, found, err)` | +| `Open`, `Size`, `Stat` | Direct file access | +| `Remove(path)` | Deletes the file, returns its size | +| `FilepathList(progress)` | Parallel walk (`saracen/walker`) of the whole pool for GC; returns sorted relative paths | +| `GenerateTempPath(name)` | Returns a random path *inside the pool root* — `///` — so a downloaded file sits on the same filesystem as its final location and `Import` can **hard-link** it instead of copying | +| `Link`, `Symlink`, `FullPath` | Used by `files.PublishedStorage.LinkFromPool` for hardlink/symlink publishing | + +### Checksum cache + +Verifying a multi-terabyte pool by re-hashing would be prohibitive. `ensureChecksums` looks +up `C` in the DB (`ChecksumCollection`) and only hashes the file when absent, then +stores `ChecksumInfo{Size, MD5, SHA1, SHA256, SHA512}`. Consequence: **if you replace a pool +file behind aptly's back with different bytes of the same size, aptly will not notice** until +the cache entry is dropped. + +### What "mirror update" and "repo add" do with the pool + +* **Mirror update** first asks `Verify` for each file of each package; only missing/mismatching + files are queued for download (`Package.DownloadList`). Downloads land in temp paths from + `GenerateTempPath`, are checksum-verified, then `Import`ed with `move=true`. +* **repo add/include/`POST /api/repos/:name/file/...`** (`deb.ImportPackageFiles`) call `Import` + with the user's file as the source and `move=false`; removal of successfully imported source + files is a separate step: CLI `-remove-files` (add) / `-no-remove-files` (include); REST removes them + after import unless `noRemove=1` / `noRemoveFiles=1`. The file appears in the pool exactly once regardless of how many repos add it. + +## 4.2 The Azure package pool (`azure/package_pool.go`) + +Selecting + +```yaml +packagepool_storage: + type: azure + account_name: … + account_key: … + container: pool + prefix: "" + endpoint: "" +``` + +stores pool files as blobs with **the same relative path scheme** as the local pool. Differences +from the local pool: + +* No `LocalPackagePool` methods → files cannot be hard-linked or symlinked; publishing to a + filesystem endpoint must use `link_method: copy`, and the cloud back-ends stream the blob + through `Open`. +* No legacy (MD5) layout (`LegacyPath` returns an empty path). +* `Import` uploads (the `move` flag is ignored); `GenerateTempPath` isn't available so the + mirror downloader uses `os.CreateTemp`. +* `Remove`/`FilepathList` use blob APIs. + +## 4.3 Published storage interface + +```go +MkDir(path) RemoveDirs(path, progress) Remove(path) +PutFile(path, sourceFilename) RenameFile(old, new) Filelist(prefix) +LinkFromPool(prefix, relPath, fileName, sourcePool, sourcePath, checksums, force) +SymLink(src, dst) HardLink(src, dst) FileExists(path) ReadLink(path) +``` + +* Paths are relative to the storage root (plus the configured `prefix` for cloud back-ends). +* `LinkFromPool` is the *only* way package files reach the published tree; it is responsible + for idempotency: if the destination already exists it must decide **same / different** + (inode for hard links, MD5 or size for copies, MD5 metadata for object stores). Different + content is an error unless `force` (`-force-overwrite`). +* `RenameFile` provides the "atomic-ish" swap of `*.tmp` index files during re-publish. + Object stores implement it as copy + delete. +* `SymLink/HardLink/ReadLink` exist for `by-hash` and multi-distribution helpers; object + stores emulate them (copy + metadata / text marker). + +### Naming published storage + +The first argument (`[storage:]prefix`) of publish commands is parsed by `deb.ParsePrefix`: + +| Value | Storage | Prefix | +|-------|---------|--------| +| `debian` | default (`""` → `rootDir/public`, hardlink) | `debian` | +| `.` | default | `.` (root) | +| `filesystem:mirror1:ubuntu` | `FileSystemPublishRoots["mirror1"]` | `ubuntu` | +| `s3:prod:` | `S3PublishRoots["prod"]` | `.` | +| `gcs:x:…`, `swift:x:…`, `azure:x:…`, `jfrog:x:…` | respective map | | + +Parsing splits at the **last** colon; an empty prefix after a colon means `.`. Leading/trailing +`/` are trimmed. `NewPublishedRepo` additionally rejects prefixes containing `.`/`..` path +components. `context.GetPublishedStorage(name)` instantiates each named storage once per +process and caches it. + +## 4.4 Filesystem published storage (`files/public.go`) + +Config (`filesystem_publish_endpoints.`): `root_dir`, `link_method` +(`hardlink` default | `symlink` | `copy`), `verify_method` (`md5` default | `size`, only for +`copy`). The implicit default storage is `rootDir/public` with `hardlink`. + +`LinkFromPool` logic: + +1. Compute `destination = root/prefix/relPath/fileName` and `mkdir -p` its directory. +2. Non-copy methods require the source pool to be a `LocalPackagePool`. +3. If destination exists: + * `copy`: compare source size to destination size (`size` mode) or compare + destination MD5 to the source MD5 (`md5` mode); equal ⇒ nothing to do; + * link/symlink: equal **inode and device** ⇒ nothing to do (a symlink to a different + filesystem could reuse inode numbers, hence the device check); + * otherwise error, unless `force`, in which case the destination is removed first. +4. Create: for `copy` stream from `sourcePool.Open` then **`fsync`** (to surface `ENOSPC`); + `symlink` → `pool.Symlink`; default → `pool.Link` (hard link). + +Hard links are the default and make a publication essentially free in disk space and instant, +**but** they require the pool and the publish root to be on the same filesystem, and a hard +link keeps the data alive even after `db cleanup` removes it from the pool — which is +actually what protects still-served files. + +`RemoveDirs` uses `os.RemoveAll`; `Filelist` walks in parallel and returns sorted relative +file names. + +## 4.5 Object-store back-ends + +All object stores share a pattern: + +* `MkDir` is a no-op (flat namespaces). +* Uploads are keyed `prefix + path`. +* `LinkFromPool` = *stream from `sourcePool.Open()` to `PutObject`* with skip-if-same logic + keyed on **MD5**, because MD5 is available as ETag or custom metadata. +* Directory removal = list by prefix + batch delete. + +| Back-end | Package | Config section | Specifics | +|----------|---------|----------------|-----------| +| **Amazon S3 / S3-compatible** | `s3` (AWS SDK v2) | `s3_publish_endpoints` | `bucket`, `region`, `endpoint` (MinIO etc.), `prefix`, `acl` (default `private`; use `public-read` for direct serving; `none` to omit), `storage_class`, `encryption_method` (`AES256`/`aws:kms`), `plus_workaround`, `disable_multidel`, `force_virtualhosted_style`, `debug`; credentials from config or the SDK's default chain (env, IAM role…). **Path cache**: on first `LinkFromPool` under a prefix it lists `prefix/pool` once and caches path → MD5 to avoid a HEAD per file. If the ETag is not a plain MD5 (multipart or SSE-KMS) it falls back to a `HEAD` reading the `Md5` user-metadata that aptly stored. `plus_workaround` writes a second copy of any key containing `+` with `+`→space, because some S3 front-ends decode `+` in URLs as a space. `RenameFile` = `CopyObject` + `DeleteObject`. `SymLink` = copy with metadata | +| **Google Cloud Storage** | `gcs` | `gcs_publish_endpoints` | `bucket`, `prefix`, `credentials_file` or inline `service_account_json`, `project`, `endpoint`, `acl`, `storage_class`, `encryption_key`, `disable_multidel`. Same MD5-based skip logic | +| **Azure Blob** | `azure` | `azure_publish_endpoints` | `account_name`, `account_key`, `container`, `prefix`, `endpoint`; `HardLink` implemented as symlink emulation; `SymLink` writes link metadata; per-prefix path→MD5 cache | +| **OpenStack Swift** | `swift` | `swift_publish_endpoints` | Keystone authentication (`username`, `password`, `auth_url`, `tenant`, `tenant_id`, `domain`, `domain_id`, `tenant_domain`, `tenant_domain_id`), `container`, `prefix`. Uses the object `Hash` (MD5) to compare | +| **JFrog Artifactory** | `jfrog` | `jfrog_publish_endpoints` | `repository`, `url`, one of `user`+`password` / `api_key` / `access_token` (also from env `JFROG_USERNAME`, `JFROG_PASSWORD`, `JFROG_APIKEY`, `JFROG_ACCESSTOKEN`), `prefix`, `plus_workaround`, `debug`. Materialises non-local pool sources in a temp file before upload | + +Because object stores are "serve by URL", the resulting bucket/container is what you point +`apt` at (with public read or a CDN/proxy). + +## 4.6 Where files end up in a published tree + +``` +// +├── dists// +│ ├── Release Release.gpg InRelease +│ ├── Contents-.gz (legacy, top-level) +│ └── / +│ ├── binary-/{Packages,Packages.gz,Packages.bz2,Release} +│ ├── debian-installer/binary-/… (udebs) +│ ├── source/{Sources,Sources.gz,Sources.bz2,Release} +│ ├── Contents-.gz +│ ├── installer-/current/{images|legacy-images}/… +│ └── (skeleton files / AppStream dep11 files) +└── pool//// (multi-dist: pool///…) +``` + +`` is the first letter of the source package name, or the first four characters for +`lib*` sources (`Package.PoolDirectory`). If `Source` carries a version (`foo (1.2)`), only the +name part is used. `MultiDist` publications place files under `pool//…` so the +same *filename* with different contents can coexist in different distributions of one prefix. +Details of generation are in [chapter 6](06-publishing.md). + +## 4.7 Cleaning up + +* **Published storage**: after `publish update/switch`, `CleanupPrefixComponentFiles` deletes + pool files under `prefix/pool//` that are no longer referenced by *any* + publication sharing the same storage+prefix (skippable with `-skip-cleanup`). `publish drop` + removes `dists/` and unreferenced pool files (whole prefix removed if it was the last + publication). +* **Pool**: `aptly db cleanup` (mark-and-sweep, [chapter 5](05-workflows.md#56-db-cleanup)). + +Both sweeps are conservative: they compute "referenced" sets from the DB rather than +inspecting file contents. diff --git a/documentation/05-workflows.md b/documentation/05-workflows.md new file mode 100644 index 000000000..e5223762f --- /dev/null +++ b/documentation/05-workflows.md @@ -0,0 +1,410 @@ +# 5. Core Workflows (Step-by-Step Internals) + +Each section follows the actual call sequence in the code, naming files and functions. +CLI entry points live in `cmd/`; REST entry points in `api/` reuse the same `deb` methods. + +Contents: +[5.1 Mirror create & update](#51-mirror-update) · +[5.2 Local repo: add / include / copy / move](#52-local-repositories) · +[5.3 Snapshots: create / filter / merge / pull / diff / verify](#53-snapshots) · +[5.4 Publish snapshot or repo](#54-publish) · +[5.5 Switch / update / drop](#55-switch-update-drop) · +[5.6 DB cleanup](#56-db-cleanup) · +[5.7 Other commands](#57-other-commands) + +--- + +## 5.1 Mirror update + +### 5.1.1 `aptly mirror create [components…]` (`cmd/mirror_create.go`) + +1. Parse `ppa:user/project` shorthand via `deb.ParsePPA` (uses `ppa_distributor_id`, + `ppa_codename`, `ppa_baseurl`), otherwise take URL/distribution/components from args. +2. `deb.NewRemoteRepo(...)` — assigns a UUID, normalises the URL (`prepare()` adds a trailing + `/`), detects **flat repositories** (distribution ends with `/` or starts with `.`), and + rejects incompatible options (components/udebs/AppStream with flat repos). +3. Copy filter, `-filter-with-deps`, `-force-components`, `-force-architectures`; a filter + string is **parsed immediately** with `query.Parse` so syntax errors surface at creation. +4. `repo.Fetch(downloader, verifier, ignoreSignatures)` — see below. This validates + architectures/components against the archive's `Release`. +5. Save via `RemoteRepoCollection.Add` (no package list yet). + +### 5.1.2 `RemoteRepo.Fetch` — obtain and verify `Release` + +``` +if ignoreSignatures: + download Release ─ok→ use it + └fail→ download InRelease, strip clear-signature (no verification) +else: + 1. download InRelease; VerifyClearsigned; extract text ── ok → done + 2. else download Release + Release.gpg; VerifyDetachedSignature ── ok → done +``` + +Then it parses the first stanza of the release: + +* `Architectures:` → sorted; `source` removed. If the mirror has no architectures configured, + all are adopted; otherwise each requested architecture must be a subset (error suggests + `-force-architectures`). +* `Components:` (non-flat): for distributions with a path (`updates/main`) the leading + `dist/` is stripped from component names. Same subset logic (`-force-components`). +* All of `MD5Sum`, `SHA1`, `SHA256`, `SHA512` blocks are parsed into + `repo.ReleaseFiles[path] = ChecksumInfo{Size, hashes…}` — the **expected checksums for every + index file**, later used to validate `Packages*.{bz2,gz,xz}` downloads. +* Remaining fields become `repo.Meta` (`Origin`, `Label`, `Suite`, `Codename`, `NotAutomatic`, + `ButAutomaticUpgrades`, …), which snapshots inherit. + +### 5.1.3 `aptly mirror update ` (`cmd/mirror_update.go`) + +```mermaid +sequenceDiagram + autonumber + participant CLI as aptly mirror update + participant R as RemoteRepo + participant DL as Downloader + participant DB as LevelDB + participant P as PackagePool + + CLI->>DB: load mirror (LoadComplete → old ref list) + CLI->>R: CheckLock() (unless -force) + CLI->>DL: Fetch Release (verify signature) + CLI->>DL: DownloadPackageIndexes (Packages/Sources/…) + Note over R: parse stanzas → Package objects → in-memory PackageList + opt AppStream enabled + CLI->>DL: DownloadAppStreamFiles → pool + end + opt filter set + CLI->>R: ApplyFilter(query, with deps) + end + CLI->>P: BuildDownloadQueue: Verify each file → tasks for missing ones + CLI->>DB: MarkAsUpdating (status + PID), then CLOSE database + par download_concurrency workers + DL->>DL: DownloadWithChecksum → temp path in pool + end + CLI->>DB: REOPEN database + CLI->>P: Import each downloaded file (move=true) + CLI->>R: FinalizeDownload: one transaction writes all Package records + R->>DB: commit + new ref list ; Update(mirror) (Status=idle) +``` + +Detailed steps: + +1. **Lock check.** `CheckLock()` returns *"mirror is locked by update operation, PID n"* only if + `Status == Updating` **and** `WorkerPID` is a live process (`kill -0`). Stale locks from + crashed runs are ignored automatically. +2. **`DownloadPackageIndexes`** builds a list of `(path, kind, component, arch)`: + * flat: `Packages` (+ `Sources` if `-with-sources`); + * normal: for every component × architecture `Packages` (binary), plus `debian-installer` + Packages (udeb), plus `installer-/current/images/SHA256SUMS` (installer), plus one + `Sources` per component. + For each path `http.DownloadTryCompression` tries **`.bz2`, `.gz`, `.xz`, then plain**; + a variant is only attempted if it appears in `ReleaseFiles` (unless `-ignore-checksums`); + 404/403 ⇒ try the next one. The download is checksum-verified against `ReleaseFiles` and + transparently decompressed. Installer indexes have a special fallback: if not listed in + `Release`, a plain `SHA256SUMS` plus detached `.gpg` is fetched and verified. +3. Stanzas are read with `ControlFileReader` → `NewPackageFromControlFile` / + `NewUdebPackage…` / `NewSourcePackage…` / `NewInstallerPackage…` and added to + `repo.packageList`. Duplicate name+version+arch produce a coloured *skipping package* + warning, not an error. +4. **Filter.** If `repo.Filter != ""`: parse, `PrepareIndex()`, then `PackageList.Filter` with + `WithDependencies = repo.FilterWithDeps`, `WithSources = repo.DownloadSources`, + `Source = empty list` (so dependencies are searched *within the mirror only*) and the + mirror's architectures. Output: `Packages filtered: old -> new`. +5. **`BuildDownloadQueue`**: `-latest` first reduces the list to the newest version per + name/arch (`FilterLatest`). With `-skip-existing-packages`, packages whose key is already in + the mirror's *old* ref list are skipped and their file info loaded from the DB (saves + thousands of `stat` calls). Otherwise for each package `DownloadList` calls + `packagePool.Verify(...)`; missing/bad files become tasks. Identical URLs are + de-duplicated (extra tasks are attached as `Additional`). +6. **Mark updating and close the DB.** `MarkAsUpdating` (status + own PID) → persisted → + `context.CloseDatabase()`. A deferred function reopens the DB and marks the mirror idle on + *any* exit path (including ^C). + Reason: a multi-hour download shouldn't hold the LevelDB lock; other aptly commands can run. +7. **Parallel download.** `download_concurrency` workers pull task indexes from a channel. + Each worker: obtain temp path (`GenerateTempPath` for a local pool, else `os.CreateTemp`) → + `DownloadWithChecksum(url, tmp, expected, ignoreChecksums)` → `task.Done = true`. Errors are + collected (mutex), not fatal. `context.GoContextHandleSignals()` makes ^C cancel the context; + a second ^C kills immediately. +8. **Reopen DB, import.** Sequentially `PackagePool.Import(tmp, filename, &checksums, move=true, + checksumStorage)`; the resulting `PoolPath` and completed checksums are propagated to + `Additional` tasks. Temp files are removed by a deferred cleanup. If interrupted or any + download failed the command returns an error **before** finalisation — *the mirror's old + content is untouched*, and re-running only fetches what's still missing. +9. **`FinalizeDownload`.** Opens **one DB transaction**, `UpdateInTransaction` for every package + (writes `P`, `xF`, `xD`, `xE`), builds the new `PackageRefList` from the list, sets + `LastDownloadDate`, commits. Then `RemoteRepoCollection.Update(repo)` stores the mirror + record and new ref list (`E`) and — via the deferred function — status idle. + +**Atomicity property:** a mirror's visible contents (its ref list) switch from old to new in +a single batch write, and only after all files are safely in the pool. A snapshot taken at any +time sees either the complete old or the complete new state. + +**Why files are verified twice:** checksum on download (integrity in flight), and `Verify` +on the next update (integrity at rest, via checksum cache). + +### 5.1.4 Other mirror commands + +* `mirror edit` changes `-archive-url`, `-filter` (`@file`/`@-` allowed), `-filter-with-deps`, + `-ignore-signatures`, the `-with-sources/-udebs/-installer/-appstream` toggles and keyrings; + `mirror rename`; `mirror drop` (refuses while snapshots were created from it unless + `-force`); `mirror show [-with-packages] [-json]`; `mirror search`; `mirror list [-raw|-json]`. +* REST equivalents: `POST /api/mirrors`, `PUT /api/mirrors/:name` (update, asynchronous + task), `POST /api/mirrors/:name` (edit), etc. (see [chapter 7](07-rest-api-and-tasks.md)). + +--- + +## 5.2 Local repositories + +### `aptly repo add ` (`cmd/repo_add.go`, `deb/import.go`) + +1. Load repo (`LoadComplete`) and its package list `NewPackageListFromRefList`. +2. `deb.CollectPackageFiles` walks arguments (directories recursively, in parallel): + `*.deb *.udeb *.ddeb *.dsc` are package candidates, `*.buildinfo` are "other files" (removed + together with processed files but not imported). +3. `deb.ImportPackageFiles(list, files, forceReplace, verifier, pool, packageCollection, + reporter, restriction, checksumStorageProvider)` — for each file: + * **Read metadata**: `.dsc` → `GetControlFileFromDsc(file, verifier)` (also verifies the + signature if present and required); `.deb`/`.udeb` → `GetControlFileFromDeb` (ar → control + tarball → `control`). + * Build `Package` (`NewSourcePackageFromControlFile` for `.dsc`, with `Source` renamed to + `Package`; otherwise `NewPackageFromControlFile`/`NewUdebPackageFromControlFile`). + * Validate name, version, architecture non-empty. + * Compute checksums; `pool.Import(file, base, &checksums, move=false, …)`. + * For source packages, every file listed in the `.dsc` is looked for **next to the `.dsc`** + and imported; if absent, `pool.Verify("", name, &checksums)` looks in the pool (so + `orig.tar.gz` can be shared between uploads). + * Update package files/PoolPaths; apply optional `restriction` query (used by + uploader rules); `PackageCollection.Update(p)` (transactional write). + * `-force-replace`: search the list for same name/version/arch and remove first. + * `list.Add(p)` — fails with `PackageConflictError` if same name/version/arch already exists + with a different file set (reported as a failure). + * Per-file failures are *warnings*; other files continue. `ImportPackageFiles` returns + processed vs failed file lists. +4. `repo.UpdateRefList(NewPackageRefListFromPackageList(list))` + `LocalRepoCollection.Update`. +5. `-remove-files` deletes processed files (`.dsc`'s companions and `.buildinfo` included). + Exit code is non-zero if any file failed. + +### `aptly repo include ` (`cmd/repo_include.go`, `deb/changes.go`) + +Import driven by Debian `.changes` files (the output of `dpkg-buildpackage`, `dput`, …): + +1. `CollectChangesFiles` finds `*.changes`. +2. `NewChanges` copies the `.changes` into a temp dir; `VerifyAndParse` verifies the PGP + signature (unless `-ignore-signatures`; unsigned rejected unless `-accept-unsigned`) and + parses `Files`/`Checksums-*`, `Distribution`, `Source`, `Binary`, `Architecture`. +3. `Prepare` copies every file listed in the `.changes` from the directory of the `.changes` + file into the temp dir (files in other directories are rejected) and verifies **size, MD5, + SHA-1 and SHA-256** against the `.changes` — any mismatch aborts that upload. +4. **Repository selection** uses the template `-repo="{{.Distribution}}"` (Go `text/template` + over the parsed changes, e.g. `{{.Distribution}}`, `{{.Source}}`); the result names the + local repo. +5. **Uploaders ACL** (`-uploaders-file` or the `Uploaders` stored on the repo at + `repo create -uploaders-file`): `Uploaders.IsAllowed(changes)` walks `rules` in order. A rule + applies if its `condition` (query language, evaluated against the `.changes` fields) matches. + For an applicable rule: if any signature key of the `.changes` matches an entry of `deny` + (after expanding `groups`; `"*"` matches everyone) the upload is **denied immediately**; + else if any key matches `allow` it is **accepted**; otherwise evaluation continues with the + next rule. If no rule accepts: *"denied as no rule matches"*. +6. Files are imported through the same `ImportPackageFiles` (with the ACL as `restriction`), + then processed files are removed unless `-no-remove-files`. + +### `repo copy / move / import / remove` + +* `copy`/`move `: load both lists, `Filter` the source list + (`-with-deps` pulls dependencies not yet in the destination); add to destination (`move` + additionally removes from source). `-dry-run` reports only. +* `import `: same idea, but source is a **mirror** (this is how you + cherry-pick packages from a mirror into a local repo — package files are already in the pool, + so nothing is downloaded). +* `remove `: `list.Remove` for each match; the package **files stay in the pool** + until `db cleanup`. +* REST: `POST /api/repos/:name/packages` and `DELETE …` take explicit package *keys*; + `POST /api/repos/:name/copy/:src/:file` is the API for `copy`. + +--- + +## 5.3 Snapshots + +### Create (`cmd/snapshot_create.go`) + +| Form | Function | Notes | +|------|----------|-------| +| `snapshot create S from mirror M` | `NewSnapshotFromRepository` | Fails with *"mirror not updated"* if the mirror has no ref list; also refuses if `CheckLock()` says an update is running. Copies `Origin/NotAutomatic/ButAutomaticUpgrades` and AppStream file map from the mirror `Meta` | +| `snapshot create S from repo R` | `NewSnapshotFromLocalRepo` | Uses the repo's ref list (empty list if none) | +| `snapshot create S empty` | `NewSnapshotFromPackageList(name, nil, empty, …)` | A blank canvas for `pull` | + +Creating a snapshot writes just `S` and `E` — no package data moves. + +### Filter (`snapshot filter `) + +Load package list → `PrepareIndex` → determine architectures (`-architectures` or all in the +list; required non-empty only with `-with-deps`) → `PackageList.Filter(queries ORed, WithDependencies=-with-deps, +Source=nil)` → new snapshot from the resulting list. Queries can be `@file`/`@-` +(`GetStringOrFileContent`). + +### Merge (`snapshot merge …`) + +```go +result := sources[0].RefList() +for i := 1..n: result = result.Merge(sources[i].RefList(), overrideMatching, false) +if latest: result.FilterLatestRefs() +``` + +`overrideMatching = !latest && !no-remove`. Default: **the last source wins per +(architecture, name)** — the version from a later snapshot *replaces* every version from +earlier ones. `-no-remove` keeps all versions; `-latest` keeps the highest version of each. +It never loads packages except for `-latest` comparisons (pure ref operations). + +### Pull (`snapshot pull `) + +The "controlled upgrade" primitive (`cmd/snapshot_pull.go`): + +1. Load both snapshots into `PackageList`s and index them. +2. Architectures: `-architectures` or all architectures found in `` (excluding source). +3. Each query is ANDed with `$Architecture ∈ archs`. +4. `sourcePackageList.Filter(queries, WithDependencies = !no-deps, Source = packageList)`: + dependency closure is computed against *source ∪ result*, but dependencies **already + satisfied by the destination snapshot** are not pulled again. +5. Merge into the destination list, grouped by `(arch,name)`: + * unless `-no-remove`, all existing packages with that arch+name are **removed** first + (the `[-] removed` lines); + * unless `-all-matches`, only the first (= highest version, given index ordering) match per + arch+name is added (`[+] added`). +6. `-dry-run` prints the plan; otherwise a new snapshot is created from the list with + description *"Pulled into 'X' with 'Y' as source, pull request was: '…'"* and both snapshots + as parents. + +### Diff / Verify / Drop / Rename + +* `diff A B` → `RefList.Diff` (two-pointer over refs; only loads packages for entries that differ + to distinguish *added / removed / version-changed*). `-only-matching` limits to + packages present in both. +* `verify S [source…]` → `PackageList.VerifyDependencies` over the snapshot (with the other + snapshots as extra dependency *sources*) and prints unsatisfied dependencies per architecture. +* `drop S` **always** refuses if the snapshot is published (it lists the publications), and + refuses if it is a *source of other snapshots* unless `-force`; `rename` rewrites metadata only. +* `search`/`show -with-packages` list contents; `aptly graph` renders the DAG of mirrors → + snapshots → publications with Graphviz. + +--- + +## 5.4 Publish + +### `aptly publish snapshot|repo [flags] … [[:]]` +(`cmd/publish_snapshot.go` — one function serves both `snapshot` and `repo`, distinguished by +`cmd.Name()`). + +1. Parse arguments: number of positional names must match the number of `-component` values + (comma-separated); optional last argument is `[storage:]prefix` → `ParsePrefix`. +2. Load each source (`LoadComplete`); warn when a source is empty (architectures then must be + given explicitly because they can't be inferred and can't be changed later). +3. `NewPublishedRepo(storage, prefix, distribution, architectures, components, sources, …)`: + * source kind determined by the first source; + * for empty `-distribution`/`-component`: `walkUpTree` climbs snapshot parents to root + mirrors/local repos and collects their `Distribution`/`Components`/`DefaultDistribution`/ + `DefaultComponent`. If all roots agree the value is used; a single-source default component is + `main`; a conflicting/absent distribution → *"unable to guess distribution name, please + specify explicitly"*; + * duplicate component names rejected; **Origin/NotAutomatic/ButAutomaticUpgrades** are + inherited from snapshots only if they all agree; + * prefix sanitised (`.`/`..` rejected). +4. Apply flag overrides (`-origin`, `-label`, `-suite`, `-codename`, `-notautomatic`, + `-butautomaticupgrades`, `-skip-contents`, `-skip-bz2`, `-acquire-by-hash`, `-signed-by`, + `-version`, `-multi-dist`) — config defaults `skip_contents_publishing` / `skip_bz2_publishing` + apply when flags are unset. +5. `CheckDuplicate` — *(storage, prefix, distribution)* already in use ⇒ error. +6. `getSigner(flags)` builds the GPG signer (see [chapter 10](10-signing-and-verification.md)); + nil signer when `-skip-signing` or `gpg_disable_sign`. +7. **`published.Publish(...)`** — the big one, [chapter 6](06-publishing.md). +8. `PublishedRepoCollection.Add(published)` — for `local` sources also stores the ref lists + `E`. +9. Print the `deb http://your-server/ ` line and, for filesystem + storages, the directory to serve. + +--- + +## 5.5 Switch, update, drop + +### `publish switch [[endpoint:]prefix] ` (`cmd/publish_switch.go`) + +*Snapshot* publications only. Component→snapshot mapping is updated in memory +(`UpdateSnapshot` sets `rePublishing = true`), then `Publish()` regenerates everything. +Because `rePublishing` is set: + +* architectures **are not recomputed** (kept from the first publication); +* index files are uploaded with a `.tmp` suffix and renamed into place at the very end + (`indexes.RenameFiles()`), which is what makes the switch "near-atomic" for clients. + +Afterwards `PublishedRepoCollection.Update` persists and — unless `-skip-cleanup` — +`CleanupPrefixComponentFiles` deletes pool files orphaned by the change. + +### `publish update [[endpoint:]prefix]` (`cmd/publish_update.go`) + +* **Local-repo publication:** re-reads the *current* ref list of each source local repo + (`UpdateLocalRepo`) — publishes new state of the repo. +* **Snapshot publication:** without staged sources there's nothing to change (snapshots + are immutable) — the command is used to apply the *staged revision* below or to re-generate + metadata with new flags (`-skip-contents`, `-origin`, `-label`, `-multi-dist`, + `-signed-by`, `-version`, …). + +### Staged source editing: `publish source …` (`cmd/publish_source_*.go`) + +`add`, `drop`, `list`, `remove`, `replace`, `update` manipulate `PublishedRepo.Revision` +(a *pending* component→source-name map) without touching the published files. +`publish update` then calls `PublishedRepo.Update` which computes +**Added / Updated / Removed** components from `Revision` vs current `Sources` +(`PublishedRepoUpdateResult`), applies them, publishes, and cleans up exactly the affected +components. `publish source drop` discards the pending revision (`DropRevision`). REST: +`/api/publish/:prefix/:distribution/sources[/:component]` and `/update`. + +### `publish drop [[endpoint:]prefix]` + +`PublishedRepoCollection.Remove`: + +1. Determine if other publications share the same *(storage, prefix)*; if none, the whole + `dists/` and `pool/` are removed; otherwise only `dists/` plus pool components not used by + siblings. +2. Delete pool files of shared components that are no longer referenced + (`CleanupPrefixComponentFiles`) unless `-skip-cleanup`. +3. Delete `U…` and (for local-repo sources) `E` keys. +4. `-force-drop` continues on storage errors (leaving orphans). + +--- + +## 5.6 DB cleanup + +`aptly db cleanup` (`cmd/db_cleanup.go`, `api/db.go`) is a **mark-and-sweep garbage collector**: + +``` +mark 1: existing := ∪ ref lists of all mirrors, local repos, snapshots, + and published *local-repo* components (Merge(…, false, true)) +sweep 1: for every key in "P*" not in existing → delete P, xF, xD, xE (one DB batch) +mark 2: referencedFiles := ⋃ pkg.FilepathList(pool) for pkg in existing (sorted) +sweep 2: for every file in pool.FilepathList() not in referencedFiles → pool.Remove() +finish : db.CompactDB() +``` + +Points to know: + +* It needs **exclusive access** — CLI holds the DB lock; the API takes resource `__all__` so no + other task runs concurrently. +* `-dry-run` prints what would be deleted; `-verbose` lists objects. +* Files are removed from the **pool only**; hard-linked copies in published trees survive. +* Package rows for local-repo *publications* are included in "existing" because those refs are + frozen copies (`E`), independent of the live local repo. +* Reports total space freed (`utils.HumanBytes`); the API stores progress in a `task.Detail` + (`TotalNumberOfPackagesToDelete`, `RemainingNumberOfPackagesToDelete`). + +--- + +## 5.7 Other commands + +| Command | What it does internally | +|---------|-------------------------| +| `aptly serve` (`cmd/serve.go`) | Requires ≥ 1 publication. Prints a suggested `deb`/`deb-src` line for every publication, then **closes the database** (`ShutdownContext`) and serves `rootDir/public` (the *default* storage only — `filesystem:` endpoints are not served) with Go's `http.FileServer` (directory listing). The docs call it "not suitable for real production usage" | +| `aptly api serve` | Runs the REST server (chapter 7) | +| `aptly graph [-format=png\|svg\|pdf…] [-layout=horizontal\|vertical] [-output=file]` | `deb.BuildGraph` creates a Graphviz digraph of mirrors, repos, snapshots, publications; renders with the external Graphviz `dot` binary; without `-output` the result is opened in a viewer | +| `aptly package search|show [-with-files -with-references]` | Queries the global `PackageCollection` (all `P…` keys); `show -with-references` lists every container referencing the package | +| `aptly db recover` | `goleveldb.RecoverDB` then checks each local repo for dangling refs (`FindDanglingReferences`) and removes them | +| `aptly config show [-yaml]` | Prints effective configuration (also converts legacy JSON → YAML) | +| `aptly task run (-filename=f \| cmd , cmd …)` | Runs several aptly commands sequentially in **one process** ("running as single thread"). Input: a file (one command per line), interactive stdin (blank line to finish), or command-line args where commands are separated by a trailing comma (`aptly task run repo create a, repo add a pkg`). For each command the DB is reopened, the command runs through the normal `Run()` path, and the context is partially cleaned up; **after the first failing command all remaining ones are skipped** and the overall result is an error | +| `aptly version` | Prints embedded version | diff --git a/documentation/06-publishing.md b/documentation/06-publishing.md new file mode 100644 index 000000000..b4bd70163 --- /dev/null +++ b/documentation/06-publishing.md @@ -0,0 +1,222 @@ +# 6. Publishing Internals + +Publishing turns "a list of package refs per component" into a directory tree that `apt` can +consume. Implemented by `PublishedRepo.Publish` (`deb/publish.go`), `indexFiles`/`indexFile` +(`deb/index_files.go`) and `ContentsIndex` (`deb/contents.go`). + +## 6.1 Inputs and outputs + +**Inputs** (all in the `PublishedRepo`): + +* `Storage`, `Prefix`, `Distribution`; +* one source per **component** (`sourceItems[component]` → snapshot or local repo, with + its `PackageRefList`); +* metadata: `Origin`, `Label`, `Suite`, `Codename`, `Version`, `NotAutomatic`, + `ButAutomaticUpgrades`, `SignedBy`, `Architectures`; +* switches: `SkipContents`, `SkipBz2`, `AcquireByHash`, `MultiDist`, `rePublishing` (internal); +* collaborators: `PackagePool`, `PublishedStorageProvider`, `CollectionFactory`, + `pgp.Signer` (nil = unsigned), `Progress`, `forceOverwrite`, `skelDir`. + +**Outputs** on the published storage: the tree in [§4.6](04-storage.md#46-where-files-end-up-in-a-published-tree). + +## 6.2 The algorithm, step by step + +`Publish()` (annotated with the sub-steps in the source): + +```mermaid +flowchart TD + A[GetPublishedStorage by name] --> B[MkDir prefix/pool and prefix/dists/dist] + B --> C[CreateTemporary DB
for Contents] + C --> D[For each component:
NewPackageListFromRefList] + D --> E{first publication?} + E -- yes --> F[Architectures := union of list architectures
error if none; sort + dedupe] + E -- "re-publish" --> G[keep stored Architectures
suffix = '.tmp'] + F --> H + G --> H[temp dir + newIndexFiles] + H --> I[For each component / list:
pre-create Packages index per arch] + I --> J[list.PrepareIndex; ForEachIndexed pkg] + J --> K[LinkFromPool for the first matching arch] + J --> L[push contents to temp DB] + J --> M[write stanza to Packages/Sources of each matching arch] + M --> N[write Contents indexes, skeleton files, AppStream files] + N --> O[write per-arch Release files] + O --> P[FinalizeAll: compress, checksum, upload, by-hash, sign] + P --> Q[build top-level Release with checksum lists] + Q --> R[Finalize Release: Release, Release.gpg, InRelease] + R --> S[RenameFiles: *.tmp → final] +``` + +### Step details + +1. **Storage & directories.** `publishedStorage.MkDir(prefix/pool)` and `…/dists/` (no-ops + on object stores). +2. **Temporary DB.** `collectionFactory.TemporaryDB()` (a scratch LevelDB, or a prefixed keyspace + in etcd) holds the *contents index* while packages stream through. Closed **and dropped** in a + `defer`. +3. **Load packages.** `NewPackageListFromRefList(p.RefList(component), PackageCollection, progress)` + per component — this is the heavy I/O phase ("Loading packages…"). +4. **Architectures.** On first publish, if none were specified, the union of + `list.Architectures(true)` (including `source`) over all components; empty ⇒ *"unable to figure + out list of architectures, please supply explicit list"*. Sorted, de-duplicated. On re-publish + the stored list is kept (so an emptied source can't erase architectures). +5. **Index writers.** `newIndexFiles(storage, basePath=prefix/dists/dist, tempDir, suffix, + acquireByHash, skipBz2)`; `suffix = ".tmp"` when re-publishing. +6. **Per component, per architecture:** `indexes.PackageIndex(component, arch, false, false, + dist)` pre-registers `Packages` files so **empty** architectures still get a file. +7. **Per package** (`list.ForEachIndexed` — sorted by name, version descending, arch): + * For the **first architecture that the package matches** (`MatchesArchitecture`: an + `Architecture: all` package matches every architecture except `source`; everything else + requires equality — source packages have architecture `source`) + compute the pool-relative directory: + * normal: `pool//` (`PoolDirectory` = `l/libfoo` or `f/foo`); + * `MultiDist`: `pool///`; + * installer packages: `dists///installer-/current/images` + (`legacy-images` when distribution is `focal`), + and call `pkg.LinkFromPool(publishedStorage, packagePool, prefix, relPath, forceOverwrite)` + which, for each `PackageFile`, calls `publishedStorage.LinkFromPool(...)`. It happens once + per package, not once per architecture. + * Open a DB **batch** on the temp DB. For **each matching architecture**: + * unless `SkipContents` or installer: compute `pkg.Contents(pool, progress)` — reads the + `.deb` from the pool, extracts the file list from `data.tar.*`, and caches it in + `xC` (persistent, reused by later publishes); push `(path, qualifiedName)` pairs + into two `ContentsIndex`es: per-component-per-arch and the *legacy* global one; + * append `pkg.Stanza()` to that architecture's `Packages`/`Sources` writer followed by a + blank line. The stanza contains extra control fields (`xE`), dependencies (`xD`), pool + `Filename:` (`pool//…/`), `Size`, `MD5sum`, `SHA1`, `SHA256`, `SHA512` + (`xF`), in canonical Debian field order (`format.go: canonicalCase` + order table). + * Free `files/deps/extra/contents` of the package immediately (bounded memory) and flush the + batch. +8. **Contents.** After all packages: for every arch × {deb,udeb} with a non-empty index, write + `/Contents-[.gz]` (`FILE LOCATION` header, sorted paths, comma-joined + qualified names, gzip only) and, after all components, the legacy top-level + `Contents-.gz` / `Contents-udeb-.gz` covering *all* components. + `ContentsIndex.WriteTo` scans the temp DB in key order, merging consecutive keys with the + same path — key layout is `uuidPrefix ‖ path ‖ 0x00 ‖ qualifiedName`. +9. **Skeleton files** (`rootDir/skel//dists///…`): arbitrary + operator-provided files copied verbatim into the component directory. +10. **AppStream (DEP-11)**: for snapshots with `AppStreamFiles`, files for the matching + component are copied out of the *package pool* into + `/dep11/…` (mirrors made with `-with-appstream`). +11. **Per-arch Release** (`/binary-/Release`, `source/Release`): tiny stanza + with `Archive`, `Architecture`, `Component`, `Origin`, `Label`, `Suite`, `Codename`, and optionally + `Acquire-By-Hash`, `Signed-By`, `Version`. Created for udebs only if `hadUdebs`. +12. **FinalizeAll** (`indexFile.Finalize` for every registered file): + * flush; if `compressable`, compress to `.gz` and `.bz2` (`utils.CompressFile` with + `pgzip`; `.bz2` skipped with `SkipBz2`; the legacy Contents are gzip only); + * compute **MD5, SHA-1, SHA-256, SHA-512, size** for the plain and each compressed + variant → stored in `generatedFiles[path]` (used for the top-level Release); + * `MkDir` the directory; if `acquireByHash`, ensure `by-hash/{MD5Sum,SHA1,SHA256,SHA512}`; + * `PutFile` each variant to `` (i.e. `Packages.tmp`, `Packages.gz.tmp` when + re-publishing) and record `renameMap[tmp] = final`; + * `acquireByHash` → `packageIndexByHash` hard-links the just-uploaded file to + `by-hash//` (if not yet present), keeps the previous index as + `.old` symlink, deletes the *older* physical file, and repoints `` → new digest; + * installer `SHA256SUMS` are **detach-signed** (`.gpg`). +13. **Top-level `Release`** (`dists//Release`): + ``` + Origin, [NotAutomatic], [ButAutomaticUpgrades], Label, Suite, Codename, + Date: , + Architectures: , + [Acquire-By-Hash: yes], [Signed-By: … + Valid-Until: now+100y], [Version], + Description: Generated by aptly, + Components: , + MD5Sum:/SHA1:/SHA256:/SHA512: one line per generated file: " " + ``` + Paths are sorted; sizes right-aligned in 8 characters. +14. **Sign Release.** `releaseFile.Finalize(signer)` writes `Release`, detached `Release.gpg` + and clearsigned `InRelease` (`detachedSign` and `clearSign` both true for the top-level file). + If `signer == nil` these are skipped and clients need `[trusted=yes]` or + `allow-insecure`. +15. **Swap in.** `indexes.RenameFiles()` renames every `*.tmp` to its final name (only when + re-publishing; first publication wrote final names directly). + +### The consistency window + +Re-publishing is *not* a transactional swap. The new index files are uploaded first under +`.tmp` names, but `RenameFiles` iterates the rename map (a Go map — **order unspecified**) and each +rename is separate. On a filesystem, `rename(2)` is atomic per file, so a client sees, at any +instant, some mix of new and old index files; a mismatch between `Release` checksums and a +`Packages.gz` can produce apt's *"Hash Sum mismatch"* error for clients updating in that +moment. Mitigations: + +* enable **`-acquire-by-hash`** so clients fetch immutable, content-addressed copies; +* keep the window short (index files are small; package files are already linked before); +* put the published tree behind a proxy/CDN with consistent-snapshot semantics or + publish to a new prefix and switch a symlink/DNS. + +## 6.3 Package files: link, copy or upload + +`LinkFromPool` is called **before** the index that references the file is visible, so clients +never see a stanza pointing at a missing file. Behaviour by target: + +| Target | What happens | Idempotency check | +|--------|--------------|------------------| +| Filesystem + `hardlink` | `link(2)` pool file → published path | same inode & device | +| Filesystem + `symlink` | `symlink` → absolute pool path | same inode & device of resolved target | +| Filesystem + `copy` | stream + `fsync` | size or MD5 (`verify_method`) | +| S3 / GCS / Azure / Swift / JFrog | upload stream | MD5 vs cached/remote MD5 | + +When two publications (same storage+prefix) contain the *same file* they link to the same +destination; the idempotency check makes that a no-op. If two **different** files would land +on the same published path (same source name & filename, different bytes — legal in Debian archives +that reuse `orig` tarball names across distributions), the second publish fails with *"file +already exists and is different"* — use `-force-overwrite` (dangerous: it can break the other +publication) or, better, **`-multi-dist`** to isolate pools per distribution. + +## 6.4 What is regenerated when + +| Command | Package lists reloaded | Index files rewritten | Pool linking | Cleanup | +|---------|-----------------------|-----------------------|--------------|---------| +| `publish snapshot/repo` | all components | all | all files | none | +| `publish switch` | all components (after replacing chosen sources) | all | all files (already-linked = no-op) | orphaned pool files of changed components | +| `publish update` (local repo) | all components (fresh ref lists) | all | all files | orphaned files of updated/removed components | +| `publish update` with staged sources | same | all | all | affected components | +| `publish drop` | — | removed | removed | see §4.7 | + +There is **no incremental index generation**: even a one-package change re-emits every `Packages` +file. The costs are dominated by package loading and gzip/bz2 compression; contents are cached +per package (`xC`) so only new packages are re-inspected. + +## 6.5 Publishing knobs and their effect + +| Option | CLI flag | Config | Effect | +|--------|----------|--------|--------| +| Skip `Contents-*` | `-skip-contents` | `skip_contents_publishing` | Saves opening every `.deb`; `apt-file` won't work | +| Skip `.bz2` | `-skip-bz2` | `skip_bz2_publishing` | Faster/smaller; only `.gz` (+plain) | +| By-hash | `-acquire-by-hash` | – | `Acquire-By-Hash: yes`; `by-hash/` trees | +| Multi-dist | `-multi-dist` | – | Distribution-scoped `pool//…` | +| Signed-By | `-signed-by=fpr1,fpr2` | – | `Signed-By` + `Valid-Until` (+100 years) in Release | +| Version | `-version=` | – | `Version:` field | +| Label/Origin/Suite/Codename | `-label -origin -suite -codename` | – | Release metadata (Suite/Codename default to the distribution) | +| Not automatic | `-notautomatic=yes -butautomaticupgrades=yes` | – | apt pinning hints (e.g. backports) | +| Skip signing | `-skip-signing` | `gpg_disable_sign` | No `Release.gpg`/`InRelease` | +| Reproducible | env `SOURCE_DATE_EPOCH` | – | Fixed `Date:` | + +## 6.6 Cleanup logic (`CleanupPrefixComponentFiles`) + +After `switch`/`update`, for each **changed** component: + +1. Components no longer published by *any* non-multi-dist publication with the same + storage+prefix are removed wholesale (`RemoveDirs(prefix/pool/)`). +2. For surviving components, compute + `referenced = ⋃ (poolDir/filename for every file of every package in every publication + sharing prefix+component)` — `listReferencedFilesByComponent` loads the ref lists of all such + publications (deduplicating repeated refs across publications with `processedComponentRefs`). +3. `existing = storage.Filelist(prefix/pool/)`; delete `existing − referenced`. + +This is why publishing many distributions into one shared prefix makes `switch` slower: it +must consider every sibling publication. + +`MultiDist` publications keep their pool under `pool//` and are cleaned +independently. + +## 6.7 Consuming what was published + +``` +# /etc/apt/sources.list.d/internal.list +deb [signed-by=/etc/apt/keyrings/internal.gpg] http://repo.example.com/internal stable main +deb-src [signed-by=/etc/apt/keyrings/internal.gpg] http://repo.example.com/internal stable main +``` + +Export the public key with `gpg --export --armor KEYID > internal.gpg` (aptly does not publish it +for you — see [chapter 10](10-signing-and-verification.md)). diff --git a/documentation/07-rest-api-and-tasks.md b/documentation/07-rest-api-and-tasks.md new file mode 100644 index 000000000..9ba1852dc --- /dev/null +++ b/documentation/07-rest-api-and-tasks.md @@ -0,0 +1,334 @@ +# 7. REST API & Task System + +`aptly api serve` exposes nearly every CLI capability over HTTP/JSON. Handlers live in +`api/`; the router is `api.Router(ctx)` (`api/router.go`), built on +[gin](https://github.com/gin-gonic/gin). This chapter explains request handling, the task +system that serialises mutations, and gives an endpoint catalogue. + +> The generated OpenAPI/Swagger spec is produced from the `@Summary/@Router/@Param` comments +> above every handler (`make swagger`, output in `docs/`). Enable the live UI with +> `enable_swagger_endpoint: true` (served at `/docs.html` and `/docs/index.html`). + +## 7.1 Starting the server + +``` +aptly api serve [-listen=:8080 | -listen=unix:///run/aptly.sock] [-no-lock] +``` + +* Verifies `rootDir` is accessible (`utils.DirIsAccessible`). +* If systemd passed exactly one listener fd (`systemd/activation`), it is used; more than one + panics ("not supported"). +* Otherwise listens on TCP or a Unix socket (the socket file is removed and recreated). +* On `SIGINT/SIGTERM`: prints *"Shutdown signal received, waiting for background tasks…"*, + `TaskList().Wait()`, then `server.Shutdown`. +* `-no-lock`: the server does **not** hold the LevelDB lock while idle (see §7.5). + +> **Security:** the API has **no authentication or authorisation** of its own. Anyone who can +> reach the port can create, delete and publish repositories, upload files and add GPG keys. +> Bind to localhost / a Unix socket, or put it behind a reverse proxy providing TLS + auth (see +> [Operations](12-operations.md#124-securing-the-api)). Uploaded files and paths are +> sanitised (`verifyPath` rejects `.`/`..`, `utils.SanitizePath`), but the trust boundary is the +> network. + +## 7.2 Request pipeline + +``` +HTTP request + → gin.Logger (or JSONLogger when log_format=json) → gin.Recovery → gin.ErrorLogger + → [swagger redirect] (/docs → /docs.html, only if enabled) + → [metrics collectors] (only if enable_metrics_endpoint) + → /api group + → [-no-lock DB acquire/release middleware] + → handler + 1. bind + validate JSON (c.Bind), path/query params + 2. cheap validation & 404 checks using a *fresh CollectionFactory* + 3. maybeRunTaskInBackground(name, resources, func(out, detail){ … }) + - inside the closure: NEW collection factory, re-load objects, mutate, save + 4. respond: 200/201 + JSON (sync) or 202 + Task JSON (async) +``` + +Key conventions: + +* Handlers get a **shallow** view for routing/404 decisions, then **re-load inside the task** + after resource locks are acquired. This avoids TOCTOU races between concurrent requests. +* Errors use `AbortWithJSONError(c, status, err)` → JSON `{"error": "message"}`. +* `router.UseRawPath = true`: percent-encoded slashes in path parameters are preserved. + For publish endpoints the *prefix* parameter uses an escaping scheme (`slashEscape`): + `_` → `/` and `__` → `_`; prefix `.` denotes the root. e.g. + `/api/publish/:./stable` (root prefix), `/api/publish/debian_pool/bookworm` for prefix + `debian/pool`. Use `storage:prefix` form (e.g. `s3:mybucket:prod`) for non-default storage. +* `?_async=true` (`1`, `yes`, … — anything that is not `n/no/f/false/0/off`) runs the + operation in the background. The obsolete config key `async_api` sets the default. + +## 7.3 The task system + +Package `task` (`list.go`, `task.go`, `resources.go`, `output.go`). + +### Why tasks exist + +Mutating operations touch shared state (a local repo's ref list, a mirror's contents, the +published tree, the package pool). Two concurrent `repo add` calls on the same repo would lose +updates. The task list **serialises conflicting operations** while allowing unrelated ones to +run in parallel. + +### Data model + +```go +type Task struct { + Name string ; ID int ; State State // IDLE | RUNNING | SUCCEEDED | FAILED + Resources []string // exclusive resource keys + output *Output // thread-safe text buffer (aptly.Progress impl) + detail *Detail // atomic.Value with structured progress + processReturnValue *ProcessReturnValue // {Code int, Value interface{}} + err error +} +type Process func(out aptly.Progress, detail *Detail) (*ProcessReturnValue, error) +``` + +* `Output` implements `aptly.Progress`, so the same `deb.*` code that draws progress bars in + the terminal writes plain text into a buffer retrievable at `/api/tasks/:id/output` + (bars are no-ops; `Printf` appends lines). +* `Detail` carries machine-readable progress (e.g. remaining packages during cleanup, publish + counters) at `/api/tasks/:id/detail`. + +### Scheduling algorithm + +``` +RunTaskInBackground(name, resources, process): + lock list + task := new Task{ID: ++counter, State: IDLE} + append to list + if no running task uses any of `resources` (ResourcesSet.UsedBy == ∅): + mark resources in use by task ; send task to the `queue` channel + else: + leave IDLE (waits) + return copy of task +consumer goroutine: + for task := range queue: set RUNNING ; go run process() + on completion: set SUCCEEDED/FAILED, append "Task succeeded/failed…" to output, + free its resources, Done() wait-groups, + then scan IDLE tasks in order — the first one whose resources are now + free is marked in use and enqueued +``` + +Consequences: + +* **Conflicting requests wait; they are not rejected.** (`runTaskInBackground` has a + `ResourceConflictError` return path that the current `List` never uses; a 409 is therefore + not produced in practice.) A synchronous HTTP request simply blocks until its turn. +* Scheduling is FIFO among *runnable* tasks with a "first-fit" scan on each completion. Only one + waiting task is released per completion event, so long queues of independent tasks progress as + completions occur. +* IDs are process-local counters (reset on restart); tasks are kept in memory until + `DELETE /api/tasks/:id` or `POST /api/tasks-clear`. Synchronous requests delete their own task + after returning the result. + +### Resource keys + +| Key | Meaning | Used by | +|-----|---------|---------| +| `R` (`RemoteRepo.Key()`) | one mirror | mirror update/edit/drop | +| `L` (`LocalRepo.Key()`) | one local repo | repo add/remove/copy/edit/drop/include | +| `S` | one snapshot | snapshot ops | +| `U>>` (`PublishedRepo.Key()`) | one publication | publish ops | +| filesystem paths of the upload dir being consumed | uploads | `repos/:name/file/...` includes the source path as a resource, so the same upload dir can't be imported twice concurrently | +| `__alllocalrepos__` | every local repo | `include` with a templated repo name | +| `__all__` | everything | `db cleanup` | + +`ResourcesSet.UsedBy` implements the wildcards: requesting `__all__` conflicts with any used +resource; requesting `__alllocalrepos__` conflicts with any key beginning with `L`; and if +someone already holds a wildcard, everything conflicts with it. + +### Sync vs async responses + +| Mode | Response | How | +|------|----------|-----| +| **Synchronous** (default) | Final status/body of the operation (`200`, `201`, `4xx/5xx`) | `maybeRunTaskInBackground` schedules the task, `WaitForTaskByID`, reads `ProcessReturnValue{Code,Value}` and the error, deletes the task, replies | +| **Asynchronous** (`_async=true`) | `202 Accepted` + JSON of the `Task` (`ID`, `Name`, `State`, `Resources`) | client polls `/api/tasks/:id`, or blocks on `/api/tasks/:id/wait`, then reads `/output`, `/return_value` | + +Example: + +```bash +# start a long mirror update without holding the connection +TASK=$(curl -s -X PUT -H 'Content-Type: application/json' -d '{}' \ + 'http://localhost:8080/api/mirrors/bookworm-main?_async=true' | jq .ID) +curl -s http://localhost:8080/api/tasks/$TASK/wait # blocks until done → task JSON +curl -s http://localhost:8080/api/tasks/$TASK/output # text log +curl -s http://localhost:8080/api/tasks/$TASK/return_value # {"Code":..,"Value":..} +curl -s -X DELETE http://localhost:8080/api/tasks/$TASK +``` + +## 7.4 Uploading files + +The API cannot read your disk, so package files are first **uploaded** into +`rootDir/upload//`: + +```bash +curl -X POST -F file=@pkg_1.0_amd64.deb http://localhost:8080/api/files/mydir +curl http://localhost:8080/api/files/mydir # list files +curl -X DELETE http://localhost:8080/api/files/mydir/pkg_1.0_amd64.deb +``` + +Then ingest with `POST /api/repos/:name/file/:dir[/:file]` (like `repo add`; removes the +files afterwards unless `?noRemove=1`) or `POST /api/repos/:name/include/:dir[/:file]` (like +`repo include`, for `.changes`; `noRemoveFiles`, `forceReplace`, `acceptUnsigned`, +`ignoreSignature`). `PUT /api/files/:dir/:file` uploads a single file by name. +`apiFilesUpload` streams each part to disk and `fsync`s it (`syncFile`) so `ENOSPC` is reported. +The Prometheus counter `aptly_api_files_uploaded_total{directory=…}` tracks uploads. + +## 7.5 Database locking in API mode + +LevelDB allows one opener per directory. Modes: + +| Mode | Behaviour | +|------|-----------| +| default | The server opens the DB at first use and keeps it open until exit. No other aptly process can use the same `rootDir` concurrently (they retry for `db-open-attempts` × ~10 s) | +| `-no-lock` | A middleware wraps every `/api` request: `acquiredb` opens the DB if there are 0 clients, `releasedb` closes it when the last in-flight client leaves (`acquireDatabase` goroutine with a `dbRequests` channel). Tasks additionally acquire/release a DB reference (`runTaskInBackground`). This lets you run CLI commands *between* API bursts, at the cost of re-opening LevelDB often | +| etcd backend | No exclusive lock; multiple servers/CLIs can share state — resource locks are still **per process** (tasks in different processes don't see each other), so coordinate externally | + +## 7.6 Endpoint catalogue + +Base path `/api`. `{name}` = object name; JSON bodies use PascalCase keys. All mutating +endpoints accept `?_async=true`. + +### Status & meta + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/api/version` | `{"Version": "…"}` | +| GET | `/api/ready` | 200 `Aptly is ready` once the router is built, else 503 | +| GET | `/api/healthy` | 200 always (liveness) | +| GET | `/api/storage` | Disk usage of `rootDir` filesystem: `Total`, `Free` (MiB), `PercentFull` | +| GET | `/api/metrics` | Prometheus metrics (if enabled); refreshes per-repo package counts first | +| GET | `/docs.html`, `/docs/*` | Swagger UI (if enabled) | + +### Local repositories + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/repos` | list | +| POST | `/api/repos` | `{"Name","Comment","DefaultDistribution","DefaultComponent","FromSnapshot"}` — `FromSnapshot` seeds from a snapshot | +| GET/PUT/DELETE | `/api/repos/{name}` | show / edit (`Name`,`Comment`, defaults — pointer fields, only provided ones change) / drop (`?force=1` when used as a source; refused if published) | +| GET | `/api/repos/{name}/packages` | `?q=` `?withDeps=1` `?format=details` `?maximumVersion=1` | +| POST/DELETE | `/api/repos/{name}/packages` | body `{"PackageRefs":[keys…]}` add/remove by key | +| POST | `/api/repos/{name}/file/{dir}[/{file}]` | import uploaded `.deb/.dsc` (`noRemove`, `forceReplace`) | +| POST | `/api/repos/{name}/include/{dir}[/{file}]` | import via `.changes` | +| POST | `/api/repos/{name}/copy/{src}/{file}` | copy package(s) matching query `file` from repo `src` (`{"with-deps":bool,"dry-run":bool}`) | +| POST | `/api/repos/{name}/snapshots` | snapshot of the repo | + +### Mirrors + +| Method | Path | Notes | +|--------|------|-------| +| GET/POST | `/api/mirrors` | list / create (`Name, ArchiveURL, Distribution, Components, Architectures, Filter, FilterWithDeps, DownloadSources/Udebs/Installer/AppStream, Keyrings, SkipComponentCheck, SkipArchitectureCheck, IgnoreSignatures`, `ppa:` URLs allowed) | +| GET | `/api/mirrors/{name}` | show | +| POST | `/api/mirrors/{name}` | **edit** (POST!) | +| PUT | `/api/mirrors/{name}` | **update** = download (long-running; use `_async`). Body (all optional): `Name` (rename), `Keyrings`, `IgnoreChecksums`, `IgnoreSignatures`, `ForceUpdate`, `SkipExistingPackages`, `LatestOnly` — the same switches as `mirror update` | +| DELETE | `/api/mirrors/{name}` | drop (`?force=1`) | +| GET | `/api/mirrors/{name}/packages` | like repo packages | +| POST | `/api/mirrors/{name}/snapshots` | snapshot of the mirror | + +### Snapshots + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/snapshots` | list (`?sort=name` (default) or `?sort=time`) | +| POST | `/api/snapshots` | `{"Name","Description","SourceSnapshots":[…],"PackageRefs":[…]}` — merge sources and/or add explicit package refs | +| GET/PUT/DELETE | `/api/snapshots/{name}` | show / rename & describe / drop (`?force=1`) | +| GET | `/api/snapshots/{name}/packages` | search/list | +| GET | `/api/snapshots/{name}/diff/{withSnapshot}` | `[{"Left":key\|null,"Right":key\|null}]`; `?onlyMatching=1` | +| POST | `/api/snapshots/{name}/merge` | body `{"Sources":[…]}` — the URL snapshot is the **destination name**; options are **query parameters**: `?latest=1`, `?no-remove=1` | +| POST | `/api/snapshots/{name}/pull` | body `{"Source","Destination","Queries":[…],"Architectures":[…]}`; options are **query parameters**: `?no-deps=1`, `?no-remove=1`, `?all-matches=1`, `?dry-run=1` | + +### Publishing + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/publish` | list all publications | +| GET | `/api/publish/{prefix}/{distribution}` | show | +| POST | `/api/publish[/{prefix}]` | create: `{"SourceKind":"snapshot\|local","Sources":[{"Component","Name"}],"Distribution","Architectures","Label","Origin","Signing":{…},"SkipContents","SkipBz2","SkipCleanup","AcquireByHash","SignedBy","MultiDist","Version","NotAutomatic","ButAutomaticUpgrades","ForceOverwrite"}`; 201 | +| PUT | `/api/publish/{prefix}/{distribution}` | switch (snapshots: `{"Snapshots":[{"Component","Name"}]}`) or refresh local repo; same flags | +| DELETE | `/api/publish/{prefix}/{distribution}` | drop (`?force=1`, `?SkipCleanup=1`) | +| POST/GET/PUT/DELETE | `/api/publish/{prefix}/{distribution}/sources` | staged sources: add one / list pending changes / replace all / discard pending | +| PUT/DELETE | `/api/publish/{prefix}/{distribution}/sources/{component}` | update / remove one staged component | +| POST | `/api/publish/{prefix}/{distribution}/update` | apply staged revision and publish | + +The `Signing` object: `{"Skip":bool,"GpgKey":"ID1, ID2","Keyring":"","SecretKeyring":"", +"Passphrase":"","PassphraseFile":""}`. The API **always** runs GPG in batch mode (no prompt), +so an agent or `PassphraseFile`/`Passphrase` is required for protected keys. + +### Files, packages, GPG, cloud listing, graph, cleanup + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/files` | list upload directories | +| POST/GET/DELETE | `/api/files/{dir}` | upload (multipart `file=`) / list files / delete dir | +| PUT/DELETE | `/api/files/{dir}/{file}` | upload one file / delete one | +| GET | `/api/packages` | search across the whole DB (`?q=`, `?format=details`) | +| GET | `/api/packages/{key}` | full detail of one package by key (URL-escaped) | +| GET | `/api/gpg/keys` | list keys in a keyring (`?keyring=`, default `trustedkeys.gpg`) | +| POST/DELETE | `/api/gpg/key` | body `{"Keyring", "GpgKeyArmor" \| "Keyserver"+"GpgKeyID"}` to import; delete takes `Keyring` + key IDs — used to trust mirror signing keys | +| GET | `/api/s3`, `/api/gcs`, `/api/jfrog` | list configured endpoints | +| GET | `/api/graph.{ext}` | Graphviz render of the object graph by shelling out to `dot -T` (`ext` = `png`, `svg`, …; `dot`/`gv` return the source); `?layout=horizontal\|vertical` | +| POST | `/api/db/cleanup` | GC (see §5.6); takes resource `__all__` | + +### Tasks + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/tasks` | all tasks | +| POST | `/api/tasks-clear` | drop finished ones | +| GET | `/api/tasks-wait` | block until *all* tasks complete | +| GET | `/api/tasks/{id}` | task JSON | +| GET | `/api/tasks/{id}/wait` | block until it completes | +| GET | `/api/tasks/{id}/output` | plain text log | +| GET | `/api/tasks/{id}/detail` | structured progress | +| GET | `/api/tasks/{id}/return_value` | `{Code, Value}` | +| DELETE | `/api/tasks/{id}` | only finished tasks | + +### Built-in file serving (`serve_in_api_mode: true`) + +Outside `/api`: `GET /repos/` lists configured *filesystem* endpoints as HTML and +`GET /repos/{storage}/{path…}` serves published files from them, so a single process can be +both API and (basic) repository server. (`aptly serve`, in contrast, exposes only the default +`public/` tree.) + +## 7.7 Metrics (`api/metrics.go`) + +Enabled with `enable_metrics_endpoint: true`; scraped at `/api/metrics` and registered through +`MetricsCollectorRegistrar`: + +| Metric | Type | Labels | +|--------|------|--------| +| `aptly_api_http_requests_in_flight` | gauge | method, path | +| `aptly_api_http_requests_total` | counter | code, method, path | +| `aptly_api_http_request_size_bytes` / `response_size_bytes` | summary | code, method, path | +| `aptly_api_http_request_duration_seconds` | histogram | code, method, path | +| `aptly_build_info` | gauge (=1) | version, goversion | +| `aptly_api_files_uploaded_total` | counter | directory | +| `aptly_repos_package_count` | gauge | source, distribution, component (published packages) | + +The `path` label uses only the first two path segments (`/api/repos`) to bound cardinality. + +## 7.8 Client recipes + +**Publish new package end-to-end (synchronous):** + +```bash +H='Content-Type: application/json' +curl -X POST -F file=@app_1.2_amd64.deb localhost:8080/api/files/incoming +curl -X POST localhost:8080/api/repos/app/file/incoming +curl -X POST -H "$H" -d '{"Name":"app-1.2"}' localhost:8080/api/repos/app/snapshots +curl -X PUT -H "$H" -d '{"Snapshots":[{"Component":"main","Name":"app-1.2"}],"Signing":{"GpgKey":"ABCD1234","PassphraseFile":"/etc/aptly.pass"}}' \ + localhost:8080/api/publish/prod/stable +``` + +**Poll a long mirror update:** + +```bash +curl -X PUT -H "$H" -d '{}' 'localhost:8080/api/mirrors/deb?_async=true' +# → {"Name":"Update mirror deb","ID":7,"State":0,"Resources":["Rabc…"]} +until [ "$(curl -s localhost:8080/api/tasks/7 | jq .State)" -ge 2 ]; do sleep 5; done +``` + +`State`: `0` idle/queued, `1` running, `2` succeeded, `3` failed. diff --git a/documentation/08-cli-and-context.md b/documentation/08-cli-and-context.md new file mode 100644 index 000000000..51cee0e2c --- /dev/null +++ b/documentation/08-cli-and-context.md @@ -0,0 +1,177 @@ +# 8. CLI & Application Context + +## 8.1 Entry point and command dispatch + +```go +// main.go +//go:embed VERSION var Version string +//go:embed debian/aptly.conf var AptlyConf []byte +func main() { + aptly.Version = Version ; aptly.AptlyConf = AptlyConf + os.Exit(cmd.Run(cmd.RootCommand(), os.Args[1:], true)) +} +``` + +The CLI framework is [`smira/commander`](https://github.com/smira/commander) (a fork of +`gonuts/commander`) with [`smira/flag`](https://github.com/smira/flag) (a fork of the standard +`flag` package that additionally supports `IsSet`, so "flag not given" can be distinguished from +"flag given with the default value" — vital for *config default vs. explicit flag* logic). + +`cmd.Run(cmd, args, initContext)` (`cmd/run.go`): + +``` +defer recover(): a *context.FatalError panic → print "ERROR: msg" to stderr, return its exit code +flags, args, err := cmd.ParseFlags(cmdArgs) // global + subcommand flags +if initContext: InitContext(flags) (creates AptlyContext) ; defer ShutdownContext() +context.UpdateFlags(flags) // context.flags = flags of the leaf command +err = cmd.Dispatch(args) // walks the command tree, calls Run(cmd,args) +if err != nil: Fatal(err) // exit 1 (2 for usage / flag errors) +``` + +Exit codes: `0` success, `1` runtime error, `2` command/flag usage error +(`commander.ErrCommandError`, `ErrFlagError`). + +### Command tree (`cmd/*.go`; every command is built by a `makeCmd…()` function) + +``` +aptly +├── api serve +├── config show +├── db cleanup | recover +├── graph +├── mirror create | list | show | drop | update | rename | edit | search +├── package search | show +├── publish snapshot | repo | list | show | drop | update | switch +│ └─ source add | drop | list | remove | replace | update +├── repo create | add | copy | drop | edit | import | list | move | remove +│ | show | rename | search | include +├── serve +├── snapshot create | list | show | verify | pull | diff | merge | drop | rename +│ | search | filter +├── task run +└── version +``` + +Each `makeCmd…` returns a `*commander.Command{Run, UsageLine, Short, Long, Flag}`; help text is +what `aptly help ` prints, and `man/aptly.1` is generated from the same strings +(`_man/gen.go` + `man/aptly.1.ronn.tmpl`). Bash/zsh completion scripts live in `completion.d/`. + +### Global flags (`cmd.RootCommand`) + +| Flag | Meaning | +|------|---------| +| `-config=path` | Config file (else `~/.aptly.conf`, `/usr/local/etc/aptly.conf`, `/etc/aptly.conf`) | +| `-architectures=a,b` | Override configured default architectures | +| `-dep-follow-suggests`, `-dep-follow-recommends`, `-dep-follow-all-variants`, `-dep-follow-source`, `-dep-verbose-resolve` | Dependency-resolution switches (override config) | +| `-db-open-attempts=N` | Retries while the DB is locked by another process (default 10) | +| `-gpg-provider=gpg\|gpg1\|gpg2\|internal` | PGP implementation | +| `-cpuprofile -memprofile -memstats -meminterval` | Only when built with `EnableDebug` | + +Precedence rule used everywhere: **explicit flag → config value → built-in default** +(`LookupOption(default, flags, name)` returns the flag value only if `flags.IsSet(name)`). + +### Reading queries from files + +Package queries and mirror filters may be given as `@file` or `@-` (stdin) via +`GetStringOrFileContent` / `AddStringOrFileFlag` (`cmd/string_or_file_flag.go`). + +### Output conventions + +Human-readable text goes through `context.Progress()` (`Printf`, `ColoredPrintf`), while +`-json` / `-raw` flags on `list/show` commands print machine-readable output directly. `ColoredPrintf` +understands `@r @g @y @!` colour markup (see `console/progress.go`). + +## 8.2 `AptlyContext` — the composition root (`context/context.go`) + +`AptlyContext` owns every shared resource and creates each **lazily on first use**, guarded by a +single mutex (a `Lock()`ed public method calls an unlocked `_private` twin so internal callers +don't deadlock). It embeds a `context.Context` so it can be passed straight to `Download(ctx,…)` +and be cancelled by signals. + +| Accessor | Lazily creates | Notes | +|----------|----------------|-------| +| `Config()` | `utils.Config` | see §8.3 | +| `Progress()` | `console.Progress` | started on creation; shut down in `Shutdown()` | +| `Downloader()` / `NewDownloader(p)` | `http.NewDownloader` or `NewGrabDownloader` | choice from `-downloader` flag, else `downloader` config; limit from `-download-limit` (KiB/s) else `download_limit`; tries = `-max-tries` else `download_retries + 1` | +| `Database()` | LevelDB or etcd `database.Storage` | `Open()` with retry on `resource temporarily unavailable`: up to `-db-open-attempts` (or config `database_open_attempts` unless it is `-1`) with `10s ± 1s` Gaussian jitter | +| `NewCollectionFactory()` | `deb.CollectionFactory` over `Database()` | one per command/request; `Fatal`s if DB can't open | +| `PackagePool()` | local or Azure pool | path from `packagepool_storage.path` else `rootDir/pool` | +| `GetPublishedStorage(name)` | one per configured endpoint | `""` → `rootDir/public` hardlink; `filesystem:x`, `s3:x`, `gcs:x`, `swift:x`, `azure:x`, `jfrog:x`; cached in a map | +| `GetSigner()`, `GetVerifier()` | `pgp.GoSigner`/`GoVerifier` or `GpgSigner`/`GpgVerifier` | provider from `-gpg-provider` else `gpg_provider` (`gpg`, `gpg1`, `gpg2`, `internal`); unknown ⇒ Fatal | +| `TaskList()` | `task.NewList()` | starts the consumer goroutine | +| `DependencyOptions()` | bitmask | computed once from flags/config (`deb.DepFollow*`) | +| `ArchitecturesList()` | `[]string` | `-architectures` else config | +| `UploadPath()`, `SkelPath()`, `DBPath()` | paths | `rootDir/upload`, `rootDir/skel`, `rootDir/db` | +| `CloseDatabase()` / `ReOpenDatabase()` | | temporarily release / reacquire the DB lock | +| `GoContextHandleSignals()` | wraps the embedded `Context` in `WithCancel`; first SIGINT/SIGTERM cancels | second signal kills default handling (`signal.Stop`) | +| `Shutdown()` | | stops task list, closes DB, drops downloader, shuts down progress, writes debug profiles | +| `Cleanup()` | | partial shutdown (downloader + progress) used between `task run` commands | + +`FatalError`/`Fatal(err)` panic with a struct carrying exit code and message; `cmd.Run` (or any +`recover`) turns it into `ERROR: …` + exit status. + +Note the `api` package keeps its own package-level `context` variable pointing to the same +`AptlyContext` (set in `Router(c)`), and `cmd` keeps one in `cmd/context.go`. + +## 8.3 Configuration loading + +`context.config()` (once, memoised by `configLoaded`): + +1. If `-config` given → `utils.LoadConfig(path, &utils.Config)`; any error is fatal. +2. Else try in order `~/.aptly.conf`, `/usr/local/etc/aptly.conf`, `/etc/aptly.conf`: + skip *not exist / permission denied*; stop at first success; any other error is fatal. +3. If nothing was found: print *"Config file not found, creating default config at ~/.aptly.conf"* and + write the embedded `debian/aptly.conf` (or `root_dir: ""` if not embedded), then load it. +4. `utils.LoadConfig` first tries **JSON** (with comments/trailing commas tolerated by + `JsonConfigReader`), and on failure re-reads as **YAML**; if both fail the error mentions both. + `utils.Config` starts as a struct of defaults (see `utils/config.go`), so keys omitted in the file keep + defaults. +5. Logging is initialised from `log_level`/`log_format` (`utils.SetupJSONLogger` or `SetupDefaultLogger`), + and `context.StructuredLogging(true)` disables terminal progress bars. + +`aptly config show [-yaml]` prints the effective config (default JSON; converting old JSON configs to +YAML is `aptly config show -yaml > aptly.yaml`). `ConfigStructure.GetRootDir()` expands a leading `~`. + +Full key reference: [chapter 9](09-configuration.md). + +## 8.4 Lifecycle of a typical command + +``` +aptly snapshot create s1 from mirror m1 + main → cmd.Run + ParseFlags (global flags + "snapshot create" flags) + InitContext NewContext(flags) — nothing opened yet + UpdateFlags(leaf flags) + Dispatch → aptlySnapshotCreate + context.NewCollectionFactory() → Config() loads/creates config + → Database() opens LevelDB (lock; retry if busy) + RemoteRepoCollection.ByName("m1") ← reads R… keys + repo.CheckLock ; LoadComplete ← reads E + deb.NewSnapshotFromRepository + SnapshotCollection.Add → batch: Put S, Put E + fmt.Printf("Snapshot … created") + ShutdownContext → Database().Close(), Progress.Shutdown() + os.Exit(0) +``` + +## 8.5 The `console` package (`console/progress.go`) + +`console.Progress` implements `aptly.Progress` with **one worker goroutine** draining a buffered +channel (capacity 100) of print tasks (`codePrint`, `codePrintStdErr`, `codeProgress`, +`codeHideProgress`, `codeFlush`, `codeStop`, `codeBarEnabled/Disabled`). Concurrent goroutines +(e.g. the download workers) therefore never interleave output. A `progressWorkerFactroy` picks a +terminal worker or a plain/structured-logging worker depending on `structuredLogging`. + +* Progress bars use `cheggaaa/pb` and are created **only when stdout is a terminal** + (`utils.RunningOnTerminal()`); otherwise `InitBar` is a no-op and only text is printed. + `Write(p)` (the `io.Writer` side) advances the bar by bytes, which is how the downloader feeds it. +* `Printf` while a bar is displayed hides the bar, prints, then redraws it. +* `ColoredPrintf` handles `@r @g @y @b @!` … `@|` colour markup through `wsxiaoys/terminal/color`. +* `Flush()` blocks until all queued messages are written — called before spawning `gpg`, which may + prompt on the terminal. + +The REST counterpart is `task.Output` / `task.PublishOutput` (`task/output.go`): a mutex-protected +text buffer whose bar methods are no-ops, except that `PublishOutput` counts down +`RemainingNumberOfPackages` for the `BarPublishGeneratePackageFiles` bar so API clients can poll +publish progress from `/api/tasks/:id/detail`. `ColoredPrintf` there appends the text plus a newline +without colours. diff --git a/documentation/09-configuration.md b/documentation/09-configuration.md new file mode 100644 index 000000000..8fd2108a5 --- /dev/null +++ b/documentation/09-configuration.md @@ -0,0 +1,210 @@ +# 9. Configuration Reference + +aptly is configured with a single file, **YAML (preferred since 1.6.0) or JSON** (comments and +trailing commas tolerated). The Go definition is `utils.ConfigStructure` (`utils/config.go`); each +field carries both a `json:` and a `yaml:` tag. The annotated template shipped with the package is +[`debian/aptly.conf`](../debian/aptly.conf) — it is also **embedded in the binary** and written to +`~/.aptly.conf` the first time aptly runs without any config. + +Location search order (first hit wins): `-config=`, `~/.aptly.conf`, +`/usr/local/etc/aptly.conf`, `/etc/aptly.conf`. Inspect the effective values with +`aptly config show` (JSON) or `aptly config show -yaml`. + +> Legacy JSON keys are camelCase (`rootDir`, `downloadConcurrency`, `S3PublishEndpoints`, …); the YAML +> keys are snake_case (`root_dir`, `download_concurrency`, `s3_publish_endpoints`, …). The tables below +> list the **YAML key**, then the JSON key in parentheses. + +## 9.1 General + +| YAML (JSON) | Default | Meaning | +|-------------|---------|---------| +| `root_dir` (`rootDir`) | `~/.aptly` | Root for `db/`, `pool/`, `public/`, `upload/`, `skel/`. `~` is expanded | +| `log_level` (`logLevel`) | `info` | `debug`, `info`, `warning`, `error` | +| `log_format` (`logFormat`) | `default` | `default` (text) or `json`. `json` also makes the API access log JSON and disables terminal bars | +| `database_open_attempts` (`databaseOpenAttempts`) | `-1` | Retries when the DB is locked. `-1` = use the `-db-open-attempts` flag (default 10); otherwise this value overrides the flag | +| `architectures` | `[]` | Default architecture list (empty = "all available"). Overridable with `-architectures` | +| `skip_legacy_pool` (`skipLegacyPool`) | `false` (template: `true`) | OBSOLETE. Stop looking for MD5-based pre-1.1 pool paths | + +## 9.2 Dependency following + +Used by `snapshot pull`, `snapshot/mirror/repo … -with-deps`, `mirror` filters with deps, and the API's +`withDeps`. Each has a matching command-line flag. + +| YAML (JSON) | Default | Effect | +|-------------|---------|--------| +| `dep_follow_suggests` (`dependencyFollowSuggests`) | false | Also resolve `Suggests:` | +| `dep_follow_recommends` (`dependencyFollowRecommends`) | false | Also resolve `Recommends:` | +| `dep_follow_all_variants` (`dependencyFollowAllVariants`) | false | For `a \| b` pull both alternatives (template spells it `dep_follow_allvariants`; the code reads `dep_follow_all_variants`) | +| `dep_follow_source` (`dependencyFollowSource`) | false | From a binary package also pull its source package | +| `dep_verboseresolve` (`dependencyVerboseResolve`) | false | Log resolver decisions (`Injecting package`, `Unsatisfied dependency`, …) | + +> Watch the key spellings: struct tags are `dep_follow_all_variants` and `dep_verboseresolve` +> (see `utils/config.go`), whereas the commented template uses slightly different spellings. Verify +> with `aptly config show -yaml` after editing. + +## 9.3 PPA short-URL expansion + +| YAML | Default | Meaning | +|------|---------|---------| +| `ppa_distributor_id` | `ubuntu` | Used in `ppa:user/project` → URL | +| `ppa_codename` | *(empty → output of `lsb_release`)* | Distribution for expansion | +| `ppa_baseurl` | `http://ppa.launchpad.net` | Base URL | + +## 9.4 Server + +| YAML | Default | Meaning | +|------|---------|---------| +| `serve_in_api_mode` | false | API server also serves `/repos/…` from *filesystem* publish endpoints | +| `enable_metrics_endpoint` | false | Prometheus `/api/metrics` | +| `enable_swagger_endpoint` | false | Swagger UI on `/docs.html` | +| `async_api` | false | OBSOLETE. Use `?_async=true` per request | + +## 9.5 Database + +```yaml +database_backend: + type: leveldb # or: etcd + db_path: "" # leveldb only; "" → /db + # url: "127.0.0.1:2379" # etcd only +``` + +## 9.6 Mirroring + +| YAML | Default | Meaning | +|------|---------|---------| +| `downloader` | `default` | `default` (net/http, streaming) or `grab` (`cavaliergopher/grab`, resumable, more robust on flaky links). Flag: `-downloader` | +| `download_concurrency` | `4` | Parallel download workers in `mirror update` | +| `download_limit` | `0` | KiB/s cap (0 = none). Flag: `-download-limit` | +| `download_retries` | `0` | Additional tries per file (`max-tries = retries + 1`). Flag: `-max-tries` | +| `download_sourcepackages` | false | Default for `-with-sources` at mirror creation | + +## 9.7 Signing & verification + +| YAML | Default | Meaning | +|------|---------|---------| +| `gpg_provider` | `gpg` | `gpg`/`gpg1`/`gpg2` (external binaries; see chapter 10) or `internal` (pure Go OpenPGP) | +| `gpg_disable_sign` | false | Never sign published repos | +| `gpg_disable_verify` | false | Never verify remote `Release`/`.changes` signatures | +| `gpg_keys` | `[]` | Key IDs used to sign Release when `-gpg-key` isn't provided (multiple keys ⇒ multiple signatures) | + +## 9.8 Publishing defaults + +| YAML | Default | Meaning | +|------|---------|---------| +| `skip_contents_publishing` | false | Default for `-skip-contents` at *first* publish | +| `skip_bz2_publishing` | false | Default for `-skip-bz2` at first publish | + +(Later `publish update/switch` can change these per publication with the same flags; they are +persisted in the publication record.) + +## 9.9 Published-storage endpoints + +An **endpoint** is a named target; you select it on the command line/API as `type:name:prefix` +(e.g. `s3:prod:`). The default endpoint (`""`) is `/public` with hard links and needs no +configuration. + +### filesystem + +```yaml +filesystem_publish_endpoints: + web1: + root_dir: /srv/apt # required + link_method: hardlink # hardlink | symlink | copy + verify_method: md5 # md5 | size (only relevant for copy) +``` +`aptly publish snapshot s filesystem:web1:ubuntu` + +### s3 + +```yaml +s3_publish_endpoints: + prod: + region: eu-central-1 + bucket: my-apt + prefix: "" + acl: public-read # private (default) | public-read | none + access_key_id: "" # else AWS SDK default chain / env vars / IAM role + secret_access_key: "" + session_token: "" + endpoint: "" # S3-compatible service URL (MinIO, Ceph, …) + storage_class: STANDARD + encryption_method: "" # AES256 | aws:kms + plus_workaround: false + disable_multidel: false + force_sigv2: false # accepted but currently ignored by the AWS SDK v2 based client + force_virtualhosted_style: false + debug: false +``` + +### gcs, azure, swift, jfrog + +See the annotated blocks in `debian/aptly.conf`; field lists are in +[chapter 4 §4.5](04-storage.md#45-object-store-back-ends). Keys: + +* `gcs_publish_endpoints.`: `bucket, prefix, credentials_file, service_account_json, project, endpoint, acl, storage_class, encryption_key, disable_multidel, debug` +* `azure_publish_endpoints.`: `container, prefix, account_name, account_key, endpoint` +* `swift_publish_endpoints.`: `container, prefix, username, password, auth_url, tenant, tenant_id, domain, domain_id, tenant_domain, tenant_domain_id` (env fallbacks `OS_USERNAME`, `OS_PASSWORD`) +* `jfrog_publish_endpoints.`: `url, repository, username, password, api_key, access_token, prefix, plus_workaround, debug` — the Artifactory repository must be of the **generic** type, not "debian". + +## 9.10 Package pool storage + +```yaml +packagepool_storage: + type: local # local (default) | azure + path: "" # "" → /pool +# --- or --- +packagepool_storage: + type: azure + account_name: … + account_key: … + container: pool1 + prefix: "" + endpoint: "" +``` + +Unknown `type` values fail config loading with `unknown pool storage type`. + +## 9.11 Complete example + +```yaml +root_dir: /var/lib/aptly +log_level: info +architectures: [amd64, arm64] + +gpg_provider: gpg +gpg_keys: ["0xABCDEF1234567890"] + +downloader: grab +download_concurrency: 8 +download_retries: 3 + +enable_metrics_endpoint: true +enable_swagger_endpoint: false + +database_backend: + type: leveldb + +filesystem_publish_endpoints: + frontend: + root_dir: /srv/apt-public + link_method: hardlink # requires /srv on the same fs as /var/lib/aptly/pool, + # otherwise use: copy + +s3_publish_endpoints: + cdn: + region: us-east-1 + bucket: acme-apt + acl: public-read +``` + +## 9.12 Environment variables that influence aptly + +| Variable | Used by | +|----------|---------| +| `HOME` | default config location (`~/.aptly.conf`), `root_dir` default (`~/.aptly`), `~` expansion | +| `GNUPGHOME` | internal PGP provider: keyring directory (default `~/.gnupg`) | +| `SOURCE_DATE_EPOCH` | fixes the `Date:` field in published `Release` (reproducible publishes) | +| `AWS_*`, `OS_USERNAME`/`OS_PASSWORD`, `JFROG_USERNAME`/`JFROG_PASSWORD`/`JFROG_APIKEY`/`JFROG_ACCESSTOKEN` | credentials fall-backs | +| `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` | the downloader honours the standard Go proxy environment (`http.ProxyFromEnvironment`) | +| `TERM=dumb` | set by `aptly-api.service` so terminal features are avoided | +| `TMPDIR` | temp files (index generation, `.changes` handling, temporary DB for Contents) | diff --git a/documentation/10-signing-and-verification.md b/documentation/10-signing-and-verification.md new file mode 100644 index 000000000..becdd1826 --- /dev/null +++ b/documentation/10-signing-and-verification.md @@ -0,0 +1,143 @@ +# 10. Signing & Verification (`pgp/`) + +aptly touches OpenPGP in two directions: + +| Direction | Purpose | Interface | Where used | +|-----------|---------|-----------|------------| +| **Verify** (inbound) | Trust remote `Release`/`InRelease`, source `.dsc`, `.changes` uploads, installer `SHA256SUMS` | `pgp.Verifier` | `mirror create/update`, `repo add` (`.dsc`), `repo include` | +| **Sign** (outbound) | Sign the repositories aptly publishes | `pgp.Signer` | `publish snapshot/repo/switch/update`, API publish | + +```go +type Signer interface { + Init() error ; SetKey(keyRef string) ; SetKeyRing(keyring, secretKeyring string) + SetPassphrase(passphrase, passphraseFile string) ; SetBatch(bool) + DetachedSign(src, dst string) error ; ClearSign(src, dst string) error +} +type Verifier interface { + InitKeyring(verbose bool) error ; AddKeyring(keyring string) + VerifyDetachedSignature(sig, cleartext io.Reader, showKeyTip bool) error + IsClearSigned(io.Reader) (bool, error) + VerifyClearsigned(io.Reader, showKeyTip bool) (*KeyInfo, error) + ExtractClearsigned(io.Reader) (*os.File, error) // NO verification +} +``` + +## 10.1 Providers + +Selected by `gpg_provider` (config) or `-gpg-provider` (global flag). `AptlyContext.GetSigner()` / +`GetVerifier()` instantiate: + +| Provider | Signer | Verifier | Notes | +|----------|--------|----------|-------| +| `gpg` (default) | `GpgSigner` (runs `gpg`) | `GpgVerifier` (runs `gpgv`) | binaries located by `GPGDefaultFinder` (`gpg`, `gpgv`); auto-detects gnupg 1.x vs 2.x | +| `gpg1` | same | same | requires `gpg`/`gpg1` and `gpgv`/`gpgv1` in `$PATH` that are GnuPG **1.x** | +| `gpg2` | same | same | requires `gpg`/`gpg2`, `gpgv`/`gpgv2` = **2.x** | +| `internal` | `GoSigner` | `GoVerifier` | pure Go (`golang.org/x/crypto/openpgp`, plus extended signature-checking helpers in `pgp/openpgp.go` that can validate *several* signers of one detached signature and track missing keys); no external binaries | + +`gpg` and `gpgv` versions must match, otherwise `NewGpgVerifier` panics *"gpg and gpgv versions +don't match"*. Test fixtures for this logic are in `pgp/test-bins`. + +## 10.2 Verifying remote repositories + +### Trust store + +* External provider: `gpgv --keyring trustedkeys.gpg …` — i.e. the **`trustedkeys.gpg`** keyring in + the GnuPG home of the user running aptly (aptly does *not* use the general `pubring`). Extra keyrings + are added with `-keyring=` (repeatable); if any are given they *replace* the default. +* Internal provider: keyring files are loaded by `loadKeyRing`; names without `/` are relative to + `$GNUPGHOME` (default `~/.gnupg`). Missing default keyring is tolerated (empty); a missing explicit + one prints a warning. +* API: `POST /api/gpg/key` (import armored key text or fetch `GpgKeyID`s from a `Keyserver`), + `GET /api/gpg/keys`, `DELETE /api/gpg/key` — implemented by shelling out to + `gpg --no-default-keyring --keyring …` (keyring path is sanitised). + +Typical bootstrap on Debian/Ubuntu (printed by aptly when the keyring is empty): + +```bash +gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \ + | gpg --no-default-keyring --keyring trustedkeys.gpg --import +# or a single key: +gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 0xKEYID +wget -O - https://vendor.example/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import +``` + +### What is verified, in which order (`RemoteRepo.Fetch`) + +1. `InRelease` (clearsigned) — `VerifyClearsigned`; text extracted with `ExtractClearsigned`. +2. If missing/invalid → `Release` + `Release.gpg` — `VerifyDetachedSignature`. +3. With `-ignore-signatures` (or `gpg_disable_verify`) **no verification** takes place; the payload is + still parsed (an `InRelease` is stripped of its armour via `ExtractClearsigned`). + +Integrity below `Release` relies on hashes, not signatures: every `Packages*` file is checked against +`Release`'s checksums (`ReleaseFiles`), and every `.deb` against the `Packages` stanza checksums +(`DownloadWithChecksum`). Disabling checksum checks (`-ignore-checksums`) weakens the chain. + +Failure output (`runGpgv`): prints which keys were **good** vs **missing** (`--status-fd 3`), and +suggests `--recv-keys` for missing key IDs (`KeyInfo{GoodKeys, MissingKeys}`). +`repo add` verifies `.dsc` signatures when present; `repo include` verifies `.changes` +(unless `-ignore-signatures`, `-accept-unsigned`); the signing keys are then used for +uploaders ACL (see [`repo include`](05-workflows.md#52-local-repositories)) decisions. + +## 10.3 Signing published repositories + +### Which files are signed + +| File | Signature | +|------|-----------| +| `dists//Release` | detached armored → `Release.gpg`; and clearsigned → `InRelease` | +| installer `…/installer-/current/images/SHA256SUMS` | detached `SHA256SUMS.gpg` | +| everything else (`Packages`, `Contents`, per-arch `Release`) | *not* individually signed — covered by hashes in the signed top-level `Release` | + +Digest algorithm for the external provider is forced to **SHA256** (`--digest-algo SHA256`). + +### Choosing the key + +Resolved by `getSigner(flags)` (CLI, `cmd/publish.go`) / `getSigner(&signingParams)` (API): + +1. `-skip-signing` (or `gpg_disable_sign: true`, or API `Signing.Skip`) ⇒ **no signer**; nothing signed. +2. Keys: `-gpg-key` (repeatable) → else config `gpg_keys` → else *gpg's default key* + (`GpgSigner` passes each as `-u `; several keys produce several signatures — useful during key + rotation. API: `Signing.GpgKey = "ID1, ID2"`.) +3. Keyrings: `-keyring`, `-secret-keyring` (GnuPG 1 only for secret keyring). +4. Passphrase: `-passphrase`, `-passphrase-file` (both **insecure** in shell history/process list; prefer a + file with restrictive mode or gpg-agent), or interactive prompt. +5. `-batch` ⇒ `--no-tty --batch` (never prompt). **The API always uses batch mode** (an interactive prompt + would block the server); so protected keys need `gpg-agent` with cached passphrase, or a passphrase + file/param. +6. `signer.Init()` checks that `gpg --list-keys` works and that *some* key exists (when no explicit + keyring is given). + +### GoSigner specifics + +`Init()` reads `pubring.gpg` and `secring.gpg` (legacy keyring formats) from `$GNUPGHOME` unless paths +are given; picks the first valid secret key or the one matching `keyRef` (key ID, or substring of a user +ID); decrypts it with the supplied/prompted passphrase (3 attempts interactive). If `SOURCE_DATE_EPOCH` +is set, signatures are created with that timestamp and *deterministic* settings. **Limitation:** modern +GnuPG (2.1+) stores secret keys in `private-keys-v1.d`/keybox, which `internal` cannot read — export the +key into legacy keyring files, or use the `gpg` provider. + +### Publishing your public key + +aptly never publishes your public key. Export it yourself and host it next to the repository +(skeleton files from `rootDir/skel` are copied *into component directories*, not the prefix root, so +they are not a convenient place for it): + +```bash +gpg --armor --export KEYID > /srv/apt/public.key +# client: +curl -fsSL https://repo.example.com/public.key | sudo gpg --dearmor -o /etc/apt/keyrings/example.gpg +# sources.list: deb [signed-by=/etc/apt/keyrings/example.gpg] https://repo.example.com/ stable main +``` + +`-signed-by=` on publish adds `Signed-By:` (+ `Valid-Until` 100 years ahead) to the top-level +`Release`, letting clients pin acceptable keys for future updates. + +## 10.4 Security notes + +* **Passphrases on the command line** are visible in `ps` and shell history; the API + `Signing.Passphrase` field is stored nowhere but travels in the request body — use TLS. +* The private key must be available on the machine running aptly; consider a dedicated *repository* + subkey with expiry, and rotate using multiple `-gpg-key`s. +* Verification via `trustedkeys.gpg` is only as strong as its contents: import vendor keys from their + official keyrings, not via TOFU keyserver fetches, when possible. +* Never use `-ignore-signatures`/`-ignore-checksums` on untrusted networks. diff --git a/documentation/11-queries-and-dependencies.md b/documentation/11-queries-and-dependencies.md new file mode 100644 index 000000000..b2f67195c --- /dev/null +++ b/documentation/11-queries-and-dependencies.md @@ -0,0 +1,189 @@ +# 11. Queries, Dependencies & Version Comparison + +## 11.1 Package query language + +Used by `mirror -filter`, `snapshot filter/pull`, `repo copy/move/remove/import`, `package search`, +`*/search` commands, `uploaders.json` conditions and the API `?q=`. Parsed by +`query.Parse(string)` (`query/lex.go` → `query/syntax.go`) into a tree of `deb.PackageQuery` +(`deb/query.go`). + +### Grammar (recursive descent, from `syntax.go`) + +``` +Query := A | A '|' Query OR (lowest precedence) +A := B | B ',' A AND +B := C | '!' B NOT +C := '(' Query ')' | D +D := [condition] | [condition] [arch] | __ +condition := '(' [operator] value ')' +operator := '<<' | '<' | '<=' | '>' | '>>' | '>=' | '=' | '%' | '~' +arch := '{' architecture '}' +``` + +Tokens: `( ) | , ! << <= < >> >= > = % ~ { }` and strings. +Note **precedence**: `,` (AND) binds tighter than `|` (OR); use parentheses. + +### Term kinds + +| Term | Example | Becomes | Meaning | +|------|---------|---------|---------| +| Package name | `nginx` | `DependencyQuery` | any version of package named `nginx` (any architecture) | +| Name + version condition | `nginx (>= 1.24)` | `DependencyQuery` | uses Debian dependency semantics (see below) | +| Name + arch | `nginx {amd64}` | `DependencyQuery` | limited to architecture | +| Exact package key | `nginx_1.24.0-1_amd64` | `PkgQuery` | matches name, version, arch exactly. Recognised when there is no condition and the string has two `_` | +| Field query | `Section (= web)` | `FieldQuery` | compare a control field. Field name must start with an **upper-case** letter (and contain no `_`) or with `$` | +| Regexp | `Name (~ ^lib.*-dev$)` / `nginx (~ ^1\.2)` | `FieldQuery`/`DependencyQuery` with `Regexp` | `~` | +| Pattern (glob) | `Name (% nginx-*)` | glob | `%` uses shell-style `filepath.Match` | +| Special fields | `$Version`, `$Architecture`, `$PackageType`, `$Source`, `$SourceVersion` | | see below | + +**Operators.** Following Debian dependency syntax: `<<` strictly less, `>>` strictly greater; the +lexer maps single `<` to *less-or-equal* and `>` to *greater-or-equal* (`<=`, `>=` explicitly the +same) — the historical Debian meaning. `=` equal (default when no operator given inside +parentheses), `%` glob pattern, `~` regular expression (Go RE2). + +### Field names available + +`Package.GetField` (`deb/package.go`) answers, case-sensitively (for source packages +`Architecture` returns the real `Architecture:` of the source, not `source`): +`Name`, `Version`, `Architecture`, `Source`, `Depends`, `Pre-Depends`, `Suggests`, `Recommends`, +`Provides`, `Build-Depends`, `Build-Depends-Indep`, plus any other control field from the stanza +(`Section`, `Priority`, `Maintainer`, `Description`, `Filename` …), and specials: + +| Special | Value | +|---------|-------| +| `$Version` | the package version, evaluated like a dependency on the package's own name (Debian version ordering) | +| `$Source` | source package name (own name if there is no `Source:`); **empty for source packages** | +| `$SourceVersion` | version from `Source: name (version)`, else the package's own version; empty for source packages | +| `$Architecture` | the architecture; an `=` comparison goes through `MatchesArchitecture`, so `$Architecture (= amd64)` also matches `Architecture: all` packages | +| `$PackageType` | `deb`, `udeb` or `source` (installer pseudo-packages report `deb`) | + +> **Quirk:** every field comparison except `%` and `~` (and "field is non-empty" when no operator is +> given) is evaluated with `CompareVersions`, i.e. with *Debian version ordering*, even for non-version +> fields such as `Section (= web)`. Equality behaves as expected; `<`/`>` on textual fields follow version +> rules, which is rarely what you want. + +### Examples + +``` +nginx all versions of package nginx +nginx (>= 1.20), !Name (nginx-extras) nginx ≥ 1.20, and never the nginx-extras package +Priority (required) | Priority (important) +$PackageType (source) +Name (~ ^python3-.*), $Version (>= 3.0) +(Section (= web) | Section (= net)), !Depends (~ apache2) +Maintainer (% *@example.com) +libssl3_3.0.11-1_amd64 one exact binary package +``` + +Errors are reported by `Parse` as `parsing failed: …` (e.g. *unexpected token … expecting end of query*) +before any DB access; mirror filters are parsed at creation and at update. + +### Evaluation strategy (`PackageList.Scan / Query`) + +Each `PackageQuery` has `Matches(PackageLike)`, `Fast(catalog)` and `Query(catalog) *PackageList`: + +* `DependencyQuery.Fast()` is true when the catalog supports searching by dependency — a + `PackageList` with an index. Then `Query` uses **binary search on the name index** (`Search`) and, + because `providesIndex` exists, also virtual packages (`Provides`). +* `AndQuery`: fast if **either** side is fast — evaluate the fast side, then `Scan` that (small) + result with the other side. +* `OrQuery`: fast only if **both** sides are fast; results are unioned. Otherwise `Scan`. +* `NotQuery`, `FieldQuery`, `MatchAllQuery`: full `Scan` calling `Matches` per package. +* `PkgQuery` (`name_version_arch`) is a direct key lookup (`SearchByKey`). + +## 11.2 Dependencies (`deb/package_deps.go`, `deb/list.go`) + +### Parsing + +`ParseDependency("libc6 (>= 2.34)")` → `Dependency{Pkg, Relation, Version, Architecture}`. +Alternatives `a | b` are split by `ParseDependencyVariants`. Architecture may be given as multiarch +`pkg:arch` (`pkg:any` = no restriction) or with aptly's own suffix `pkg (= 1.0) {arch}` (used by generated +source dependencies such as `foo (= 1.0) {source}`). Relations inside parentheses: `<<` (strictly less), +`<`/`<=` (both = less-or-equal), `=`, `>`/`>=` (both = greater-or-equal), `>>`; no operator = equal. +Relations: `VersionDontCare, Less, LessOrEqual, Equal, GreaterOrEqual, Greater, PatternMatch, Regexp`. + +### Which fields are followed (`Package.GetDependencies(options)`) + +Always `Depends` + `Pre-Depends`; optionally (bitmask `deb.DepFollow*`, set from flags/config): + +| Option | Adds | +|--------|------| +| `DepFollowRecommends` | `Recommends` | +| `DepFollowSuggests` | `Suggests` | +| `DepFollowBuild` (internal) | `Build-Depends`, `Build-Depends-Indep` | +| `DepFollowSource` | a dependency on the package's own *source* package: `src (= version) {source}` | +| `DepFollowAllVariants` | for `a \| b` keep looking for every alternative even if one is already satisfied | +| `DepVerboseResolve` | log decisions | + +### Verification & closure + +`PackageList.VerifyDependencies(options, architectures, sources, progress)`: + +``` +for arch in architectures: + cache := {} # dependency-hash → satisfied? + for p in list where p.MatchesArchitecture(arch): + for depString in p.GetDependencies(options): + variants := ParseDependencyVariants(depString) # alternatives + for v in variants: # arch defaults to `arch` + satisfied := sources.Search(v, allMatches=false, searchProvided=true) != nil (cached) + if satisfied and !FollowAllVariants: variantsMissing = nil ; break + if !satisfied: variantsMissing += v + missing += variantsMissing +missing = dedupe(missing) +``` + +`PackageList.Search(dep, allMatches, searchProvided)` binary-searches the name index and filters by +architecture (`MatchesDependency`: an `all` package satisfies any architecture but `source`), version +relation (`versionSatisfiesDependency` via `CompareVersions`) and, when `searchProvided`, virtual packages +from `Provides`. In `Provides`, only `=` versions are allowed (others are logged as a warning and ignored); +an **unversioned** `Provides: foo` is treated as providing `foo` at the *providing package's own version*. + +`PackageList.Filter(FilterOptions)` implements **dependency closure**: + +``` +result := ⋃ query.Query(list) for each query +if WithSources: add matching source packages (via Source: field / SourceRegex) +if WithDependencies: + depSource := Source ∪ result + loop while something was added: + missing := result.VerifyDependencies(opts, arch, depSource) + for dep in missing: + if !FollowAllVariants and result.Search(dep) != nil: continue # satisfied now + for p in list.Search(dep, allMatches=true, searchProvided=true): + add p to result and depSource ; (unless FollowAllVariants → only first) +``` + +The **`Source`** in `FilterOptions` is the set that dependencies are considered *already satisfied* +by (e.g. `snapshot pull` passes the destination snapshot, `repo copy -with-deps` passes the target repo, +mirror filters pass an empty list so *everything* needed is pulled in). + +### Practical consequences + +* Filtering a mirror with `-filter-with-deps` will pull `Depends`/`Pre-Depends` closure only — use + `-dep-follow-recommends` etc. globally if you want `apt install` with default recommends to work offline. +* Missing dependencies are **not an error** during filtering (they are logged only with + `-dep-verbose-resolve`); use `aptly snapshot verify` to audit a snapshot afterwards. +* Virtual packages resolved through `Provides` may pick arbitrary providers; with + `-dep-follow-all-variants` *all* providers/alternatives are included (larger result). + +## 11.3 Debian version comparison (`deb/version.go`) + +`CompareVersions(a, b)` → `-1, 0, 1` implements the Debian Policy algorithm +`[epoch:]upstream_version[-debian_revision]`: + +1. Split epoch (text before the first `:`), then the revision — `parseVersion` splits at the **first** + `-` (Debian Policy says the *last* hyphen; the two differ only for upstream versions that themselves + contain hyphens) — leaving the upstream part. +2. Compare epoch (numeric; missing = 0), then upstream, then revision, each with `compareVersionPart`: + alternate **non-digit run** (compared with `compareLexicographic`) and **digit run** (compared + numerically), left to right. +3. `compareLexicographic`: letters sort before non-letters; `~` sorts before everything, even the end of + the string (so `1.0~rc1 < 1.0`); otherwise ASCII order. + +`PackageList` orders packages by name ascending then **version descending** (newest first), which is +why `snapshot pull` "first match wins" picks the highest version, and `FilterLatest`/ +`FilterLatestRefs` keep only the maximum. + +Note the distinction between **key order** (byte-wise on `P `, used inside +`PackageRefList`) and **version order** (`CompareVersions`, used when semantics matter). diff --git a/documentation/12-operations.md b/documentation/12-operations.md new file mode 100644 index 000000000..d442a622e --- /dev/null +++ b/documentation/12-operations.md @@ -0,0 +1,163 @@ +# 12. Operations Guide + +## 12.1 Directory layout on disk + +``` +/ (default ~/.aptly; owner: the aptly user) +├── db/ LevelDB files (CURRENT, LOCK, MANIFEST-*, *.ldb, *.log) +├── pool/ content-addressed package files (pool/ab/cd/<28 hex>_) +├── public/ default published storage (hard links into pool/) +├── upload/ API upload area (upload//) +└── skel/ optional skeleton files copied into dists/// +``` + +`~/.aptly.conf` (or `/etc/aptly.conf`) holds configuration; GnuPG state lives in `~/.gnupg`. +Temp files use `$TMPDIR` (`/tmp`): index generation, `.changes` handling, temporary Contents DB. + +## 12.2 Deployment patterns + +### A. CLI on a build/release host +Simple: cron or CI job runs `aptly mirror update …; aptly snapshot create …; aptly publish switch …`. +Use `aptly task run` to run several steps in one process. + +### B. API server (systemd) +Debian packaging provides `aptly-api` (`debian/aptly-api.service`, `debian/aptly-api.default`): + +``` +aptly-api.service User=aptly-api Group=aptly-api WorkingDirectory=~ LimitNOFILE=32768 Environment=TERM=dumb +/etc/default/aptly-api LISTEN_ADDRESS='localhost:8080' APTLY_OPTIONS="-no-lock" +ExecStart=/usr/bin/aptly api serve -config=/etc/aptly.conf ${APTLY_OPTIONS} -listen=${LISTEN_ADDRESS} +``` + +The post-install script creates the `aptly-api` system user (home `/var/lib/aptly-api`) and makes +`/etc/aptly.conf` `root:aptly-api 0640` because the config may contain secrets (cloud credentials, +passphrases). Note that the `aptly` package's `/etc/aptly.conf` therefore has **restrictive permissions** +(keep them so). + +* The packaged default listens on **localhost** only and uses `-no-lock` (so administrators can still run + `aptly` CLI commands while the service is idle). +* No socket unit is shipped, but `aptly api serve` adopts a single listener fd passed by systemd (you can + write your own `aptly-api.socket` with `ListenStream=…`). + +### C. Docker +`system/Dockerfile` builds a *development/test* image; for production, build a minimal image containing the +static binary, mount `root_dir` as a volume, and expose 8080 behind a reverse proxy. + +### D. Serving the published repository + +| Option | Notes | +|--------|-------| +| nginx/Apache over `public/` | Most common. Enable directory listing (autoindex) only if you want browsing; `Cache-Control: no-cache` (or short TTL) for `dists/`, long TTL for `pool/`. Because hard links are used, `public/` must live on the same filesystem as `pool/` or use `link_method: copy` | +| Object store (S3/GCS/Azure/Swift) | Publish directly with an endpoint; serve through the store's website endpoint or a CDN. Set `acl: public-read` or use signed access | +| `aptly serve` | Development convenience only | +| API-mode `/repos/…` | Convenience for filesystem endpoints | + +Minimal nginx snippet: + +```nginx +server { + listen 80; + root /var/lib/aptly/public; + location / { autoindex on; } + location ~ /dists/ { add_header Cache-Control "no-cache"; } +} +``` + +## 12.3 Concurrency & locking summary + +| Situation | What happens | +|-----------|--------------| +| Two CLI commands at once | Second waits: retries opening LevelDB every ~10 s for `database_open_attempts` (default 10) tries, then fails *"unable to reopen the DB, maximum number of retries reached"*. Use `-db-open-attempts=0` to fail immediately (`0` loops zero times after the first try) | +| CLI while `aptly mirror update` is in its download phase | Allowed — the updater closed the DB, so other commands (listing, publishing, repo work) proceed. Commands touching *that mirror* are guarded by `CheckLock`: e.g. `snapshot create … from mirror` and another `mirror update` refuse while it is being updated | +| CLI while `aptly api serve` (default) | Blocked until the server stops | +| CLI while `aptly api serve -no-lock` | Works whenever the API is idle; API requests during a CLI command wait for the DB (they retry to open) | +| Two API mutations on the same object | Serialised by the task list (queue) | +| Two mirrors updating in parallel via API | Allowed (different resource keys) but they share the pool and network | +| `db cleanup` | Exclusive: no other API tasks run; don't run publish/update concurrently in CLI | +| Crash during mirror update | `Status=updating` with a dead PID is ignored by `CheckLock`; re-run `mirror update` (only missing files are refetched). Leftover temp files can remain in `pool///…` random names — `db cleanup` removes files unreferenced by any package | +| Crash during `publish` | Package files linked, indexes partially uploaded (`*.tmp` when re-publishing). Re-run the same publish command/`publish update`; stale `.tmp` files are overwritten | + +## 12.4 Securing the API + +The API has no built-in authentication. Recommendations: + +1. Bind to `localhost` or a Unix socket (`-listen=unix:///run/aptly/aptly.sock`, restrict permissions with + directory ownership). +2. Front with nginx/Envoy/Traefik providing **TLS + auth** (basic auth, mTLS, OIDC proxy). +3. Restrict methods/paths per role at the proxy (e.g. CI may only `POST /api/files/*`, + `POST /api/repos/*/file/*`, `PUT /api/publish/*`). +4. Don't put passphrases in requests if the proxy logs bodies; prefer `gpg-agent` or `PassphraseFile`. +5. Limit upload size at the proxy (`client_max_body_size`); uploads are written to disk in `upload/`. +6. Run as a dedicated unprivileged user; its `~/.gnupg` holds the signing key. + +## 12.5 Backup, restore, migration + +* **What to back up:** `root_dir/db` (metadata), `root_dir/pool` (package files), GnuPG keys, config. + `public/` is *derived* (re-publishable) — but the exact bytes of indexes matter to clients that pinned + hashes, so many operators back it up too. +* **Consistent backup:** stop aptly (or make sure no CLI/API task runs); LevelDB files are only + consistent when the DB is closed. Copy `db/` first, then `pool/` (extra files in pool are harmless; missing files are not). +* **Restore:** put back both; run `aptly db cleanup -dry-run` to check consistency; `aptly db recover` if + the DB reports corruption (rebuilds LevelDB and drops dangling refs from local repos). +* **Moving to another host:** copy `db/` and `pool/`; publishes with hard links need to be regenerated + (`aptly publish update`/`switch`) since `public/` links won't survive a cross-fs copy. +* **LevelDB → etcd:** `system/leveldb2etcd.py` helper. +* **Reproducibility:** *snapshots are the backup of intent*. Keep snapshot names and publication commands in + version control. + +## 12.6 Disk & performance + +* Disk = pool (dominant) + DB + (public if using `copy` links). Mirrors of full archives are tens to + hundreds of GB per suite/arch; use `-filter`, `-architectures`, skip `-with-sources`, and `mirror update -latest`. +* `download_concurrency` 4–16 for fast links; `downloader: grab` for flaky networks; `download_limit` to be + polite; `-skip-existing-packages` to speed up repeated updates of big mirrors. +* Publishing: `-skip-contents` and `-skip-bz2` drastically reduce time on large repos. +* Many publications sharing one prefix slow down `switch`/`update` (cleanup scans siblings) — prefer distinct + prefixes or `-multi-dist` for big estates. +* LevelDB: `db cleanup` compacts. `aptly db cleanup` on multi-million-package DBs can take minutes; + schedule off-peak. +* Memory: package lists hold all packages of the sources in RAM, so large merges/publishes need + proportionally more memory; a build with `EnableDebug` adds `-memprofile`/`-memstats` flags for + investigation. +* Put `root_dir` on SSD: LevelDB & index generation are IO-bound. + +## 12.7 Monitoring + +* `GET /api/ready`, `GET /api/healthy` (liveness/readiness probes). +* `GET /api/storage` (disk usage of root_dir filesystem) — alert at e.g. 85 %. +* Prometheus: enable `enable_metrics_endpoint`; alert on `aptly_api_http_requests_total{code=~"5.."}` and on + `aptly_repos_package_count` drops. +* Logs: `log_format: json` for machine parsing; task output via `/api/tasks/:id/output`. + +## 12.8 Routine maintenance checklist + +| Frequency | Task | +|-----------|------| +| Each mirror refresh | `mirror update` → `snapshot create` → `snapshot verify` → publish/switch | +| Weekly | `aptly db cleanup` after deleting old snapshots; check `/api/storage` | +| On key rotation | Publish with two `-gpg-key`s, update clients' keyrings, then drop the old key | +| Before upgrades | Back up `db/`; upgrade aptly (DB formats are auto-migrated on read, see §3.3) | +| Quarterly | Test restore procedure; prune obsolete snapshots/publications | + +## 12.9 Troubleshooting + +| Symptom | Likely cause / fix | +|---------|-------------------| +| `unable to reopen the DB, maximum number of retries reached` | Another aptly process (or `api serve` without `-no-lock`) holds the DB. Stop it, or use `-no-lock` / etcd | +| `mirror is locked by update operation, PID N` | Update in progress; if stale (dead PID) it is ignored automatically; use `-force` otherwise | +| `Looks like your keyring with trusted keys is empty` / `Release signature verification failed` | Import archive keys into **`trustedkeys.gpg`** (see chapter 10) or use `-keyring` | +| `unable to figure out list of architectures, please supply explicit list` | Publishing from an empty source: add `-architectures=amd64` | +| `error linking file to …: file already exists and is different` | Different bytes with same pool path in a shared prefix. Use `-multi-dist`, distinct prefixes, or (carefully) `-force-overwrite` | +| `cannot link … from non-local pool` | Azure pool + hardlink/symlink publish endpoint ⇒ set `link_method: copy` | +| `Hash Sum mismatch` on clients after re-publish | Client fetched during the rename window: enable `-acquire-by-hash`; or retry | +| `file … already exists` on `Import` with different size | Corrupt/partial file in the pool; delete the pool file and re-run the update | +| `prefix/distribution already used by another published repo` | Use `publish switch/update` or choose another prefix | +| `unable to publish: … no such file` in pool | Pool file removed manually; `mirror update` re-downloads missing files (it verifies), `repo` packages must be re-added | +| Publishing prompts for a passphrase / hangs (API) | API is batch-only; use gpg-agent or `Signing.PassphraseFile` | +| Huge memory in `snapshot merge/pull` | Load of all packages; use `-latest`/filters, more RAM | +| `aptly` says `Config file not found, creating default config…` | Expected the first run; edit `~/.aptly.conf` | +| Old `.deb` files linger in `public/pool` | `publish update` w/o `-skip-cleanup` removes orphans; drop old publication; hard-linked files persist after `db cleanup` until publications are cleaned | +| `Filter … unsatisfied dependency` (verbose) | Enable `-dep-verbose-resolve`; add source snapshot providing it | + +Add `log_level: debug` (or `-dep-verbose-resolve`) for more diagnostics; S3/GCS/JFrog endpoints have +`debug: true` to dump requests. diff --git a/documentation/13-development.md b/documentation/13-development.md new file mode 100644 index 000000000..950b3b365 --- /dev/null +++ b/documentation/13-development.md @@ -0,0 +1,221 @@ +# 13. Development Guide + +## 13.1 Repository map + +``` +aptly/ interfaces (PackagePool, PublishedStorage, Downloader, Progress, …) +api/ REST handlers + router (gin) + swagger annotations +azure/ gcs/ s3/ swift/ jfrog/ cloud storage implementations +cmd/ CLI commands (commander) +console/ terminal progress +context/ AptlyContext +database/ KV interface + goleveldb + etcddb +deb/ Debian domain logic +docs/ swagger sources & generated docs package +files/ local pool + local published storage +http/ downloaders, compression fallback +pgp/ GPG (external / internal) +query/ query language parser +task/ task queue +utils/ config, checksums, compression, helpers +system/ Python system (functional) test suite +debian/ man/ completion.d/ _man/ systemd/ packaging, docs, completion, activation helper +``` + +Go version: see `go.mod` (`go 1.26`). Key dependencies: `gin-gonic/gin` (HTTP), `smira/commander` + +`smira/flag` (CLI), `syndtr/goleveldb` and `go.etcd.io/etcd/client/v3` (DB), `ugorji/go/codec` (msgpack), +`aws/aws-sdk-go-v2`, `cloud.google.com/go/storage`, `Azure/azure-sdk-for-go` (azblob), `ncw/swift`, `jfrog-client-go` (storage), +`klauspost/compress` + `pgzip` (compression), `rs/zerolog` (logs), `prometheus/client_golang`, +`swaggo/*` (OpenAPI), `awalterschulze/gographviz`, `cheggaaa/pb`, `google/uuid`, `saracen/walker`, +`gopkg.in/check.v1` (gocheck test framework), `gopkg.in/yaml.v3`. + +## 13.2 Building + +```bash +make prepare # go mod verify/tidy + `go generate` (creates the VERSION file embedded in the binary) +make build # swagger + `go build -o build/aptly` +make binaries # cross-built static binaries +make dpkg # Debian packages +make docker-build # everything inside the dev container (system/Dockerfile) +make help # all documented targets +``` + +`VERSION` comes from `make version`: the top `debian/changelog` version, plus a +`+.` suffix for non-tag (CI) builds (`releasetype`). + +## 13.3 Tests + +### Unit tests + +```bash +make test # go test ./... with gocheck; runs under `faketime` +make test TEST=SnapshotSuite # -check.f regex to select suites/tests +make bench # go test ./deb -run=nothing -bench=. -benchmem +``` + +* Written with **gocheck** (`. "gopkg.in/check.v1"`): a suite type + `Test…` methods, hooked with + `func Test(t *testing.T) { TestingT(t) }` in each package's `*_test.go`. +* `faketime` pins the clock (`TEST_FAKETIME`) because some fixtures use expired certificates/keys. +* Fakes: `http.NewFakeDownloader`, `files/mocks.go`, temp directories, `goleveldb.NewOpenDB(tempdir)`, + and a fake S3 (`s3/server_test.go`). +* Tests for `cmd`/`context` are minimal; behaviour is covered by the system tests below. + +### System (functional) tests — `system/` + +Python 3 test-runner exercising the **real binary** through the CLI and the REST API. + +```bash +make system-test # build test binary + run everything (long tests) +make docker-system-test TEST=t04_mirror # one directory +make docker-system-test TEST=UpdateMirror26Test # one test class +make docker-system-test CAPTURE=1 TEST=… # regenerate gold files +COVERAGE_SKIP=1 make system-test # skip coverage plumbing +``` + +Structure: + +* `system/tNN_name/` — a test *package* per feature: `t01_version`, `t02_config`, `t03_help`, + `t04_mirror`, `t05_snapshot`, `t06_publish`, `t07_serve`, `t08_db`, `t09_repo`, `t10_task`, + `t11_package`, `t12_api`, `t13_etcd`, `t14_graph`. Storage back-ends are exercised through helper base classes + (`s3_lib.py`, `swift_lib.py`, `azure_lib.py`, `gcs_lib.py`, `jfrog_lib.py`, `fs_endpoint_lib.py`). +* Each `*.py` defines classes deriving from `BaseTest` (`system/lib.py`) or a specialised base + (`APITest` in `api_lib.py`, `S3Test`, `SwiftTest`, `AzureTest`, `FileSystemEndpointTest`). + Key attributes: `runCmd` (the command line), `expectedCode`, `fixtureDB`/`fixturePool`/`fixtureGpg`/`fixtureWebServer`, + `configOverride`, `longTest`, `requiresGPG1/2`, `requiresDot`, `databaseType` (`leveldb`/`etcd`), `sortOutput`, + `outputMatchPrepare` (regexp normalisers). +* **Gold files** `_gold` hold expected stdout; `*_mirror_show`, etc. hold expected follow-up + command outputs. `CAPTURE=1` rewrites them — review diffs carefully. +* Fixtures come from separate repos (`aptly-fixture-db`, `aptly-fixture-pool`) and `system/files` (test keys, + packages). A local HTTP server (`fixtureWebServer`) hosts fake remote mirrors so no internet is needed. +* The runner builds a **coverage-instrumented test binary** using `main_test.go` (`//go:build testruncli`, + `TestRunMain` calls `cmd.Run` and prints `EXIT: n`), enabling coverage of CLI paths. +* Lint for tests: `make flake8` (`.flake8`). + +### CI (`.github/workflows/ci.yml`, `golangci-lint.yml`) + +Jobs: **Unit Tests** (Docker image, coverage) → **System Test** (Ubuntu runner with +`graphviz gnupg2 gpgv2 faketime python3-…`, flake8, Azurite for Azure, benchmarks) → **Upload Coverage** +(merged, to Codecov via `codecov.yml`) → **Build** matrix producing packages for Debian 11/12/13 and Ubuntu 20.04–26.04 +(artifact upload via `.github/workflows/scripts/upload-artifacts.sh`). `golangci-lint.yml` runs linting separately. + +### Linting + +`make lint` runs golangci-lint (version pinned in the Makefile). Code style: standard `gofmt`; exported +identifiers documented; `// nolint:` markers only where justified. + +## 13.4 Generating API docs + +Handlers carry swag annotations: + +```go +// @Summary Create Repository +// @Description **Create a local repository** … +// @Tags Repos +// @Consume json +// @Param request body repoCreateParams true "Parameters" +// @Produce json +// @Success 201 {object} deb.LocalRepo +// @Failure 400 {object} Error "Bad Request" +// @Router /api/repos [post] +func apiReposCreate(c *gin.Context) { … } +``` + +`make swagger` runs `swag init --propertyStrategy pascalcase --parseDependency --parseInternal +--markdownFiles docs --generalInfo docs/swagger.conf` and regenerates `docs/docs.go`, `swagger.json/yaml`. +The `docs/*.md` files (`Repos.md`, `Snapshots.md`, …) provide the tag descriptions shown in the UI. `docs/index.go` +embeds `docs.html` served by the router. (This `documentation/` folder is intentionally separate and +is **not** consumed by swag.) + +## 13.5 Recipes for common changes + +### Add a new CLI command + +1. Create `cmd/_.go`: + ```go + func aptlyFooBar(cmd *commander.Command, args []string) error { + if len(args) != 1 { cmd.Usage(); return commander.ErrCommandError } + collectionFactory := context.NewCollectionFactory() + … // use deb.* APIs + return nil + } + func makeCmdFooBar() *commander.Command { + c := &commander.Command{Run: aptlyFooBar, UsageLine: "bar ", Short: "…", Long: `…`, + Flag: *flag.NewFlagSet("aptly-foo-bar", flag.ExitOnError)} + c.Flag.Bool("dry-run", false, "…") + return c + } + ``` +2. Register it in the group's `Subcommands` (e.g. `cmd/snapshot.go`) or add a new group in `RootCommand`. +3. Read flags through `context.Flags().Lookup("dry-run").Value.Get().(bool)`; use `LookupOption` when a + config default exists. +4. Update help golds (`system/t03_help`), man page (`make man`), completions (`completion.d/*`), and add + system tests (+ gold files). + +### Add a REST endpoint + +1. Add params struct with `binding:"required"`/`json:` tags and swag comments in `api/.go`. +2. Handler skeleton: bind → shallow validation/404 → `maybeRunTaskInBackground(c, name, resources, func(out, detail){ fresh + factory; LoadComplete; mutate; Update; return &task.ProcessReturnValue{Code, Value}, err })`. + **Choose resource keys** = keys of every object you mutate (`repo.Key()` …); use `task.AllResourcesKey` for + global operations. +3. Register in `api/router.go` (respect canonical lock order in `api/api.go` comments). +4. `make swagger`; add tests in `api/*_test.go` (gin test router + fake context, see `api/api_test.go`) and system + tests in `system/t12_api`. + +### Add a published-storage backend + +1. New package `foo/` with `PublishedStorage` implementing `aptly.PublishedStorage` (all 11 methods; make + `MkDir` a no-op for flat namespaces). Cache remote listings (`pathCache`) if `LinkFromPool` needs MD5 lookups. + Store MD5 as metadata if the store's ETag isn't MD5. +2. Config struct in `utils/config.go` (`FooPublishRoots map[string]FooPublishRoot` + json/yaml tags, initialise the + map in `Config`), documented in `debian/aptly.conf`. +3. Wire in `context.GetPublishedStorage`: `strings.HasPrefix(name, "foo:")` → constructor. +4. Optional: an API listing endpoint (like `api/s3.go`), tests (`foo/public_test.go` with a fake/emulator), and a system + test base class in `system/`. + +### Add a package pool backend + +Implement `aptly.PackagePool` (+ `LocalPackagePool` if files live on a local FS); extend `PackagePoolStorage` (custom +`UnmarshalJSON/YAML` discriminator on `type`) and `context.PackagePool()`. + +### Add a database backend + +Implement `database.Storage` (`Reader/Writer/PrefixReader/Batch/Transaction/CreateTemporary/CompactDB/Drop`), reuse the +contract tests pattern from `database/goleveldb/database_test.go`, and select it in `context._database()` by +`database_backend.type`. Keep keys **byte-ordered prefix scannable** — the entire data model depends on it. + +### Extend the query language + +`query/lex.go` (tokens), `query/syntax.go` (grammar → `deb.PackageQuery`), `deb/query.go` (new `PackageQuery` type +implementing `Matches/Fast/Query/String`). Add unit tests in `query/*_test.go` and `deb/query_test.go`; document in the +website + man page (`docs`). + +### Add a package field to the model + +Fields in `deb.Package` are msgpack-serialised: **adding** exported fields is backward compatible; renaming/removing is +not. Bump handling in `PackageCollection.ByKey` if the format truly changes (see the `0xC1 0x01` marker and `oldPackage`). +Remember `PackageFiles.Hash()` participates in the package key — changing what it hashes changes keys of *all* packages. + +## 13.6 Debugging tips + +* `-dep-verbose-resolve` for dependency resolution; `log_level: debug` for storage back-ends and API. +* Build with `aptly/version.go: EnableDebug = true` to get `-cpuprofile/-memprofile/-memstats` and gin debug mode. +* Inspect the DB: LevelDB keys are readable text prefixes; use a LevelDB tool or write a small Go program with + `goleveldb.NewOpenDB(path)` and `ProcessByPrefix([]byte("S"), …)`. **Stop aptly first** (lock). +* `aptly package show -with-references -with-files 'Pamd64 nginx 1.24.0-1 8f3a91c2'` shows which mirrors/snapshots/repos + reference a package and the pool file paths. +* `aptly graph -output=g.svg` visualises the DAG. +* Reproduce publish output deterministically with `SOURCE_DATE_EPOCH`. +* Use `aptly task run` in system tests when scripting multiple commands against one DB. + +## 13.7 Releasing + +See [`Releasing.md`](../Releasing.md): create `release/1.x.y`, update `debian/changelog`, merge, tag `v` on master +(`git tag -a v$version …`), publish the generated `docs/swagger.json` to the website repository, update the download page and +announce. `make version` derives the version string embedded in the binary. + +## 13.8 Contributing + +Read [`CONTRIBUTING.md`](../CONTRIBUTING.md) and [`CODE_OF_CONDUCT.md`](../CODE_OF_CONDUCT.md). PRs should include unit +and/or system tests (a *failing* system test reproducing a bug is a welcome first PR), pass `make lint`, and update the +man page/docs when behaviour changes. diff --git a/documentation/14-glossary.md b/documentation/14-glossary.md new file mode 100644 index 000000000..061eb45ce --- /dev/null +++ b/documentation/14-glossary.md @@ -0,0 +1,47 @@ +# 14. Glossary + +| Term | Meaning in aptly | +|------|------------------| +| **Acquire-By-Hash** | apt feature: index files fetched from `by-hash//` so they're immutable; enabled with `-acquire-by-hash` | +| **Architecture** | `amd64`, `arm64`, `all` (arch-independent), `source`, … Package files of arch `all` are published into every `binary-` index | +| **Batch** | `database.Batch`: several writes applied atomically | +| **Checksum cache** | DB keys `C` remembering `ChecksumInfo` of pool files | +| **Collection** | Repository object (`RemoteRepoCollection`, `SnapshotCollection`, …) providing CRUD over one entity kind; obtained via `CollectionFactory` | +| **Component** | Subdivision of a distribution (`main`, `contrib`, `non-free`, …). Each published component has its own source | +| **Contents index** | `Contents-[.gz]` mapping file paths to packages (for `apt-file`) | +| **Control file / Stanza** | RFC-822-like `Field: value` block describing a package/release (`deb.Stanza`, `ControlFileReader`) | +| **Dangling reference** | Ref in a ref list whose package record no longer exists | +| **Distribution** | Release name (`bookworm`, `stable`); directory `dists/` | +| **Endpoint** | Named published-storage target from the config (`s3:prod`, `filesystem:web1`) | +| **Flat repository** | Archive without `dists/`/components: `Packages` next to the debs; distribution written `./` or `sub/` | +| **Installer package** | Synthetic package representing debian-installer images in a mirror (`-with-installer`) | +| **InRelease / Release.gpg** | Clearsigned / detached-signed forms of `Release` | +| **Key (package key)** | `P `; unique identity of a package record | +| **Legacy pool** | Pre-1.1 MD5-based pool layout, read-only support | +| **Local repository** | User-managed list of packages (`deb.LocalRepo`) | +| **Mirror** | Configuration + snapshot-in-time list of a remote archive (`deb.RemoteRepo`) | +| **MultiDist** | Publication mode with per-distribution pool directories | +| **Offloaded fields** | Package data (files, deps, extra, contents) stored in separate keys `xF/xD/xE/xC` | +| **Package pool** | Content-addressed store for package files | +| **PackageList** | In-memory set of `Package`s, indexable and searchable | +| **PackageRefList** | Sorted list of package keys; the "contents" of mirror/repo/snapshot | +| **Prefix** | Path under a published storage where a publication lives (`.` = root) | +| **Progress** | Abstraction for progress bars and messages (`console.Progress`, `task.Output`) | +| **Publication / Published repo** | Binding (storage, prefix, distribution) → sources per component, plus generated files | +| **Pull** | Copy packages (+dependencies) from one snapshot into another, creating a new snapshot | +| **Query** | Package selection expression (see chapter 11) | +| **Resource (task)** | Key naming an object a task needs exclusively (`L`, `__all__`, …) | +| **Revision (publish)** | Staged, not yet applied set of component→source changes on a publication | +| **rePublishing** | Internal flag: regeneration of an existing publication (uses `.tmp` names + rename) | +| **rootDir** | Base directory for db, pool, public, upload, skel | +| **Skeleton files** | Extra files under `rootDir/skel//dists///` copied into the published component | +| **Snapshot** | Immutable ref list with provenance; source for publishing | +| **Source package** | Debian source (`.dsc` + tarballs); architecture `source` | +| **Stanza** | See *Control file* | +| **Switch** | Re-point a snapshot publication to different snapshots and regenerate | +| **Task** | Unit of work in the API's task list, with state, output and resource locks | +| **trustedkeys.gpg** | Keyring aptly's external verifier (`gpgv`) uses for remote repositories | +| **udeb** | Micro-package used by debian-installer (`-with-udebs`) | +| **Uploaders** | ACL (`uploaders.json`) governing which signing keys may `repo include` which packages | +| **Upload directory** | `rootDir/upload/`: staging area for API uploads | +| **UUID** | Random identifier of a persisted entity; basis for DB keys | diff --git a/documentation/README.md b/documentation/README.md new file mode 100644 index 000000000..c944c3b04 --- /dev/null +++ b/documentation/README.md @@ -0,0 +1,48 @@ +# aptly Documentation + +Complete, code-level documentation of **aptly**, the Debian repository management tool. +It explains what the tool does, how it is built, how every subsystem works internally, +and how to run, extend and test it. + +> These documents describe the code in this repository. File and function names are given +> so you can jump straight to the source. Statements were verified against the code at the +> time of writing; when in doubt, the source is authoritative. + +## Reading order + +| # | Document | Read it to learn... | +|---|----------|---------------------| +| 1 | [Overview & Concepts](01-overview-and-concepts.md) | What aptly is, the object model (mirror / local repo / snapshot / published repo), typical workflows | +| 2 | [Architecture](02-architecture.md) | Layers, Go packages, dependency graph, runtime modes, key design decisions | +| 3 | [Data Model & Database](03-data-model-and-database.md) | Key/value layout, encodings, `PackageRefList`, collections, DB backends | +| 4 | [Package Pool & Published Storage](04-storage.md) | Content-addressed pool, publishing back-ends (filesystem, S3, GCS, Azure, Swift, JFrog) | +| 5 | [Core Workflows](05-workflows.md) | Step-by-step internals: mirror update, snapshot, merge/pull, add/include, cleanup | +| 6 | [Publishing Internals](06-publishing.md) | How `Packages`, `Release`, `Contents`, by-hash files and signatures are generated | +| 7 | [REST API & Task System](07-rest-api-and-tasks.md) | Router, sync/async execution, resource locking, endpoint catalogue | +| 8 | [CLI & Context](08-cli-and-context.md) | Command tree, `AptlyContext`, lifecycle of a command, DB open/close | +| 9 | [Configuration Reference](09-configuration.md) | Every config key, defaults, examples for all storage back-ends | +| 10 | [Signing & Verification](10-signing-and-verification.md) | GPG providers, Release signing, mirror signature verification | +| 11 | [Queries, Dependencies & Versions](11-queries-and-dependencies.md) | Query language grammar, dependency resolution, Debian version comparison | +| 12 | [Operations Guide](12-operations.md) | Deployment, locking, backups, recovery, performance, troubleshooting | +| 13 | [Development Guide](13-development.md) | Building, unit & system tests, CI, how to add commands / endpoints / back-ends | +| 14 | [Glossary](14-glossary.md) | Terms used throughout | + +## One-paragraph summary + +aptly keeps **metadata** (which packages exist, which repository/snapshot contains which +package) in an embedded key/value database and **package files** in a de-duplicated, +content-addressed *pool* on disk (or Azure Blob storage). Everything you can do — mirror a +remote archive, build a local repository from `.deb`/`.dsc`/`.changes` files, freeze state as +an immutable *snapshot*, filter/merge/pull between snapshots, and *publish* a snapshot as a +signed, apt-consumable repository to a directory, S3, GCS, Azure, Swift or Artifactory — is a +manipulation of lists of package *references* plus, at publish time, a generation of +`Packages`/`Release` index files and links/copies of pool files. It is driven either by the +`aptly` CLI or by the same logic exposed through a REST API served by `aptly api serve`. + +## Other documentation in this repository + +* [`docs/`](../docs) — API reference fragments used to generate the Swagger/OpenAPI spec + (`make swagger`) and the embedded `/docs` page. +* [`files/README.md`](../files/README.md) — details of the on-disk pool layout. +* `man/aptly.1` — man page generated from the CLI help (`_man/gen.go`). +* Upstream user docs: .