diff --git a/README.rst b/README.rst index 90c911522..f401eae0c 100644 --- a/README.rst +++ b/README.rst @@ -21,6 +21,9 @@ Aptly is a swiss army knife for Debian repository management. Documentation is available at `http://www.aptly.info/ `_. For support please use open `issues `_ or `discussions `_. +In-depth documentation of the architecture, data model, workflows and internals lives in the +`documentation/ `_ directory. + Aptly features: * make mirrors of remote Debian/Ubuntu repositories, limiting by components/architectures diff --git a/documentation/01-overview-and-concepts.md b/documentation/01-overview-and-concepts.md new file mode 100644 index 000000000..0f26587c1 --- /dev/null +++ b/documentation/01-overview-and-concepts.md @@ -0,0 +1,140 @@ +# 1. Overview & Concepts + +## 1.1 What problem aptly solves + +`apt` clients consume *repositories*: a directory tree containing `dists//…` +index files (`Release`, `Packages`, `Sources`, `Contents`) and a `pool/` of package files. +Running one of those trees in production raises hard questions: + +* How do I get a **partial copy** of `archive.ubuntu.com` (only some components/architectures, + only some packages and their dependencies)? +* How do I make installation **repeatable** — i.e. freeze "what the archive looked like on + Tuesday" and ship exactly that to 500 machines? +* How do I **publish my own packages** as a proper, signed repository? +* How do I **promote** a tested set of packages from *staging* to *production*, atomically, + with the ability to roll back? + +aptly answers with a small set of composable objects and operations. Its stated goal (from the +CLI help in `cmd/cmd.go`) is "to establish repeatability and controlled changes in a +package-centric environment". + +## 1.2 The object model + +``` + (remote archive on the internet) + │ aptly mirror create / update + ▼ + ┌───────────────┐ snapshot ┌────────────┐ filter / merge / pull ┌────────────┐ + │ Mirror │ ──────────────► │ Snapshot │ ──────────────────────► │ Snapshot │ + │ (RemoteRepo) │ │ (immutable)│ │ (immutable)│ + └───────────────┘ └─────┬──────┘ └─────┬──────┘ + │ │ + ┌───────────────┐ snapshot │ aptly publish snapshot │ + │ Local repo │ ──────────────────────┘ │ │ + │ (LocalRepo) │◄── aptly repo add / include / import ▼ ▼ + └───────┬───────┘ ┌────────────────────────────────┐ + │ aptly publish repo (direct) │ Published repository │ + └──────────────────────────────────────────► │ (PublishedRepo) │ + │ = dists/ + pool/ on a storage │ + └────────────────────────────────┘ +``` + +| Object | Go type | Mutable? | What it holds | +|--------|---------|----------|---------------| +| **Package** | `deb.Package` | no | One `.deb`/`.udeb`/source package: name, version, architecture, dependencies, extra control fields, list of files with checksums | +| **Mirror** | `deb.RemoteRepo` | yes (via `update`) | Configuration of a remote archive (URL, distribution, components, architectures, filter) + the list of packages found at last update | +| **Local repository** | `deb.LocalRepo` | yes | A user-managed list of packages (added from files, or copied from other places) | +| **Snapshot** | `deb.Snapshot` | **no** | An immutable list of packages + provenance ("created from mirror X", "merge of A and B", …) | +| **Published repository** | `deb.PublishedRepo` | re-publishable | A binding *(storage, prefix, distribution)* → one source per *component* (snapshot or local repo), plus the generated files | +| **Package pool** | `aptly.PackagePool` | append/GC | De-duplicated storage of the actual package files | +| **Published storage** | `aptly.PublishedStorage` | | Where the published tree lives (directory, S3, GCS, Azure, Swift, JFrog) | + +Two crucial facts drive everything else: + +1. **Repositories, mirrors and snapshots do not contain packages — they contain *references* + to packages.** A reference is a short byte string (`"Pamd64 nginx 1.24.0-1 8f3a91c2"`); + the package record itself lives once in the database. Creating a snapshot of a + 100,000-package mirror therefore copies a sorted list of 100,000 short keys, not + 100,000 packages (see [chapter 3](03-data-model-and-database.md)). +2. **Package *files* are stored once**, in a pool addressed by SHA-256, no matter how many + mirrors/snapshots/publications use them (see [chapter 4](04-storage.md)). + +## 1.3 Life cycle of a package + +1. **Discovered.** Either parsed from a remote `Packages`/`Sources` index (mirror update) or + read from a `.deb`/`.dsc` on disk (`repo add`, `repo include`). +2. **Registered.** A `deb.Package` is written to the database under a key derived from + *(architecture, name, version, hash-of-files)*. Large/rarely used fields (file list, + dependency lists, extra control fields) are stored under separate "offload" keys. +3. **Downloaded/imported.** The package file(s) are downloaded (mirror) or copied (local repo) + into the pool at a path derived from the file's SHA-256. +4. **Referenced.** The package key is inserted into a `PackageRefList` belonging to a mirror, + local repo or snapshot. +5. **Published.** When a repository containing it is published, the file is hard-linked + (or symlinked/copied/uploaded) from the pool into `pool////…` on the + published storage, and a stanza describing it is written to `Packages`. +6. **Garbage collected.** After all references disappear (mirror dropped, snapshot deleted, + …) `aptly db cleanup` removes the package record and — if no other package references it — + the pool file. + +## 1.4 Typical workflows + +### A. Mirror upstream, freeze, publish + +```bash +aptly mirror create -architectures=amd64 -filter='nginx | curl' -filter-with-deps \ + bookworm-main http://deb.debian.org/debian bookworm main +aptly mirror update bookworm-main # download indexes + packages +aptly snapshot create bookworm-2026-09-30 from mirror bookworm-main +aptly publish snapshot -distribution=bookworm bookworm-2026-09-30 debian +# clients: deb http://host/debian bookworm main +``` + +### B. Publish your own packages + +```bash +aptly repo create -distribution=stable -component=main myrepo +aptly repo add myrepo ./build/*.deb # or: aptly repo include ./build/x.changes +aptly publish repo myrepo internal # publishes directly from the local repo +aptly repo add myrepo ./build/newer.deb +aptly publish update stable internal # re-publish (only changed state) +``` + +### C. Controlled upgrade (`pull`) and atomic switch + +```bash +# take one package (+dependencies) from a newer snapshot into the current one +aptly snapshot pull -no-remove prod-2026-09-01 bookworm-2026-09-30 prod-2026-10-01 nginx +aptly publish switch bookworm prod-2026-10-01 debian # atomically re-point the publication +# rollback: aptly publish switch bookworm prod-2026-09-01 debian +``` + +### D. Multi-component / multi-source publishing + +```bash +aptly publish snapshot -component=main,contrib -distribution=stable snap-main snap-contrib +``` + +Each component of a published repository is fed by its *own* source (snapshot or local repo). +All components of one publication must have the same source kind. + +### E. REST-driven automation (CI/CD) + +```bash +curl -F file=@app_1.0_amd64.deb http://aptly:8080/api/files/ci-upload +curl -X POST http://aptly:8080/api/repos/myrepo/file/ci-upload +curl -X PUT -H 'Content-Type: application/json' \ + -d '{"Signing":{"Passphrase":"…"}}' \ + http://aptly:8080/api/publish/internal/stable +``` + +## 1.5 What aptly deliberately does *not* do + +* It is **not a package build system** (it only ingests finished `.deb`/`.dsc`/`.changes`). +* It is **not a long-running daemon by default**: the CLI is a batch tool; `aptly api serve` + is the (optional) server mode. +* It does **not** support RPM/YUM repositories (the `aptly` Go package comment mentions + "doesn't depend directly on Debian or CentOS", but only Debian formats are implemented). +* It is **not multi-writer**: with the default LevelDB backend only one process may open the + database at a time (see [Operations](12-operations.md)). The `etcd` backend and the + server's `-no-lock` mode relax this. diff --git a/documentation/02-architecture.md b/documentation/02-architecture.md new file mode 100644 index 000000000..cc9435d4b --- /dev/null +++ b/documentation/02-architecture.md @@ -0,0 +1,246 @@ +# 2. Architecture + +## 2.1 System context + +```mermaid +flowchart LR + subgraph Operators + CLI["aptly CLI
(cmd/)"] + HTTPC["REST clients
curl, CI, dashboards"] + end + + subgraph aptly process + CMD["cmd: commander tree"] + API["api: gin router + handlers"] + CTX["context.AptlyContext
(wiring / lifecycle)"] + DEB["deb: domain model & algorithms"] + TASK["task: resource-aware
task queue"] + end + + subgraph Local state under rootDir + DB[("db/ LevelDB
(or etcd)")] + POOL[("pool/
package files")] + UP[("upload/")] + PUB[("public/")] + end + + UPSTREAM["Upstream Debian/Ubuntu
archives (HTTP/HTTPS/FTP)"] + STORES["Published storage:
S3 · GCS · Azure · Swift · JFrog · FS"] + GPG["GnuPG / internal PGP"] + APT["apt clients"] + + CLI --> CMD --> CTX + HTTPC --> API --> CTX + API --> TASK + CTX --> DEB + DEB --> DB + DEB --> POOL + DEB --> UPSTREAM + DEB --> STORES + DEB --> GPG + API --> UP + PUB --> APT + STORES --> APT +``` + +## 2.2 Layered view + +aptly is a single Go module (`github.com/aptly-dev/aptly`, `go 1.26`) producing a single +static binary. Its packages form clean layers; arrows below point from a package to what it +depends on. + +``` +┌───────────────────────────────────────────────────────────────────────────┐ +│ main.go (embeds VERSION and debian/aptly.conf, calls cmd.Run) │ +├───────────────────────────────────────────────────────────────────────────┤ +│ cmd/ CLI commands (commander) api/ REST handlers (gin) │ +├───────────────────────────────────────────────────────────────────────────┤ +│ context/ AptlyContext: config, DB, pool, storages, downloader, signer │ +│ task/ task list, resource locking, task output │ +├───────────────────────────────────────────────────────────────────────────┤ +│ deb/ domain: Package, PackageList, PackageRefList, RemoteRepo, │ +│ LocalRepo, Snapshot, PublishedRepo, collections, index files, │ +│ Changes, query engine, dependency & version logic │ +│ query/ parser for the package query language → deb.PackageQuery │ +├───────────────────────────────────────────────────────────────────────────┤ +│ aptly/ interfaces: PackagePool, PublishedStorage, Downloader, Progress, │ +│ ChecksumStorage, PublishedStorageProvider │ +├───────────────────────────────────────────────────────────────────────────┤ +│ Implementations of the interfaces │ +│ database/{goleveldb,etcddb} files/ (fs pool + fs publish) │ +│ s3/ gcs/ azure/ swift/ jfrog/ (publish; azure also pool) │ +│ http/ (downloader) pgp/ (signer/verifier) console/ (progress) │ +├───────────────────────────────────────────────────────────────────────────┤ +│ utils/ config, checksums, compression, logging, small helpers │ +│ systemd/activation socket activation for `api serve` │ +└───────────────────────────────────────────────────────────────────────────┘ +``` + +### Package responsibilities + +| Package | Key files | Responsibility | +|---------|-----------|----------------| +| `main` | `main.go` | Embeds `VERSION` and `debian/aptly.conf` (default config template); `os.Exit(cmd.Run(...))` | +| `cmd` | `cmd.go`, `run.go`, `mirror_*.go`, `repo_*.go`, `snapshot_*.go`, `publish_*.go`, `db_*.go`, `serve.go`, `api_serve.go`, `task_run.go`, `graph.go` … | One file per (sub)command; builds the `commander` tree; parses flags; calls into `deb`/`context` | +| `context` | `context.go` | `AptlyContext`: lazy singletons for config, DB, pool, published storages, downloader, progress, task list, GPG signer/verifier; signal handling; shutdown | +| `api` | `router.go`, `api.go`, `repos.go`, `mirror.go`, `snapshot.go`, `publish.go`, `files.go`, `packages.go`, `db.go`, `task.go`, `gpg.go`, `graph.go`, `s3.go`, `gcs.go`, `jfrog.go`, `storage.go`, `metrics.go`, `middleware.go`, `error.go` | HTTP handlers, request structs (with swagger annotations), sync/async task wrappers | +| `task` | `list.go`, `task.go`, `resources.go`, `output.go` | Background task queue with **resource-based mutual exclusion** | +| `deb` | see §2.4 | Everything Debian-specific and all repository logic | +| `query` | `lex.go`, `syntax.go`, `query.go` | Lexer and recursive-descent parser for query strings | +| `aptly` | `interfaces.go`, `conf.go`, `const.go`, `report.go`, `version.go` | Backend-neutral interfaces & globals (`Version`, `AptlyConf`, `EnableDebug`) | +| `database` | `database.go` + `goleveldb/`, `etcddb/` | KV `Storage` abstraction; two implementations | +| `files` | `package_pool.go`, `public.go` | Local filesystem pool + local filesystem published storage | +| `s3`, `gcs`, `azure`, `swift`, `jfrog` | `public.go`, plus `package_pool.go` in `azure` | Cloud publish targets; Azure can also host the pool | +| `http` | `download.go`, `grab.go`, `compression.go`, `temp.go`, `gcp_auth.go`, `fake.go` | Downloaders (default & grab), "try .xz/.bz2/.gz" logic, `ar+https` GCP Artifact Registry auth | +| `pgp` | `gnupg.go`, `internal.go`, `openpgp.go`, `gnupg_finder.go` | `Signer`/`Verifier` implemented via external `gpg` or pure Go | +| `console` | `progress.go` | Terminal `aptly.Progress` with bars and coloured output | +| `utils` | `config.go`, `checksum.go`, `compress.go`, `copyfile.go`, `logging.go`, `human.go`, `list.go`, `utils.go` | Shared helpers | +| `systemd/activation` | `listeners.go`, `files.go`, `packetconns.go` | Socket-activation helper used by `aptly api serve` | +| `docs` | `index.go`, `*.md`, generated `docs.go`/`swagger.*` | Swagger sources and embedded docs page | + +## 2.3 Dependency inversion: the `aptly` interfaces + +The `aptly` package (`aptly/interfaces.go`) is the seam that keeps the domain code +storage-agnostic. `deb` depends **only** on these interfaces: + +```go +type PackagePool interface { // where package *files* live + Verify(...) ; Import(...) ; LegacyPath(...) ; Size(...) ; Open(...) + FilepathList(...) ; Remove(...) +} +type LocalPackagePool interface { // optional capability: pool is on a local FS + Stat ; GenerateTempPath ; Link ; Symlink ; FullPath +} +type PublishedStorage interface { // where the *published tree* lives + MkDir ; PutFile ; RemoveDirs ; Remove ; LinkFromPool ; Filelist + RenameFile ; SymLink ; HardLink ; FileExists ; ReadLink +} +type PublishedStorageProvider interface { GetPublishedStorage(name string) (PublishedStorage, error) } +type Downloader interface { Download ; DownloadWithChecksum ; GetProgress ; GetLength } +type Progress interface { … bars + Printf/ColoredPrintf … } +type ChecksumStorage interface { Get(path) ; Update(path, *ChecksumInfo) } +``` + +Consequences: + +* Adding a new publish target means implementing `PublishedStorage` (see + [Development](13-development.md#add-a-published-storage-backend)). +* The Azure pool implements `PackagePool` but **not** `LocalPackagePool`, so hard-linking or + symlinking pool files into a published tree is impossible; `LinkFromPool` then must + copy/upload (`files.PublishedStorage.LinkFromPool` errors with *"cannot link … from + non-local pool"* unless `link_method: copy`). +* `Progress` lets the same algorithm run with a terminal progress bar (CLI) or a buffered + text log (`task.Output`, REST) — see [chapter 7](07-rest-api-and-tasks.md). +* `AptlyContext` itself implements `PublishedStorageProvider`, so `deb.PublishedRepo.Publish` + can lazily obtain the storage named in its `Storage` field (`""`, `filesystem:x`, + `s3:x`, `gcs:x`, `swift:x`, `azure:x`, `jfrog:x`). + +## 2.4 The `deb` package (the heart) + +| Concept | File(s) | Notes | +|---------|---------|-------| +| Parsing of RFC-822-like control data | `format.go` | `Stanza`, `ControlFileReader`; canonical field ordering when writing | +| `Package` | `package.go`, `package_files.go`, `package_deps.go`, `package_collection.go` | In-memory model with lazy "offloaded" fields; `PackageCollection` persists them | +| `PackageList` | `list.go` | In-memory set of packages, indexed for search, dependency verification, filter | +| `PackageRefList` | `reflist.go` | Sorted `[][]byte` of package keys; Diff/Merge/Subtract; msgpack-encoded | +| Mirror | `remote.go`, `ppa.go` | `RemoteRepo` + `RemoteRepoCollection`; index download & parsing; `ppa:` URL expansion | +| Local repo | `local.go`, `import.go`, `changes.go`, `uploaders.go` | `LocalRepo`; file collection/import; `.changes` handling; uploader ACL rules | +| Snapshot | `snapshot.go` | `Snapshot` + `SnapshotCollection` | +| Publication | `publish.go`, `index_files.go`, `contents.go` | `PublishedRepo` and `Publish()`; generation/signing/atomic rename of index files; contents index | +| Query engine | `query.go` | `PackageQuery` implementations (`AndQuery`, `OrQuery`, `NotQuery`, `FieldQuery`, `DependencyQuery`, `PkgQuery`, `MatchAllQuery`) | +| Version/deps | `version.go`, `package_deps.go` | Debian version comparison, dependency parsing | +| Factory | `collections.go` | `CollectionFactory` hands out one collection per type over a shared DB | +| Misc | `graph.go`, `find_dangling.go`, `checksum_collection.go`, `debian.go`, `deb.go` | Graphviz export, dangling-ref detection, checksum cache, `.deb`/`.dsc` readers | + +### Collections pattern + +Each persistent type has a `*Collection` (e.g. `SnapshotCollection`) taking a +`database.Storage`. A collection offers `Add`, `Update`, `Drop`, `ByName`, `ByUUID`, +`ForEach`, `Len`, and — importantly — `LoadComplete`. Objects are loaded **shallow** +(metadata only, cheap) and their `PackageRefList` is fetched from a separate key only when +`LoadComplete` is called. This keeps `aptly snapshot list` fast even with huge snapshots. + +`CollectionFactory` (`deb/collections.go`) is created once per command/request and caches +one instance of each collection, guarded by a mutex. `Flush()` drops them to reclaim memory. + +> **Lock order.** `api/api.go` documents the canonical acquisition order for collections to +> avoid deadlocks: `RemoteRepoCollection → LocalRepoCollection → SnapshotCollection → +> PublishedRepoCollection`. + +## 2.5 Runtime modes + +| Mode | Entry | Process model | DB handling | +|------|-------|---------------|-------------| +| **CLI batch** | `aptly …` | One process per command | Opens LevelDB (exclusive file lock) on first use; closes on exit. Long operations (mirror update) *close* the DB during downloading so other commands can run | +| **API server** | `aptly api serve -listen=:8080` | Long-lived HTTP server (gin); mutating requests run as tasks | DB stays open for the life of the server. With `-no-lock` the server closes the DB whenever there are zero in-flight requests | +| **Static file server** | `aptly serve -listen=:8080` | Long-lived; serves `rootDir/public` (and prints recommended `sources.list` lines) | Read-only lookup of published repos at start | +| **Task script** | `aptly task run -filename=cmds.txt` or stdin | Runs many aptly commands in a single process (one DB open) | Avoids repeated open/close cost | +| **Systemd-managed API** | `debian/aptly-api.service` | Runs `aptly api serve -config=/etc/aptly.conf … -listen=…` as user `aptly-api`. `aptly api serve` can also adopt a single listener fd passed by systemd socket activation (`systemd/activation`) | Same as API server | + +## 2.6 Concurrency model + +* **CLI**: essentially single-threaded orchestration; parallelism is confined to + * package downloading (`download_concurrency` goroutines in `cmd/mirror_update.go`), + * filesystem walking (`saracen/walker`), + * compression (`klauspost/pgzip`). +* **API**: gin serves each HTTP request in its own goroutine, but every operation that + mutates state is wrapped in a **task** (`task.List`). Tasks declare the *resources* they + need (e.g. the key of a local repo `"L"`); the list guarantees no two tasks with + intersecting resources run concurrently, and special keys + `__all__` / `__alllocalrepos__` express "everything"/"all local repos". See + [chapter 7](07-rest-api-and-tasks.md#73-the-task-system). +* **Database**: LevelDB is safe for concurrent goroutines inside one process. Cross-process + safety is provided by the LevelDB file lock ("resource temporarily unavailable" ⇒ retry with + jittered ~10 s sleeps up to `database_open_attempts`). +* **Mirror update guard**: a `RemoteRepo` records `Status` and `WorkerPID` in the DB; + `CheckLock()` refuses a second update when that PID is alive (override: `-force`). + +## 2.7 Cross-cutting concerns + +| Concern | Where | Behaviour | +|---------|-------|-----------| +| Configuration | `utils/config.go`, `context.config()` | YAML or JSON (with comments). Search order: `-config` flag, `~/.aptly.conf`, `/usr/local/etc/aptly.conf`, `/etc/aptly.conf`. If none found, a default (the embedded `debian/aptly.conf`) is written to `~/.aptly.conf` | +| Logging | `utils/logging.go` (zerolog) | `log_level`, `log_format: default|json`; JSON mode also makes gin log JSON (`api.JSONLogger`) | +| Error handling (CLI) | `context.Fatal` | Commands return `error`; `cmd.Run` converts to exit code 1 (2 for usage errors) via a panic/recover on `FatalError` | +| Error handling (API) | `api.AbortWithJSONError` | Sets JSON content type and aborts with status; gin's `ErrorLogger` middleware renders `{"error": "..."}` | +| Metrics | `api/metrics.go` | Prometheus counters/gauges/histograms (opt-in `enable_metrics_endpoint`) incl. per-repo package counts | +| Signals | `context.GoContextHandleSignals`, `cmd/api_serve.go` | First `^C` cancels the Go context (downloads stop, cleanup runs); second `^C` aborts immediately. API server waits for running tasks before shutdown | +| Reproducible builds | `deb/publish.go` | `SOURCE_DATE_EPOCH` env var overrides `Date:` in the top-level `Release` file | +| Debug tooling | `aptly/version.go` (`const EnableDebug`) | Compile-time constant (default `false`); when set to `true` at build time it adds `-cpuprofile`, `-memprofile`, `-memstats`/`-meminterval` flags and gin debug mode | + +## 2.8 Key design decisions and their rationale + +1. **References, not copies.** `PackageRefList` (sorted list of package keys) makes + snapshot/merge/diff operations linear merges over sorted byte slices and makes snapshots + nearly free. The price: deleting data needs a mark-and-sweep GC (`db cleanup`). +2. **Content-addressed pool.** Files are keyed by SHA-256, so identical files from different + mirrors/versions are stored once, and integrity can be re-verified cheaply. A MD5-keyed + *legacy* layout (aptly ≤ 1.0) is still read (unless `skip_legacy_pool`). +3. **Package key includes a hash of the file set** (`FilesHash`). Two "same name+version" + packages with different bytes (e.g. rebuilt without version bump) become *distinct keys*, + so a snapshot can never silently change content. +4. **Offloaded package fields.** The base `Package` record is small; `Files`, `Depends…` and + `Extra` are separate keys (`xF`, `xD`, `xE` prefixes). Listing, diffing and merging rarely + need them, so they are loaded lazily. +5. **Publishing writes to temp names and renames.** Index files are generated in a temp dir, + uploaded as `*.tmp` when re-publishing, then renamed — minimising the time clients could + see a half-written state (see [chapter 6](06-publishing.md)). +6. **One codebase, two front-ends.** CLI commands and REST handlers are separate thin + layers over the same `deb` operations; the REST layer adds tasks, JSON and locking. +7. **Interfaces at every I/O boundary** (DB, pool, published storage, downloader, signer, + progress) — unit tests inject fakes (`http/fake.go`, `files/mocks.go`, `console`), and new + back-ends are additive. +8. **Explicit DB open/close.** The CLI can release the LevelDB lock during long network + phases (`context.CloseDatabase()` / `ReOpenDatabase()`), which is why a mirror update + collects all download tasks first, closes the DB, downloads, reopens, then imports. + +## 2.9 Build & release artefacts + +* `go build` → `aptly` binary. `main.go` embeds `VERSION` (generated by `go generate` from + `make version`, based on `debian/changelog` + git commit for CI builds) and + `debian/aptly.conf`. +* Debian packaging (`debian/`): packages `aptly` (binary, man page, bash completion), + `aptly-api` (systemd unit + `/etc/aptly.conf` handling) and `aptly-dbg`. +* `completion.d/` — bash and zsh completions; `man/` + `_man/gen.go` — man page generation from + command help text. +* `docs/` + `make swagger` — OpenAPI generation via `swag` from handler annotations. diff --git a/documentation/03-data-model-and-database.md b/documentation/03-data-model-and-database.md new file mode 100644 index 000000000..74ae58457 --- /dev/null +++ b/documentation/03-data-model-and-database.md @@ -0,0 +1,279 @@ +# 3. Data Model & Database + +aptly's entire *metadata* state lives in an ordered key/value store. Package *files* do not +(see [chapter 4](04-storage.md)). This chapter describes the key space, value encodings, the +in-memory model built from them, and the storage back-ends. + +## 3.1 The KV abstraction (`database/database.go`) + +```go +type Storage interface { + Reader // Get(key) ([]byte, error) → database.ErrNotFound + Writer // Put(key, value) ; Delete(key) + PrefixReader // HasPrefix ; ProcessByPrefix ; KeysByPrefix ; FetchByPrefix + CreateBatch() Batch // buffered writes, applied by Write() + OpenTransaction() (Transaction, error) // Commit() / Discard() + CreateTemporary() (Storage, error) // scratch DB of the same type + Open() ; Close() ; CompactDB() ; Drop() +} +``` + +Everything else in aptly is expressed using only these operations, notably **prefix scans** +(all snapshots = every key starting with `S`). + +### Back-ends + +| Backend | Package | Selected by | Notes | +|---------|---------|-------------|-------| +| **LevelDB** (default) | `database/goleveldb` | `database_backend.type: leveldb` (or unset) | Embedded, single-process (exclusive file lock), bloom filter (10 bits), 256 cached open files. Path: `rootDir/db` or `database_backend.db_path`. `Put` skips writing if the value is unchanged. Temporary DBs are created in the OS temp dir with throttled compaction settings (`CompactionL0Trigger=32`, `WriteL0Pause=96`, `WriteL0Slowdown=64`) | +| **etcd** | `database/etcddb` | `database_backend.type: etcd`, `url: host:2379` | Network KV; allows several aptly processes to share state. Message limits raised to ~2 GiB, 30 s dial timeout. Temporary DBs are emulated by a UUID key prefix in the same cluster | + +`aptly db recover` (`cmd/db_recover.go`) runs `goleveldb.RecoverDB` to rebuild a corrupted +LevelDB. `system/leveldb2etcd.py` migrates LevelDB → etcd. + +## 3.2 Key space + +| First byte(s) | Meaning | Key format | Value | +|---------------|---------|-----------|-------| +| `P` | **Package** (core record) | `P ` e.g. `Pamd64 nginx 1.24.0-1 8f3a91c2` | `0xC1 0x01` marker + msgpack(`Package` public fields) | +| `xF` + package key | Package **files** (offloaded) | `xF` + full package key | msgpack(`PackageFiles`) | +| `xD` + package key | Package **dependencies** (offloaded) | `xD` + key | msgpack(`PackageDependencies`: Depends, PreDepends, Suggests, Recommends, Build-Depends…) | +| `xE` + package key | Package **extra** control fields (offloaded) | `xE` + key | msgpack(`Stanza`, i.e. `map[string]string`) | +| `xC` + package key | Cached **contents** (list of file paths in the `.deb`) | `xC` + key | msgpack(`[]string`); computed lazily on first publish with Contents enabled | +| `R` | **Remote repo** (mirror) | `R` | msgpack(`RemoteRepo`) — metadata only | +| `L` | **Local repo** | `L` | msgpack(`LocalRepo`) | +| `S` | **Snapshot** | `S` | msgpack(`Snapshot`) | +| `U` | **Published repo** | `U>>` (`StoragePrefix()` + `">>"` + `Distribution`) | msgpack(`PublishedRepo`) | +| `E` | **Package ref list** for a mirror/local repo/snapshot | `E` | msgpack(`PackageRefList{Refs [][]byte}`) | +| `E` (published) | Ref list *frozen at publish time* for a **local-repo** publication | `E` | msgpack(`PackageRefList`) | +| `C` | **Checksum cache** for pool files | `C` | msgpack(`utils.ChecksumInfo`) | + +Notes: + +* UUIDs are random v4 (`google/uuid`), assigned at creation; **names are only unique + indexes** enforced by the collections (renames just rewrite the metadata record). +* Because a ref list key is `E` and each mirror/local repo/snapshot has its own UUID, + the three never collide. Published local-repo ref lists append the component name to + the published repo's own UUID. +* The `P` scan is what makes "all packages in the DB" cheap: `PackageCollection.AllPackageRefs()` + is simply `KeysByPrefix("P")` — the very first byte deliberately excludes `xF/xD/xE/xC` + keys (they start with `x`). +* Published repos are **all loaded into memory** on first use (`PublishedRepoCollection.loadList`) + because they are few and lookups by *(storage, prefix, distribution)* / by source are + common. + +### Entity relationships + +```mermaid +erDiagram + PACKAGE ||--o{ PACKAGE_FILE : "has (xF)" + PACKAGE_FILE }o--|| POOL_FILE : "PoolPath (sha256 addressed)" + REMOTE_REPO ||--|| REFLIST : "E" + LOCAL_REPO ||--|| REFLIST : "E" + SNAPSHOT ||--|| REFLIST : "E" + REFLIST }o--o{ PACKAGE : "refs (P keys)" + SNAPSHOT }o--o{ SNAPSHOT : "SourceIDs (merge/pull/filter)" + SNAPSHOT }o--o| REMOTE_REPO : "SourceKind=repo" + SNAPSHOT }o--o| LOCAL_REPO : "SourceKind=local" + PUBLISHED_REPO }o--o{ SNAPSHOT : "Sources[component] (kind=snapshot)" + PUBLISHED_REPO }o--o{ LOCAL_REPO : "Sources[component] (kind=local)" + PUBLISHED_REPO ||--o{ REFLIST : "E (kind=local only)" +``` + +## 3.3 The Package record + +```go +type Package struct { + Name, Version, Architecture string + SourceArchitecture string // for source packages: real Architecture: field + Source string // binary → its source package name (+ optional version) + Provides []string + FilesHash uint64 // FNV-64a over the sorted file list + checksums + IsInstaller, IsSource, IsUdeb, V06Plus bool + // offloaded (lazy; not in the core record): + deps *PackageDependencies ; extra *Stanza ; files *PackageFiles ; contents []string + collection *PackageCollection +} +``` + +### Package key + +``` +P +``` + +* `V06Plus == true` (all packages created by aptly ≥ 0.6) includes `FilesHash`; older + packages use the *short key* `P `. +* `FilesHash = FNV-64a( for each file sorted by name: filename ‖ size ‖ MD5 ‖ SHA1 ‖ SHA256 )` + (`deb/package_files.go: PackageFiles.Hash`). Same name/version/arch but different bytes ⇒ + **different key**, so two mirrors can legitimately hold "different" `foo_1.0_amd64`. +* Keys sort lexicographically by architecture, then name, then version-as-string, then hash. + This ordering is **not** Debian version order; version comparisons use `CompareVersions`. + +### Where package data comes from + +| Source | Constructor | Notes | +|--------|-------------|-------| +| `Packages` stanza (mirror) | `NewPackageFromControlFile` | Pulls `Filename`, `Size`, `MD5sum`, `SHA1`, `SHA256`, `SHA512`; the `Filename` directory is remembered as `downloadPath` (for download only, not stored) | +| `Sources` stanza (mirror) | `NewSourcePackageFromControlFile` | Parses `Files`, `Checksums-Sha1/256/512`; `Architecture` is set to `source` | +| `.deb` file | `deb/deb.go` `GetControlFileFromDeb` → `NewPackageFromControlFile` | `ar` archive → `control.tar.{gz,xz,zst,…}` → `control` | +| `.udeb` | `NewUdebPackageFromControlFile` | Same but `IsUdeb` | +| `.dsc` | `deb/debian.go` `GetControlFileFromDsc` → source constructor | Also verifies referenced files exist next to it | +| Debian-installer images | `NewInstallerPackageFromControlFile` | Parses the installer `SHA256SUMS` file of a component/architecture and creates one synthetic package named `installer` (`IsInstaller`) whose files are the listed images; file sizes are obtained with `GetLength` (HTTP HEAD). Published under `dists///installer-/current/images` (`legacy-images` for `focal`) | + +### Offloading and lazy loading + +`Package.Files()`, `.Deps()`, `.Extra()` and `.Contents()` transparently load their data from +`xF/xD/xE/xC` keys through the `collection` back-pointer the first time they are used. The +publish loop explicitly drops (`pkg.files = nil` …) these after writing each stanza to keep +memory bounded on huge repositories. + +### Migration of old databases + +`PackageCollection.ByKey` detects the encoding by the `0xC1 0x01` marker: + +* Present → current format (msgpack of the split record). +* Absent → aptly < 0.4 "all fields in one struct" (`oldPackage`); it is decoded, converted and + **immediately rewritten** in the new format. + +Similar tolerant decoding exists in `RemoteRepo.Decode` (Go < 1.2 time encoding, codec +`TimeNotBuiltin`) and `PublishedRepo.Decode` (pre-0.6 single `SnapshotUUID`+`Component` fields +→ `Sources` map; default `SourceKind = snapshot`). + +## 3.4 PackageRefList — the unit of "repository content" + +```go +type PackageRefList struct { Refs [][]byte } // ALWAYS sorted lexicographically +``` + +Refs are package keys **without** any prefix other than the leading `P` (`p.Key("")`). +Because the slice is sorted, all set-like operations are linear two-pointer merges: + +| Operation | Method | Used by | +|-----------|--------|---------| +| Membership | `Has(p)` — binary search | conflict checks | +| Set difference | `Subtract(r)` | `db cleanup` (all keys − referenced keys) | +| Diff with details | `Diff(r, packageCollection)` | `snapshot diff` (CLI and `GET /api/snapshots/:name/diff/:with`); loads packages to classify entries as *added / removed / changed* | +| Merge | `Merge(r, overrideMatching, ignoreConflicting)` | `snapshot merge`, `db cleanup`, repo add | +| Latest per name | `FilterLatestRefs()` | `snapshot merge -latest` | +| Iterate | `ForEach` | everything | +| Build from list | `NewPackageRefListFromPackageList` | after filtering/pull | + +### Merge semantics (`deb/reflist.go: Merge`) + +`Merge(l, r, overrideMatching, ignoreConflicting)` walks both lists in key order: + +* identical refs → keep one; +* `overrideMatching=true` → when `(arch,name)` is equal on both sides, the **right** + package replaces the left one ("last wins"); +* `ignoreConflicting=false` → same `(arch,name,version)` but different file-hash: the right + one wins (conflict resolution); +* `ignoreConflicting=true, overrideMatching=false` → keep everything, a plain union. + +Callers: `snapshot merge` calls `Merge(right, overrideMatching, false)` with +`overrideMatching = !latest && !no-remove`, and afterwards calls `FilterLatestRefs()` for +`-latest` (`-latest` and `-no-remove` are mutually exclusive). `db cleanup` calls +`Merge(refs, false, true)` to build the union of all referenced packages. The API snapshot +create/pull handlers use `Merge(…, true, false)`. + +### Encoding + +`Encode()` → msgpack via `ugorji/go/codec` (`MsgpackHandle`). `Decode()` enables the codec's `ZeroCopy` +option, so decoded refs may alias the input buffer rather than copying it. + +## 3.5 PackageList — the in-memory working set + +`PackageList` (`deb/list.go`) is a **map** `key → *Package` plus optional indexes: + +* `packagesIndex` — packages sorted by name ascending, then version **descending** (newest first, + via `CompareVersions`), then architecture, for binary search (`PrepareIndex()`; required before + `Search/Filter`); +* `providesIndex` — virtual package name → providers. + +Two key functions exist: the *short key* (no file hash) prevents adding two packages of the +same name/version/arch — a `PackageConflictError` is returned (mirror update logs +"skipping package … duplicate in packages index"); the *full key* variant +(`NewPackageListWithDuplicates(true, …)`) allows duplicates. + +Typical life: `NewPackageListFromRefList(reflist, packageCollection, progress)` loads every +referenced `Package` from the DB (progress bar `BarGeneralBuildPackageList`), then callers +`PrepareIndex()`, `Filter`, `Search`, `VerifyDependencies`, `ForEachIndexed`, +`Append`, `Remove`… + +## 3.6 The persisted entities + +### RemoteRepo (`R…`) + +`UUID, Name, ArchiveRoot, Distribution, Components[], Architectures[], Meta (Release fields), +LastDownloadDate, ReleaseFiles (map path → ChecksumInfo), Filter, FilterWithDeps, +Status (idle/updating), WorkerPID, SkipComponentCheck, SkipArchitectureCheck, +DownloadSources, DownloadUdebs, DownloadInstaller, DownloadAppStream, AppStreamFiles`. +Ref list under `E`. *Flat* repositories (distribution ending in `/` or starting with +`.`) are detected in `NewRemoteRepo` and have no components. + +### LocalRepo (`L…`) + +`UUID, Name, Comment, DefaultDistribution, DefaultComponent, Uploaders (json ACL)`; ref list +`E`. + +### Snapshot (`S…`) + +`UUID, Name, CreatedAt, SourceKind ("repo"|"local"|"snapshot"), SourceIDs[], Description, +Origin, NotAutomatic, ButAutomaticUpgrades, AppStreamFiles`; ref list `E`. +Snapshots are immutable by convention: the only mutating operations are **rename** and +**description edit**. Provenance (`SourceKind`+`SourceIDs`) forms a DAG that +`aptly graph` (Graphviz) and `walkUpTree` (used to infer default distribution/component) +traverse. + +### PublishedRepo (`U…`) + +``` +UUID, Storage, Prefix, Distribution, +Origin, NotAutomatic, ButAutomaticUpgrades, Label, Suite, Codename, Version, +Architectures[], SourceKind ("snapshot"|"local"), +Sources: map[component]sourceUUID, +SkipContents, SkipBz2, AcquireByHash, SignedBy, MultiDist, +Revision {Sources: map[component]name} // staged, not-yet-applied source changes +``` + +* `(Storage, Prefix, Distribution)` must be unique (`CheckDuplicate`). +* For `SourceKind == "local"` the *ref list is copied* at publish time into + `E`; the publication therefore represents the local repo **as of + publish time** and only changes on `publish update`. +* `Revision` implements the "`publish source add/remove/update/…`" staging workflow: edit the + set of sources first, then apply them with `publish update` + (`PublishedRepo.Update` consumes/`DropRevision`s it). + +### Checksum cache (`C`) + +Pool files are validated against their DB-cached checksums rather than re-hashed on every +run. `ChecksumCollection` implements `aptly.ChecksumStorage`; `PackagePool.ensureChecksums` +fills it on demand. + +## 3.7 Transactions, batches, and consistency + +* **Package writes** use a transaction (`PackageCollection.Update` / + `UpdateInTransaction`) so the four keys of a package (`P`, `xF`, `xD`, `xE`) are atomic. + Mirror update opens **one transaction for the whole package set** in `FinalizeDownload` + (see [chapter 5](05-workflows.md#51-mirror-update)). +* **Entity + ref list** writes use a `Batch` (`collection.Update` → `Put(Key) ; Put(RefKey)`), + applied atomically by LevelDB's write batch. +* There is **no cross-entity transaction**: e.g. dropping a snapshot deletes `S…` and + `E…` in one batch, but the "is it published?" check is a prior read. In the API this is + protected by task resource locks, in the CLI by the process-exclusive DB lock. +* Referential integrity is **soft**: a snapshot references package keys that must exist. + `aptly db cleanup` never deletes a package referenced by any mirror, local repo, snapshot + or local-repo publication. The reverse problem — refs pointing to packages that no longer exist — + is repaired for **local repos** by `aptly db recover` (`deb/find_dangling.go`, + `FindDanglingReferences` → the refs are removed from the repo). + +## 3.8 Performance characteristics + +* A package record is small (core) plus a few KB of offloaded fields; a snapshot of N packages + costs on the order of N short keys, not N packages. +* Loading a ref list into a `PackageList` reads every referenced package from the DB (several + Gets per package) — this is the dominant cost of `snapshot merge/pull/filter` and `publish`, + and the reason those operations show progress bars. Ref-only operations (`Merge`, + `Subtract`, `Has`) never touch package records. +* Sizes vary with the mirror (long `Description` fields make the `xE` records the largest part + of the DB); measure your own mirrors with `du -sh $rootDir/db $rootDir/pool`. diff --git a/documentation/04-storage.md b/documentation/04-storage.md new file mode 100644 index 000000000..317a67522 --- /dev/null +++ b/documentation/04-storage.md @@ -0,0 +1,209 @@ +# 4. Package Pool & Published Storage + +aptly separates two storage concepts that are often conflated: + +| | **Package pool** | **Published storage** | +|-|------------------|-----------------------| +| Purpose | Authoritative, de-duplicated store of *every package file aptly knows about* | The tree that apt clients actually download (`dists/`, `pool/`) | +| Interface | `aptly.PackagePool` (+ optional `LocalPackagePool`) | `aptly.PublishedStorage` | +| Count | Exactly **one** per aptly instance | **Any number** (named endpoints) | +| Addressing | Content-addressed (SHA-256) | Human/apt layout (`pool////file.deb`) | +| Back-ends | Local filesystem, Azure Blob | Filesystem, S3, GCS, Azure Blob, Swift, JFrog Artifactory | +| Written by | mirror update, `repo add/include` | `publish snapshot/repo/update/switch` | +| Removed by | `db cleanup` | `publish drop`, `publish update/switch` cleanup | + +## 4.1 The local package pool (`files/package_pool.go`) + +### Layout + +For a file with SHA-256 `476e0cdac6bc757dd2b78bacc1325323b09c45ecb41d4562deec2a1c7c148405` +and name `my-package_1.2.3_all.deb`: + +``` +/47/6e/0cdac6bc757dd2b78bacc13253_my-package_1.2.3_all.deb + │ │ └── sha256[4:32] + "_" + basename + │ └────── sha256[2:4] + └───────── sha256[0:2] +``` + +(`buildPoolPath`: `hash[0:2]/hash[2:4]/hash[4:32] + "_" + filename`.) The pool root is +`rootDir/pool` unless `packagepool_storage.path` says otherwise. See also +[`files/README.md`](../files/README.md). + +**Legacy layout** (aptly ≤ 1.0): `//`. It is still *read* unless +`skip_legacy_pool: true` (the default for fresh configs); new files are never written that +way. Package records keep the `PoolPath` they were imported with, so mixed pools work. + +### Operations + +| Method | Behaviour | +|--------|-----------| +| `Import(src, basename, checksums, move, checksumStorage)` | Serialised by a mutex. Computes any missing checksums (MD5+SHA-256 mandatory; recomputes if size mismatched). If the target exists **with identical size**, it is trusted after `ensureChecksums`, and its path is returned (de-duplication); if size differs → error *"file … already exists"*. If a legacy path exists with same size, that path is returned. Otherwise it creates the directories and **tries `os.Link` (hard link) when source and pool are on the same device**, falling back to a copy. Finally it records checksums in the checksum cache and, if `move`, removes the source | +| `Verify(poolPath, basename, checksums, checksumStorage)` | Existence + size + checksum check. With no `poolPath` it *guesses* candidates from SHA-256 (modern) and MD5 (legacy). Fills back the full `ChecksumInfo` into the argument on success. Returns `(path, found, err)` | +| `Open`, `Size`, `Stat` | Direct file access | +| `Remove(path)` | Deletes the file, returns its size | +| `FilepathList(progress)` | Parallel walk (`saracen/walker`) of the whole pool for GC; returns sorted relative paths | +| `GenerateTempPath(name)` | Returns a random path *inside the pool root* — `///` — so a downloaded file sits on the same filesystem as its final location and `Import` can **hard-link** it instead of copying | +| `Link`, `Symlink`, `FullPath` | Used by `files.PublishedStorage.LinkFromPool` for hardlink/symlink publishing | + +### Checksum cache + +Verifying a multi-terabyte pool by re-hashing would be prohibitive. `ensureChecksums` looks +up `C` in the DB (`ChecksumCollection`) and only hashes the file when absent, then +stores `ChecksumInfo{Size, MD5, SHA1, SHA256, SHA512}`. Consequence: **if you replace a pool +file behind aptly's back with different bytes of the same size, aptly will not notice** until +the cache entry is dropped. + +### What "mirror update" and "repo add" do with the pool + +* **Mirror update** first asks `Verify` for each file of each package; only missing/mismatching + files are queued for download (`Package.DownloadList`). Downloads land in temp paths from + `GenerateTempPath`, are checksum-verified, then `Import`ed with `move=true`. +* **repo add/include/`POST /api/repos/:name/file/...`** (`deb.ImportPackageFiles`) call `Import` + with the user's file as the source and `move=false`; removal of successfully imported source + files is a separate step: CLI `-remove-files` (add) / `-no-remove-files` (include); REST removes them + after import unless `noRemove=1` / `noRemoveFiles=1`. The file appears in the pool exactly once regardless of how many repos add it. + +## 4.2 The Azure package pool (`azure/package_pool.go`) + +Selecting + +```yaml +packagepool_storage: + type: azure + account_name: … + account_key: … + container: pool + prefix: "" + endpoint: "" +``` + +stores pool files as blobs with **the same relative path scheme** as the local pool. Differences +from the local pool: + +* No `LocalPackagePool` methods → files cannot be hard-linked or symlinked; publishing to a + filesystem endpoint must use `link_method: copy`, and the cloud back-ends stream the blob + through `Open`. +* No legacy (MD5) layout (`LegacyPath` returns an empty path). +* `Import` uploads (the `move` flag is ignored); `GenerateTempPath` isn't available so the + mirror downloader uses `os.CreateTemp`. +* `Remove`/`FilepathList` use blob APIs. + +## 4.3 Published storage interface + +```go +MkDir(path) RemoveDirs(path, progress) Remove(path) +PutFile(path, sourceFilename) RenameFile(old, new) Filelist(prefix) +LinkFromPool(prefix, relPath, fileName, sourcePool, sourcePath, checksums, force) +SymLink(src, dst) HardLink(src, dst) FileExists(path) ReadLink(path) +``` + +* Paths are relative to the storage root (plus the configured `prefix` for cloud back-ends). +* `LinkFromPool` is the *only* way package files reach the published tree; it is responsible + for idempotency: if the destination already exists it must decide **same / different** + (inode for hard links, MD5 or size for copies, MD5 metadata for object stores). Different + content is an error unless `force` (`-force-overwrite`). +* `RenameFile` provides the "atomic-ish" swap of `*.tmp` index files during re-publish. + Object stores implement it as copy + delete. +* `SymLink/HardLink/ReadLink` exist for `by-hash` and multi-distribution helpers; object + stores emulate them (copy + metadata / text marker). + +### Naming published storage + +The first argument (`[storage:]prefix`) of publish commands is parsed by `deb.ParsePrefix`: + +| Value | Storage | Prefix | +|-------|---------|--------| +| `debian` | default (`""` → `rootDir/public`, hardlink) | `debian` | +| `.` | default | `.` (root) | +| `filesystem:mirror1:ubuntu` | `FileSystemPublishRoots["mirror1"]` | `ubuntu` | +| `s3:prod:` | `S3PublishRoots["prod"]` | `.` | +| `gcs:x:…`, `swift:x:…`, `azure:x:…`, `jfrog:x:…` | respective map | | + +Parsing splits at the **last** colon; an empty prefix after a colon means `.`. Leading/trailing +`/` are trimmed. `NewPublishedRepo` additionally rejects prefixes containing `.`/`..` path +components. `context.GetPublishedStorage(name)` instantiates each named storage once per +process and caches it. + +## 4.4 Filesystem published storage (`files/public.go`) + +Config (`filesystem_publish_endpoints.`): `root_dir`, `link_method` +(`hardlink` default | `symlink` | `copy`), `verify_method` (`md5` default | `size`, only for +`copy`). The implicit default storage is `rootDir/public` with `hardlink`. + +`LinkFromPool` logic: + +1. Compute `destination = root/prefix/relPath/fileName` and `mkdir -p` its directory. +2. Non-copy methods require the source pool to be a `LocalPackagePool`. +3. If destination exists: + * `copy`: compare source size to destination size (`size` mode) or compare + destination MD5 to the source MD5 (`md5` mode); equal ⇒ nothing to do; + * link/symlink: equal **inode and device** ⇒ nothing to do (a symlink to a different + filesystem could reuse inode numbers, hence the device check); + * otherwise error, unless `force`, in which case the destination is removed first. +4. Create: for `copy` stream from `sourcePool.Open` then **`fsync`** (to surface `ENOSPC`); + `symlink` → `pool.Symlink`; default → `pool.Link` (hard link). + +Hard links are the default and make a publication essentially free in disk space and instant, +**but** they require the pool and the publish root to be on the same filesystem, and a hard +link keeps the data alive even after `db cleanup` removes it from the pool — which is +actually what protects still-served files. + +`RemoveDirs` uses `os.RemoveAll`; `Filelist` walks in parallel and returns sorted relative +file names. + +## 4.5 Object-store back-ends + +All object stores share a pattern: + +* `MkDir` is a no-op (flat namespaces). +* Uploads are keyed `prefix + path`. +* `LinkFromPool` = *stream from `sourcePool.Open()` to `PutObject`* with skip-if-same logic + keyed on **MD5**, because MD5 is available as ETag or custom metadata. +* Directory removal = list by prefix + batch delete. + +| Back-end | Package | Config section | Specifics | +|----------|---------|----------------|-----------| +| **Amazon S3 / S3-compatible** | `s3` (AWS SDK v2) | `s3_publish_endpoints` | `bucket`, `region`, `endpoint` (MinIO etc.), `prefix`, `acl` (default `private`; use `public-read` for direct serving; `none` to omit), `storage_class`, `encryption_method` (`AES256`/`aws:kms`), `plus_workaround`, `disable_multidel`, `force_virtualhosted_style`, `debug`; credentials from config or the SDK's default chain (env, IAM role…). **Path cache**: on first `LinkFromPool` under a prefix it lists `prefix/pool` once and caches path → MD5 to avoid a HEAD per file. If the ETag is not a plain MD5 (multipart or SSE-KMS) it falls back to a `HEAD` reading the `Md5` user-metadata that aptly stored. `plus_workaround` writes a second copy of any key containing `+` with `+`→space, because some S3 front-ends decode `+` in URLs as a space. `RenameFile` = `CopyObject` + `DeleteObject`. `SymLink` = copy with metadata | +| **Google Cloud Storage** | `gcs` | `gcs_publish_endpoints` | `bucket`, `prefix`, `credentials_file` or inline `service_account_json`, `project`, `endpoint`, `acl`, `storage_class`, `encryption_key`, `disable_multidel`. Same MD5-based skip logic | +| **Azure Blob** | `azure` | `azure_publish_endpoints` | `account_name`, `account_key`, `container`, `prefix`, `endpoint`; `HardLink` implemented as symlink emulation; `SymLink` writes link metadata; per-prefix path→MD5 cache | +| **OpenStack Swift** | `swift` | `swift_publish_endpoints` | Keystone authentication (`username`, `password`, `auth_url`, `tenant`, `tenant_id`, `domain`, `domain_id`, `tenant_domain`, `tenant_domain_id`), `container`, `prefix`. Uses the object `Hash` (MD5) to compare | +| **JFrog Artifactory** | `jfrog` | `jfrog_publish_endpoints` | `repository`, `url`, one of `user`+`password` / `api_key` / `access_token` (also from env `JFROG_USERNAME`, `JFROG_PASSWORD`, `JFROG_APIKEY`, `JFROG_ACCESSTOKEN`), `prefix`, `plus_workaround`, `debug`. Materialises non-local pool sources in a temp file before upload | + +Because object stores are "serve by URL", the resulting bucket/container is what you point +`apt` at (with public read or a CDN/proxy). + +## 4.6 Where files end up in a published tree + +``` +// +├── dists// +│ ├── Release Release.gpg InRelease +│ ├── Contents-.gz (legacy, top-level) +│ └── / +│ ├── binary-/{Packages,Packages.gz,Packages.bz2,Release} +│ ├── debian-installer/binary-/… (udebs) +│ ├── source/{Sources,Sources.gz,Sources.bz2,Release} +│ ├── Contents-.gz +│ ├── installer-/current/{images|legacy-images}/… +│ └── (skeleton files / AppStream dep11 files) +└── pool//// (multi-dist: pool///…) +``` + +`` is the first letter of the source package name, or the first four characters for +`lib*` sources (`Package.PoolDirectory`). If `Source` carries a version (`foo (1.2)`), only the +name part is used. `MultiDist` publications place files under `pool//…` so the +same *filename* with different contents can coexist in different distributions of one prefix. +Details of generation are in [chapter 6](06-publishing.md). + +## 4.7 Cleaning up + +* **Published storage**: after `publish update/switch`, `CleanupPrefixComponentFiles` deletes + pool files under `prefix/pool//` that are no longer referenced by *any* + publication sharing the same storage+prefix (skippable with `-skip-cleanup`). `publish drop` + removes `dists/` and unreferenced pool files (whole prefix removed if it was the last + publication). +* **Pool**: `aptly db cleanup` (mark-and-sweep, [chapter 5](05-workflows.md#56-db-cleanup)). + +Both sweeps are conservative: they compute "referenced" sets from the DB rather than +inspecting file contents. diff --git a/documentation/05-workflows.md b/documentation/05-workflows.md new file mode 100644 index 000000000..e5223762f --- /dev/null +++ b/documentation/05-workflows.md @@ -0,0 +1,410 @@ +# 5. Core Workflows (Step-by-Step Internals) + +Each section follows the actual call sequence in the code, naming files and functions. +CLI entry points live in `cmd/`; REST entry points in `api/` reuse the same `deb` methods. + +Contents: +[5.1 Mirror create & update](#51-mirror-update) · +[5.2 Local repo: add / include / copy / move](#52-local-repositories) · +[5.3 Snapshots: create / filter / merge / pull / diff / verify](#53-snapshots) · +[5.4 Publish snapshot or repo](#54-publish) · +[5.5 Switch / update / drop](#55-switch-update-drop) · +[5.6 DB cleanup](#56-db-cleanup) · +[5.7 Other commands](#57-other-commands) + +--- + +## 5.1 Mirror update + +### 5.1.1 `aptly mirror create [components…]` (`cmd/mirror_create.go`) + +1. Parse `ppa:user/project` shorthand via `deb.ParsePPA` (uses `ppa_distributor_id`, + `ppa_codename`, `ppa_baseurl`), otherwise take URL/distribution/components from args. +2. `deb.NewRemoteRepo(...)` — assigns a UUID, normalises the URL (`prepare()` adds a trailing + `/`), detects **flat repositories** (distribution ends with `/` or starts with `.`), and + rejects incompatible options (components/udebs/AppStream with flat repos). +3. Copy filter, `-filter-with-deps`, `-force-components`, `-force-architectures`; a filter + string is **parsed immediately** with `query.Parse` so syntax errors surface at creation. +4. `repo.Fetch(downloader, verifier, ignoreSignatures)` — see below. This validates + architectures/components against the archive's `Release`. +5. Save via `RemoteRepoCollection.Add` (no package list yet). + +### 5.1.2 `RemoteRepo.Fetch` — obtain and verify `Release` + +``` +if ignoreSignatures: + download Release ─ok→ use it + └fail→ download InRelease, strip clear-signature (no verification) +else: + 1. download InRelease; VerifyClearsigned; extract text ── ok → done + 2. else download Release + Release.gpg; VerifyDetachedSignature ── ok → done +``` + +Then it parses the first stanza of the release: + +* `Architectures:` → sorted; `source` removed. If the mirror has no architectures configured, + all are adopted; otherwise each requested architecture must be a subset (error suggests + `-force-architectures`). +* `Components:` (non-flat): for distributions with a path (`updates/main`) the leading + `dist/` is stripped from component names. Same subset logic (`-force-components`). +* All of `MD5Sum`, `SHA1`, `SHA256`, `SHA512` blocks are parsed into + `repo.ReleaseFiles[path] = ChecksumInfo{Size, hashes…}` — the **expected checksums for every + index file**, later used to validate `Packages*.{bz2,gz,xz}` downloads. +* Remaining fields become `repo.Meta` (`Origin`, `Label`, `Suite`, `Codename`, `NotAutomatic`, + `ButAutomaticUpgrades`, …), which snapshots inherit. + +### 5.1.3 `aptly mirror update ` (`cmd/mirror_update.go`) + +```mermaid +sequenceDiagram + autonumber + participant CLI as aptly mirror update + participant R as RemoteRepo + participant DL as Downloader + participant DB as LevelDB + participant P as PackagePool + + CLI->>DB: load mirror (LoadComplete → old ref list) + CLI->>R: CheckLock() (unless -force) + CLI->>DL: Fetch Release (verify signature) + CLI->>DL: DownloadPackageIndexes (Packages/Sources/…) + Note over R: parse stanzas → Package objects → in-memory PackageList + opt AppStream enabled + CLI->>DL: DownloadAppStreamFiles → pool + end + opt filter set + CLI->>R: ApplyFilter(query, with deps) + end + CLI->>P: BuildDownloadQueue: Verify each file → tasks for missing ones + CLI->>DB: MarkAsUpdating (status + PID), then CLOSE database + par download_concurrency workers + DL->>DL: DownloadWithChecksum → temp path in pool + end + CLI->>DB: REOPEN database + CLI->>P: Import each downloaded file (move=true) + CLI->>R: FinalizeDownload: one transaction writes all Package records + R->>DB: commit + new ref list ; Update(mirror) (Status=idle) +``` + +Detailed steps: + +1. **Lock check.** `CheckLock()` returns *"mirror is locked by update operation, PID n"* only if + `Status == Updating` **and** `WorkerPID` is a live process (`kill -0`). Stale locks from + crashed runs are ignored automatically. +2. **`DownloadPackageIndexes`** builds a list of `(path, kind, component, arch)`: + * flat: `Packages` (+ `Sources` if `-with-sources`); + * normal: for every component × architecture `Packages` (binary), plus `debian-installer` + Packages (udeb), plus `installer-/current/images/SHA256SUMS` (installer), plus one + `Sources` per component. + For each path `http.DownloadTryCompression` tries **`.bz2`, `.gz`, `.xz`, then plain**; + a variant is only attempted if it appears in `ReleaseFiles` (unless `-ignore-checksums`); + 404/403 ⇒ try the next one. The download is checksum-verified against `ReleaseFiles` and + transparently decompressed. Installer indexes have a special fallback: if not listed in + `Release`, a plain `SHA256SUMS` plus detached `.gpg` is fetched and verified. +3. Stanzas are read with `ControlFileReader` → `NewPackageFromControlFile` / + `NewUdebPackage…` / `NewSourcePackage…` / `NewInstallerPackage…` and added to + `repo.packageList`. Duplicate name+version+arch produce a coloured *skipping package* + warning, not an error. +4. **Filter.** If `repo.Filter != ""`: parse, `PrepareIndex()`, then `PackageList.Filter` with + `WithDependencies = repo.FilterWithDeps`, `WithSources = repo.DownloadSources`, + `Source = empty list` (so dependencies are searched *within the mirror only*) and the + mirror's architectures. Output: `Packages filtered: old -> new`. +5. **`BuildDownloadQueue`**: `-latest` first reduces the list to the newest version per + name/arch (`FilterLatest`). With `-skip-existing-packages`, packages whose key is already in + the mirror's *old* ref list are skipped and their file info loaded from the DB (saves + thousands of `stat` calls). Otherwise for each package `DownloadList` calls + `packagePool.Verify(...)`; missing/bad files become tasks. Identical URLs are + de-duplicated (extra tasks are attached as `Additional`). +6. **Mark updating and close the DB.** `MarkAsUpdating` (status + own PID) → persisted → + `context.CloseDatabase()`. A deferred function reopens the DB and marks the mirror idle on + *any* exit path (including ^C). + Reason: a multi-hour download shouldn't hold the LevelDB lock; other aptly commands can run. +7. **Parallel download.** `download_concurrency` workers pull task indexes from a channel. + Each worker: obtain temp path (`GenerateTempPath` for a local pool, else `os.CreateTemp`) → + `DownloadWithChecksum(url, tmp, expected, ignoreChecksums)` → `task.Done = true`. Errors are + collected (mutex), not fatal. `context.GoContextHandleSignals()` makes ^C cancel the context; + a second ^C kills immediately. +8. **Reopen DB, import.** Sequentially `PackagePool.Import(tmp, filename, &checksums, move=true, + checksumStorage)`; the resulting `PoolPath` and completed checksums are propagated to + `Additional` tasks. Temp files are removed by a deferred cleanup. If interrupted or any + download failed the command returns an error **before** finalisation — *the mirror's old + content is untouched*, and re-running only fetches what's still missing. +9. **`FinalizeDownload`.** Opens **one DB transaction**, `UpdateInTransaction` for every package + (writes `P`, `xF`, `xD`, `xE`), builds the new `PackageRefList` from the list, sets + `LastDownloadDate`, commits. Then `RemoteRepoCollection.Update(repo)` stores the mirror + record and new ref list (`E`) and — via the deferred function — status idle. + +**Atomicity property:** a mirror's visible contents (its ref list) switch from old to new in +a single batch write, and only after all files are safely in the pool. A snapshot taken at any +time sees either the complete old or the complete new state. + +**Why files are verified twice:** checksum on download (integrity in flight), and `Verify` +on the next update (integrity at rest, via checksum cache). + +### 5.1.4 Other mirror commands + +* `mirror edit` changes `-archive-url`, `-filter` (`@file`/`@-` allowed), `-filter-with-deps`, + `-ignore-signatures`, the `-with-sources/-udebs/-installer/-appstream` toggles and keyrings; + `mirror rename`; `mirror drop` (refuses while snapshots were created from it unless + `-force`); `mirror show [-with-packages] [-json]`; `mirror search`; `mirror list [-raw|-json]`. +* REST equivalents: `POST /api/mirrors`, `PUT /api/mirrors/:name` (update, asynchronous + task), `POST /api/mirrors/:name` (edit), etc. (see [chapter 7](07-rest-api-and-tasks.md)). + +--- + +## 5.2 Local repositories + +### `aptly repo add ` (`cmd/repo_add.go`, `deb/import.go`) + +1. Load repo (`LoadComplete`) and its package list `NewPackageListFromRefList`. +2. `deb.CollectPackageFiles` walks arguments (directories recursively, in parallel): + `*.deb *.udeb *.ddeb *.dsc` are package candidates, `*.buildinfo` are "other files" (removed + together with processed files but not imported). +3. `deb.ImportPackageFiles(list, files, forceReplace, verifier, pool, packageCollection, + reporter, restriction, checksumStorageProvider)` — for each file: + * **Read metadata**: `.dsc` → `GetControlFileFromDsc(file, verifier)` (also verifies the + signature if present and required); `.deb`/`.udeb` → `GetControlFileFromDeb` (ar → control + tarball → `control`). + * Build `Package` (`NewSourcePackageFromControlFile` for `.dsc`, with `Source` renamed to + `Package`; otherwise `NewPackageFromControlFile`/`NewUdebPackageFromControlFile`). + * Validate name, version, architecture non-empty. + * Compute checksums; `pool.Import(file, base, &checksums, move=false, …)`. + * For source packages, every file listed in the `.dsc` is looked for **next to the `.dsc`** + and imported; if absent, `pool.Verify("", name, &checksums)` looks in the pool (so + `orig.tar.gz` can be shared between uploads). + * Update package files/PoolPaths; apply optional `restriction` query (used by + uploader rules); `PackageCollection.Update(p)` (transactional write). + * `-force-replace`: search the list for same name/version/arch and remove first. + * `list.Add(p)` — fails with `PackageConflictError` if same name/version/arch already exists + with a different file set (reported as a failure). + * Per-file failures are *warnings*; other files continue. `ImportPackageFiles` returns + processed vs failed file lists. +4. `repo.UpdateRefList(NewPackageRefListFromPackageList(list))` + `LocalRepoCollection.Update`. +5. `-remove-files` deletes processed files (`.dsc`'s companions and `.buildinfo` included). + Exit code is non-zero if any file failed. + +### `aptly repo include ` (`cmd/repo_include.go`, `deb/changes.go`) + +Import driven by Debian `.changes` files (the output of `dpkg-buildpackage`, `dput`, …): + +1. `CollectChangesFiles` finds `*.changes`. +2. `NewChanges` copies the `.changes` into a temp dir; `VerifyAndParse` verifies the PGP + signature (unless `-ignore-signatures`; unsigned rejected unless `-accept-unsigned`) and + parses `Files`/`Checksums-*`, `Distribution`, `Source`, `Binary`, `Architecture`. +3. `Prepare` copies every file listed in the `.changes` from the directory of the `.changes` + file into the temp dir (files in other directories are rejected) and verifies **size, MD5, + SHA-1 and SHA-256** against the `.changes` — any mismatch aborts that upload. +4. **Repository selection** uses the template `-repo="{{.Distribution}}"` (Go `text/template` + over the parsed changes, e.g. `{{.Distribution}}`, `{{.Source}}`); the result names the + local repo. +5. **Uploaders ACL** (`-uploaders-file` or the `Uploaders` stored on the repo at + `repo create -uploaders-file`): `Uploaders.IsAllowed(changes)` walks `rules` in order. A rule + applies if its `condition` (query language, evaluated against the `.changes` fields) matches. + For an applicable rule: if any signature key of the `.changes` matches an entry of `deny` + (after expanding `groups`; `"*"` matches everyone) the upload is **denied immediately**; + else if any key matches `allow` it is **accepted**; otherwise evaluation continues with the + next rule. If no rule accepts: *"denied as no rule matches"*. +6. Files are imported through the same `ImportPackageFiles` (with the ACL as `restriction`), + then processed files are removed unless `-no-remove-files`. + +### `repo copy / move / import / remove` + +* `copy`/`move `: load both lists, `Filter` the source list + (`-with-deps` pulls dependencies not yet in the destination); add to destination (`move` + additionally removes from source). `-dry-run` reports only. +* `import `: same idea, but source is a **mirror** (this is how you + cherry-pick packages from a mirror into a local repo — package files are already in the pool, + so nothing is downloaded). +* `remove `: `list.Remove` for each match; the package **files stay in the pool** + until `db cleanup`. +* REST: `POST /api/repos/:name/packages` and `DELETE …` take explicit package *keys*; + `POST /api/repos/:name/copy/:src/:file` is the API for `copy`. + +--- + +## 5.3 Snapshots + +### Create (`cmd/snapshot_create.go`) + +| Form | Function | Notes | +|------|----------|-------| +| `snapshot create S from mirror M` | `NewSnapshotFromRepository` | Fails with *"mirror not updated"* if the mirror has no ref list; also refuses if `CheckLock()` says an update is running. Copies `Origin/NotAutomatic/ButAutomaticUpgrades` and AppStream file map from the mirror `Meta` | +| `snapshot create S from repo R` | `NewSnapshotFromLocalRepo` | Uses the repo's ref list (empty list if none) | +| `snapshot create S empty` | `NewSnapshotFromPackageList(name, nil, empty, …)` | A blank canvas for `pull` | + +Creating a snapshot writes just `S` and `E` — no package data moves. + +### Filter (`snapshot filter `) + +Load package list → `PrepareIndex` → determine architectures (`-architectures` or all in the +list; required non-empty only with `-with-deps`) → `PackageList.Filter(queries ORed, WithDependencies=-with-deps, +Source=nil)` → new snapshot from the resulting list. Queries can be `@file`/`@-` +(`GetStringOrFileContent`). + +### Merge (`snapshot merge …`) + +```go +result := sources[0].RefList() +for i := 1..n: result = result.Merge(sources[i].RefList(), overrideMatching, false) +if latest: result.FilterLatestRefs() +``` + +`overrideMatching = !latest && !no-remove`. Default: **the last source wins per +(architecture, name)** — the version from a later snapshot *replaces* every version from +earlier ones. `-no-remove` keeps all versions; `-latest` keeps the highest version of each. +It never loads packages except for `-latest` comparisons (pure ref operations). + +### Pull (`snapshot pull `) + +The "controlled upgrade" primitive (`cmd/snapshot_pull.go`): + +1. Load both snapshots into `PackageList`s and index them. +2. Architectures: `-architectures` or all architectures found in `` (excluding source). +3. Each query is ANDed with `$Architecture ∈ archs`. +4. `sourcePackageList.Filter(queries, WithDependencies = !no-deps, Source = packageList)`: + dependency closure is computed against *source ∪ result*, but dependencies **already + satisfied by the destination snapshot** are not pulled again. +5. Merge into the destination list, grouped by `(arch,name)`: + * unless `-no-remove`, all existing packages with that arch+name are **removed** first + (the `[-] removed` lines); + * unless `-all-matches`, only the first (= highest version, given index ordering) match per + arch+name is added (`[+] added`). +6. `-dry-run` prints the plan; otherwise a new snapshot is created from the list with + description *"Pulled into 'X' with 'Y' as source, pull request was: '…'"* and both snapshots + as parents. + +### Diff / Verify / Drop / Rename + +* `diff A B` → `RefList.Diff` (two-pointer over refs; only loads packages for entries that differ + to distinguish *added / removed / version-changed*). `-only-matching` limits to + packages present in both. +* `verify S [source…]` → `PackageList.VerifyDependencies` over the snapshot (with the other + snapshots as extra dependency *sources*) and prints unsatisfied dependencies per architecture. +* `drop S` **always** refuses if the snapshot is published (it lists the publications), and + refuses if it is a *source of other snapshots* unless `-force`; `rename` rewrites metadata only. +* `search`/`show -with-packages` list contents; `aptly graph` renders the DAG of mirrors → + snapshots → publications with Graphviz. + +--- + +## 5.4 Publish + +### `aptly publish snapshot|repo [flags] … [[:]]` +(`cmd/publish_snapshot.go` — one function serves both `snapshot` and `repo`, distinguished by +`cmd.Name()`). + +1. Parse arguments: number of positional names must match the number of `-component` values + (comma-separated); optional last argument is `[storage:]prefix` → `ParsePrefix`. +2. Load each source (`LoadComplete`); warn when a source is empty (architectures then must be + given explicitly because they can't be inferred and can't be changed later). +3. `NewPublishedRepo(storage, prefix, distribution, architectures, components, sources, …)`: + * source kind determined by the first source; + * for empty `-distribution`/`-component`: `walkUpTree` climbs snapshot parents to root + mirrors/local repos and collects their `Distribution`/`Components`/`DefaultDistribution`/ + `DefaultComponent`. If all roots agree the value is used; a single-source default component is + `main`; a conflicting/absent distribution → *"unable to guess distribution name, please + specify explicitly"*; + * duplicate component names rejected; **Origin/NotAutomatic/ButAutomaticUpgrades** are + inherited from snapshots only if they all agree; + * prefix sanitised (`.`/`..` rejected). +4. Apply flag overrides (`-origin`, `-label`, `-suite`, `-codename`, `-notautomatic`, + `-butautomaticupgrades`, `-skip-contents`, `-skip-bz2`, `-acquire-by-hash`, `-signed-by`, + `-version`, `-multi-dist`) — config defaults `skip_contents_publishing` / `skip_bz2_publishing` + apply when flags are unset. +5. `CheckDuplicate` — *(storage, prefix, distribution)* already in use ⇒ error. +6. `getSigner(flags)` builds the GPG signer (see [chapter 10](10-signing-and-verification.md)); + nil signer when `-skip-signing` or `gpg_disable_sign`. +7. **`published.Publish(...)`** — the big one, [chapter 6](06-publishing.md). +8. `PublishedRepoCollection.Add(published)` — for `local` sources also stores the ref lists + `E`. +9. Print the `deb http://your-server/ ` line and, for filesystem + storages, the directory to serve. + +--- + +## 5.5 Switch, update, drop + +### `publish switch [[endpoint:]prefix] ` (`cmd/publish_switch.go`) + +*Snapshot* publications only. Component→snapshot mapping is updated in memory +(`UpdateSnapshot` sets `rePublishing = true`), then `Publish()` regenerates everything. +Because `rePublishing` is set: + +* architectures **are not recomputed** (kept from the first publication); +* index files are uploaded with a `.tmp` suffix and renamed into place at the very end + (`indexes.RenameFiles()`), which is what makes the switch "near-atomic" for clients. + +Afterwards `PublishedRepoCollection.Update` persists and — unless `-skip-cleanup` — +`CleanupPrefixComponentFiles` deletes pool files orphaned by the change. + +### `publish update [[endpoint:]prefix]` (`cmd/publish_update.go`) + +* **Local-repo publication:** re-reads the *current* ref list of each source local repo + (`UpdateLocalRepo`) — publishes new state of the repo. +* **Snapshot publication:** without staged sources there's nothing to change (snapshots + are immutable) — the command is used to apply the *staged revision* below or to re-generate + metadata with new flags (`-skip-contents`, `-origin`, `-label`, `-multi-dist`, + `-signed-by`, `-version`, …). + +### Staged source editing: `publish source …` (`cmd/publish_source_*.go`) + +`add`, `drop`, `list`, `remove`, `replace`, `update` manipulate `PublishedRepo.Revision` +(a *pending* component→source-name map) without touching the published files. +`publish update` then calls `PublishedRepo.Update` which computes +**Added / Updated / Removed** components from `Revision` vs current `Sources` +(`PublishedRepoUpdateResult`), applies them, publishes, and cleans up exactly the affected +components. `publish source drop` discards the pending revision (`DropRevision`). REST: +`/api/publish/:prefix/:distribution/sources[/:component]` and `/update`. + +### `publish drop [[endpoint:]prefix]` + +`PublishedRepoCollection.Remove`: + +1. Determine if other publications share the same *(storage, prefix)*; if none, the whole + `dists/` and `pool/` are removed; otherwise only `dists/` plus pool components not used by + siblings. +2. Delete pool files of shared components that are no longer referenced + (`CleanupPrefixComponentFiles`) unless `-skip-cleanup`. +3. Delete `U…` and (for local-repo sources) `E` keys. +4. `-force-drop` continues on storage errors (leaving orphans). + +--- + +## 5.6 DB cleanup + +`aptly db cleanup` (`cmd/db_cleanup.go`, `api/db.go`) is a **mark-and-sweep garbage collector**: + +``` +mark 1: existing := ∪ ref lists of all mirrors, local repos, snapshots, + and published *local-repo* components (Merge(…, false, true)) +sweep 1: for every key in "P*" not in existing → delete P, xF, xD, xE (one DB batch) +mark 2: referencedFiles := ⋃ pkg.FilepathList(pool) for pkg in existing (sorted) +sweep 2: for every file in pool.FilepathList() not in referencedFiles → pool.Remove() +finish : db.CompactDB() +``` + +Points to know: + +* It needs **exclusive access** — CLI holds the DB lock; the API takes resource `__all__` so no + other task runs concurrently. +* `-dry-run` prints what would be deleted; `-verbose` lists objects. +* Files are removed from the **pool only**; hard-linked copies in published trees survive. +* Package rows for local-repo *publications* are included in "existing" because those refs are + frozen copies (`E`), independent of the live local repo. +* Reports total space freed (`utils.HumanBytes`); the API stores progress in a `task.Detail` + (`TotalNumberOfPackagesToDelete`, `RemainingNumberOfPackagesToDelete`). + +--- + +## 5.7 Other commands + +| Command | What it does internally | +|---------|-------------------------| +| `aptly serve` (`cmd/serve.go`) | Requires ≥ 1 publication. Prints a suggested `deb`/`deb-src` line for every publication, then **closes the database** (`ShutdownContext`) and serves `rootDir/public` (the *default* storage only — `filesystem:` endpoints are not served) with Go's `http.FileServer` (directory listing). The docs call it "not suitable for real production usage" | +| `aptly api serve` | Runs the REST server (chapter 7) | +| `aptly graph [-format=png\|svg\|pdf…] [-layout=horizontal\|vertical] [-output=file]` | `deb.BuildGraph` creates a Graphviz digraph of mirrors, repos, snapshots, publications; renders with the external Graphviz `dot` binary; without `-output` the result is opened in a viewer | +| `aptly package search|show [-with-files -with-references]` | Queries the global `PackageCollection` (all `P…` keys); `show -with-references` lists every container referencing the package | +| `aptly db recover` | `goleveldb.RecoverDB` then checks each local repo for dangling refs (`FindDanglingReferences`) and removes them | +| `aptly config show [-yaml]` | Prints effective configuration (also converts legacy JSON → YAML) | +| `aptly task run (-filename=f \| cmd , cmd …)` | Runs several aptly commands sequentially in **one process** ("running as single thread"). Input: a file (one command per line), interactive stdin (blank line to finish), or command-line args where commands are separated by a trailing comma (`aptly task run repo create a, repo add a pkg`). For each command the DB is reopened, the command runs through the normal `Run()` path, and the context is partially cleaned up; **after the first failing command all remaining ones are skipped** and the overall result is an error | +| `aptly version` | Prints embedded version | diff --git a/documentation/06-publishing.md b/documentation/06-publishing.md new file mode 100644 index 000000000..b4bd70163 --- /dev/null +++ b/documentation/06-publishing.md @@ -0,0 +1,222 @@ +# 6. Publishing Internals + +Publishing turns "a list of package refs per component" into a directory tree that `apt` can +consume. Implemented by `PublishedRepo.Publish` (`deb/publish.go`), `indexFiles`/`indexFile` +(`deb/index_files.go`) and `ContentsIndex` (`deb/contents.go`). + +## 6.1 Inputs and outputs + +**Inputs** (all in the `PublishedRepo`): + +* `Storage`, `Prefix`, `Distribution`; +* one source per **component** (`sourceItems[component]` → snapshot or local repo, with + its `PackageRefList`); +* metadata: `Origin`, `Label`, `Suite`, `Codename`, `Version`, `NotAutomatic`, + `ButAutomaticUpgrades`, `SignedBy`, `Architectures`; +* switches: `SkipContents`, `SkipBz2`, `AcquireByHash`, `MultiDist`, `rePublishing` (internal); +* collaborators: `PackagePool`, `PublishedStorageProvider`, `CollectionFactory`, + `pgp.Signer` (nil = unsigned), `Progress`, `forceOverwrite`, `skelDir`. + +**Outputs** on the published storage: the tree in [§4.6](04-storage.md#46-where-files-end-up-in-a-published-tree). + +## 6.2 The algorithm, step by step + +`Publish()` (annotated with the sub-steps in the source): + +```mermaid +flowchart TD + A[GetPublishedStorage by name] --> B[MkDir prefix/pool and prefix/dists/dist] + B --> C[CreateTemporary DB
for Contents] + C --> D[For each component:
NewPackageListFromRefList] + D --> E{first publication?} + E -- yes --> F[Architectures := union of list architectures
error if none; sort + dedupe] + E -- "re-publish" --> G[keep stored Architectures
suffix = '.tmp'] + F --> H + G --> H[temp dir + newIndexFiles] + H --> I[For each component / list:
pre-create Packages index per arch] + I --> J[list.PrepareIndex; ForEachIndexed pkg] + J --> K[LinkFromPool for the first matching arch] + J --> L[push contents to temp DB] + J --> M[write stanza to Packages/Sources of each matching arch] + M --> N[write Contents indexes, skeleton files, AppStream files] + N --> O[write per-arch Release files] + O --> P[FinalizeAll: compress, checksum, upload, by-hash, sign] + P --> Q[build top-level Release with checksum lists] + Q --> R[Finalize Release: Release, Release.gpg, InRelease] + R --> S[RenameFiles: *.tmp → final] +``` + +### Step details + +1. **Storage & directories.** `publishedStorage.MkDir(prefix/pool)` and `…/dists/` (no-ops + on object stores). +2. **Temporary DB.** `collectionFactory.TemporaryDB()` (a scratch LevelDB, or a prefixed keyspace + in etcd) holds the *contents index* while packages stream through. Closed **and dropped** in a + `defer`. +3. **Load packages.** `NewPackageListFromRefList(p.RefList(component), PackageCollection, progress)` + per component — this is the heavy I/O phase ("Loading packages…"). +4. **Architectures.** On first publish, if none were specified, the union of + `list.Architectures(true)` (including `source`) over all components; empty ⇒ *"unable to figure + out list of architectures, please supply explicit list"*. Sorted, de-duplicated. On re-publish + the stored list is kept (so an emptied source can't erase architectures). +5. **Index writers.** `newIndexFiles(storage, basePath=prefix/dists/dist, tempDir, suffix, + acquireByHash, skipBz2)`; `suffix = ".tmp"` when re-publishing. +6. **Per component, per architecture:** `indexes.PackageIndex(component, arch, false, false, + dist)` pre-registers `Packages` files so **empty** architectures still get a file. +7. **Per package** (`list.ForEachIndexed` — sorted by name, version descending, arch): + * For the **first architecture that the package matches** (`MatchesArchitecture`: an + `Architecture: all` package matches every architecture except `source`; everything else + requires equality — source packages have architecture `source`) + compute the pool-relative directory: + * normal: `pool//` (`PoolDirectory` = `l/libfoo` or `f/foo`); + * `MultiDist`: `pool///`; + * installer packages: `dists///installer-/current/images` + (`legacy-images` when distribution is `focal`), + and call `pkg.LinkFromPool(publishedStorage, packagePool, prefix, relPath, forceOverwrite)` + which, for each `PackageFile`, calls `publishedStorage.LinkFromPool(...)`. It happens once + per package, not once per architecture. + * Open a DB **batch** on the temp DB. For **each matching architecture**: + * unless `SkipContents` or installer: compute `pkg.Contents(pool, progress)` — reads the + `.deb` from the pool, extracts the file list from `data.tar.*`, and caches it in + `xC` (persistent, reused by later publishes); push `(path, qualifiedName)` pairs + into two `ContentsIndex`es: per-component-per-arch and the *legacy* global one; + * append `pkg.Stanza()` to that architecture's `Packages`/`Sources` writer followed by a + blank line. The stanza contains extra control fields (`xE`), dependencies (`xD`), pool + `Filename:` (`pool//…/`), `Size`, `MD5sum`, `SHA1`, `SHA256`, `SHA512` + (`xF`), in canonical Debian field order (`format.go: canonicalCase` + order table). + * Free `files/deps/extra/contents` of the package immediately (bounded memory) and flush the + batch. +8. **Contents.** After all packages: for every arch × {deb,udeb} with a non-empty index, write + `/Contents-[.gz]` (`FILE LOCATION` header, sorted paths, comma-joined + qualified names, gzip only) and, after all components, the legacy top-level + `Contents-.gz` / `Contents-udeb-.gz` covering *all* components. + `ContentsIndex.WriteTo` scans the temp DB in key order, merging consecutive keys with the + same path — key layout is `uuidPrefix ‖ path ‖ 0x00 ‖ qualifiedName`. +9. **Skeleton files** (`rootDir/skel//dists///…`): arbitrary + operator-provided files copied verbatim into the component directory. +10. **AppStream (DEP-11)**: for snapshots with `AppStreamFiles`, files for the matching + component are copied out of the *package pool* into + `/dep11/…` (mirrors made with `-with-appstream`). +11. **Per-arch Release** (`/binary-/Release`, `source/Release`): tiny stanza + with `Archive`, `Architecture`, `Component`, `Origin`, `Label`, `Suite`, `Codename`, and optionally + `Acquire-By-Hash`, `Signed-By`, `Version`. Created for udebs only if `hadUdebs`. +12. **FinalizeAll** (`indexFile.Finalize` for every registered file): + * flush; if `compressable`, compress to `.gz` and `.bz2` (`utils.CompressFile` with + `pgzip`; `.bz2` skipped with `SkipBz2`; the legacy Contents are gzip only); + * compute **MD5, SHA-1, SHA-256, SHA-512, size** for the plain and each compressed + variant → stored in `generatedFiles[path]` (used for the top-level Release); + * `MkDir` the directory; if `acquireByHash`, ensure `by-hash/{MD5Sum,SHA1,SHA256,SHA512}`; + * `PutFile` each variant to `` (i.e. `Packages.tmp`, `Packages.gz.tmp` when + re-publishing) and record `renameMap[tmp] = final`; + * `acquireByHash` → `packageIndexByHash` hard-links the just-uploaded file to + `by-hash//` (if not yet present), keeps the previous index as + `.old` symlink, deletes the *older* physical file, and repoints `` → new digest; + * installer `SHA256SUMS` are **detach-signed** (`.gpg`). +13. **Top-level `Release`** (`dists//Release`): + ``` + Origin, [NotAutomatic], [ButAutomaticUpgrades], Label, Suite, Codename, + Date: , + Architectures: , + [Acquire-By-Hash: yes], [Signed-By: … + Valid-Until: now+100y], [Version], + Description: Generated by aptly, + Components: , + MD5Sum:/SHA1:/SHA256:/SHA512: one line per generated file: " " + ``` + Paths are sorted; sizes right-aligned in 8 characters. +14. **Sign Release.** `releaseFile.Finalize(signer)` writes `Release`, detached `Release.gpg` + and clearsigned `InRelease` (`detachedSign` and `clearSign` both true for the top-level file). + If `signer == nil` these are skipped and clients need `[trusted=yes]` or + `allow-insecure`. +15. **Swap in.** `indexes.RenameFiles()` renames every `*.tmp` to its final name (only when + re-publishing; first publication wrote final names directly). + +### The consistency window + +Re-publishing is *not* a transactional swap. The new index files are uploaded first under +`.tmp` names, but `RenameFiles` iterates the rename map (a Go map — **order unspecified**) and each +rename is separate. On a filesystem, `rename(2)` is atomic per file, so a client sees, at any +instant, some mix of new and old index files; a mismatch between `Release` checksums and a +`Packages.gz` can produce apt's *"Hash Sum mismatch"* error for clients updating in that +moment. Mitigations: + +* enable **`-acquire-by-hash`** so clients fetch immutable, content-addressed copies; +* keep the window short (index files are small; package files are already linked before); +* put the published tree behind a proxy/CDN with consistent-snapshot semantics or + publish to a new prefix and switch a symlink/DNS. + +## 6.3 Package files: link, copy or upload + +`LinkFromPool` is called **before** the index that references the file is visible, so clients +never see a stanza pointing at a missing file. Behaviour by target: + +| Target | What happens | Idempotency check | +|--------|--------------|------------------| +| Filesystem + `hardlink` | `link(2)` pool file → published path | same inode & device | +| Filesystem + `symlink` | `symlink` → absolute pool path | same inode & device of resolved target | +| Filesystem + `copy` | stream + `fsync` | size or MD5 (`verify_method`) | +| S3 / GCS / Azure / Swift / JFrog | upload stream | MD5 vs cached/remote MD5 | + +When two publications (same storage+prefix) contain the *same file* they link to the same +destination; the idempotency check makes that a no-op. If two **different** files would land +on the same published path (same source name & filename, different bytes — legal in Debian archives +that reuse `orig` tarball names across distributions), the second publish fails with *"file +already exists and is different"* — use `-force-overwrite` (dangerous: it can break the other +publication) or, better, **`-multi-dist`** to isolate pools per distribution. + +## 6.4 What is regenerated when + +| Command | Package lists reloaded | Index files rewritten | Pool linking | Cleanup | +|---------|-----------------------|-----------------------|--------------|---------| +| `publish snapshot/repo` | all components | all | all files | none | +| `publish switch` | all components (after replacing chosen sources) | all | all files (already-linked = no-op) | orphaned pool files of changed components | +| `publish update` (local repo) | all components (fresh ref lists) | all | all files | orphaned files of updated/removed components | +| `publish update` with staged sources | same | all | all | affected components | +| `publish drop` | — | removed | removed | see §4.7 | + +There is **no incremental index generation**: even a one-package change re-emits every `Packages` +file. The costs are dominated by package loading and gzip/bz2 compression; contents are cached +per package (`xC`) so only new packages are re-inspected. + +## 6.5 Publishing knobs and their effect + +| Option | CLI flag | Config | Effect | +|--------|----------|--------|--------| +| Skip `Contents-*` | `-skip-contents` | `skip_contents_publishing` | Saves opening every `.deb`; `apt-file` won't work | +| Skip `.bz2` | `-skip-bz2` | `skip_bz2_publishing` | Faster/smaller; only `.gz` (+plain) | +| By-hash | `-acquire-by-hash` | – | `Acquire-By-Hash: yes`; `by-hash/` trees | +| Multi-dist | `-multi-dist` | – | Distribution-scoped `pool//…` | +| Signed-By | `-signed-by=fpr1,fpr2` | – | `Signed-By` + `Valid-Until` (+100 years) in Release | +| Version | `-version=` | – | `Version:` field | +| Label/Origin/Suite/Codename | `-label -origin -suite -codename` | – | Release metadata (Suite/Codename default to the distribution) | +| Not automatic | `-notautomatic=yes -butautomaticupgrades=yes` | – | apt pinning hints (e.g. backports) | +| Skip signing | `-skip-signing` | `gpg_disable_sign` | No `Release.gpg`/`InRelease` | +| Reproducible | env `SOURCE_DATE_EPOCH` | – | Fixed `Date:` | + +## 6.6 Cleanup logic (`CleanupPrefixComponentFiles`) + +After `switch`/`update`, for each **changed** component: + +1. Components no longer published by *any* non-multi-dist publication with the same + storage+prefix are removed wholesale (`RemoveDirs(prefix/pool/)`). +2. For surviving components, compute + `referenced = ⋃ (poolDir/filename for every file of every package in every publication + sharing prefix+component)` — `listReferencedFilesByComponent` loads the ref lists of all such + publications (deduplicating repeated refs across publications with `processedComponentRefs`). +3. `existing = storage.Filelist(prefix/pool/)`; delete `existing − referenced`. + +This is why publishing many distributions into one shared prefix makes `switch` slower: it +must consider every sibling publication. + +`MultiDist` publications keep their pool under `pool//` and are cleaned +independently. + +## 6.7 Consuming what was published + +``` +# /etc/apt/sources.list.d/internal.list +deb [signed-by=/etc/apt/keyrings/internal.gpg] http://repo.example.com/internal stable main +deb-src [signed-by=/etc/apt/keyrings/internal.gpg] http://repo.example.com/internal stable main +``` + +Export the public key with `gpg --export --armor KEYID > internal.gpg` (aptly does not publish it +for you — see [chapter 10](10-signing-and-verification.md)). diff --git a/documentation/07-rest-api-and-tasks.md b/documentation/07-rest-api-and-tasks.md new file mode 100644 index 000000000..9ba1852dc --- /dev/null +++ b/documentation/07-rest-api-and-tasks.md @@ -0,0 +1,334 @@ +# 7. REST API & Task System + +`aptly api serve` exposes nearly every CLI capability over HTTP/JSON. Handlers live in +`api/`; the router is `api.Router(ctx)` (`api/router.go`), built on +[gin](https://github.com/gin-gonic/gin). This chapter explains request handling, the task +system that serialises mutations, and gives an endpoint catalogue. + +> The generated OpenAPI/Swagger spec is produced from the `@Summary/@Router/@Param` comments +> above every handler (`make swagger`, output in `docs/`). Enable the live UI with +> `enable_swagger_endpoint: true` (served at `/docs.html` and `/docs/index.html`). + +## 7.1 Starting the server + +``` +aptly api serve [-listen=:8080 | -listen=unix:///run/aptly.sock] [-no-lock] +``` + +* Verifies `rootDir` is accessible (`utils.DirIsAccessible`). +* If systemd passed exactly one listener fd (`systemd/activation`), it is used; more than one + panics ("not supported"). +* Otherwise listens on TCP or a Unix socket (the socket file is removed and recreated). +* On `SIGINT/SIGTERM`: prints *"Shutdown signal received, waiting for background tasks…"*, + `TaskList().Wait()`, then `server.Shutdown`. +* `-no-lock`: the server does **not** hold the LevelDB lock while idle (see §7.5). + +> **Security:** the API has **no authentication or authorisation** of its own. Anyone who can +> reach the port can create, delete and publish repositories, upload files and add GPG keys. +> Bind to localhost / a Unix socket, or put it behind a reverse proxy providing TLS + auth (see +> [Operations](12-operations.md#124-securing-the-api)). Uploaded files and paths are +> sanitised (`verifyPath` rejects `.`/`..`, `utils.SanitizePath`), but the trust boundary is the +> network. + +## 7.2 Request pipeline + +``` +HTTP request + → gin.Logger (or JSONLogger when log_format=json) → gin.Recovery → gin.ErrorLogger + → [swagger redirect] (/docs → /docs.html, only if enabled) + → [metrics collectors] (only if enable_metrics_endpoint) + → /api group + → [-no-lock DB acquire/release middleware] + → handler + 1. bind + validate JSON (c.Bind), path/query params + 2. cheap validation & 404 checks using a *fresh CollectionFactory* + 3. maybeRunTaskInBackground(name, resources, func(out, detail){ … }) + - inside the closure: NEW collection factory, re-load objects, mutate, save + 4. respond: 200/201 + JSON (sync) or 202 + Task JSON (async) +``` + +Key conventions: + +* Handlers get a **shallow** view for routing/404 decisions, then **re-load inside the task** + after resource locks are acquired. This avoids TOCTOU races between concurrent requests. +* Errors use `AbortWithJSONError(c, status, err)` → JSON `{"error": "message"}`. +* `router.UseRawPath = true`: percent-encoded slashes in path parameters are preserved. + For publish endpoints the *prefix* parameter uses an escaping scheme (`slashEscape`): + `_` → `/` and `__` → `_`; prefix `.` denotes the root. e.g. + `/api/publish/:./stable` (root prefix), `/api/publish/debian_pool/bookworm` for prefix + `debian/pool`. Use `storage:prefix` form (e.g. `s3:mybucket:prod`) for non-default storage. +* `?_async=true` (`1`, `yes`, … — anything that is not `n/no/f/false/0/off`) runs the + operation in the background. The obsolete config key `async_api` sets the default. + +## 7.3 The task system + +Package `task` (`list.go`, `task.go`, `resources.go`, `output.go`). + +### Why tasks exist + +Mutating operations touch shared state (a local repo's ref list, a mirror's contents, the +published tree, the package pool). Two concurrent `repo add` calls on the same repo would lose +updates. The task list **serialises conflicting operations** while allowing unrelated ones to +run in parallel. + +### Data model + +```go +type Task struct { + Name string ; ID int ; State State // IDLE | RUNNING | SUCCEEDED | FAILED + Resources []string // exclusive resource keys + output *Output // thread-safe text buffer (aptly.Progress impl) + detail *Detail // atomic.Value with structured progress + processReturnValue *ProcessReturnValue // {Code int, Value interface{}} + err error +} +type Process func(out aptly.Progress, detail *Detail) (*ProcessReturnValue, error) +``` + +* `Output` implements `aptly.Progress`, so the same `deb.*` code that draws progress bars in + the terminal writes plain text into a buffer retrievable at `/api/tasks/:id/output` + (bars are no-ops; `Printf` appends lines). +* `Detail` carries machine-readable progress (e.g. remaining packages during cleanup, publish + counters) at `/api/tasks/:id/detail`. + +### Scheduling algorithm + +``` +RunTaskInBackground(name, resources, process): + lock list + task := new Task{ID: ++counter, State: IDLE} + append to list + if no running task uses any of `resources` (ResourcesSet.UsedBy == ∅): + mark resources in use by task ; send task to the `queue` channel + else: + leave IDLE (waits) + return copy of task +consumer goroutine: + for task := range queue: set RUNNING ; go run process() + on completion: set SUCCEEDED/FAILED, append "Task succeeded/failed…" to output, + free its resources, Done() wait-groups, + then scan IDLE tasks in order — the first one whose resources are now + free is marked in use and enqueued +``` + +Consequences: + +* **Conflicting requests wait; they are not rejected.** (`runTaskInBackground` has a + `ResourceConflictError` return path that the current `List` never uses; a 409 is therefore + not produced in practice.) A synchronous HTTP request simply blocks until its turn. +* Scheduling is FIFO among *runnable* tasks with a "first-fit" scan on each completion. Only one + waiting task is released per completion event, so long queues of independent tasks progress as + completions occur. +* IDs are process-local counters (reset on restart); tasks are kept in memory until + `DELETE /api/tasks/:id` or `POST /api/tasks-clear`. Synchronous requests delete their own task + after returning the result. + +### Resource keys + +| Key | Meaning | Used by | +|-----|---------|---------| +| `R` (`RemoteRepo.Key()`) | one mirror | mirror update/edit/drop | +| `L` (`LocalRepo.Key()`) | one local repo | repo add/remove/copy/edit/drop/include | +| `S` | one snapshot | snapshot ops | +| `U>>` (`PublishedRepo.Key()`) | one publication | publish ops | +| filesystem paths of the upload dir being consumed | uploads | `repos/:name/file/...` includes the source path as a resource, so the same upload dir can't be imported twice concurrently | +| `__alllocalrepos__` | every local repo | `include` with a templated repo name | +| `__all__` | everything | `db cleanup` | + +`ResourcesSet.UsedBy` implements the wildcards: requesting `__all__` conflicts with any used +resource; requesting `__alllocalrepos__` conflicts with any key beginning with `L`; and if +someone already holds a wildcard, everything conflicts with it. + +### Sync vs async responses + +| Mode | Response | How | +|------|----------|-----| +| **Synchronous** (default) | Final status/body of the operation (`200`, `201`, `4xx/5xx`) | `maybeRunTaskInBackground` schedules the task, `WaitForTaskByID`, reads `ProcessReturnValue{Code,Value}` and the error, deletes the task, replies | +| **Asynchronous** (`_async=true`) | `202 Accepted` + JSON of the `Task` (`ID`, `Name`, `State`, `Resources`) | client polls `/api/tasks/:id`, or blocks on `/api/tasks/:id/wait`, then reads `/output`, `/return_value` | + +Example: + +```bash +# start a long mirror update without holding the connection +TASK=$(curl -s -X PUT -H 'Content-Type: application/json' -d '{}' \ + 'http://localhost:8080/api/mirrors/bookworm-main?_async=true' | jq .ID) +curl -s http://localhost:8080/api/tasks/$TASK/wait # blocks until done → task JSON +curl -s http://localhost:8080/api/tasks/$TASK/output # text log +curl -s http://localhost:8080/api/tasks/$TASK/return_value # {"Code":..,"Value":..} +curl -s -X DELETE http://localhost:8080/api/tasks/$TASK +``` + +## 7.4 Uploading files + +The API cannot read your disk, so package files are first **uploaded** into +`rootDir/upload//`: + +```bash +curl -X POST -F file=@pkg_1.0_amd64.deb http://localhost:8080/api/files/mydir +curl http://localhost:8080/api/files/mydir # list files +curl -X DELETE http://localhost:8080/api/files/mydir/pkg_1.0_amd64.deb +``` + +Then ingest with `POST /api/repos/:name/file/:dir[/:file]` (like `repo add`; removes the +files afterwards unless `?noRemove=1`) or `POST /api/repos/:name/include/:dir[/:file]` (like +`repo include`, for `.changes`; `noRemoveFiles`, `forceReplace`, `acceptUnsigned`, +`ignoreSignature`). `PUT /api/files/:dir/:file` uploads a single file by name. +`apiFilesUpload` streams each part to disk and `fsync`s it (`syncFile`) so `ENOSPC` is reported. +The Prometheus counter `aptly_api_files_uploaded_total{directory=…}` tracks uploads. + +## 7.5 Database locking in API mode + +LevelDB allows one opener per directory. Modes: + +| Mode | Behaviour | +|------|-----------| +| default | The server opens the DB at first use and keeps it open until exit. No other aptly process can use the same `rootDir` concurrently (they retry for `db-open-attempts` × ~10 s) | +| `-no-lock` | A middleware wraps every `/api` request: `acquiredb` opens the DB if there are 0 clients, `releasedb` closes it when the last in-flight client leaves (`acquireDatabase` goroutine with a `dbRequests` channel). Tasks additionally acquire/release a DB reference (`runTaskInBackground`). This lets you run CLI commands *between* API bursts, at the cost of re-opening LevelDB often | +| etcd backend | No exclusive lock; multiple servers/CLIs can share state — resource locks are still **per process** (tasks in different processes don't see each other), so coordinate externally | + +## 7.6 Endpoint catalogue + +Base path `/api`. `{name}` = object name; JSON bodies use PascalCase keys. All mutating +endpoints accept `?_async=true`. + +### Status & meta + +| Method | Path | Purpose | +|--------|------|---------| +| GET | `/api/version` | `{"Version": "…"}` | +| GET | `/api/ready` | 200 `Aptly is ready` once the router is built, else 503 | +| GET | `/api/healthy` | 200 always (liveness) | +| GET | `/api/storage` | Disk usage of `rootDir` filesystem: `Total`, `Free` (MiB), `PercentFull` | +| GET | `/api/metrics` | Prometheus metrics (if enabled); refreshes per-repo package counts first | +| GET | `/docs.html`, `/docs/*` | Swagger UI (if enabled) | + +### Local repositories + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/repos` | list | +| POST | `/api/repos` | `{"Name","Comment","DefaultDistribution","DefaultComponent","FromSnapshot"}` — `FromSnapshot` seeds from a snapshot | +| GET/PUT/DELETE | `/api/repos/{name}` | show / edit (`Name`,`Comment`, defaults — pointer fields, only provided ones change) / drop (`?force=1` when used as a source; refused if published) | +| GET | `/api/repos/{name}/packages` | `?q=` `?withDeps=1` `?format=details` `?maximumVersion=1` | +| POST/DELETE | `/api/repos/{name}/packages` | body `{"PackageRefs":[keys…]}` add/remove by key | +| POST | `/api/repos/{name}/file/{dir}[/{file}]` | import uploaded `.deb/.dsc` (`noRemove`, `forceReplace`) | +| POST | `/api/repos/{name}/include/{dir}[/{file}]` | import via `.changes` | +| POST | `/api/repos/{name}/copy/{src}/{file}` | copy package(s) matching query `file` from repo `src` (`{"with-deps":bool,"dry-run":bool}`) | +| POST | `/api/repos/{name}/snapshots` | snapshot of the repo | + +### Mirrors + +| Method | Path | Notes | +|--------|------|-------| +| GET/POST | `/api/mirrors` | list / create (`Name, ArchiveURL, Distribution, Components, Architectures, Filter, FilterWithDeps, DownloadSources/Udebs/Installer/AppStream, Keyrings, SkipComponentCheck, SkipArchitectureCheck, IgnoreSignatures`, `ppa:` URLs allowed) | +| GET | `/api/mirrors/{name}` | show | +| POST | `/api/mirrors/{name}` | **edit** (POST!) | +| PUT | `/api/mirrors/{name}` | **update** = download (long-running; use `_async`). Body (all optional): `Name` (rename), `Keyrings`, `IgnoreChecksums`, `IgnoreSignatures`, `ForceUpdate`, `SkipExistingPackages`, `LatestOnly` — the same switches as `mirror update` | +| DELETE | `/api/mirrors/{name}` | drop (`?force=1`) | +| GET | `/api/mirrors/{name}/packages` | like repo packages | +| POST | `/api/mirrors/{name}/snapshots` | snapshot of the mirror | + +### Snapshots + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/snapshots` | list (`?sort=name` (default) or `?sort=time`) | +| POST | `/api/snapshots` | `{"Name","Description","SourceSnapshots":[…],"PackageRefs":[…]}` — merge sources and/or add explicit package refs | +| GET/PUT/DELETE | `/api/snapshots/{name}` | show / rename & describe / drop (`?force=1`) | +| GET | `/api/snapshots/{name}/packages` | search/list | +| GET | `/api/snapshots/{name}/diff/{withSnapshot}` | `[{"Left":key\|null,"Right":key\|null}]`; `?onlyMatching=1` | +| POST | `/api/snapshots/{name}/merge` | body `{"Sources":[…]}` — the URL snapshot is the **destination name**; options are **query parameters**: `?latest=1`, `?no-remove=1` | +| POST | `/api/snapshots/{name}/pull` | body `{"Source","Destination","Queries":[…],"Architectures":[…]}`; options are **query parameters**: `?no-deps=1`, `?no-remove=1`, `?all-matches=1`, `?dry-run=1` | + +### Publishing + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/publish` | list all publications | +| GET | `/api/publish/{prefix}/{distribution}` | show | +| POST | `/api/publish[/{prefix}]` | create: `{"SourceKind":"snapshot\|local","Sources":[{"Component","Name"}],"Distribution","Architectures","Label","Origin","Signing":{…},"SkipContents","SkipBz2","SkipCleanup","AcquireByHash","SignedBy","MultiDist","Version","NotAutomatic","ButAutomaticUpgrades","ForceOverwrite"}`; 201 | +| PUT | `/api/publish/{prefix}/{distribution}` | switch (snapshots: `{"Snapshots":[{"Component","Name"}]}`) or refresh local repo; same flags | +| DELETE | `/api/publish/{prefix}/{distribution}` | drop (`?force=1`, `?SkipCleanup=1`) | +| POST/GET/PUT/DELETE | `/api/publish/{prefix}/{distribution}/sources` | staged sources: add one / list pending changes / replace all / discard pending | +| PUT/DELETE | `/api/publish/{prefix}/{distribution}/sources/{component}` | update / remove one staged component | +| POST | `/api/publish/{prefix}/{distribution}/update` | apply staged revision and publish | + +The `Signing` object: `{"Skip":bool,"GpgKey":"ID1, ID2","Keyring":"","SecretKeyring":"", +"Passphrase":"","PassphraseFile":""}`. The API **always** runs GPG in batch mode (no prompt), +so an agent or `PassphraseFile`/`Passphrase` is required for protected keys. + +### Files, packages, GPG, cloud listing, graph, cleanup + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/files` | list upload directories | +| POST/GET/DELETE | `/api/files/{dir}` | upload (multipart `file=`) / list files / delete dir | +| PUT/DELETE | `/api/files/{dir}/{file}` | upload one file / delete one | +| GET | `/api/packages` | search across the whole DB (`?q=`, `?format=details`) | +| GET | `/api/packages/{key}` | full detail of one package by key (URL-escaped) | +| GET | `/api/gpg/keys` | list keys in a keyring (`?keyring=`, default `trustedkeys.gpg`) | +| POST/DELETE | `/api/gpg/key` | body `{"Keyring", "GpgKeyArmor" \| "Keyserver"+"GpgKeyID"}` to import; delete takes `Keyring` + key IDs — used to trust mirror signing keys | +| GET | `/api/s3`, `/api/gcs`, `/api/jfrog` | list configured endpoints | +| GET | `/api/graph.{ext}` | Graphviz render of the object graph by shelling out to `dot -T` (`ext` = `png`, `svg`, …; `dot`/`gv` return the source); `?layout=horizontal\|vertical` | +| POST | `/api/db/cleanup` | GC (see §5.6); takes resource `__all__` | + +### Tasks + +| Method | Path | Notes | +|--------|------|-------| +| GET | `/api/tasks` | all tasks | +| POST | `/api/tasks-clear` | drop finished ones | +| GET | `/api/tasks-wait` | block until *all* tasks complete | +| GET | `/api/tasks/{id}` | task JSON | +| GET | `/api/tasks/{id}/wait` | block until it completes | +| GET | `/api/tasks/{id}/output` | plain text log | +| GET | `/api/tasks/{id}/detail` | structured progress | +| GET | `/api/tasks/{id}/return_value` | `{Code, Value}` | +| DELETE | `/api/tasks/{id}` | only finished tasks | + +### Built-in file serving (`serve_in_api_mode: true`) + +Outside `/api`: `GET /repos/` lists configured *filesystem* endpoints as HTML and +`GET /repos/{storage}/{path…}` serves published files from them, so a single process can be +both API and (basic) repository server. (`aptly serve`, in contrast, exposes only the default +`public/` tree.) + +## 7.7 Metrics (`api/metrics.go`) + +Enabled with `enable_metrics_endpoint: true`; scraped at `/api/metrics` and registered through +`MetricsCollectorRegistrar`: + +| Metric | Type | Labels | +|--------|------|--------| +| `aptly_api_http_requests_in_flight` | gauge | method, path | +| `aptly_api_http_requests_total` | counter | code, method, path | +| `aptly_api_http_request_size_bytes` / `response_size_bytes` | summary | code, method, path | +| `aptly_api_http_request_duration_seconds` | histogram | code, method, path | +| `aptly_build_info` | gauge (=1) | version, goversion | +| `aptly_api_files_uploaded_total` | counter | directory | +| `aptly_repos_package_count` | gauge | source, distribution, component (published packages) | + +The `path` label uses only the first two path segments (`/api/repos`) to bound cardinality. + +## 7.8 Client recipes + +**Publish new package end-to-end (synchronous):** + +```bash +H='Content-Type: application/json' +curl -X POST -F file=@app_1.2_amd64.deb localhost:8080/api/files/incoming +curl -X POST localhost:8080/api/repos/app/file/incoming +curl -X POST -H "$H" -d '{"Name":"app-1.2"}' localhost:8080/api/repos/app/snapshots +curl -X PUT -H "$H" -d '{"Snapshots":[{"Component":"main","Name":"app-1.2"}],"Signing":{"GpgKey":"ABCD1234","PassphraseFile":"/etc/aptly.pass"}}' \ + localhost:8080/api/publish/prod/stable +``` + +**Poll a long mirror update:** + +```bash +curl -X PUT -H "$H" -d '{}' 'localhost:8080/api/mirrors/deb?_async=true' +# → {"Name":"Update mirror deb","ID":7,"State":0,"Resources":["Rabc…"]} +until [ "$(curl -s localhost:8080/api/tasks/7 | jq .State)" -ge 2 ]; do sleep 5; done +``` + +`State`: `0` idle/queued, `1` running, `2` succeeded, `3` failed. diff --git a/documentation/08-cli-and-context.md b/documentation/08-cli-and-context.md new file mode 100644 index 000000000..51cee0e2c --- /dev/null +++ b/documentation/08-cli-and-context.md @@ -0,0 +1,177 @@ +# 8. CLI & Application Context + +## 8.1 Entry point and command dispatch + +```go +// main.go +//go:embed VERSION var Version string +//go:embed debian/aptly.conf var AptlyConf []byte +func main() { + aptly.Version = Version ; aptly.AptlyConf = AptlyConf + os.Exit(cmd.Run(cmd.RootCommand(), os.Args[1:], true)) +} +``` + +The CLI framework is [`smira/commander`](https://github.com/smira/commander) (a fork of +`gonuts/commander`) with [`smira/flag`](https://github.com/smira/flag) (a fork of the standard +`flag` package that additionally supports `IsSet`, so "flag not given" can be distinguished from +"flag given with the default value" — vital for *config default vs. explicit flag* logic). + +`cmd.Run(cmd, args, initContext)` (`cmd/run.go`): + +``` +defer recover(): a *context.FatalError panic → print "ERROR: msg" to stderr, return its exit code +flags, args, err := cmd.ParseFlags(cmdArgs) // global + subcommand flags +if initContext: InitContext(flags) (creates AptlyContext) ; defer ShutdownContext() +context.UpdateFlags(flags) // context.flags = flags of the leaf command +err = cmd.Dispatch(args) // walks the command tree, calls Run(cmd,args) +if err != nil: Fatal(err) // exit 1 (2 for usage / flag errors) +``` + +Exit codes: `0` success, `1` runtime error, `2` command/flag usage error +(`commander.ErrCommandError`, `ErrFlagError`). + +### Command tree (`cmd/*.go`; every command is built by a `makeCmd…()` function) + +``` +aptly +├── api serve +├── config show +├── db cleanup | recover +├── graph +├── mirror create | list | show | drop | update | rename | edit | search +├── package search | show +├── publish snapshot | repo | list | show | drop | update | switch +│ └─ source add | drop | list | remove | replace | update +├── repo create | add | copy | drop | edit | import | list | move | remove +│ | show | rename | search | include +├── serve +├── snapshot create | list | show | verify | pull | diff | merge | drop | rename +│ | search | filter +├── task run +└── version +``` + +Each `makeCmd…` returns a `*commander.Command{Run, UsageLine, Short, Long, Flag}`; help text is +what `aptly help ` prints, and `man/aptly.1` is generated from the same strings +(`_man/gen.go` + `man/aptly.1.ronn.tmpl`). Bash/zsh completion scripts live in `completion.d/`. + +### Global flags (`cmd.RootCommand`) + +| Flag | Meaning | +|------|---------| +| `-config=path` | Config file (else `~/.aptly.conf`, `/usr/local/etc/aptly.conf`, `/etc/aptly.conf`) | +| `-architectures=a,b` | Override configured default architectures | +| `-dep-follow-suggests`, `-dep-follow-recommends`, `-dep-follow-all-variants`, `-dep-follow-source`, `-dep-verbose-resolve` | Dependency-resolution switches (override config) | +| `-db-open-attempts=N` | Retries while the DB is locked by another process (default 10) | +| `-gpg-provider=gpg\|gpg1\|gpg2\|internal` | PGP implementation | +| `-cpuprofile -memprofile -memstats -meminterval` | Only when built with `EnableDebug` | + +Precedence rule used everywhere: **explicit flag → config value → built-in default** +(`LookupOption(default, flags, name)` returns the flag value only if `flags.IsSet(name)`). + +### Reading queries from files + +Package queries and mirror filters may be given as `@file` or `@-` (stdin) via +`GetStringOrFileContent` / `AddStringOrFileFlag` (`cmd/string_or_file_flag.go`). + +### Output conventions + +Human-readable text goes through `context.Progress()` (`Printf`, `ColoredPrintf`), while +`-json` / `-raw` flags on `list/show` commands print machine-readable output directly. `ColoredPrintf` +understands `@r @g @y @!` colour markup (see `console/progress.go`). + +## 8.2 `AptlyContext` — the composition root (`context/context.go`) + +`AptlyContext` owns every shared resource and creates each **lazily on first use**, guarded by a +single mutex (a `Lock()`ed public method calls an unlocked `_private` twin so internal callers +don't deadlock). It embeds a `context.Context` so it can be passed straight to `Download(ctx,…)` +and be cancelled by signals. + +| Accessor | Lazily creates | Notes | +|----------|----------------|-------| +| `Config()` | `utils.Config` | see §8.3 | +| `Progress()` | `console.Progress` | started on creation; shut down in `Shutdown()` | +| `Downloader()` / `NewDownloader(p)` | `http.NewDownloader` or `NewGrabDownloader` | choice from `-downloader` flag, else `downloader` config; limit from `-download-limit` (KiB/s) else `download_limit`; tries = `-max-tries` else `download_retries + 1` | +| `Database()` | LevelDB or etcd `database.Storage` | `Open()` with retry on `resource temporarily unavailable`: up to `-db-open-attempts` (or config `database_open_attempts` unless it is `-1`) with `10s ± 1s` Gaussian jitter | +| `NewCollectionFactory()` | `deb.CollectionFactory` over `Database()` | one per command/request; `Fatal`s if DB can't open | +| `PackagePool()` | local or Azure pool | path from `packagepool_storage.path` else `rootDir/pool` | +| `GetPublishedStorage(name)` | one per configured endpoint | `""` → `rootDir/public` hardlink; `filesystem:x`, `s3:x`, `gcs:x`, `swift:x`, `azure:x`, `jfrog:x`; cached in a map | +| `GetSigner()`, `GetVerifier()` | `pgp.GoSigner`/`GoVerifier` or `GpgSigner`/`GpgVerifier` | provider from `-gpg-provider` else `gpg_provider` (`gpg`, `gpg1`, `gpg2`, `internal`); unknown ⇒ Fatal | +| `TaskList()` | `task.NewList()` | starts the consumer goroutine | +| `DependencyOptions()` | bitmask | computed once from flags/config (`deb.DepFollow*`) | +| `ArchitecturesList()` | `[]string` | `-architectures` else config | +| `UploadPath()`, `SkelPath()`, `DBPath()` | paths | `rootDir/upload`, `rootDir/skel`, `rootDir/db` | +| `CloseDatabase()` / `ReOpenDatabase()` | | temporarily release / reacquire the DB lock | +| `GoContextHandleSignals()` | wraps the embedded `Context` in `WithCancel`; first SIGINT/SIGTERM cancels | second signal kills default handling (`signal.Stop`) | +| `Shutdown()` | | stops task list, closes DB, drops downloader, shuts down progress, writes debug profiles | +| `Cleanup()` | | partial shutdown (downloader + progress) used between `task run` commands | + +`FatalError`/`Fatal(err)` panic with a struct carrying exit code and message; `cmd.Run` (or any +`recover`) turns it into `ERROR: …` + exit status. + +Note the `api` package keeps its own package-level `context` variable pointing to the same +`AptlyContext` (set in `Router(c)`), and `cmd` keeps one in `cmd/context.go`. + +## 8.3 Configuration loading + +`context.config()` (once, memoised by `configLoaded`): + +1. If `-config` given → `utils.LoadConfig(path, &utils.Config)`; any error is fatal. +2. Else try in order `~/.aptly.conf`, `/usr/local/etc/aptly.conf`, `/etc/aptly.conf`: + skip *not exist / permission denied*; stop at first success; any other error is fatal. +3. If nothing was found: print *"Config file not found, creating default config at ~/.aptly.conf"* and + write the embedded `debian/aptly.conf` (or `root_dir: ""` if not embedded), then load it. +4. `utils.LoadConfig` first tries **JSON** (with comments/trailing commas tolerated by + `JsonConfigReader`), and on failure re-reads as **YAML**; if both fail the error mentions both. + `utils.Config` starts as a struct of defaults (see `utils/config.go`), so keys omitted in the file keep + defaults. +5. Logging is initialised from `log_level`/`log_format` (`utils.SetupJSONLogger` or `SetupDefaultLogger`), + and `context.StructuredLogging(true)` disables terminal progress bars. + +`aptly config show [-yaml]` prints the effective config (default JSON; converting old JSON configs to +YAML is `aptly config show -yaml > aptly.yaml`). `ConfigStructure.GetRootDir()` expands a leading `~`. + +Full key reference: [chapter 9](09-configuration.md). + +## 8.4 Lifecycle of a typical command + +``` +aptly snapshot create s1 from mirror m1 + main → cmd.Run + ParseFlags (global flags + "snapshot create" flags) + InitContext NewContext(flags) — nothing opened yet + UpdateFlags(leaf flags) + Dispatch → aptlySnapshotCreate + context.NewCollectionFactory() → Config() loads/creates config + → Database() opens LevelDB (lock; retry if busy) + RemoteRepoCollection.ByName("m1") ← reads R… keys + repo.CheckLock ; LoadComplete ← reads E + deb.NewSnapshotFromRepository + SnapshotCollection.Add → batch: Put S, Put E + fmt.Printf("Snapshot … created") + ShutdownContext → Database().Close(), Progress.Shutdown() + os.Exit(0) +``` + +## 8.5 The `console` package (`console/progress.go`) + +`console.Progress` implements `aptly.Progress` with **one worker goroutine** draining a buffered +channel (capacity 100) of print tasks (`codePrint`, `codePrintStdErr`, `codeProgress`, +`codeHideProgress`, `codeFlush`, `codeStop`, `codeBarEnabled/Disabled`). Concurrent goroutines +(e.g. the download workers) therefore never interleave output. A `progressWorkerFactroy` picks a +terminal worker or a plain/structured-logging worker depending on `structuredLogging`. + +* Progress bars use `cheggaaa/pb` and are created **only when stdout is a terminal** + (`utils.RunningOnTerminal()`); otherwise `InitBar` is a no-op and only text is printed. + `Write(p)` (the `io.Writer` side) advances the bar by bytes, which is how the downloader feeds it. +* `Printf` while a bar is displayed hides the bar, prints, then redraws it. +* `ColoredPrintf` handles `@r @g @y @b @!` … `@|` colour markup through `wsxiaoys/terminal/color`. +* `Flush()` blocks until all queued messages are written — called before spawning `gpg`, which may + prompt on the terminal. + +The REST counterpart is `task.Output` / `task.PublishOutput` (`task/output.go`): a mutex-protected +text buffer whose bar methods are no-ops, except that `PublishOutput` counts down +`RemainingNumberOfPackages` for the `BarPublishGeneratePackageFiles` bar so API clients can poll +publish progress from `/api/tasks/:id/detail`. `ColoredPrintf` there appends the text plus a newline +without colours. diff --git a/documentation/09-configuration.md b/documentation/09-configuration.md new file mode 100644 index 000000000..8fd2108a5 --- /dev/null +++ b/documentation/09-configuration.md @@ -0,0 +1,210 @@ +# 9. Configuration Reference + +aptly is configured with a single file, **YAML (preferred since 1.6.0) or JSON** (comments and +trailing commas tolerated). The Go definition is `utils.ConfigStructure` (`utils/config.go`); each +field carries both a `json:` and a `yaml:` tag. The annotated template shipped with the package is +[`debian/aptly.conf`](../debian/aptly.conf) — it is also **embedded in the binary** and written to +`~/.aptly.conf` the first time aptly runs without any config. + +Location search order (first hit wins): `-config=`, `~/.aptly.conf`, +`/usr/local/etc/aptly.conf`, `/etc/aptly.conf`. Inspect the effective values with +`aptly config show` (JSON) or `aptly config show -yaml`. + +> Legacy JSON keys are camelCase (`rootDir`, `downloadConcurrency`, `S3PublishEndpoints`, …); the YAML +> keys are snake_case (`root_dir`, `download_concurrency`, `s3_publish_endpoints`, …). The tables below +> list the **YAML key**, then the JSON key in parentheses. + +## 9.1 General + +| YAML (JSON) | Default | Meaning | +|-------------|---------|---------| +| `root_dir` (`rootDir`) | `~/.aptly` | Root for `db/`, `pool/`, `public/`, `upload/`, `skel/`. `~` is expanded | +| `log_level` (`logLevel`) | `info` | `debug`, `info`, `warning`, `error` | +| `log_format` (`logFormat`) | `default` | `default` (text) or `json`. `json` also makes the API access log JSON and disables terminal bars | +| `database_open_attempts` (`databaseOpenAttempts`) | `-1` | Retries when the DB is locked. `-1` = use the `-db-open-attempts` flag (default 10); otherwise this value overrides the flag | +| `architectures` | `[]` | Default architecture list (empty = "all available"). Overridable with `-architectures` | +| `skip_legacy_pool` (`skipLegacyPool`) | `false` (template: `true`) | OBSOLETE. Stop looking for MD5-based pre-1.1 pool paths | + +## 9.2 Dependency following + +Used by `snapshot pull`, `snapshot/mirror/repo … -with-deps`, `mirror` filters with deps, and the API's +`withDeps`. Each has a matching command-line flag. + +| YAML (JSON) | Default | Effect | +|-------------|---------|--------| +| `dep_follow_suggests` (`dependencyFollowSuggests`) | false | Also resolve `Suggests:` | +| `dep_follow_recommends` (`dependencyFollowRecommends`) | false | Also resolve `Recommends:` | +| `dep_follow_all_variants` (`dependencyFollowAllVariants`) | false | For `a \| b` pull both alternatives (template spells it `dep_follow_allvariants`; the code reads `dep_follow_all_variants`) | +| `dep_follow_source` (`dependencyFollowSource`) | false | From a binary package also pull its source package | +| `dep_verboseresolve` (`dependencyVerboseResolve`) | false | Log resolver decisions (`Injecting package`, `Unsatisfied dependency`, …) | + +> Watch the key spellings: struct tags are `dep_follow_all_variants` and `dep_verboseresolve` +> (see `utils/config.go`), whereas the commented template uses slightly different spellings. Verify +> with `aptly config show -yaml` after editing. + +## 9.3 PPA short-URL expansion + +| YAML | Default | Meaning | +|------|---------|---------| +| `ppa_distributor_id` | `ubuntu` | Used in `ppa:user/project` → URL | +| `ppa_codename` | *(empty → output of `lsb_release`)* | Distribution for expansion | +| `ppa_baseurl` | `http://ppa.launchpad.net` | Base URL | + +## 9.4 Server + +| YAML | Default | Meaning | +|------|---------|---------| +| `serve_in_api_mode` | false | API server also serves `/repos/…` from *filesystem* publish endpoints | +| `enable_metrics_endpoint` | false | Prometheus `/api/metrics` | +| `enable_swagger_endpoint` | false | Swagger UI on `/docs.html` | +| `async_api` | false | OBSOLETE. Use `?_async=true` per request | + +## 9.5 Database + +```yaml +database_backend: + type: leveldb # or: etcd + db_path: "" # leveldb only; "" → /db + # url: "127.0.0.1:2379" # etcd only +``` + +## 9.6 Mirroring + +| YAML | Default | Meaning | +|------|---------|---------| +| `downloader` | `default` | `default` (net/http, streaming) or `grab` (`cavaliergopher/grab`, resumable, more robust on flaky links). Flag: `-downloader` | +| `download_concurrency` | `4` | Parallel download workers in `mirror update` | +| `download_limit` | `0` | KiB/s cap (0 = none). Flag: `-download-limit` | +| `download_retries` | `0` | Additional tries per file (`max-tries = retries + 1`). Flag: `-max-tries` | +| `download_sourcepackages` | false | Default for `-with-sources` at mirror creation | + +## 9.7 Signing & verification + +| YAML | Default | Meaning | +|------|---------|---------| +| `gpg_provider` | `gpg` | `gpg`/`gpg1`/`gpg2` (external binaries; see chapter 10) or `internal` (pure Go OpenPGP) | +| `gpg_disable_sign` | false | Never sign published repos | +| `gpg_disable_verify` | false | Never verify remote `Release`/`.changes` signatures | +| `gpg_keys` | `[]` | Key IDs used to sign Release when `-gpg-key` isn't provided (multiple keys ⇒ multiple signatures) | + +## 9.8 Publishing defaults + +| YAML | Default | Meaning | +|------|---------|---------| +| `skip_contents_publishing` | false | Default for `-skip-contents` at *first* publish | +| `skip_bz2_publishing` | false | Default for `-skip-bz2` at first publish | + +(Later `publish update/switch` can change these per publication with the same flags; they are +persisted in the publication record.) + +## 9.9 Published-storage endpoints + +An **endpoint** is a named target; you select it on the command line/API as `type:name:prefix` +(e.g. `s3:prod:`). The default endpoint (`""`) is `/public` with hard links and needs no +configuration. + +### filesystem + +```yaml +filesystem_publish_endpoints: + web1: + root_dir: /srv/apt # required + link_method: hardlink # hardlink | symlink | copy + verify_method: md5 # md5 | size (only relevant for copy) +``` +`aptly publish snapshot s filesystem:web1:ubuntu` + +### s3 + +```yaml +s3_publish_endpoints: + prod: + region: eu-central-1 + bucket: my-apt + prefix: "" + acl: public-read # private (default) | public-read | none + access_key_id: "" # else AWS SDK default chain / env vars / IAM role + secret_access_key: "" + session_token: "" + endpoint: "" # S3-compatible service URL (MinIO, Ceph, …) + storage_class: STANDARD + encryption_method: "" # AES256 | aws:kms + plus_workaround: false + disable_multidel: false + force_sigv2: false # accepted but currently ignored by the AWS SDK v2 based client + force_virtualhosted_style: false + debug: false +``` + +### gcs, azure, swift, jfrog + +See the annotated blocks in `debian/aptly.conf`; field lists are in +[chapter 4 §4.5](04-storage.md#45-object-store-back-ends). Keys: + +* `gcs_publish_endpoints.`: `bucket, prefix, credentials_file, service_account_json, project, endpoint, acl, storage_class, encryption_key, disable_multidel, debug` +* `azure_publish_endpoints.`: `container, prefix, account_name, account_key, endpoint` +* `swift_publish_endpoints.`: `container, prefix, username, password, auth_url, tenant, tenant_id, domain, domain_id, tenant_domain, tenant_domain_id` (env fallbacks `OS_USERNAME`, `OS_PASSWORD`) +* `jfrog_publish_endpoints.`: `url, repository, username, password, api_key, access_token, prefix, plus_workaround, debug` — the Artifactory repository must be of the **generic** type, not "debian". + +## 9.10 Package pool storage + +```yaml +packagepool_storage: + type: local # local (default) | azure + path: "" # "" → /pool +# --- or --- +packagepool_storage: + type: azure + account_name: … + account_key: … + container: pool1 + prefix: "" + endpoint: "" +``` + +Unknown `type` values fail config loading with `unknown pool storage type`. + +## 9.11 Complete example + +```yaml +root_dir: /var/lib/aptly +log_level: info +architectures: [amd64, arm64] + +gpg_provider: gpg +gpg_keys: ["0xABCDEF1234567890"] + +downloader: grab +download_concurrency: 8 +download_retries: 3 + +enable_metrics_endpoint: true +enable_swagger_endpoint: false + +database_backend: + type: leveldb + +filesystem_publish_endpoints: + frontend: + root_dir: /srv/apt-public + link_method: hardlink # requires /srv on the same fs as /var/lib/aptly/pool, + # otherwise use: copy + +s3_publish_endpoints: + cdn: + region: us-east-1 + bucket: acme-apt + acl: public-read +``` + +## 9.12 Environment variables that influence aptly + +| Variable | Used by | +|----------|---------| +| `HOME` | default config location (`~/.aptly.conf`), `root_dir` default (`~/.aptly`), `~` expansion | +| `GNUPGHOME` | internal PGP provider: keyring directory (default `~/.gnupg`) | +| `SOURCE_DATE_EPOCH` | fixes the `Date:` field in published `Release` (reproducible publishes) | +| `AWS_*`, `OS_USERNAME`/`OS_PASSWORD`, `JFROG_USERNAME`/`JFROG_PASSWORD`/`JFROG_APIKEY`/`JFROG_ACCESSTOKEN` | credentials fall-backs | +| `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` | the downloader honours the standard Go proxy environment (`http.ProxyFromEnvironment`) | +| `TERM=dumb` | set by `aptly-api.service` so terminal features are avoided | +| `TMPDIR` | temp files (index generation, `.changes` handling, temporary DB for Contents) | diff --git a/documentation/10-signing-and-verification.md b/documentation/10-signing-and-verification.md new file mode 100644 index 000000000..becdd1826 --- /dev/null +++ b/documentation/10-signing-and-verification.md @@ -0,0 +1,143 @@ +# 10. Signing & Verification (`pgp/`) + +aptly touches OpenPGP in two directions: + +| Direction | Purpose | Interface | Where used | +|-----------|---------|-----------|------------| +| **Verify** (inbound) | Trust remote `Release`/`InRelease`, source `.dsc`, `.changes` uploads, installer `SHA256SUMS` | `pgp.Verifier` | `mirror create/update`, `repo add` (`.dsc`), `repo include` | +| **Sign** (outbound) | Sign the repositories aptly publishes | `pgp.Signer` | `publish snapshot/repo/switch/update`, API publish | + +```go +type Signer interface { + Init() error ; SetKey(keyRef string) ; SetKeyRing(keyring, secretKeyring string) + SetPassphrase(passphrase, passphraseFile string) ; SetBatch(bool) + DetachedSign(src, dst string) error ; ClearSign(src, dst string) error +} +type Verifier interface { + InitKeyring(verbose bool) error ; AddKeyring(keyring string) + VerifyDetachedSignature(sig, cleartext io.Reader, showKeyTip bool) error + IsClearSigned(io.Reader) (bool, error) + VerifyClearsigned(io.Reader, showKeyTip bool) (*KeyInfo, error) + ExtractClearsigned(io.Reader) (*os.File, error) // NO verification +} +``` + +## 10.1 Providers + +Selected by `gpg_provider` (config) or `-gpg-provider` (global flag). `AptlyContext.GetSigner()` / +`GetVerifier()` instantiate: + +| Provider | Signer | Verifier | Notes | +|----------|--------|----------|-------| +| `gpg` (default) | `GpgSigner` (runs `gpg`) | `GpgVerifier` (runs `gpgv`) | binaries located by `GPGDefaultFinder` (`gpg`, `gpgv`); auto-detects gnupg 1.x vs 2.x | +| `gpg1` | same | same | requires `gpg`/`gpg1` and `gpgv`/`gpgv1` in `$PATH` that are GnuPG **1.x** | +| `gpg2` | same | same | requires `gpg`/`gpg2`, `gpgv`/`gpgv2` = **2.x** | +| `internal` | `GoSigner` | `GoVerifier` | pure Go (`golang.org/x/crypto/openpgp`, plus extended signature-checking helpers in `pgp/openpgp.go` that can validate *several* signers of one detached signature and track missing keys); no external binaries | + +`gpg` and `gpgv` versions must match, otherwise `NewGpgVerifier` panics *"gpg and gpgv versions +don't match"*. Test fixtures for this logic are in `pgp/test-bins`. + +## 10.2 Verifying remote repositories + +### Trust store + +* External provider: `gpgv --keyring trustedkeys.gpg …` — i.e. the **`trustedkeys.gpg`** keyring in + the GnuPG home of the user running aptly (aptly does *not* use the general `pubring`). Extra keyrings + are added with `-keyring=` (repeatable); if any are given they *replace* the default. +* Internal provider: keyring files are loaded by `loadKeyRing`; names without `/` are relative to + `$GNUPGHOME` (default `~/.gnupg`). Missing default keyring is tolerated (empty); a missing explicit + one prints a warning. +* API: `POST /api/gpg/key` (import armored key text or fetch `GpgKeyID`s from a `Keyserver`), + `GET /api/gpg/keys`, `DELETE /api/gpg/key` — implemented by shelling out to + `gpg --no-default-keyring --keyring …` (keyring path is sanitised). + +Typical bootstrap on Debian/Ubuntu (printed by aptly when the keyring is empty): + +```bash +gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \ + | gpg --no-default-keyring --keyring trustedkeys.gpg --import +# or a single key: +gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keyserver.ubuntu.com --recv-keys 0xKEYID +wget -O - https://vendor.example/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import +``` + +### What is verified, in which order (`RemoteRepo.Fetch`) + +1. `InRelease` (clearsigned) — `VerifyClearsigned`; text extracted with `ExtractClearsigned`. +2. If missing/invalid → `Release` + `Release.gpg` — `VerifyDetachedSignature`. +3. With `-ignore-signatures` (or `gpg_disable_verify`) **no verification** takes place; the payload is + still parsed (an `InRelease` is stripped of its armour via `ExtractClearsigned`). + +Integrity below `Release` relies on hashes, not signatures: every `Packages*` file is checked against +`Release`'s checksums (`ReleaseFiles`), and every `.deb` against the `Packages` stanza checksums +(`DownloadWithChecksum`). Disabling checksum checks (`-ignore-checksums`) weakens the chain. + +Failure output (`runGpgv`): prints which keys were **good** vs **missing** (`--status-fd 3`), and +suggests `--recv-keys` for missing key IDs (`KeyInfo{GoodKeys, MissingKeys}`). +`repo add` verifies `.dsc` signatures when present; `repo include` verifies `.changes` +(unless `-ignore-signatures`, `-accept-unsigned`); the signing keys are then used for +uploaders ACL (see [`repo include`](05-workflows.md#52-local-repositories)) decisions. + +## 10.3 Signing published repositories + +### Which files are signed + +| File | Signature | +|------|-----------| +| `dists//Release` | detached armored → `Release.gpg`; and clearsigned → `InRelease` | +| installer `…/installer-/current/images/SHA256SUMS` | detached `SHA256SUMS.gpg` | +| everything else (`Packages`, `Contents`, per-arch `Release`) | *not* individually signed — covered by hashes in the signed top-level `Release` | + +Digest algorithm for the external provider is forced to **SHA256** (`--digest-algo SHA256`). + +### Choosing the key + +Resolved by `getSigner(flags)` (CLI, `cmd/publish.go`) / `getSigner(&signingParams)` (API): + +1. `-skip-signing` (or `gpg_disable_sign: true`, or API `Signing.Skip`) ⇒ **no signer**; nothing signed. +2. Keys: `-gpg-key` (repeatable) → else config `gpg_keys` → else *gpg's default key* + (`GpgSigner` passes each as `-u `; several keys produce several signatures — useful during key + rotation. API: `Signing.GpgKey = "ID1, ID2"`.) +3. Keyrings: `-keyring`, `-secret-keyring` (GnuPG 1 only for secret keyring). +4. Passphrase: `-passphrase`, `-passphrase-file` (both **insecure** in shell history/process list; prefer a + file with restrictive mode or gpg-agent), or interactive prompt. +5. `-batch` ⇒ `--no-tty --batch` (never prompt). **The API always uses batch mode** (an interactive prompt + would block the server); so protected keys need `gpg-agent` with cached passphrase, or a passphrase + file/param. +6. `signer.Init()` checks that `gpg --list-keys` works and that *some* key exists (when no explicit + keyring is given). + +### GoSigner specifics + +`Init()` reads `pubring.gpg` and `secring.gpg` (legacy keyring formats) from `$GNUPGHOME` unless paths +are given; picks the first valid secret key or the one matching `keyRef` (key ID, or substring of a user +ID); decrypts it with the supplied/prompted passphrase (3 attempts interactive). If `SOURCE_DATE_EPOCH` +is set, signatures are created with that timestamp and *deterministic* settings. **Limitation:** modern +GnuPG (2.1+) stores secret keys in `private-keys-v1.d`/keybox, which `internal` cannot read — export the +key into legacy keyring files, or use the `gpg` provider. + +### Publishing your public key + +aptly never publishes your public key. Export it yourself and host it next to the repository +(skeleton files from `rootDir/skel` are copied *into component directories*, not the prefix root, so +they are not a convenient place for it): + +```bash +gpg --armor --export KEYID > /srv/apt/public.key +# client: +curl -fsSL https://repo.example.com/public.key | sudo gpg --dearmor -o /etc/apt/keyrings/example.gpg +# sources.list: deb [signed-by=/etc/apt/keyrings/example.gpg] https://repo.example.com/ stable main +``` + +`-signed-by=` on publish adds `Signed-By:` (+ `Valid-Until` 100 years ahead) to the top-level +`Release`, letting clients pin acceptable keys for future updates. + +## 10.4 Security notes + +* **Passphrases on the command line** are visible in `ps` and shell history; the API + `Signing.Passphrase` field is stored nowhere but travels in the request body — use TLS. +* The private key must be available on the machine running aptly; consider a dedicated *repository* + subkey with expiry, and rotate using multiple `-gpg-key`s. +* Verification via `trustedkeys.gpg` is only as strong as its contents: import vendor keys from their + official keyrings, not via TOFU keyserver fetches, when possible. +* Never use `-ignore-signatures`/`-ignore-checksums` on untrusted networks. diff --git a/documentation/11-queries-and-dependencies.md b/documentation/11-queries-and-dependencies.md new file mode 100644 index 000000000..b2f67195c --- /dev/null +++ b/documentation/11-queries-and-dependencies.md @@ -0,0 +1,189 @@ +# 11. Queries, Dependencies & Version Comparison + +## 11.1 Package query language + +Used by `mirror -filter`, `snapshot filter/pull`, `repo copy/move/remove/import`, `package search`, +`*/search` commands, `uploaders.json` conditions and the API `?q=`. Parsed by +`query.Parse(string)` (`query/lex.go` → `query/syntax.go`) into a tree of `deb.PackageQuery` +(`deb/query.go`). + +### Grammar (recursive descent, from `syntax.go`) + +``` +Query := A | A '|' Query OR (lowest precedence) +A := B | B ',' A AND +B := C | '!' B NOT +C := '(' Query ')' | D +D := [condition] | [condition] [arch] | __ +condition := '(' [operator] value ')' +operator := '<<' | '<' | '<=' | '>' | '>>' | '>=' | '=' | '%' | '~' +arch := '{' architecture '}' +``` + +Tokens: `( ) | , ! << <= < >> >= > = % ~ { }` and strings. +Note **precedence**: `,` (AND) binds tighter than `|` (OR); use parentheses. + +### Term kinds + +| Term | Example | Becomes | Meaning | +|------|---------|---------|---------| +| Package name | `nginx` | `DependencyQuery` | any version of package named `nginx` (any architecture) | +| Name + version condition | `nginx (>= 1.24)` | `DependencyQuery` | uses Debian dependency semantics (see below) | +| Name + arch | `nginx {amd64}` | `DependencyQuery` | limited to architecture | +| Exact package key | `nginx_1.24.0-1_amd64` | `PkgQuery` | matches name, version, arch exactly. Recognised when there is no condition and the string has two `_` | +| Field query | `Section (= web)` | `FieldQuery` | compare a control field. Field name must start with an **upper-case** letter (and contain no `_`) or with `$` | +| Regexp | `Name (~ ^lib.*-dev$)` / `nginx (~ ^1\.2)` | `FieldQuery`/`DependencyQuery` with `Regexp` | `~` | +| Pattern (glob) | `Name (% nginx-*)` | glob | `%` uses shell-style `filepath.Match` | +| Special fields | `$Version`, `$Architecture`, `$PackageType`, `$Source`, `$SourceVersion` | | see below | + +**Operators.** Following Debian dependency syntax: `<<` strictly less, `>>` strictly greater; the +lexer maps single `<` to *less-or-equal* and `>` to *greater-or-equal* (`<=`, `>=` explicitly the +same) — the historical Debian meaning. `=` equal (default when no operator given inside +parentheses), `%` glob pattern, `~` regular expression (Go RE2). + +### Field names available + +`Package.GetField` (`deb/package.go`) answers, case-sensitively (for source packages +`Architecture` returns the real `Architecture:` of the source, not `source`): +`Name`, `Version`, `Architecture`, `Source`, `Depends`, `Pre-Depends`, `Suggests`, `Recommends`, +`Provides`, `Build-Depends`, `Build-Depends-Indep`, plus any other control field from the stanza +(`Section`, `Priority`, `Maintainer`, `Description`, `Filename` …), and specials: + +| Special | Value | +|---------|-------| +| `$Version` | the package version, evaluated like a dependency on the package's own name (Debian version ordering) | +| `$Source` | source package name (own name if there is no `Source:`); **empty for source packages** | +| `$SourceVersion` | version from `Source: name (version)`, else the package's own version; empty for source packages | +| `$Architecture` | the architecture; an `=` comparison goes through `MatchesArchitecture`, so `$Architecture (= amd64)` also matches `Architecture: all` packages | +| `$PackageType` | `deb`, `udeb` or `source` (installer pseudo-packages report `deb`) | + +> **Quirk:** every field comparison except `%` and `~` (and "field is non-empty" when no operator is +> given) is evaluated with `CompareVersions`, i.e. with *Debian version ordering*, even for non-version +> fields such as `Section (= web)`. Equality behaves as expected; `<`/`>` on textual fields follow version +> rules, which is rarely what you want. + +### Examples + +``` +nginx all versions of package nginx +nginx (>= 1.20), !Name (nginx-extras) nginx ≥ 1.20, and never the nginx-extras package +Priority (required) | Priority (important) +$PackageType (source) +Name (~ ^python3-.*), $Version (>= 3.0) +(Section (= web) | Section (= net)), !Depends (~ apache2) +Maintainer (% *@example.com) +libssl3_3.0.11-1_amd64 one exact binary package +``` + +Errors are reported by `Parse` as `parsing failed: …` (e.g. *unexpected token … expecting end of query*) +before any DB access; mirror filters are parsed at creation and at update. + +### Evaluation strategy (`PackageList.Scan / Query`) + +Each `PackageQuery` has `Matches(PackageLike)`, `Fast(catalog)` and `Query(catalog) *PackageList`: + +* `DependencyQuery.Fast()` is true when the catalog supports searching by dependency — a + `PackageList` with an index. Then `Query` uses **binary search on the name index** (`Search`) and, + because `providesIndex` exists, also virtual packages (`Provides`). +* `AndQuery`: fast if **either** side is fast — evaluate the fast side, then `Scan` that (small) + result with the other side. +* `OrQuery`: fast only if **both** sides are fast; results are unioned. Otherwise `Scan`. +* `NotQuery`, `FieldQuery`, `MatchAllQuery`: full `Scan` calling `Matches` per package. +* `PkgQuery` (`name_version_arch`) is a direct key lookup (`SearchByKey`). + +## 11.2 Dependencies (`deb/package_deps.go`, `deb/list.go`) + +### Parsing + +`ParseDependency("libc6 (>= 2.34)")` → `Dependency{Pkg, Relation, Version, Architecture}`. +Alternatives `a | b` are split by `ParseDependencyVariants`. Architecture may be given as multiarch +`pkg:arch` (`pkg:any` = no restriction) or with aptly's own suffix `pkg (= 1.0) {arch}` (used by generated +source dependencies such as `foo (= 1.0) {source}`). Relations inside parentheses: `<<` (strictly less), +`<`/`<=` (both = less-or-equal), `=`, `>`/`>=` (both = greater-or-equal), `>>`; no operator = equal. +Relations: `VersionDontCare, Less, LessOrEqual, Equal, GreaterOrEqual, Greater, PatternMatch, Regexp`. + +### Which fields are followed (`Package.GetDependencies(options)`) + +Always `Depends` + `Pre-Depends`; optionally (bitmask `deb.DepFollow*`, set from flags/config): + +| Option | Adds | +|--------|------| +| `DepFollowRecommends` | `Recommends` | +| `DepFollowSuggests` | `Suggests` | +| `DepFollowBuild` (internal) | `Build-Depends`, `Build-Depends-Indep` | +| `DepFollowSource` | a dependency on the package's own *source* package: `src (= version) {source}` | +| `DepFollowAllVariants` | for `a \| b` keep looking for every alternative even if one is already satisfied | +| `DepVerboseResolve` | log decisions | + +### Verification & closure + +`PackageList.VerifyDependencies(options, architectures, sources, progress)`: + +``` +for arch in architectures: + cache := {} # dependency-hash → satisfied? + for p in list where p.MatchesArchitecture(arch): + for depString in p.GetDependencies(options): + variants := ParseDependencyVariants(depString) # alternatives + for v in variants: # arch defaults to `arch` + satisfied := sources.Search(v, allMatches=false, searchProvided=true) != nil (cached) + if satisfied and !FollowAllVariants: variantsMissing = nil ; break + if !satisfied: variantsMissing += v + missing += variantsMissing +missing = dedupe(missing) +``` + +`PackageList.Search(dep, allMatches, searchProvided)` binary-searches the name index and filters by +architecture (`MatchesDependency`: an `all` package satisfies any architecture but `source`), version +relation (`versionSatisfiesDependency` via `CompareVersions`) and, when `searchProvided`, virtual packages +from `Provides`. In `Provides`, only `=` versions are allowed (others are logged as a warning and ignored); +an **unversioned** `Provides: foo` is treated as providing `foo` at the *providing package's own version*. + +`PackageList.Filter(FilterOptions)` implements **dependency closure**: + +``` +result := ⋃ query.Query(list) for each query +if WithSources: add matching source packages (via Source: field / SourceRegex) +if WithDependencies: + depSource := Source ∪ result + loop while something was added: + missing := result.VerifyDependencies(opts, arch, depSource) + for dep in missing: + if !FollowAllVariants and result.Search(dep) != nil: continue # satisfied now + for p in list.Search(dep, allMatches=true, searchProvided=true): + add p to result and depSource ; (unless FollowAllVariants → only first) +``` + +The **`Source`** in `FilterOptions` is the set that dependencies are considered *already satisfied* +by (e.g. `snapshot pull` passes the destination snapshot, `repo copy -with-deps` passes the target repo, +mirror filters pass an empty list so *everything* needed is pulled in). + +### Practical consequences + +* Filtering a mirror with `-filter-with-deps` will pull `Depends`/`Pre-Depends` closure only — use + `-dep-follow-recommends` etc. globally if you want `apt install` with default recommends to work offline. +* Missing dependencies are **not an error** during filtering (they are logged only with + `-dep-verbose-resolve`); use `aptly snapshot verify` to audit a snapshot afterwards. +* Virtual packages resolved through `Provides` may pick arbitrary providers; with + `-dep-follow-all-variants` *all* providers/alternatives are included (larger result). + +## 11.3 Debian version comparison (`deb/version.go`) + +`CompareVersions(a, b)` → `-1, 0, 1` implements the Debian Policy algorithm +`[epoch:]upstream_version[-debian_revision]`: + +1. Split epoch (text before the first `:`), then the revision — `parseVersion` splits at the **first** + `-` (Debian Policy says the *last* hyphen; the two differ only for upstream versions that themselves + contain hyphens) — leaving the upstream part. +2. Compare epoch (numeric; missing = 0), then upstream, then revision, each with `compareVersionPart`: + alternate **non-digit run** (compared with `compareLexicographic`) and **digit run** (compared + numerically), left to right. +3. `compareLexicographic`: letters sort before non-letters; `~` sorts before everything, even the end of + the string (so `1.0~rc1 < 1.0`); otherwise ASCII order. + +`PackageList` orders packages by name ascending then **version descending** (newest first), which is +why `snapshot pull` "first match wins" picks the highest version, and `FilterLatest`/ +`FilterLatestRefs` keep only the maximum. + +Note the distinction between **key order** (byte-wise on `P `, used inside +`PackageRefList`) and **version order** (`CompareVersions`, used when semantics matter). diff --git a/documentation/12-operations.md b/documentation/12-operations.md new file mode 100644 index 000000000..d442a622e --- /dev/null +++ b/documentation/12-operations.md @@ -0,0 +1,163 @@ +# 12. Operations Guide + +## 12.1 Directory layout on disk + +``` +/ (default ~/.aptly; owner: the aptly user) +├── db/ LevelDB files (CURRENT, LOCK, MANIFEST-*, *.ldb, *.log) +├── pool/ content-addressed package files (pool/ab/cd/<28 hex>_) +├── public/ default published storage (hard links into pool/) +├── upload/ API upload area (upload//) +└── skel/ optional skeleton files copied into dists/// +``` + +`~/.aptly.conf` (or `/etc/aptly.conf`) holds configuration; GnuPG state lives in `~/.gnupg`. +Temp files use `$TMPDIR` (`/tmp`): index generation, `.changes` handling, temporary Contents DB. + +## 12.2 Deployment patterns + +### A. CLI on a build/release host +Simple: cron or CI job runs `aptly mirror update …; aptly snapshot create …; aptly publish switch …`. +Use `aptly task run` to run several steps in one process. + +### B. API server (systemd) +Debian packaging provides `aptly-api` (`debian/aptly-api.service`, `debian/aptly-api.default`): + +``` +aptly-api.service User=aptly-api Group=aptly-api WorkingDirectory=~ LimitNOFILE=32768 Environment=TERM=dumb +/etc/default/aptly-api LISTEN_ADDRESS='localhost:8080' APTLY_OPTIONS="-no-lock" +ExecStart=/usr/bin/aptly api serve -config=/etc/aptly.conf ${APTLY_OPTIONS} -listen=${LISTEN_ADDRESS} +``` + +The post-install script creates the `aptly-api` system user (home `/var/lib/aptly-api`) and makes +`/etc/aptly.conf` `root:aptly-api 0640` because the config may contain secrets (cloud credentials, +passphrases). Note that the `aptly` package's `/etc/aptly.conf` therefore has **restrictive permissions** +(keep them so). + +* The packaged default listens on **localhost** only and uses `-no-lock` (so administrators can still run + `aptly` CLI commands while the service is idle). +* No socket unit is shipped, but `aptly api serve` adopts a single listener fd passed by systemd (you can + write your own `aptly-api.socket` with `ListenStream=…`). + +### C. Docker +`system/Dockerfile` builds a *development/test* image; for production, build a minimal image containing the +static binary, mount `root_dir` as a volume, and expose 8080 behind a reverse proxy. + +### D. Serving the published repository + +| Option | Notes | +|--------|-------| +| nginx/Apache over `public/` | Most common. Enable directory listing (autoindex) only if you want browsing; `Cache-Control: no-cache` (or short TTL) for `dists/`, long TTL for `pool/`. Because hard links are used, `public/` must live on the same filesystem as `pool/` or use `link_method: copy` | +| Object store (S3/GCS/Azure/Swift) | Publish directly with an endpoint; serve through the store's website endpoint or a CDN. Set `acl: public-read` or use signed access | +| `aptly serve` | Development convenience only | +| API-mode `/repos/…` | Convenience for filesystem endpoints | + +Minimal nginx snippet: + +```nginx +server { + listen 80; + root /var/lib/aptly/public; + location / { autoindex on; } + location ~ /dists/ { add_header Cache-Control "no-cache"; } +} +``` + +## 12.3 Concurrency & locking summary + +| Situation | What happens | +|-----------|--------------| +| Two CLI commands at once | Second waits: retries opening LevelDB every ~10 s for `database_open_attempts` (default 10) tries, then fails *"unable to reopen the DB, maximum number of retries reached"*. Use `-db-open-attempts=0` to fail immediately (`0` loops zero times after the first try) | +| CLI while `aptly mirror update` is in its download phase | Allowed — the updater closed the DB, so other commands (listing, publishing, repo work) proceed. Commands touching *that mirror* are guarded by `CheckLock`: e.g. `snapshot create … from mirror` and another `mirror update` refuse while it is being updated | +| CLI while `aptly api serve` (default) | Blocked until the server stops | +| CLI while `aptly api serve -no-lock` | Works whenever the API is idle; API requests during a CLI command wait for the DB (they retry to open) | +| Two API mutations on the same object | Serialised by the task list (queue) | +| Two mirrors updating in parallel via API | Allowed (different resource keys) but they share the pool and network | +| `db cleanup` | Exclusive: no other API tasks run; don't run publish/update concurrently in CLI | +| Crash during mirror update | `Status=updating` with a dead PID is ignored by `CheckLock`; re-run `mirror update` (only missing files are refetched). Leftover temp files can remain in `pool///…` random names — `db cleanup` removes files unreferenced by any package | +| Crash during `publish` | Package files linked, indexes partially uploaded (`*.tmp` when re-publishing). Re-run the same publish command/`publish update`; stale `.tmp` files are overwritten | + +## 12.4 Securing the API + +The API has no built-in authentication. Recommendations: + +1. Bind to `localhost` or a Unix socket (`-listen=unix:///run/aptly/aptly.sock`, restrict permissions with + directory ownership). +2. Front with nginx/Envoy/Traefik providing **TLS + auth** (basic auth, mTLS, OIDC proxy). +3. Restrict methods/paths per role at the proxy (e.g. CI may only `POST /api/files/*`, + `POST /api/repos/*/file/*`, `PUT /api/publish/*`). +4. Don't put passphrases in requests if the proxy logs bodies; prefer `gpg-agent` or `PassphraseFile`. +5. Limit upload size at the proxy (`client_max_body_size`); uploads are written to disk in `upload/`. +6. Run as a dedicated unprivileged user; its `~/.gnupg` holds the signing key. + +## 12.5 Backup, restore, migration + +* **What to back up:** `root_dir/db` (metadata), `root_dir/pool` (package files), GnuPG keys, config. + `public/` is *derived* (re-publishable) — but the exact bytes of indexes matter to clients that pinned + hashes, so many operators back it up too. +* **Consistent backup:** stop aptly (or make sure no CLI/API task runs); LevelDB files are only + consistent when the DB is closed. Copy `db/` first, then `pool/` (extra files in pool are harmless; missing files are not). +* **Restore:** put back both; run `aptly db cleanup -dry-run` to check consistency; `aptly db recover` if + the DB reports corruption (rebuilds LevelDB and drops dangling refs from local repos). +* **Moving to another host:** copy `db/` and `pool/`; publishes with hard links need to be regenerated + (`aptly publish update`/`switch`) since `public/` links won't survive a cross-fs copy. +* **LevelDB → etcd:** `system/leveldb2etcd.py` helper. +* **Reproducibility:** *snapshots are the backup of intent*. Keep snapshot names and publication commands in + version control. + +## 12.6 Disk & performance + +* Disk = pool (dominant) + DB + (public if using `copy` links). Mirrors of full archives are tens to + hundreds of GB per suite/arch; use `-filter`, `-architectures`, skip `-with-sources`, and `mirror update -latest`. +* `download_concurrency` 4–16 for fast links; `downloader: grab` for flaky networks; `download_limit` to be + polite; `-skip-existing-packages` to speed up repeated updates of big mirrors. +* Publishing: `-skip-contents` and `-skip-bz2` drastically reduce time on large repos. +* Many publications sharing one prefix slow down `switch`/`update` (cleanup scans siblings) — prefer distinct + prefixes or `-multi-dist` for big estates. +* LevelDB: `db cleanup` compacts. `aptly db cleanup` on multi-million-package DBs can take minutes; + schedule off-peak. +* Memory: package lists hold all packages of the sources in RAM, so large merges/publishes need + proportionally more memory; a build with `EnableDebug` adds `-memprofile`/`-memstats` flags for + investigation. +* Put `root_dir` on SSD: LevelDB & index generation are IO-bound. + +## 12.7 Monitoring + +* `GET /api/ready`, `GET /api/healthy` (liveness/readiness probes). +* `GET /api/storage` (disk usage of root_dir filesystem) — alert at e.g. 85 %. +* Prometheus: enable `enable_metrics_endpoint`; alert on `aptly_api_http_requests_total{code=~"5.."}` and on + `aptly_repos_package_count` drops. +* Logs: `log_format: json` for machine parsing; task output via `/api/tasks/:id/output`. + +## 12.8 Routine maintenance checklist + +| Frequency | Task | +|-----------|------| +| Each mirror refresh | `mirror update` → `snapshot create` → `snapshot verify` → publish/switch | +| Weekly | `aptly db cleanup` after deleting old snapshots; check `/api/storage` | +| On key rotation | Publish with two `-gpg-key`s, update clients' keyrings, then drop the old key | +| Before upgrades | Back up `db/`; upgrade aptly (DB formats are auto-migrated on read, see §3.3) | +| Quarterly | Test restore procedure; prune obsolete snapshots/publications | + +## 12.9 Troubleshooting + +| Symptom | Likely cause / fix | +|---------|-------------------| +| `unable to reopen the DB, maximum number of retries reached` | Another aptly process (or `api serve` without `-no-lock`) holds the DB. Stop it, or use `-no-lock` / etcd | +| `mirror is locked by update operation, PID N` | Update in progress; if stale (dead PID) it is ignored automatically; use `-force` otherwise | +| `Looks like your keyring with trusted keys is empty` / `Release signature verification failed` | Import archive keys into **`trustedkeys.gpg`** (see chapter 10) or use `-keyring` | +| `unable to figure out list of architectures, please supply explicit list` | Publishing from an empty source: add `-architectures=amd64` | +| `error linking file to …: file already exists and is different` | Different bytes with same pool path in a shared prefix. Use `-multi-dist`, distinct prefixes, or (carefully) `-force-overwrite` | +| `cannot link … from non-local pool` | Azure pool + hardlink/symlink publish endpoint ⇒ set `link_method: copy` | +| `Hash Sum mismatch` on clients after re-publish | Client fetched during the rename window: enable `-acquire-by-hash`; or retry | +| `file … already exists` on `Import` with different size | Corrupt/partial file in the pool; delete the pool file and re-run the update | +| `prefix/distribution already used by another published repo` | Use `publish switch/update` or choose another prefix | +| `unable to publish: … no such file` in pool | Pool file removed manually; `mirror update` re-downloads missing files (it verifies), `repo` packages must be re-added | +| Publishing prompts for a passphrase / hangs (API) | API is batch-only; use gpg-agent or `Signing.PassphraseFile` | +| Huge memory in `snapshot merge/pull` | Load of all packages; use `-latest`/filters, more RAM | +| `aptly` says `Config file not found, creating default config…` | Expected the first run; edit `~/.aptly.conf` | +| Old `.deb` files linger in `public/pool` | `publish update` w/o `-skip-cleanup` removes orphans; drop old publication; hard-linked files persist after `db cleanup` until publications are cleaned | +| `Filter … unsatisfied dependency` (verbose) | Enable `-dep-verbose-resolve`; add source snapshot providing it | + +Add `log_level: debug` (or `-dep-verbose-resolve`) for more diagnostics; S3/GCS/JFrog endpoints have +`debug: true` to dump requests. diff --git a/documentation/13-development.md b/documentation/13-development.md new file mode 100644 index 000000000..950b3b365 --- /dev/null +++ b/documentation/13-development.md @@ -0,0 +1,221 @@ +# 13. Development Guide + +## 13.1 Repository map + +``` +aptly/ interfaces (PackagePool, PublishedStorage, Downloader, Progress, …) +api/ REST handlers + router (gin) + swagger annotations +azure/ gcs/ s3/ swift/ jfrog/ cloud storage implementations +cmd/ CLI commands (commander) +console/ terminal progress +context/ AptlyContext +database/ KV interface + goleveldb + etcddb +deb/ Debian domain logic +docs/ swagger sources & generated docs package +files/ local pool + local published storage +http/ downloaders, compression fallback +pgp/ GPG (external / internal) +query/ query language parser +task/ task queue +utils/ config, checksums, compression, helpers +system/ Python system (functional) test suite +debian/ man/ completion.d/ _man/ systemd/ packaging, docs, completion, activation helper +``` + +Go version: see `go.mod` (`go 1.26`). Key dependencies: `gin-gonic/gin` (HTTP), `smira/commander` + +`smira/flag` (CLI), `syndtr/goleveldb` and `go.etcd.io/etcd/client/v3` (DB), `ugorji/go/codec` (msgpack), +`aws/aws-sdk-go-v2`, `cloud.google.com/go/storage`, `Azure/azure-sdk-for-go` (azblob), `ncw/swift`, `jfrog-client-go` (storage), +`klauspost/compress` + `pgzip` (compression), `rs/zerolog` (logs), `prometheus/client_golang`, +`swaggo/*` (OpenAPI), `awalterschulze/gographviz`, `cheggaaa/pb`, `google/uuid`, `saracen/walker`, +`gopkg.in/check.v1` (gocheck test framework), `gopkg.in/yaml.v3`. + +## 13.2 Building + +```bash +make prepare # go mod verify/tidy + `go generate` (creates the VERSION file embedded in the binary) +make build # swagger + `go build -o build/aptly` +make binaries # cross-built static binaries +make dpkg # Debian packages +make docker-build # everything inside the dev container (system/Dockerfile) +make help # all documented targets +``` + +`VERSION` comes from `make version`: the top `debian/changelog` version, plus a +`+.` suffix for non-tag (CI) builds (`releasetype`). + +## 13.3 Tests + +### Unit tests + +```bash +make test # go test ./... with gocheck; runs under `faketime` +make test TEST=SnapshotSuite # -check.f regex to select suites/tests +make bench # go test ./deb -run=nothing -bench=. -benchmem +``` + +* Written with **gocheck** (`. "gopkg.in/check.v1"`): a suite type + `Test…` methods, hooked with + `func Test(t *testing.T) { TestingT(t) }` in each package's `*_test.go`. +* `faketime` pins the clock (`TEST_FAKETIME`) because some fixtures use expired certificates/keys. +* Fakes: `http.NewFakeDownloader`, `files/mocks.go`, temp directories, `goleveldb.NewOpenDB(tempdir)`, + and a fake S3 (`s3/server_test.go`). +* Tests for `cmd`/`context` are minimal; behaviour is covered by the system tests below. + +### System (functional) tests — `system/` + +Python 3 test-runner exercising the **real binary** through the CLI and the REST API. + +```bash +make system-test # build test binary + run everything (long tests) +make docker-system-test TEST=t04_mirror # one directory +make docker-system-test TEST=UpdateMirror26Test # one test class +make docker-system-test CAPTURE=1 TEST=… # regenerate gold files +COVERAGE_SKIP=1 make system-test # skip coverage plumbing +``` + +Structure: + +* `system/tNN_name/` — a test *package* per feature: `t01_version`, `t02_config`, `t03_help`, + `t04_mirror`, `t05_snapshot`, `t06_publish`, `t07_serve`, `t08_db`, `t09_repo`, `t10_task`, + `t11_package`, `t12_api`, `t13_etcd`, `t14_graph`. Storage back-ends are exercised through helper base classes + (`s3_lib.py`, `swift_lib.py`, `azure_lib.py`, `gcs_lib.py`, `jfrog_lib.py`, `fs_endpoint_lib.py`). +* Each `*.py` defines classes deriving from `BaseTest` (`system/lib.py`) or a specialised base + (`APITest` in `api_lib.py`, `S3Test`, `SwiftTest`, `AzureTest`, `FileSystemEndpointTest`). + Key attributes: `runCmd` (the command line), `expectedCode`, `fixtureDB`/`fixturePool`/`fixtureGpg`/`fixtureWebServer`, + `configOverride`, `longTest`, `requiresGPG1/2`, `requiresDot`, `databaseType` (`leveldb`/`etcd`), `sortOutput`, + `outputMatchPrepare` (regexp normalisers). +* **Gold files** `_gold` hold expected stdout; `*_mirror_show`, etc. hold expected follow-up + command outputs. `CAPTURE=1` rewrites them — review diffs carefully. +* Fixtures come from separate repos (`aptly-fixture-db`, `aptly-fixture-pool`) and `system/files` (test keys, + packages). A local HTTP server (`fixtureWebServer`) hosts fake remote mirrors so no internet is needed. +* The runner builds a **coverage-instrumented test binary** using `main_test.go` (`//go:build testruncli`, + `TestRunMain` calls `cmd.Run` and prints `EXIT: n`), enabling coverage of CLI paths. +* Lint for tests: `make flake8` (`.flake8`). + +### CI (`.github/workflows/ci.yml`, `golangci-lint.yml`) + +Jobs: **Unit Tests** (Docker image, coverage) → **System Test** (Ubuntu runner with +`graphviz gnupg2 gpgv2 faketime python3-…`, flake8, Azurite for Azure, benchmarks) → **Upload Coverage** +(merged, to Codecov via `codecov.yml`) → **Build** matrix producing packages for Debian 11/12/13 and Ubuntu 20.04–26.04 +(artifact upload via `.github/workflows/scripts/upload-artifacts.sh`). `golangci-lint.yml` runs linting separately. + +### Linting + +`make lint` runs golangci-lint (version pinned in the Makefile). Code style: standard `gofmt`; exported +identifiers documented; `// nolint:` markers only where justified. + +## 13.4 Generating API docs + +Handlers carry swag annotations: + +```go +// @Summary Create Repository +// @Description **Create a local repository** … +// @Tags Repos +// @Consume json +// @Param request body repoCreateParams true "Parameters" +// @Produce json +// @Success 201 {object} deb.LocalRepo +// @Failure 400 {object} Error "Bad Request" +// @Router /api/repos [post] +func apiReposCreate(c *gin.Context) { … } +``` + +`make swagger` runs `swag init --propertyStrategy pascalcase --parseDependency --parseInternal +--markdownFiles docs --generalInfo docs/swagger.conf` and regenerates `docs/docs.go`, `swagger.json/yaml`. +The `docs/*.md` files (`Repos.md`, `Snapshots.md`, …) provide the tag descriptions shown in the UI. `docs/index.go` +embeds `docs.html` served by the router. (This `documentation/` folder is intentionally separate and +is **not** consumed by swag.) + +## 13.5 Recipes for common changes + +### Add a new CLI command + +1. Create `cmd/_.go`: + ```go + func aptlyFooBar(cmd *commander.Command, args []string) error { + if len(args) != 1 { cmd.Usage(); return commander.ErrCommandError } + collectionFactory := context.NewCollectionFactory() + … // use deb.* APIs + return nil + } + func makeCmdFooBar() *commander.Command { + c := &commander.Command{Run: aptlyFooBar, UsageLine: "bar ", Short: "…", Long: `…`, + Flag: *flag.NewFlagSet("aptly-foo-bar", flag.ExitOnError)} + c.Flag.Bool("dry-run", false, "…") + return c + } + ``` +2. Register it in the group's `Subcommands` (e.g. `cmd/snapshot.go`) or add a new group in `RootCommand`. +3. Read flags through `context.Flags().Lookup("dry-run").Value.Get().(bool)`; use `LookupOption` when a + config default exists. +4. Update help golds (`system/t03_help`), man page (`make man`), completions (`completion.d/*`), and add + system tests (+ gold files). + +### Add a REST endpoint + +1. Add params struct with `binding:"required"`/`json:` tags and swag comments in `api/.go`. +2. Handler skeleton: bind → shallow validation/404 → `maybeRunTaskInBackground(c, name, resources, func(out, detail){ fresh + factory; LoadComplete; mutate; Update; return &task.ProcessReturnValue{Code, Value}, err })`. + **Choose resource keys** = keys of every object you mutate (`repo.Key()` …); use `task.AllResourcesKey` for + global operations. +3. Register in `api/router.go` (respect canonical lock order in `api/api.go` comments). +4. `make swagger`; add tests in `api/*_test.go` (gin test router + fake context, see `api/api_test.go`) and system + tests in `system/t12_api`. + +### Add a published-storage backend + +1. New package `foo/` with `PublishedStorage` implementing `aptly.PublishedStorage` (all 11 methods; make + `MkDir` a no-op for flat namespaces). Cache remote listings (`pathCache`) if `LinkFromPool` needs MD5 lookups. + Store MD5 as metadata if the store's ETag isn't MD5. +2. Config struct in `utils/config.go` (`FooPublishRoots map[string]FooPublishRoot` + json/yaml tags, initialise the + map in `Config`), documented in `debian/aptly.conf`. +3. Wire in `context.GetPublishedStorage`: `strings.HasPrefix(name, "foo:")` → constructor. +4. Optional: an API listing endpoint (like `api/s3.go`), tests (`foo/public_test.go` with a fake/emulator), and a system + test base class in `system/`. + +### Add a package pool backend + +Implement `aptly.PackagePool` (+ `LocalPackagePool` if files live on a local FS); extend `PackagePoolStorage` (custom +`UnmarshalJSON/YAML` discriminator on `type`) and `context.PackagePool()`. + +### Add a database backend + +Implement `database.Storage` (`Reader/Writer/PrefixReader/Batch/Transaction/CreateTemporary/CompactDB/Drop`), reuse the +contract tests pattern from `database/goleveldb/database_test.go`, and select it in `context._database()` by +`database_backend.type`. Keep keys **byte-ordered prefix scannable** — the entire data model depends on it. + +### Extend the query language + +`query/lex.go` (tokens), `query/syntax.go` (grammar → `deb.PackageQuery`), `deb/query.go` (new `PackageQuery` type +implementing `Matches/Fast/Query/String`). Add unit tests in `query/*_test.go` and `deb/query_test.go`; document in the +website + man page (`docs`). + +### Add a package field to the model + +Fields in `deb.Package` are msgpack-serialised: **adding** exported fields is backward compatible; renaming/removing is +not. Bump handling in `PackageCollection.ByKey` if the format truly changes (see the `0xC1 0x01` marker and `oldPackage`). +Remember `PackageFiles.Hash()` participates in the package key — changing what it hashes changes keys of *all* packages. + +## 13.6 Debugging tips + +* `-dep-verbose-resolve` for dependency resolution; `log_level: debug` for storage back-ends and API. +* Build with `aptly/version.go: EnableDebug = true` to get `-cpuprofile/-memprofile/-memstats` and gin debug mode. +* Inspect the DB: LevelDB keys are readable text prefixes; use a LevelDB tool or write a small Go program with + `goleveldb.NewOpenDB(path)` and `ProcessByPrefix([]byte("S"), …)`. **Stop aptly first** (lock). +* `aptly package show -with-references -with-files 'Pamd64 nginx 1.24.0-1 8f3a91c2'` shows which mirrors/snapshots/repos + reference a package and the pool file paths. +* `aptly graph -output=g.svg` visualises the DAG. +* Reproduce publish output deterministically with `SOURCE_DATE_EPOCH`. +* Use `aptly task run` in system tests when scripting multiple commands against one DB. + +## 13.7 Releasing + +See [`Releasing.md`](../Releasing.md): create `release/1.x.y`, update `debian/changelog`, merge, tag `v` on master +(`git tag -a v$version …`), publish the generated `docs/swagger.json` to the website repository, update the download page and +announce. `make version` derives the version string embedded in the binary. + +## 13.8 Contributing + +Read [`CONTRIBUTING.md`](../CONTRIBUTING.md) and [`CODE_OF_CONDUCT.md`](../CODE_OF_CONDUCT.md). PRs should include unit +and/or system tests (a *failing* system test reproducing a bug is a welcome first PR), pass `make lint`, and update the +man page/docs when behaviour changes. diff --git a/documentation/14-glossary.md b/documentation/14-glossary.md new file mode 100644 index 000000000..061eb45ce --- /dev/null +++ b/documentation/14-glossary.md @@ -0,0 +1,47 @@ +# 14. Glossary + +| Term | Meaning in aptly | +|------|------------------| +| **Acquire-By-Hash** | apt feature: index files fetched from `by-hash//` so they're immutable; enabled with `-acquire-by-hash` | +| **Architecture** | `amd64`, `arm64`, `all` (arch-independent), `source`, … Package files of arch `all` are published into every `binary-` index | +| **Batch** | `database.Batch`: several writes applied atomically | +| **Checksum cache** | DB keys `C` remembering `ChecksumInfo` of pool files | +| **Collection** | Repository object (`RemoteRepoCollection`, `SnapshotCollection`, …) providing CRUD over one entity kind; obtained via `CollectionFactory` | +| **Component** | Subdivision of a distribution (`main`, `contrib`, `non-free`, …). Each published component has its own source | +| **Contents index** | `Contents-[.gz]` mapping file paths to packages (for `apt-file`) | +| **Control file / Stanza** | RFC-822-like `Field: value` block describing a package/release (`deb.Stanza`, `ControlFileReader`) | +| **Dangling reference** | Ref in a ref list whose package record no longer exists | +| **Distribution** | Release name (`bookworm`, `stable`); directory `dists/` | +| **Endpoint** | Named published-storage target from the config (`s3:prod`, `filesystem:web1`) | +| **Flat repository** | Archive without `dists/`/components: `Packages` next to the debs; distribution written `./` or `sub/` | +| **Installer package** | Synthetic package representing debian-installer images in a mirror (`-with-installer`) | +| **InRelease / Release.gpg** | Clearsigned / detached-signed forms of `Release` | +| **Key (package key)** | `P `; unique identity of a package record | +| **Legacy pool** | Pre-1.1 MD5-based pool layout, read-only support | +| **Local repository** | User-managed list of packages (`deb.LocalRepo`) | +| **Mirror** | Configuration + snapshot-in-time list of a remote archive (`deb.RemoteRepo`) | +| **MultiDist** | Publication mode with per-distribution pool directories | +| **Offloaded fields** | Package data (files, deps, extra, contents) stored in separate keys `xF/xD/xE/xC` | +| **Package pool** | Content-addressed store for package files | +| **PackageList** | In-memory set of `Package`s, indexable and searchable | +| **PackageRefList** | Sorted list of package keys; the "contents" of mirror/repo/snapshot | +| **Prefix** | Path under a published storage where a publication lives (`.` = root) | +| **Progress** | Abstraction for progress bars and messages (`console.Progress`, `task.Output`) | +| **Publication / Published repo** | Binding (storage, prefix, distribution) → sources per component, plus generated files | +| **Pull** | Copy packages (+dependencies) from one snapshot into another, creating a new snapshot | +| **Query** | Package selection expression (see chapter 11) | +| **Resource (task)** | Key naming an object a task needs exclusively (`L`, `__all__`, …) | +| **Revision (publish)** | Staged, not yet applied set of component→source changes on a publication | +| **rePublishing** | Internal flag: regeneration of an existing publication (uses `.tmp` names + rename) | +| **rootDir** | Base directory for db, pool, public, upload, skel | +| **Skeleton files** | Extra files under `rootDir/skel//dists///` copied into the published component | +| **Snapshot** | Immutable ref list with provenance; source for publishing | +| **Source package** | Debian source (`.dsc` + tarballs); architecture `source` | +| **Stanza** | See *Control file* | +| **Switch** | Re-point a snapshot publication to different snapshots and regenerate | +| **Task** | Unit of work in the API's task list, with state, output and resource locks | +| **trustedkeys.gpg** | Keyring aptly's external verifier (`gpgv`) uses for remote repositories | +| **udeb** | Micro-package used by debian-installer (`-with-udebs`) | +| **Uploaders** | ACL (`uploaders.json`) governing which signing keys may `repo include` which packages | +| **Upload directory** | `rootDir/upload/`: staging area for API uploads | +| **UUID** | Random identifier of a persisted entity; basis for DB keys | diff --git a/documentation/README.md b/documentation/README.md new file mode 100644 index 000000000..c944c3b04 --- /dev/null +++ b/documentation/README.md @@ -0,0 +1,48 @@ +# aptly Documentation + +Complete, code-level documentation of **aptly**, the Debian repository management tool. +It explains what the tool does, how it is built, how every subsystem works internally, +and how to run, extend and test it. + +> These documents describe the code in this repository. File and function names are given +> so you can jump straight to the source. Statements were verified against the code at the +> time of writing; when in doubt, the source is authoritative. + +## Reading order + +| # | Document | Read it to learn... | +|---|----------|---------------------| +| 1 | [Overview & Concepts](01-overview-and-concepts.md) | What aptly is, the object model (mirror / local repo / snapshot / published repo), typical workflows | +| 2 | [Architecture](02-architecture.md) | Layers, Go packages, dependency graph, runtime modes, key design decisions | +| 3 | [Data Model & Database](03-data-model-and-database.md) | Key/value layout, encodings, `PackageRefList`, collections, DB backends | +| 4 | [Package Pool & Published Storage](04-storage.md) | Content-addressed pool, publishing back-ends (filesystem, S3, GCS, Azure, Swift, JFrog) | +| 5 | [Core Workflows](05-workflows.md) | Step-by-step internals: mirror update, snapshot, merge/pull, add/include, cleanup | +| 6 | [Publishing Internals](06-publishing.md) | How `Packages`, `Release`, `Contents`, by-hash files and signatures are generated | +| 7 | [REST API & Task System](07-rest-api-and-tasks.md) | Router, sync/async execution, resource locking, endpoint catalogue | +| 8 | [CLI & Context](08-cli-and-context.md) | Command tree, `AptlyContext`, lifecycle of a command, DB open/close | +| 9 | [Configuration Reference](09-configuration.md) | Every config key, defaults, examples for all storage back-ends | +| 10 | [Signing & Verification](10-signing-and-verification.md) | GPG providers, Release signing, mirror signature verification | +| 11 | [Queries, Dependencies & Versions](11-queries-and-dependencies.md) | Query language grammar, dependency resolution, Debian version comparison | +| 12 | [Operations Guide](12-operations.md) | Deployment, locking, backups, recovery, performance, troubleshooting | +| 13 | [Development Guide](13-development.md) | Building, unit & system tests, CI, how to add commands / endpoints / back-ends | +| 14 | [Glossary](14-glossary.md) | Terms used throughout | + +## One-paragraph summary + +aptly keeps **metadata** (which packages exist, which repository/snapshot contains which +package) in an embedded key/value database and **package files** in a de-duplicated, +content-addressed *pool* on disk (or Azure Blob storage). Everything you can do — mirror a +remote archive, build a local repository from `.deb`/`.dsc`/`.changes` files, freeze state as +an immutable *snapshot*, filter/merge/pull between snapshots, and *publish* a snapshot as a +signed, apt-consumable repository to a directory, S3, GCS, Azure, Swift or Artifactory — is a +manipulation of lists of package *references* plus, at publish time, a generation of +`Packages`/`Release` index files and links/copies of pool files. It is driven either by the +`aptly` CLI or by the same logic exposed through a REST API served by `aptly api serve`. + +## Other documentation in this repository + +* [`docs/`](../docs) — API reference fragments used to generate the Swagger/OpenAPI spec + (`make swagger`) and the embedded `/docs` page. +* [`files/README.md`](../files/README.md) — details of the on-disk pool layout. +* `man/aptly.1` — man page generated from the CLI help (`_man/gen.go`). +* Upstream user docs: .