diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..e075abd
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,12 @@
+* text=auto eol=lf
+
+*.sh text eol=lf
+*.bat text eol=crlf
+*.ps1 text eol=lf
+
+*.png binary
+*.jpg binary
+*.jpeg binary
+*.gif binary
+*.ico binary
+*.pdf binary
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
index 7c36301..782c3b8 100644
--- a/.github/ISSUE_TEMPLATE/config.yml
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -3,3 +3,6 @@ contact_links:
- name: Security vulnerability
url: https://github.com/coldsofttech/BatchVault/security/advisories/new
about: Please report security problems privately.
+ - name: Questions and support
+ url: https://github.com/coldsofttech/BatchVault/blob/main/SUPPORT.md
+ about: See how to get help before opening an issue.
diff --git a/.github/social-preview.html b/.github/social-preview.html
deleted file mode 100644
index 6e0eff4..0000000
--- a/.github/social-preview.html
+++ /dev/null
@@ -1,32 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- BatchVault
- Archive, split and restore with proof
-
-
- Python 3.11+
-
- CLI
-
- AES-256
-
-
-
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index d17057d..4dc2a3e 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -25,3 +25,6 @@ jobs:
- name: Ruff
run: uv run ruff check .
+
+ - name: Mypy
+ run: uv run mypy
diff --git a/.markdownlint-cli2.yaml b/.markdownlint-cli2.yaml
new file mode 100644
index 0000000..22f0770
--- /dev/null
+++ b/.markdownlint-cli2.yaml
@@ -0,0 +1,4 @@
+config:
+ MD013: false # line length
+ MD033: false # inline HTML
+ MD041: false # first line heading (templates start with a subheading)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1e181ae..c636289 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,13 +1,27 @@
# Changelog
-All notable changes to this project are documented here.
+All notable changes to this project are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/).
## [Unreleased]
+## [0.1.1] - 2026-11-14
+
+### Added
+
+- `--resume` for `pack` and `unpack` continues an interrupted run, reusing the files that already finished and checking them first. Also in Python, the GitHub Action and Docker.
+- `pack` writes a `BATCHVAULT-README.md` next to the manifest with how the archive was made and the commands to restore it. It never holds a password or key; `--no-readme` turns it off.
+
+### Changed
+
+- Pressing Ctrl+C now stops starting new files instead of working through the whole queue.
+- The Python modules were reorganised. `import batchvault as bv`, the command and the GitHub Action work as before, but code that imported internal modules directly must update those imports.
+
## [0.1.0] - 2026-10-24
First release.
+### Added
+
- Archive many files at once into volumes of a size you choose, and restore them from a manifest, a folder, any single volume or a list.
- Two engines: a built-in one that needs nothing installed (xz, gzip, bzip2 or no compression) and an optional 7-Zip engine.
- Your folder structure is kept: archives mirror the tree and `unpack` recreates it. `list files` remembers the folder you listed, and `--root DIR` overrides where paths start.
@@ -21,6 +35,3 @@ First release.
- A Python API, a GitHub Action (`coldsofttech/BatchVault@v1`) and Docker images with and without 7-Zip.
- Sensible defaults: `--level auto`, no worker threads (`--jobs 0`) and three retries.
- Requires Python 3.11.4 or later.
-
-[Unreleased]: https://github.com/coldsofttech/BatchVault/compare/v0.1.0...HEAD
-[0.1.0]: https://github.com/coldsofttech/BatchVault/releases/tag/v0.1.0
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index af053b5..9591cc8 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -13,13 +13,14 @@ uv run pre-commit install
```
uv run ruff check .
+uv run mypy
uv run pytest -m "not integration and not e2e" -n auto
uv run pytest -m integration # needs 7-Zip installed; skipped otherwise
uv run pytest -m "e2e and not docker" # optional: end-to-end CLI and package tests
```
CI also runs the Docker and GitHub Action end-to-end tests; see
-[docs/testing.md](docs/testing.md).
+[docs/development.md](docs/development.md#testing).
- Add or update tests for the behaviour you change.
- Add a one or two line entry to `[Unreleased]` in `CHANGELOG.md`, written for
@@ -30,5 +31,6 @@ CI also runs the Docker and GitHub Action end-to-end tests; see
## Reporting problems
-Use the issue templates. For security problems, see [SECURITY.md](SECURITY.md)
-instead of opening a public issue.
+Use the issue templates, and see [SUPPORT.md](SUPPORT.md) for where to ask
+questions first. For security problems, see [SECURITY.md](SECURITY.md) instead of
+opening a public issue.
diff --git a/README.md b/README.md
index 107c9b6..8ce19d6 100644
--- a/README.md
+++ b/README.md
@@ -98,18 +98,18 @@ Things to know:
one made each file.
- Videos, photos and other already-compressed files do not shrink, whatever the
level. For them use `--format store`. See the
- [compression guide](docs/compression.md) for choosing a format and level, and
+ [compression guide](docs/guides/compression.md) for choosing a format and level, and
for memory use.
## Use
| I want to use BatchVault... | Read |
|---|---|
-| in a terminal | [Command line guide](docs/cli.md) |
-| in my own Python code | [Python guide](docs/python.md) |
-| in a GitHub workflow | [GitHub Action guide](docs/github-action.md) |
-| in a container | [Docker guide](docs/docker.md) |
-| choosing a format and level | [Compression guide](docs/compression.md) |
+| in a terminal | [Command line guide](docs/guides/cli.md) |
+| in my own Python code | [Python guide](docs/guides/python.md) |
+| in a GitHub workflow | [GitHub Action guide](docs/guides/github-action.md) |
+| in a container | [Docker guide](docs/guides/docker.md) |
+| choosing a format and level | [Compression guide](docs/guides/compression.md) |
A **source** is an original file; an **archive** is the set of volumes made from
it. `pack` turns sources into archives, `unpack` turns archives back into files.
@@ -230,7 +230,7 @@ update = true
```
Command-line options always win over the file. See
-[docs/configuration.md](docs/configuration.md) for every setting and the
+[docs/guides/configuration.md](docs/guides/configuration.md) for every setting and the
environment variables.
## Check your setup
@@ -257,9 +257,9 @@ logged, and web addresses with credentials in them are masked.
The newest 20 logs are kept. Change the folder with `BATCHVAULT_LOG_DIR`, pick a
file with `--log-file`, or turn logging off with `--no-log`.
-## Contributing and security
+## Support, contributing and security
-Contributions are welcome. Start with [CONTRIBUTING.md](CONTRIBUTING.md) and
+For help, see [SUPPORT.md](SUPPORT.md). Contributions are welcome. Start with [CONTRIBUTING.md](CONTRIBUTING.md) and
follow the [code of conduct](CODE_OF_CONDUCT.md). To report a vulnerability,
please read [SECURITY.md](SECURITY.md) first and do not open a public issue.
Technical notes live in [docs/architecture.md](docs/architecture.md) and
diff --git a/SUPPORT.md b/SUPPORT.md
new file mode 100644
index 0000000..d8c5852
--- /dev/null
+++ b/SUPPORT.md
@@ -0,0 +1,7 @@
+# Support
+
+- **Start with the docs:** the [README](README.md) covers installing and using BatchVault. For more detail see the [command line](docs/guides/cli.md), the [Python API](docs/guides/python.md), [configuration](docs/guides/configuration.md), [compression](docs/guides/compression.md), [Docker](docs/guides/docker.md) and the [GitHub Action](docs/guides/github-action.md).
+- **Check your setup:** run `batchvault doctor`. It reports your Python version, whether the xz, gzip and bzip2 libraries and 7-Zip are available, and where logs are kept.
+- **Questions and ideas:** use [GitHub Discussions](../../discussions) if enabled, otherwise open an issue with the feature request form.
+- **Bugs:** open an issue with the bug report form. Include the BatchVault version (`batchvault --version`), the engine (native or 7-Zip), your operating system and Python version, how you run it (command line, Python, Docker or the GitHub Action), the command or code, and the error output. The newest run log helps too; `batchvault doctor` shows where they are kept. Leave out passwords, key files and private data.
+- **Security problems:** see [SECURITY.md](SECURITY.md). Do not open a public issue.
diff --git a/action.yml b/action.yml
index 434ca36..cc169be 100644
--- a/action.yml
+++ b/action.yml
@@ -93,6 +93,12 @@ inputs:
public-key:
description: Public key; unpack and verify then require a valid signature.
default: ""
+ resume:
+ description: >-
+ Continue an interrupted pack or unpack in the output folder. Every
+ setting comes from the earlier run, so only output, password, sign-key,
+ public-key and seven-zip are used; the other inputs are ignored.
+ default: "false"
fail-on-error:
description: Fail the step if any file failed or was skipped.
default: "true"
@@ -153,6 +159,7 @@ runs:
AGAINST: ${{ inputs.against }}
SEVEN_ZIP: ${{ inputs.seven-zip }}
UPDATE: ${{ inputs.update }}
+ RESUME: ${{ inputs.resume }}
RETRIES: ${{ inputs.retries }}
ENCRYPT: ${{ inputs.encrypt }}
SIGN_KEY: ${{ inputs.sign-key }}
@@ -176,34 +183,45 @@ runs:
# with "-" is never read as an option.
args=(--json)
positional=()
- [ -n "$JOBS" ] && args+=(--jobs "$JOBS")
+ # A resumed run takes jobs and retries from the earlier run.
+ [ "$RESUME" != "true" ] && [ -n "$JOBS" ] && args+=(--jobs "$JOBS")
[ -n "$SEVEN_ZIP" ] && args+=(--7z "$SEVEN_ZIP")
case "$COMMAND" in
pack)
- while IFS= read -r line; do [ -n "$line" ] && positional+=("$line"); done <<< "$FILES"
- [ -n "$LIST_FILE" ] && args+=(--list "$LIST_FILE")
- [ -n "$OUTPUT" ] && args+=(--output "$OUTPUT")
- args+=(--engine "$ENGINE" --level "$LEVEL" --volume-size "$VOLUME_SIZE")
- [ -n "$FORMAT" ] && args+=(--format "$FORMAT")
- [ -n "$MANIFEST" ] && args+=(--manifest "$MANIFEST")
- [ -n "$ROOT" ] && args+=(--root "$ROOT")
- while IFS= read -r line; do [ -n "$line" ] && args+=(--include "$line"); done <<< "$INCLUDE"
- while IFS= read -r line; do [ -n "$line" ] && args+=(--exclude "$line"); done <<< "$EXCLUDE"
- [ "$VERIFY" = "true" ] && args+=(--verify)
- [ "$FORCE" = "true" ] && args+=(--force)
- [ "$UPDATE" = "true" ] && args+=(--update)
- [ -n "$RETRIES" ] && args+=(--retries "$RETRIES")
- [ "$ENCRYPT" = "true" ] && args+=(--encrypt)
- [ -n "$SIGN_KEY" ] && args+=(--sign-key "$SIGN_KEY")
+ if [ "$RESUME" = "true" ]; then
+ args+=(--resume)
+ [ -n "$OUTPUT" ] && args+=(--output "$OUTPUT")
+ [ -n "$SIGN_KEY" ] && args+=(--sign-key "$SIGN_KEY")
+ else
+ while IFS= read -r line; do [ -n "$line" ] && positional+=("$line"); done <<< "$FILES"
+ [ -n "$LIST_FILE" ] && args+=(--list "$LIST_FILE")
+ [ -n "$OUTPUT" ] && args+=(--output "$OUTPUT")
+ args+=(--engine "$ENGINE" --level "$LEVEL" --volume-size "$VOLUME_SIZE")
+ [ -n "$FORMAT" ] && args+=(--format "$FORMAT")
+ [ -n "$MANIFEST" ] && args+=(--manifest "$MANIFEST")
+ [ -n "$ROOT" ] && args+=(--root "$ROOT")
+ while IFS= read -r line; do [ -n "$line" ] && args+=(--include "$line"); done <<< "$INCLUDE"
+ while IFS= read -r line; do [ -n "$line" ] && args+=(--exclude "$line"); done <<< "$EXCLUDE"
+ [ "$VERIFY" = "true" ] && args+=(--verify)
+ [ "$FORCE" = "true" ] && args+=(--force)
+ [ "$UPDATE" = "true" ] && args+=(--update)
+ [ -n "$RETRIES" ] && args+=(--retries "$RETRIES")
+ [ "$ENCRYPT" = "true" ] && args+=(--encrypt)
+ [ -n "$SIGN_KEY" ] && args+=(--sign-key "$SIGN_KEY")
+ fi
out_dir="${OUTPUT:-archives}"
;;
unpack)
- positional+=("$SOURCE")
+ if [ "$RESUME" = "true" ]; then
+ args+=(--resume)
+ else
+ positional+=("$SOURCE")
+ [ "$FORCE" = "true" ] && args+=(--force)
+ [ "$UPDATE" = "true" ] && args+=(--update)
+ [ -n "$RETRIES" ] && args+=(--retries "$RETRIES")
+ fi
[ -n "$OUTPUT" ] && args+=(--output "$OUTPUT")
- [ "$FORCE" = "true" ] && args+=(--force)
- [ "$UPDATE" = "true" ] && args+=(--update)
- [ -n "$RETRIES" ] && args+=(--retries "$RETRIES")
[ -n "$PUBLIC_KEY" ] && args+=(--public-key "$PUBLIC_KEY")
out_dir="${OUTPUT:-restored}"
;;
diff --git a/.github/social-preview.png b/brand/social-preview.png
similarity index 100%
rename from .github/social-preview.png
rename to brand/social-preview.png
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..a04b806
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,5 @@
+# Documentation
+
+- **Guides:** [command line](guides/cli.md), [Python API](guides/python.md), [configuration](guides/configuration.md), [compression](guides/compression.md), [Docker](guides/docker.md) and the [GitHub Action](guides/github-action.md). Start with the [project README](../README.md) for installing and a first run.
+- **[Architecture](architecture.md) and [development](development.md):** how the repository fits together, how to work on it, and how it is tested and released.
+- **Help and security:** [SUPPORT.md](../SUPPORT.md) for questions and bug reports, [SECURITY.md](../SECURITY.md) for vulnerabilities.
diff --git a/docs/architecture.md b/docs/architecture.md
index 8e53ceb..f391d5a 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -5,32 +5,84 @@ for the original requirements.
## Layers
-The command-line interface is a thin wrapper over the Python API in `api.py`.
+The command-line interface is a thin wrapper over the Python API in `core/api/`.
The API never imports `rich`; it reports progress through an optional callback
and the CLI renders it.
```
src/batchvault/
- api.py pack, unpack, verify, inspect, list_files, list_archives
- cli.py argparse commands, exit codes, JSON / event output
- ui.py Rich progress, tables, panels (CLI only)
- notify.py webhook notifications (CLI only)
- config.py batchvault.toml / [tool.batchvault] -> option defaults
- models.py Status, FileResult, RunResult, ProgressEvent, InspectEntry
- options.py PackOptions, size parsing, formats
- errors.py exception hierarchy
- inputs.py list files (text/JSON/stdin), dedupe, conflict checks
- filters.py include / exclude via pathspec (gitignore-style)
- locate.py resolve a manifest / folder / volume / list to archives
- manifest.py manifest read / write / validate, SHA-256 helpers
- volumes.py volume naming, discovery, writer/reader, staging cleanup
- blocks.py block-parallel compression, format detection
- crypto.py optional AES-256-GCM record encryption (native engine)
- signing.py optional Ed25519 manifest signatures
- logs.py, security.py, eta.py, doctor.py
- engines/ base, native, sevenzip, registry
+ __init__.py public names (pack, unpack, ...), loaded on first use
+ __main__.py python -m batchvault
+ core/ the Python API and what it is built on
+ api/ the public operations
+ packing/ pack: run (the operation), plan (what to pack, reuse checks),
+ single (one file), finish (README, signature, source deletion)
+ unpacking/ unpack: run (the operation), restore (one file)
+ verifying.py verify
+ listing.py inspect, list_files, list_archives
+ _common.py retry, job splitting, progress and target helpers shared by them
+ config.py batchvault.toml / [tool.batchvault] -> option defaults
+ models.py Status, FileResult, RunResult, ProgressEvent, InspectEntry
+ options.py PackOptions, size parsing, formats
+ errors.py exception hierarchy
+ inputs.py list files (text/JSON/stdin), dedupe, conflict checks
+ filters.py include / exclude via pathspec (gitignore-style)
+ locate.py resolve a manifest / folder / volume / list to archives
+ levels.py --level auto: picks a level per file (signature, then extension)
+ data/ levels.toml: extension groups and the level for each
+ names.py, security.py, eta.py
+ archive/ the on-disk archive format
+ manifest.py manifest read / write / validate, SHA-256 helpers
+ volumes.py volume naming, discovery, writer/reader, staging cleanup
+ blocks.py block-parallel compression, format detection
+ crypto.py optional AES-256-GCM record encryption (native engine)
+ signing.py optional Ed25519 manifest signatures
+ resume.py run journals (.batchvault-pack.jsonl / -unpack.jsonl) for --resume
+ readme.py BATCHVAULT-README.md: how the archive was made and restored
+ templates/ the text of that README (string.Template files)
+ engines/ base (interface), native, sevenzip, registry (engine choice)
+ cli/ command-line interface
+ cli.py main(): config, logging, dispatch, exit codes
+ parser.py argparse definition of every command and option
+ commands/ one module per command: pack, unpack, verify, inspect, listing
+ (list files / archives), keys (keygen, sign), doctor
+ _common.py output, password and notify helpers shared by the commands
+ ui.py Rich progress, tables, panels (CLI only)
+ notify.py webhook notifications (CLI only)
+ logs.py per-run log file and redaction
+ doctor.py the checks behind `batchvault doctor`
```
+Dependencies point one way: `cli` uses `core`, and `core` uses `archive` and
+`engines`. `archive` and `engines` use only the small shared modules in `core`
+(`errors`, `options`, `names`, `security`) and never `core/api`. Only `__main__.py`
+and `cli` itself import `cli`.
+
+## Imports and start-up
+
+Imports are lazy: a module imports what it needs inside the function that uses it,
+and names used only in annotations sit under `TYPE_CHECKING`. A name that a class
+body, a default argument or a module constant needs when the module loads (a base
+class, a compiled regular expression, `MIB`) stays at the top of its module. This
+keeps `import batchvault` cheap and means a command pays only for the code it runs:
+`batchvault --help`, for example, loads neither `cryptography` nor `pathspec`.
+
+The public names (`pack`, `unpack`, `verify`, `inspect`, `list_files`,
+`list_archives`, `Status`, the error classes and so on) are loaded on first use
+through a `_LAZY` table and `__getattr__` in `batchvault/__init__.py`, and the same
+pattern is used by `core/api/__init__.py`. A new public name goes in `_LAZY`,
+`__all__` and the `TYPE_CHECKING` block.
+
+Settings that are data rather than logic live in files read on first use:
+`core/data/levels.toml` (which compression level suits which file type) and
+`archive/templates/*.md.tmpl` (the generated README text).
+
+## Tests
+
+`tests/unit/` mirrors this tree (`core/`, `core/api/`, `archive/`, `engines/`,
+`cli/`); `tests/integration/` has `native/` and `sevenzip/` runs; `tests/e2e/` tests
+the installed program. See [Testing in development.md](development.md#testing).
+
## Engines
| Engine | Format | Volumes |
@@ -52,7 +104,7 @@ src/batchvault/
JSON, `schema_version` 1: one entry per source file with name, `folder` (its
folder under the stored root, forward slashes, `""` when not in a subfolder;
-chosen by `inputs.assign_folders`: an explicit `--root`, else the root recorded by
+chosen by `core/inputs.py::assign_folders`: an explicit `--root`, else the root recorded by
`list files`, else the deepest folder the files share; validated so
it can never be absolute or contain `..`), size, SHA-256,
engine, format, level, volume size, `encrypted`, `mtime_ns` and an ordered
@@ -63,6 +115,28 @@ written atomically and merged across runs.
Because the manifest records every volume, a missing last volume is detected;
without a manifest only gaps in the numbering can be.
+## Resuming (`--resume`)
+
+`pack` and `unpack` keep a JSON Lines journal in the output folder (`archive/resume.py`):
+a header line with every setting of the run (no secrets), then one line per
+finished file (`pack`: its manifest entry; `unpack`: the path and SHA-256 of the
+file placed). Lines are appended and fsynced one at a time, so a crash loses at
+most the line being written, and a torn last line is ignored. The journal is
+removed when a run ends without failures.
+
+`--resume` reads the settings from the header instead of the command line (so
+there is nothing to mismatch) and refuses options that would change them.
+`pack` reuses a journaled file only after checking the source's size and mtime
+and re-hashing every volume against the journal; otherwise the file is made again
+through the usual staging and atomic install. `unpack` keeps a file that matches
+its manifest checksum, restores a missing one, redoes one it wrote that no longer
+matches, and never touches one it did not write. The manifest, signature and
+README are still written once at the end. Resume works file by file; continuing
+inside a single huge file is tracked separately (issue #9).
+
+When a worker raises or the user presses Ctrl+C, no further queued file is
+started (running ones finish), so each file is complete or absent.
+
## Encryption (`batchvault[crypto]`)
- **Native:** a header (magic, format, random salt and nonce prefix) followed by
@@ -83,7 +157,7 @@ without a manifest only gaps in the numbering can be.
Ed25519. `.sig` holds a base64 signature over the exact manifest
bytes. `unpack` and `verify` accept `--public-key` to require a valid signature
before touching any archive. The signature is checked on the exact bytes that are
-then parsed (`locate.resolve_targets` takes a manifest loader), so the file
+then parsed (`core/locate.py::resolve_targets` takes a manifest loader), so the file
cannot be swapped between the check and the use. A signature proves who made a
manifest, not that it is the latest one: replacing a manifest and its volumes
with an older, validly signed set is not detected. Changing the manifest without re-signing removes
@@ -113,8 +187,8 @@ without a terminal and without `--yes` nothing is deleted.
up from the current folder or given with `--config` / `BATCHVAULT_CONFIG`.
Precedence: command line, then config file, then built-in defaults. A file found
by searching (not named with `--config` / `BATCHVAULT_CONFIG`) is rejected if it
-sets any option in `config.RESTRICTED_OPTIONS` (programs, webhooks, trusted keys,
-password files, log files, deletion). See `docs/configuration.md`.
+sets any option in `core/config.py::RESTRICTED_OPTIONS` (programs, webhooks, trusted keys,
+password files, log files, deletion). See [the configuration guide](guides/configuration.md).
## Hardening notes
diff --git a/docs/development.md b/docs/development.md
index 065db23..f229a4e 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -7,22 +7,150 @@ uv sync
uv run pre-commit install
```
+The layout of the code is described in [architecture.md](architecture.md). Tests live in
+`tests/`, with `tests/unit/` mirroring `src/batchvault/`; see [Testing](#testing) below.
+
## Common tasks
```
uv run pytest -m "not integration and not e2e" -n auto # unit tests
-uv run pytest -m integration # needs 7-Zip
+uv run pytest -m integration # native always; 7-Zip tests need 7-Zip
uv run pytest -m "e2e and not docker" # CLI and package, end to end
uv run pytest -m docker # needs Docker
uv run ruff check . # lint
+uv run mypy # type check
```
-See [testing.md](testing.md) for what each layer covers and when CI runs it.
+`sweepito.toml` configures Sweepito for this repository, if you use it to clean caches and
+build output (`.venv/` is kept).
+
+## Imports
+
+Imports are lazy: a module imports what it needs inside the function that uses it, and names
+used only in annotations sit under `TYPE_CHECKING`. A name needed when the module loads (a
+base class, a compiled regular expression, a module constant) stays at the top of its module.
+The public names in `batchvault/__init__.py` are loaded on first use through `_LAZY` and
+`__getattr__`; add a new public name to `_LAZY`, `__all__` and the `TYPE_CHECKING` block.
+
+## Data files
+
+Two kinds of content are kept out of the code and read on first use:
+
+- `src/batchvault/core/data/levels.toml`: which compression level `--level auto` picks for
+ each kind of file. Each group has a `kind`, a `level` (0 to 9) and its file extensions; the
+ `incompressible` group must stay, because a compressed-file signature always gets its level.
+- `src/batchvault/archive/templates/*.md.tmpl`: the text of the `BATCHVAULT-README.md` written
+ next to a manifest. They are `string.Template` files, so a missing placeholder is an error.
+
+Both are packaged with the wheel.
+
+## Testing
+
+There are four layers. Each has a marker, a place in the repository and a trigger.
+
+| Layer | Where | Marker | What it proves |
+|---|---|---|---|
+| Unit | `tests/unit/` (mirrors `src/batchvault/`) | none | Modules behave correctly, fast, no external tools |
+| Integration | `tests/integration/native/`, `tests/integration/sevenzip/` | `integration` | Each engine and the API work end to end on larger, multi-volume data; the 7-Zip ones need a real 7-Zip (skipped without it) |
+| End-to-end | `tests/e2e/` | `e2e` | The installed program works the way users use it |
+| End-to-end, GitHub Action | `.github/workflows/e2e.yml` (job `action`) | n/a | `action.yml` works on real runners |
+
+Everything under `tests/e2e/` is marked `e2e` automatically. Two sub-markers allow narrower
+runs: `package` (builds the wheel and installs it into a fresh virtual environment) and
+`docker` (builds and runs the images).
+
+### What each end-to-end suite covers
+
+- **CLI** (`tests/e2e/test_cli_e2e.py`): runs the real `batchvault` command in a separate
+ process. Full journey (list, pack, inspect, verify, unpack, byte comparison) on both
+ engines, `--update`, standard-input pipelines, `--json-events`, exit codes, damaged
+ archives, restoring from one volume, config files, parallel runs, log files,
+ `--delete-source`, encryption and signing.
+- **Python package** (`tests/e2e/test_package_e2e.py`): builds the wheel, checks its
+ contents, installs it into a clean virtual environment, and checks the metadata, the
+ console script, `python -m batchvault`, the API (and that importing it does not load
+ `rich`), the message when the optional `crypto` extra is missing, and encryption and
+ signing once it is installed.
+- **Docker** (`tests/e2e/test_docker_e2e.py`): builds both images from the `Dockerfile` and
+ runs containers with a mounted folder: default command, non-root user, pack and unpack,
+ exit codes, encryption, config file, and 7-Zip only in the `-7zip` image.
+- **GitHub Action** (workflow job `action`): uses the action from this checkout (`uses: ./`)
+ on Linux, Windows and macOS: pack, verify, unpack, `update`, artifact upload and download,
+ encryption with right and wrong passwords, signing and a tampered manifest, failure
+ reporting, and 7-Zip where the runner has it. Assertions are shell and Python steps that
+ compare outputs and restored bytes (`tests/e2e/scripts/`).
+
+### Running them locally
+
+```
+uv run pytest -m "not integration and not e2e" -n auto # unit tests (default, fast)
+uv run pytest -m integration # 7-Zip tests need 7-Zip
+uv run pytest -m "e2e and not docker" # CLI + package (needs network for pip)
+uv run pytest -m "e2e and not docker and not package" # CLI only, quickest
+uv run pytest -m package # package only
+uv run pytest -m docker # needs Docker; builds the images
+```
+
+Do not use `-n` with the package tests; they share a wheel and environments.
+
+To test images you already built or pulled, skip the build:
+
+```
+BATCHVAULT_E2E_IMAGE=ghcr.io/coldsofttech/batchvault:latest \
+BATCHVAULT_E2E_IMAGE_7ZIP=ghcr.io/coldsofttech/batchvault:latest-7zip \
+uv run pytest -m docker
+```
+
+The Action job cannot run under pytest. To try it locally, use
+[act](https://github.com/nektos/act) (`act -j action`); Linux only, and the artifact steps may
+need extra setup. Otherwise push a branch and run the workflow by hand (below).
+
+### Platforms in CI
+
+Unit and integration tests (`tests.yml`) and the CLI and package end-to-end tests (`e2e.yml`)
+use the same matrix: Ubuntu with Python 3.11, 3.12, 3.13, 3.14 and 3.15 (3.15 is allowed to
+fail while it may be a pre-release), Windows with 3.11 and macOS with 3.11. The Docker tests
+run on Ubuntu only, and the Action tests run on Ubuntu, Windows and macOS with Python 3.12.
+
+### When they run in CI
+
+| Trigger | Unit | Integration | CLI + package | Docker | Action |
+|---|---|---|---|---|---|
+| Pull request | yes | yes | if `src/`, `tests/e2e/`, `action.yml`, `Dockerfile`, `pyproject.toml` changed | same | same |
+| Push to `main` | yes | yes | yes | yes | yes |
+| Weekly schedule (Mondays) | no | no | yes | yes | yes |
+| Manual (**Actions** tab, **End-to-end**, **Run workflow**) | no | no | yes | yes | yes |
+| Release tag (`v*.*.*`) | yes | yes | yes | yes | yes, and publishing waits for it |
+
+Unit and integration tests run from `tests.yml`; end-to-end suites run from `e2e.yml`, which
+`release.yml` calls before it publishes anything.
+
+### Adding a test
+
+- Behaviour of one module: a unit test in `tests/unit/`, in the folder that matches the
+ module (`core/`, `core/api/`, `archive/`, `engines/` or `cli/`).
+- An engine against real data or a real tool: `tests/integration/native/` or
+ `tests/integration/sevenzip/`, marked `integration`.
+- Something a user would notice when running the program: add it to
+ `tests/e2e/test_cli_e2e.py`.
+- Something about packaging or installation: `test_package_e2e.py`.
+- Something that only exists inside the image: `test_docker_e2e.py`.
+- A new Action input: add a step to the `action` job in `e2e.yml` that uses it and asserts on
+ the outputs or files.
+
+Keep end-to-end tests independent: each uses its own `tmp_path` and a private log folder, and
+never touches your real archives or log folder.
## Releasing
-1. Update the version in `pyproject.toml` and `src/batchvault/__init__.py`.
-2. Move the `[Unreleased]` changelog entries under the new version.
+1. Choose the version with [Semantic Versioning](https://semver.org/): a patch
+ (`0.1.1`) for fixes only, a minor (`0.2.0`) for new features, a major for breaking changes
+ to the public API or the command line. Before 1.0, a minor may also carry breaking changes;
+ say so under `### Changed`. Update the version in `pyproject.toml` and
+ `src/batchvault/__init__.py`.
+2. Move the `[Unreleased]` changelog entries under the new version, keeping the `### Added`,
+ `### Changed`, `### Deprecated`, `### Removed`, `### Fixed` and `### Security` headings that
+ are used.
3. Push a tag `vX.Y.Z`. The release workflow checks the versions, runs lint and
tests, publishes to PyPI via trusted publishing, and creates a GitHub release.
diff --git a/docs/cli.md b/docs/guides/cli.md
similarity index 70%
rename from docs/cli.md
rename to docs/guides/cli.md
index 3652ce6..1c2397a 100644
--- a/docs/cli.md
+++ b/docs/guides/cli.md
@@ -94,6 +94,8 @@ subfolder (see "Folders are kept" below).
| `--verify` | Test each archive after creating it | |
| `--encrypt` | Encrypt (password from `BATCHVAULT_PASSWORD` or `--password-file`) | |
| `--sign-key FILE` | Sign the manifest with this private key | |
+| `--resume` | Continue an interrupted run; takes every setting from the earlier run ([details](#resuming-an-interrupted-run)) | |
+| `--no-readme` | Do not write `BATCHVAULT-README.md` next to the manifest ([details](#the-readme-inside-the-archive)) | writes it |
| `--delete-source` | Delete each original after a full restore-and-compare proves the archive | |
| `--yes` | Do not ask before `--delete-source` deletes | |
| `--retries N` | Retry a file after an I/O or engine error | `3` |
@@ -109,6 +111,105 @@ subfolder (see "Folders are kept" below).
`--delete-source` never deletes without an explicit `--yes` unless you answer
`y` at a terminal prompt.
+### Resuming an interrupted run
+
+A run can stop part-way: Ctrl+C, a crash, a power cut, a dropped network drive or
+a CI time limit. While `pack` and `unpack` run, they keep a small journal in the
+output folder (`.batchvault-pack.jsonl` and `.batchvault-unpack.jsonl`). `--resume`
+continues from it instead of starting over:
+
+```
+batchvault pack --resume -o archives
+batchvault unpack --resume -o restored
+```
+
+**You give no other settings.** The first line of the journal records everything
+the run was started with (the files, engine, format, level, volume size,
+include/exclude patterns, `--update`, `--verify`, `--delete-source`, whether it
+is encrypted or signed, ...), so a resume cannot differ from the run it
+continues. Passing one of those options again with `--resume` is refused with
+an error that names it; options from a settings file are ignored.
+
+Options you may still give: `-o/--output` (where the journal is), `--jobs`,
+`--retries`, `--7z`, `--yes`, and the secrets below. Without `--jobs` and
+`--retries`, the earlier run's values are used.
+
+**Secrets are never stored.** If the earlier run was encrypted, give the password
+again (`BATCHVAULT_PASSWORD` or `--password-file`); if it signed the manifest,
+give `--sign-key` again; for `unpack --resume` after a `--public-key` run, give
+`--public-key` again. The password is checked against the files already made
+(the smallest intact archive is tested) so an archive never mixes passwords.
+
+With no journal, `--resume` stops with an error naming the file it looked for.
+Starting a run without `--resume` replaces any old journal in that folder.
+
+**`pack --resume`**
+
+- A file the journal lists is reused only if its source is unchanged (size and
+ modification time) and every volume exists with the recorded size and SHA-256.
+ Volumes are always re-hashed, never trusted by name: a power cut can leave files
+ that exist but are cut short. Anything else is made again, and the result says
+ why (`redone: volume x.bv.002 fails its checksum`).
+- The manifest, signature and `BATCHVAULT-README.md` are written at the end, as
+ in an uninterrupted run, from the reused and the new files together. The journal
+ is removed when no file failed; if one did, it stays so a later `--resume`
+ retries only that file.
+- A file is either complete or absent: volumes are built in a staging folder and
+ moved into place only when finished. Resume continues file by file; a file that
+ was in progress is made again from the start.
+- All source files must still exist. One that has gone is reported as failed.
+- `--delete-source` still deletes an original only after the archive is restored
+ and compared byte for byte (and confirmed), whether the archive was reused or
+ made now. A journal alone never leads to a deletion.
+- An archive made by resuming has the same files, volume layout and checksums as
+ an uninterrupted one for the built-in engine without encryption. An encrypted
+ archive gets a new random salt on every run, so only its restored files, not
+ its bytes, are identical.
+
+**`unpack --resume`**
+
+- A file already in the output folder that matches its checksum is kept; a
+ missing one is restored. Nothing reaches the output folder until it is verified:
+ files are restored in a staging folder first, as always.
+- A file that this run wrote earlier (it is in the journal) but that no longer
+ matches is restored again. A file that differs but was **not** written by this
+ run is never overwritten or deleted: it is reported as skipped (use `--force`
+ without `--resume` if you want it replaced). Other files in the folder are left
+ alone.
+- The journal remembers which manifest it was made for; if the manifest changed
+ since, resume is refused.
+
+`--resume` cannot be combined with `--force`, `--update` or `--dry-run`. In
+progress output and `--json`, reused files count as done (the bar and ETA do not
+treat them as new speed), and the result lists how many were reused and redone
+(`"resume": {"reused": 3, "redone": 1}`).
+
+### The README inside the archive
+
+After a successful `pack`, BatchVault writes `BATCHVAULT-README.md` next to the
+manifest. It is for whoever opens the archive later, perhaps years on and
+without knowing what BatchVault is. It says:
+
+- how the archive was made: BatchVault version, creation time, engine, format,
+ level, volume size, include/exclude patterns, whether it is encrypted or
+ signed, and the number of files, volumes and bytes;
+- how to install exactly that version (`pip`, `pipx`, `uv tool`, plus the Docker
+ image and GitHub Action);
+- the `unpack` and `verify` commands for this archive, including
+ `--password-file` or `--public-key` when they apply, and the other ways to
+ restore (a folder, a single volume, a list).
+
+It never contains a password, key or token: for an encrypted archive it only
+says a password is needed. It is not a volume, so `verify`, `inspect`,
+`unpack` and `--update` ignore it. After `--update` it is rewritten to describe
+the whole archive.
+
+The first line of the file is a marker. BatchVault only replaces a file that has
+it, so a file of your own called `BATCHVAULT-README.md` is never overwritten (you
+get a warning instead). Delete the marker line from the generated file if you
+want to keep your edits. Use `--no-readme` (or `readme = false` under `[pack]`
+in the config file) to turn it off.
+
### Folders are kept
Your folder structure is kept automatically; no option is needed. A file in a
@@ -166,7 +267,8 @@ checked against their checksums, and nothing is placed in the output folder if a
check fails. Existing files are skipped unless `--force` is given; `--update`
skips files that already match and replaces the rest.
-Other options: `-j`, `--retries`, `--7z`, `--password-file`, `--public-key FILE`
+Other options: `-j`, `--retries`, `--7z`, `--password-file`, `--public-key FILE`,
+`--resume` (continue an interrupted run; `SOURCE` is then not needed)
(require a valid manifest signature first), and `--max-output-size SIZE`.
`--max-output-size` (for example `50g`) refuses any archive that would expand
diff --git a/docs/compression.md b/docs/guides/compression.md
similarity index 100%
rename from docs/compression.md
rename to docs/guides/compression.md
diff --git a/docs/configuration.md b/docs/guides/configuration.md
similarity index 88%
rename from docs/configuration.md
rename to docs/guides/configuration.md
index bf4b6c4..77d2c1c 100644
--- a/docs/configuration.md
+++ b/docs/guides/configuration.md
@@ -54,11 +54,17 @@ include = ["*.mp4"]
exclude = ["tmp/"]
update = true
retries = 2
+readme = false # --no-readme: skip BATCHVAULT-README.md
[unpack]
update = true
```
+`resume = true` is also accepted under `[pack]` and `[unpack]`, but it makes every
+run a resume (an error when there is no journal), so it is better given on the
+command line. With `--resume`, settings from the file are ignored: the earlier
+run's own settings are used.
+
Never put passwords in the file. Use `BATCHVAULT_PASSWORD`,
`--password-file`, and `BATCHVAULT_KEY_PASSWORD` for an encrypted signing key.
A notification URL often contains a secret; prefer `BATCHVAULT_NOTIFY_URL`.
diff --git a/docs/docker.md b/docs/guides/docker.md
similarity index 77%
rename from docs/docker.md
rename to docs/guides/docker.md
index 68db6f5..8e720eb 100644
--- a/docs/docker.md
+++ b/docs/guides/docker.md
@@ -38,6 +38,27 @@ docker run --rm -v "$PWD:/data" ghcr.io/coldsofttech/batchvault:latest-7zip \
pack report.pdf -o archives --engine 7zip
```
+## Resuming an interrupted run
+
+The journal that `--resume` needs is written to the output folder, so keep that
+folder on a mounted volume (`-v "$PWD:/data"` as above) and it survives the
+container. After an interruption, run the same image with `--resume` and no other
+settings:
+
+```
+docker run --rm -v "$PWD:/data" ghcr.io/coldsofttech/batchvault \
+ pack --resume -o archives
+```
+
+```
+docker run --rm -v "$PWD:/data" ghcr.io/coldsofttech/batchvault \
+ unpack --resume -o restored
+```
+
+Give the password again with `-e BATCHVAULT_PASSWORD` or `--password-file` if the
+first run was encrypted (it is never stored). The source files must be mounted at
+the same paths as in the first run.
+
## Passwords and keys
Pass secrets through a mounted file or the environment, not on the command
diff --git a/docs/github-action.md b/docs/guides/github-action.md
similarity index 84%
rename from docs/github-action.md
rename to docs/guides/github-action.md
index de0db51..460b1f0 100644
--- a/docs/github-action.md
+++ b/docs/guides/github-action.md
@@ -36,6 +36,7 @@ PyPI instead.
| `root` | pack | Advanced: store paths relative to this folder instead of the automatic choice (folders are kept either way) | automatic |
| `force` | pack, unpack | Replace existing archives / files | `false` |
| `update` | pack, unpack | Skip unchanged files | `false` |
+| `resume` | pack, unpack | Continue an interrupted run in the output folder. Every setting comes from the earlier run, so only `output`, `password`, `sign-key`, `public-key` and `seven-zip` are used and the other inputs are ignored | `false` |
| `verify` | pack | Test each archive after creating it | `false` |
| `retries` | pack, unpack | Retries after an I/O or engine error | `3` |
| `manifest` | pack | Manifest location | |
@@ -125,6 +126,24 @@ Use the result in a later step:
run: echo "Archiving did not fully succeed: ${{ steps.vault.outputs.status }}"
```
+## Resuming after a timeout
+
+A run that was cut off (a job time limit, a cancelled workflow) can be continued
+if its output folder survived, for example because it is on a self-hosted runner,
+a cache or a mounted volume. Run the same step with `resume`:
+
+```yaml
+- uses: coldsofttech/BatchVault@v1
+ with:
+ command: pack
+ output: archives
+ resume: "true"
+ password: ${{ secrets.ARCHIVE_PASSWORD }} # again, if the first run was encrypted
+```
+
+`unpack` works the same way with `command: unpack`, `output` and `resume`. With no
+journal in the output folder the step fails with a clear message.
+
## Notes
- Inputs reach the command through environment variables and an argument list,
diff --git a/docs/python.md b/docs/guides/python.md
similarity index 70%
rename from docs/python.md
rename to docs/guides/python.md
index d16130d..5287798 100644
--- a/docs/python.md
+++ b/docs/guides/python.md
@@ -15,7 +15,7 @@ result = bv.pack(
verify=True,
)
-print(result.status) # Status.SUCCESS / FAILED / SKIPPED / PARTIAL
+print(result.status) # Status.SUCCESS / FAILED / SKIPPED / PARTIAL
for entry in result.entries:
print(entry.source.name, entry.status, entry.message)
```
@@ -45,7 +45,10 @@ bv.pack(
include=(), exclude=(), force=False, update=False, verify=False,
manifest=None, allow_archives=False, strict=False, dry_run=False,
password=None, retries=3, sign_key=None, sign_key_password=None,
- delete_source=False, confirm=None, progress=None,
+ delete_source=False,
+ readme=True, # write BATCHVAULT-README.md next to the manifest
+ resume=False, # continue an interrupted run (settings come from its journal)
+ confirm=None, progress=None,
)
```
@@ -53,6 +56,19 @@ bv.pack(
`"bzip2"` or `"store"` (built-in engine only).
- `password` encrypts the archives (the built-in engine needs
`batchvault[crypto]`). `sign_key` is the path of an Ed25519 private key.
+- `readme=True` writes `BATCHVAULT-README.md` next to the manifest: how the
+ archive was made, how to install BatchVault and the exact restore commands. It
+ never contains the password or keys. An existing file of that name is only
+ replaced if BatchVault wrote it (it starts with a marker line); otherwise a
+ warning is added to `RunResult.warnings`. Pass `readme=False` to skip it.
+- `resume=True` continues an interrupted run in `output` using the journal
+ `pack` keeps there (`.batchvault-pack.jsonl`). Every setting is taken from the
+ earlier run, so pass only `output` (plus `password` and `sign_key` if it used
+ them, and optionally `jobs`, `retries`, `seven_zip`, `confirm`, `progress`).
+ Passing another setting raises `OptionsError`; no journal, or a password, key or
+ source that does not match, raises `ResumeError`. Finished, unchanged files
+ whose volumes still match their checksums are reused
+ (`FileResult.resumed`); the rest are done again (`FileResult.redone` says why).
- `delete_source=True` deletes each original after a restore-and-compare proves
the archive. `confirm(paths) -> bool`, if given, is asked first; with no
`confirm`, deletion is automatic once the check passes.
@@ -67,8 +83,8 @@ bv.pack(
subfolder).
```python
- files = bv.list_files("D:/Media") # each entry remembers D:/ as its root
- bv.pack([f.path for f in files], "archives") # shared folder: keeps below D:/Media
+ files = bv.list_files("D:/Media") # each entry remembers D:/ as its root
+ bv.pack([f.path for f in files], "archives") # shared folder: keeps below D:/Media
# to keep "Media" too, pass the listing's root:
bv.pack([f.path for f in files], "archives", root=files[0].root)
```
@@ -76,14 +92,29 @@ bv.pack(
### unpack and verify
```python
-result = bv.unpack("archives", "restored", force=False, update=False,
- password=None, public_key=None, retries=3)
+result = bv.unpack(
+ "archives",
+ "restored",
+ force=False,
+ update=False,
+ password=None,
+ public_key=None,
+ retries=3,
+)
-result = bv.verify("archives", against="originals", password=None,
- public_key=None)
+result = bv.verify("archives", against="originals", password=None, public_key=None)
```
`source` can be a manifest, a folder, any volume, or a list of volumes.
+
+`bv.unpack(output="restored", resume=True)` continues an interrupted restore from
+the journal in the output folder (`.batchvault-unpack.jsonl`); omit `source`, it
+is taken from the earlier run. Files that are already restored and match their
+checksum are kept, missing ones are restored, and a file the earlier run wrote
+that no longer matches is restored again. A different file that the run did not
+write is never touched. `force` and `update` cannot be combined with `resume`;
+pass `public_key` again if the earlier run required a signature. `RunResult.resume`
+is true for a resumed run, with `RunResult.reused` and `RunResult.redone`.
`public_key` requires a valid manifest signature first. `max_output_size`
(`"50g"` or bytes) refuses an archive that would expand beyond that size before
anything is written; with a manifest, an archive can never expand beyond the
@@ -133,6 +164,7 @@ thread-safe.
def show(event: bv.ProgressEvent):
print(event.kind, event.source.name, event.status)
+
bv.pack(["a.bin", "b.bin"], "archives", progress=show)
```
@@ -146,6 +178,7 @@ it where you like:
```python
import logging
+
logging.getLogger("batchvault").addHandler(logging.StreamHandler())
logging.getLogger("batchvault").setLevel(logging.INFO)
```
@@ -168,6 +201,10 @@ bv.unpack("archives", "restored", public_key="batchvault.pub", password="...")
Free disk space safely, asking first:
```python
-bv.pack(files, "archives", delete_source=True,
- confirm=lambda paths: input(f"Delete {len(paths)} files? ") == "y")
+bv.pack(
+ files,
+ "archives",
+ delete_source=True,
+ confirm=lambda paths: input(f"Delete {len(paths)} files? ") == "y",
+)
```
diff --git a/docs/testing.md b/docs/testing.md
deleted file mode 100644
index 6bf9b90..0000000
--- a/docs/testing.md
+++ /dev/null
@@ -1,99 +0,0 @@
-# Testing
-
-There are four layers. Each has a marker, a place in the repository and a
-trigger.
-
-| Layer | Where | Marker | What it proves |
-|---|---|---|---|
-| Unit | `tests/test_*.py` | none | Modules behave correctly, fast, no external tools |
-| Integration | `tests/integration/` | `integration` | The 7-Zip engine and the API work with a real 7-Zip (skipped without it) |
-| End-to-end | `tests/e2e/` | `e2e` | The installed program works the way users use it |
-| End-to-end, GitHub Action | `.github/workflows/e2e.yml` (job `action`) | n/a | `action.yml` works on real runners |
-
-Everything under `tests/e2e/` is marked `e2e` automatically. Two sub-markers
-allow narrower runs: `package` (builds the wheel and installs it into a fresh
-virtual environment) and `docker` (builds and runs the images).
-
-## What each end-to-end suite covers
-
-- **CLI** (`tests/e2e/test_cli_e2e.py`): runs the real `batchvault` command in a
- separate process. Full journey (list, pack, inspect, verify, unpack, byte
- comparison) on both engines, `--update`, standard-input pipelines,
- `--json-events`, exit codes, damaged archives, restoring from one volume,
- config files, parallel runs, log files, `--delete-source`, encryption and
- signing.
-- **Python package** (`tests/e2e/test_package_e2e.py`): builds the wheel, checks
- its contents, installs it into a clean virtual environment, and checks the
- metadata, the console script, `python -m batchvault`, the API (and that
- importing it does not load `rich`), the message when the optional `crypto`
- extra is missing, and encryption and signing once it is installed.
-- **Docker** (`tests/e2e/test_docker_e2e.py`): builds both images from the
- `Dockerfile` and runs containers with a mounted folder: default command,
- non-root user, pack and unpack, exit codes, encryption, config file, and 7-Zip
- only in the `-7zip` image.
-- **GitHub Action** (workflow job `action`): uses the action from this checkout
- (`uses: ./`) on Linux, Windows and macOS: pack, verify, unpack, `update`,
- artifact upload and download, encryption with right and wrong passwords,
- signing and a tampered manifest, failure reporting, and 7-Zip where the
- runner has it. Assertions are shell and Python steps that compare outputs and
- restored bytes (`tests/e2e/scripts/`).
-
-## Running them locally
-
-```
-uv run pytest -m "not integration and not e2e" -n auto # unit tests (default, fast)
-uv run pytest -m integration # needs 7-Zip
-uv run pytest -m "e2e and not docker" # CLI + package (needs network for pip)
-uv run pytest -m "e2e and not docker and not package" # CLI only, quickest
-uv run pytest -m package # package only
-uv run pytest -m docker # needs Docker; builds the images
-```
-
-Do not use `-n` with the package tests; they share a wheel and environments.
-
-To test images you already built or pulled, skip the build:
-
-```
-BATCHVAULT_E2E_IMAGE=ghcr.io/coldsofttech/batchvault:latest \
-BATCHVAULT_E2E_IMAGE_7ZIP=ghcr.io/coldsofttech/batchvault:latest-7zip \
-uv run pytest -m docker
-```
-
-The Action job cannot run under pytest. To try it locally, use
-[act](https://github.com/nektos/act) (`act -j action`); Linux only, and the
-artifact steps may need extra setup. Otherwise push a branch and run the
-workflow by hand (below).
-
-## Platforms in CI
-
-Unit and integration tests (`tests.yml`) and the CLI and package end-to-end
-tests (`e2e.yml`) use the same matrix: Ubuntu with Python 3.11, 3.12, 3.13, 3.14
-and 3.15 (3.15 is allowed to fail while it may be a pre-release), Windows with
-3.11 and macOS with 3.11. The Docker tests run on Ubuntu only, and the Action
-tests run on Ubuntu, Windows and macOS with Python 3.12.
-
-## When they run in CI
-
-| Trigger | Unit | Integration | CLI + package | Docker | Action |
-|---|---|---|---|---|---|
-| Pull request | yes | yes | if `src/`, `tests/e2e/`, `action.yml`, `Dockerfile`, `pyproject.toml` changed | same | same |
-| Push to `main` | yes | yes | yes | yes | yes |
-| Weekly schedule (Mondays) | no | no | yes | yes | yes |
-| Manual (**Actions** tab, **End-to-end**, **Run workflow**) | no | no | yes | yes | yes |
-| Release tag (`v*.*.*`) | yes | yes | yes | yes | yes, and publishing waits for it |
-
-Unit and integration tests run from `tests.yml`; end-to-end suites run from
-`e2e.yml`, which `release.yml` calls before it publishes anything.
-
-## Adding a test
-
-- Behaviour of one module: a unit test in `tests/`.
-- Something a user would notice when running the program: add it to
- `tests/e2e/test_cli_e2e.py`.
-- Something about packaging or installation: `test_package_e2e.py`.
-- Something that only exists inside the image: `test_docker_e2e.py`.
-- A new Action input: add a step to the `action` job in `e2e.yml` that uses it
- and asserts on the outputs or files.
-
-Keep end-to-end tests independent: each uses its own `tmp_path` and a private
-log folder, and never touches your real archives or log folder.
diff --git a/pyproject.toml b/pyproject.toml
index 1d65651..ebeeec8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "batchvault"
-version = "0.1.0"
+version = "0.1.1"
description = "Batch archive, compress, split and restore files with a native or 7-Zip engine."
readme = "README.md"
license = { file = "LICENSE" }
@@ -16,13 +16,14 @@ dependencies = ["rich>=13", "pathspec>=0.12"]
crypto = ["cryptography>=42"]
[project.scripts]
-batchvault = "batchvault.cli:main"
+batchvault = "batchvault.cli.cli:main"
[dependency-groups]
dev = [
"pytest>=8",
"pytest-xdist>=3",
"ruff>=0.6",
+ "mypy>=1.10",
"pre-commit>=3",
"cryptography>=42",
]
@@ -44,6 +45,10 @@ markers = [
"docker: e2e tests that build and run the Docker images (need Docker)",
]
+[tool.mypy]
+files = ["src"]
+python_version = "3.11"
+
[tool.ruff]
line-length = 88
target-version = "py311"
diff --git a/src/batchvault/__init__.py b/src/batchvault/__init__.py
index 91433dc..e2c50d9 100644
--- a/src/batchvault/__init__.py
+++ b/src/batchvault/__init__.py
@@ -1,38 +1,43 @@
+from __future__ import annotations
+
import logging
+from typing import TYPE_CHECKING
-__version__ = "0.1.0"
+__version__ = "0.1.1"
# Library code only logs; the command-line interface decides where logs go.
logging.getLogger("batchvault").addHandler(logging.NullHandler())
-from batchvault.api import ( # noqa: E402
- inspect,
- list_archives,
- list_files,
- pack,
- unpack,
- verify,
-)
-from batchvault.errors import ( # noqa: E402
- BatchVaultError,
- ConfigError,
- EngineError,
- EngineNotFoundError,
- InputError,
- ManifestError,
- OptionsError,
- OutputExistsError,
- SignatureError,
- VolumeError,
-)
-from batchvault.models import ( # noqa: E402
- FileEntry,
- FileResult,
- InspectEntry,
- ProgressEvent,
- RunResult,
- Status,
-)
+if TYPE_CHECKING:
+ from batchvault.core.api import (
+ inspect,
+ list_archives,
+ list_files,
+ pack,
+ unpack,
+ verify,
+ )
+ from batchvault.core.errors import (
+ BatchVaultError,
+ ConfigError,
+ EngineError,
+ EngineNotFoundError,
+ InputError,
+ ManifestError,
+ OptionsError,
+ OutputExistsError,
+ ResumeError,
+ SignatureError,
+ VolumeError,
+ )
+ from batchvault.core.models import (
+ FileEntry,
+ FileResult,
+ InspectEntry,
+ ProgressEvent,
+ RunResult,
+ Status,
+ )
__all__ = [
"BatchVaultError",
@@ -47,6 +52,7 @@
"OptionsError",
"OutputExistsError",
"ProgressEvent",
+ "ResumeError",
"RunResult",
"SignatureError",
"Status",
@@ -59,3 +65,46 @@
"unpack",
"verify",
]
+
+# Public name to the module that defines it. Loaded on first use, so importing a
+# submodule such as `batchvault.core.options` does not pull in the whole API.
+_LAZY = {
+ "inspect": "batchvault.core.api",
+ "list_archives": "batchvault.core.api",
+ "list_files": "batchvault.core.api",
+ "pack": "batchvault.core.api",
+ "unpack": "batchvault.core.api",
+ "verify": "batchvault.core.api",
+ "BatchVaultError": "batchvault.core.errors",
+ "ConfigError": "batchvault.core.errors",
+ "EngineError": "batchvault.core.errors",
+ "EngineNotFoundError": "batchvault.core.errors",
+ "InputError": "batchvault.core.errors",
+ "ManifestError": "batchvault.core.errors",
+ "OptionsError": "batchvault.core.errors",
+ "OutputExistsError": "batchvault.core.errors",
+ "ResumeError": "batchvault.core.errors",
+ "SignatureError": "batchvault.core.errors",
+ "VolumeError": "batchvault.core.errors",
+ "FileEntry": "batchvault.core.models",
+ "FileResult": "batchvault.core.models",
+ "InspectEntry": "batchvault.core.models",
+ "ProgressEvent": "batchvault.core.models",
+ "RunResult": "batchvault.core.models",
+ "Status": "batchvault.core.models",
+}
+
+
+def __getattr__(name: str) -> object:
+ module = _LAZY.get(name)
+ if module is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ from importlib import import_module
+
+ value = getattr(import_module(module), name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(__all__)
diff --git a/src/batchvault/__main__.py b/src/batchvault/__main__.py
index 01eab0c..ddcc745 100644
--- a/src/batchvault/__main__.py
+++ b/src/batchvault/__main__.py
@@ -1,3 +1,3 @@
-from batchvault.cli import main
+from batchvault.cli.cli import main
raise SystemExit(main())
diff --git a/src/batchvault/api.py b/src/batchvault/api.py
deleted file mode 100644
index 694da22..0000000
--- a/src/batchvault/api.py
+++ /dev/null
@@ -1,884 +0,0 @@
-"""Public Python API. The command-line interface is a thin wrapper over this."""
-
-from __future__ import annotations
-
-import dataclasses
-import logging
-import os
-import tempfile
-import time
-import unicodedata
-from collections.abc import Callable, Iterable
-from concurrent.futures import ThreadPoolExecutor
-from pathlib import Path
-
-from batchvault import signing
-from batchvault.crypto import require_cryptography
-from batchvault.engines.base import Engine
-from batchvault.engines.registry import engine_for_volume, get_engine, option_warnings
-from batchvault.errors import (
- BatchVaultError,
- EngineError,
- InputError,
- ManifestError,
- OptionsError,
- OutputExistsError,
- SignatureError,
-)
-from batchvault.filters import PathFilter
-from batchvault.inputs import gather_sources
-from batchvault.levels import AUTO, LevelChoice, choose_level, parse_level
-from batchvault.locate import Target, resolve_targets
-from batchvault.manifest import (
- MANIFEST_NAME,
- Manifest,
- ManifestEntry,
- VolumeInfo,
- hash_file,
-)
-from batchvault.models import (
- FileEntry,
- FileResult,
- InspectEntry,
- ProgressEvent,
- RunResult,
- Status,
-)
-from batchvault.names import unsafe_on_windows
-from batchvault.options import PackOptions, format_size, parse_size
-from batchvault.volumes import STAGING_PREFIX, cleanup_staging, existing_volumes
-
-Progress = Callable[[ProgressEvent], None]
-Confirm = Callable[[list[Path]], bool]
-
-log = logging.getLogger(__name__)
-
-RETRY_BASE_DELAY = 1.0
-RETRY_MAX_DELAY = 30.0
-ENCRYPTED_MESSAGE = "Archive is encrypted; a password is required."
-
-
-def _log_result(action: str, outcome: FileResult) -> None:
- level = logging.WARNING if outcome.status == Status.FAILED else logging.INFO
- log.log(
- level, "%s %s: %s %s", action, outcome.source, outcome.status, outcome.message
- )
-
-
-def _split_jobs(jobs: int | None, files: int) -> tuple[int, int]:
- """Split one thread budget into (files in parallel, threads per file).
-
- A budget of 0 means no worker threads: one file at a time, compressed in
- the calling thread.
- """
- total = jobs or 0
- if total < 0:
- raise InputError("jobs cannot be negative")
- if total == 0:
- return 1, 0
- workers = max(1, min(files, total))
- return workers, max(1, total // workers)
-
-
-def _notify(progress: Progress | None, event: ProgressEvent) -> None:
- if progress:
- progress(event)
-
-
-def _relative(path: Path, base: Path) -> str:
- try:
- return Path(os.path.relpath(path, base)).as_posix()
- except ValueError: # different drive on Windows
- return str(path)
-
-
-def _retry(call, retries: int, retry_on: tuple[type[Exception], ...]):
- """Run ``call()``, retrying transient failures with exponential backoff."""
- attempt = 0
- while True:
- try:
- return call()
- except FileNotFoundError:
- raise
- except retry_on as exc:
- if attempt >= retries:
- raise
- delay = min(RETRY_BASE_DELAY * 2**attempt, RETRY_MAX_DELAY)
- log.warning(
- "Attempt %d failed (%s); retrying in %.1fs", attempt + 1, exc, delay
- )
- time.sleep(delay)
- attempt += 1
-
-
-def _format_name(backend: Engine, options: PackOptions) -> str:
- return "7z" if backend.name == "7zip" else options.format
-
-
-def _up_to_date(
- entry: ManifestEntry,
- stat: os.stat_result,
- backend: Engine,
- options: PackOptions,
- manifest_dir: Path,
-) -> bool:
- """True if an archive already matches the file and the requested settings."""
- if not entry.mtime_ns or entry.mtime_ns != stat.st_mtime_ns:
- return False
- if entry.size != stat.st_size:
- return False
- if (
- entry.engine != backend.name
- or entry.format != _format_name(backend, options)
- or entry.level != options.level
- or entry.volume_size != options.volume_size
- or entry.encrypted != bool(options.password)
- ):
- return False
-
- for volume in entry.volumes:
- path = manifest_dir / volume.file
- if not path.is_file() or path.stat().st_size != volume.size:
- return False
- return True
-
-
-def pack(
- files: Iterable[str | Path] = (),
- output: str | Path = "archives",
- *,
- list_file: str | Path | None = None,
- root: str | Path | None = None,
- engine: str = "auto",
- seven_zip: str | Path | None = None,
- format: str | None = None,
- level: int | str = AUTO,
- volume_size: str | int = "10m",
- jobs: int = 0,
- include: Iterable[str] = (),
- exclude: Iterable[str] = (),
- force: bool = False,
- update: bool = False,
- verify: bool = False,
- manifest: str | Path | None = None,
- allow_archives: bool = False,
- strict: bool = False,
- dry_run: bool = False,
- password: str | None = None,
- retries: int = 3,
- sign_key: str | Path | None = None,
- sign_key_password: str | None = None,
- delete_source: bool = False,
- confirm: Confirm | None = None,
- progress: Progress | None = None,
-) -> RunResult:
- """Archive files into split volumes and record them in a manifest.
-
- Each file gets its own folder ``//`` holding its volumes.
- Folders are kept: a file in a subfolder is stored under
- ``///`` and ``unpack`` puts it back there. The
- folder is relative to the one ``list files`` recorded (so the listed folder's
- name is kept), else to the deepest folder all the files share; ``root``
- overrides this with a folder of your choice.
-
- Existing archives are skipped unless ``force`` is set; with ``update`` they
- are skipped only if the file and settings are unchanged, and re-archived
- otherwise. ``jobs`` is the total thread budget; ``progress`` is called from
- worker threads. With ``strict``, settings the engine cannot honour raise an
- error instead of a warning. With ``dry_run`` nothing is written.
-
- ``password`` encrypts the archives (the native engine needs
- ``batchvault[crypto]``). ``retries`` retries a file that failed on an I/O or
- engine error. ``sign_key`` signs the manifest with an Ed25519 private key.
- ``delete_source`` deletes each original after a full restore-and-compare
- proves the archive is good (and ``confirm``, if given, agrees).
- """
- if retries < 0:
- raise OptionsError("retries cannot be negative")
- level = parse_level(level)
- auto_level = level == AUTO # each file gets the level that suits its type
-
- root_path = Path(root).expanduser().resolve() if root is not None else None
- sourced = gather_sources(
- files,
- list_file,
- path_filter=PathFilter(include, exclude),
- allow_archives=allow_archives,
- root=root_path,
- )
- sources = [path for path, _ in sourced]
- folders = dict(sourced)
- backend = get_engine(engine, seven_zip)
- warnings = option_warnings(backend, format, level)
- if strict and warnings:
- raise OptionsError(" ".join(warnings))
- if password and backend.name == "native":
- require_cryptography()
-
- private_key = None
- if sign_key and not dry_run:
- private_key = signing.load_private_key(Path(sign_key), sign_key_password)
-
- workers, per_file_jobs = _split_jobs(jobs, len(sources))
- options = PackOptions(
- level=5 if auto_level else level,
- volume_size=parse_size(volume_size),
- jobs=per_file_jobs,
- force=force,
- format=format or "xz",
- password=password,
- )
-
- out_root = Path(output).expanduser().resolve()
- manifest_path = (
- Path(manifest).expanduser().resolve() if manifest else out_root / MANIFEST_NAME
- )
- # Load first so a damaged manifest stops the run before any work is done.
- known = Manifest.load(manifest_path) if manifest_path.exists() else Manifest()
- if dry_run:
- return _plan_pack(
- backend, sources, folders, out_root, manifest_path.parent, options,
- warnings, known if update else None, auto_level,
- ) # fmt: skip
-
- cleanup_staging(out_root)
- out_root.mkdir(parents=True, exist_ok=True)
-
- log.info("Packing %d file(s) with %s into %s", len(sources), backend.name, out_root)
- total = sum(source.stat().st_size for source in sources)
- _notify(progress, ProgressEvent("plan", out_root, total, count=len(sources)))
-
- def work(source: Path) -> tuple[FileResult, ManifestEntry | None]:
- return _pack_one(
- backend, source, folders[source], out_root, manifest_path.parent,
- options, verify, progress, known if update else None, retries,
- auto_level,
- ) # fmt: skip
-
- with ThreadPoolExecutor(max_workers=workers) as executor:
- outcomes = list(executor.map(work, sources))
-
- result = RunResult(
- warnings=warnings, manifest=manifest_path, engine=backend.name
- )
- for file_result, entry in outcomes:
- result.entries.append(file_result)
- if entry:
- known.entries[entry.path] = entry
-
- if any(entry for _, entry in outcomes):
- known.save(manifest_path)
- _sign_or_clear(manifest_path, private_key, result)
-
- if delete_source:
- _delete_sources(backend, outcomes, password, confirm, result)
- return result
-
-
-def _sign_or_clear(manifest_path: Path, private_key, result: RunResult) -> None:
- if private_key is not None:
- signing.sign_manifest(manifest_path, private_key)
- return
-
- stale = signing.signature_path(manifest_path)
- if stale.exists():
- stale.unlink()
- result.warnings.append(
- "The manifest changed, so its old signature was removed. "
- "Sign it again with a private key."
- )
-
-
-def _file_options(
- options: PackOptions, source: Path, auto_level: bool
-) -> tuple[PackOptions, LevelChoice | None]:
- """The options for one file; with ``--level auto`` the level suits its type."""
- if not auto_level or options.format == "store": # store ignores the level
- return options, None
- choice = choose_level(source)
- return dataclasses.replace(options, level=choice.level), choice
-
-
-def _folder_path(base: Path, folder: str) -> Path:
- """``base`` joined with a stored forward-slash folder (``base`` if empty)."""
- return base.joinpath(*folder.split("/")) if folder else base
-
-
-def _plan_pack(
- backend: Engine,
- sources: list[Path],
- folders: dict[Path, str],
- out_root: Path,
- manifest_dir: Path,
- options: PackOptions,
- warnings: list[str],
- known: Manifest | None,
- auto_level: bool = False,
-) -> RunResult:
- result = RunResult(warnings=warnings, engine=backend.name)
- suffix = backend.capabilities.archive_suffix
-
- for source in sources:
- stat = source.stat()
- folder = folders[source]
- existing = existing_volumes(
- _folder_path(out_root, folder) / source.name, f"{source.name}{suffix}"
- )
- key = f"{folder}/{source.name}" if folder else source.name
- entry = known.entries.get(key) if known else None
- file_options, choice = _file_options(options, source, auto_level)
-
- if entry and _up_to_date(entry, stat, backend, file_options, manifest_dir):
- outcome = FileResult(
- source, Status.SKIPPED, "Would skip: up to date.", stat.st_size
- )
- elif existing and not (file_options.force or known is not None):
- outcome = FileResult(
- source,
- Status.SKIPPED,
- "Would skip: archive already exists.",
- stat.st_size,
- )
- else:
- message = "Would update" if existing else "Would archive"
- if choice:
- message += f" at level {choice.level} ({choice.kind})"
- outcome = FileResult(source, Status.SUCCESS, message + ".", stat.st_size)
- outcome.folder = folder
- outcome.level = file_options.level
- outcome.name = source.name
- result.entries.append(outcome)
- return result
-
-
-def _pack_summary(
- size: int, stored: int, volumes: int, verified: bool, choice: LevelChoice | None
-) -> str:
- """e.g. ``3 volumes, 12.3 MiB -> 4.1 MiB (33%), level 6 (text), verified``."""
- sizes = f"{format_size(size)} -> {format_size(stored)}"
- if size:
- sizes += f" ({stored / size:.0%})"
- parts = [f"{volumes} volume{'' if volumes == 1 else 's'}", sizes]
- if choice:
- parts.append(f"level {choice.level} ({choice.kind})")
- if verified:
- parts.append("verified")
- return ", ".join(parts)
-
-
-def _pack_one(
- backend: Engine,
- source: Path,
- folder: str,
- out_root: Path,
- manifest_dir: Path,
- options: PackOptions,
- verify: bool,
- progress: Progress | None,
- known: Manifest | None,
- retries: int,
- auto_level: bool = False,
-) -> tuple[FileResult, ManifestEntry | None]:
- stat = source.stat()
- size = stat.st_size
- _notify(progress, ProgressEvent("start", source, size))
- options, choice = _file_options(options, source, auto_level)
-
- entry = None
- try:
- key = f"{folder}/{source.name}" if folder else source.name
- previous = known.entries.get(key) if known is not None else None
- if previous and _up_to_date(previous, stat, backend, options, manifest_dir):
- outcome = FileResult(source, Status.SKIPPED, "Up to date.", size)
- else:
- # In update mode a changed file replaces its old archive.
- file_options = options
- if known is not None:
- file_options = dataclasses.replace(options, force=True)
-
- destination = _folder_path(out_root, folder) / source.name
- volumes = _retry(
- lambda: backend.pack(source, destination, file_options),
- retries,
- (OSError, EngineError),
- )
- entry = ManifestEntry(
- name=source.name,
- folder=folder,
- size=size,
- sha256=hash_file(source),
- engine=backend.name,
- format=_format_name(backend, options),
- level=options.level,
- volume_size=options.volume_size,
- volumes=[
- VolumeInfo(
- _relative(volume, manifest_dir),
- volume.stat().st_size,
- hash_file(volume),
- )
- for volume in volumes
- ],
- encrypted=bool(options.password),
- mtime_ns=stat.st_mtime_ns,
- )
- if verify:
- backend.test(volumes[0], password=options.password)
- stored = sum(volume.size for volume in entry.volumes)
- outcome = FileResult(
- source,
- Status.SUCCESS,
- _pack_summary(size, stored, len(volumes), verify, choice),
- size,
- volumes,
- archive=volumes[0],
- stored_size=stored,
- level=options.level,
- )
- except OutputExistsError as exc:
- outcome = FileResult(source, Status.SKIPPED, str(exc), size)
- except (BatchVaultError, OSError) as exc:
- outcome, entry = FileResult(source, Status.FAILED, str(exc), size), None
-
- outcome.folder = folder
- outcome.name = source.name
- _log_result("pack", outcome)
- _notify(progress, ProgressEvent("finish", source, size, outcome.status))
- return outcome, entry
-
-
-def _roundtrip_problem(
- backend: Engine, outcome: FileResult, entry: ManifestEntry, password: str | None
-) -> str | None:
- """Restore the archive and compare it with the original; None if identical."""
- try:
- parent = outcome.volumes[0].parent
- with tempfile.TemporaryDirectory(dir=parent, prefix=STAGING_PREFIX) as tmp:
- restored = backend.unpack(
- outcome.volumes[0],
- Path(tmp),
- force=True,
- password=password,
- max_bytes=entry.size,
- )
- if [path.name for path in restored] != [entry.name]:
- return "the archive does not contain the expected file"
- if hash_file(restored[0]) != entry.sha256:
- return "the restored file differs from the original"
- if hash_file(outcome.source) != entry.sha256:
- return "the original changed while it was being archived"
- except (BatchVaultError, OSError) as exc:
- return str(exc)
- return None
-
-
-def _delete_sources(
- backend: Engine,
- outcomes: list[tuple[FileResult, ManifestEntry | None]],
- password: str | None,
- confirm: Confirm | None,
- result: RunResult,
-) -> None:
- verified = []
- for outcome, entry in outcomes:
- if outcome.status != Status.SUCCESS or entry is None:
- continue
- problem = _roundtrip_problem(backend, outcome, entry, password)
- if problem:
- outcome.warnings.append(f"Original kept: {problem}.")
- log.warning("Original kept for %s: %s", outcome.source, problem)
- else:
- verified.append(outcome)
-
- if not verified:
- return
- if confirm is not None and not confirm([o.source for o in verified]):
- result.warnings.append("Original files were kept (deletion not confirmed).")
- return
-
- for outcome in verified:
- try:
- outcome.source.unlink()
- except OSError as exc:
- outcome.warnings.append(f"Could not delete the original: {exc}")
- else:
- outcome.message += "; original deleted"
- log.info("Deleted original %s", outcome.source)
-
-
-def _resolve(source: str | Path, public_key: str | Path | None) -> list[Target]:
- """Find the archives for ``source``.
-
- With ``public_key``, each manifest's signature is checked on the very bytes
- that are then parsed, so the file cannot be swapped in between.
- """
- if not public_key:
- return resolve_targets(source)
-
- key = Path(public_key)
-
- def load_signed(path: Path) -> Manifest:
- try:
- raw = path.read_bytes()
- except OSError as exc:
- raise ManifestError(f"Cannot read manifest {path}: {exc}") from exc
- signing.verify_manifest_bytes(raw, path, key)
- return Manifest.from_bytes(raw, path)
-
- targets = resolve_targets(source, load_signed)
- if any(target.manifest is None for target in targets):
- raise SignatureError("Signature check needs a manifest, but none was found.")
- return targets
-
-
-def _plan_event(where: Path, targets: list[Target]) -> ProgressEvent:
- # Sizes are only known from a manifest; without one the UI counts files.
- known = all(target.entry for target in targets)
- total = sum(target.entry.size for target in targets if target.entry)
- return ProgressEvent("plan", where, total if known else 0, count=len(targets))
-
-
-def unpack(
- source: str | Path,
- output: str | Path = "restored",
- *,
- force: bool = False,
- update: bool = False,
- jobs: int = 0,
- seven_zip: str | Path | None = None,
- password: str | None = None,
- public_key: str | Path | None = None,
- retries: int = 3,
- max_output_size: str | int | None = None,
- progress: Progress | None = None,
-) -> RunResult:
- """Restore files from a manifest, a folder, any volume, or a list of volumes.
-
- With a manifest, volumes are checked first and the restored file is checked
- against its recorded SHA-256 before it is moved into place. Existing files
- are skipped unless ``force`` is set; with ``update`` they are skipped only
- when they already match the archive, and replaced otherwise. ``public_key``
- requires a valid manifest signature before anything is restored.
-
- ``max_output_size`` refuses an archive that would expand beyond that many
- bytes (a decompression bomb) before anything is written. With a manifest an
- archive may never expand beyond the size recorded for its file.
- """
- if retries < 0:
- raise OptionsError("retries cannot be negative")
- max_bytes = parse_size(max_output_size) if max_output_size is not None else None
-
- targets = _resolve(source, public_key)
-
- out_root = Path(output).expanduser().resolve()
- cleanup_staging(out_root)
- workers, _ = _split_jobs(jobs, len(targets))
-
- log.info("Unpacking %d archive(s) into %s", len(targets), out_root)
- _notify(progress, _plan_event(out_root, targets))
-
- def work(target: Target) -> FileResult:
- size = target.entry.size if target.entry else 0
- _notify(progress, ProgressEvent("start", target.volume, size))
- outcome = _restore_one(
- target, out_root, force, update, seven_zip, password, retries, max_bytes
- )
- outcome.folder = target.entry.folder if target.entry else ""
- outcome.name = target.name
- _log_result("unpack", outcome)
- _notify(progress, ProgressEvent("finish", target.volume, size, outcome.status))
- return outcome
-
- with ThreadPoolExecutor(max_workers=workers) as executor:
- outcomes = list(executor.map(work, targets))
-
- result = RunResult(entries=outcomes)
- if any(target.entry is None for target in targets):
- result.warnings.append(
- "No manifest found for some archives; their checksums were not verified."
- )
- return result
-
-
-def _destination_folder(out_root: Path, folder: str) -> Path:
- """Where files stored under ``folder`` are restored, kept inside ``out_root``."""
- destination = _folder_path(out_root, folder)
- if not folder:
- return destination
-
- if os.name == "nt" and any(unsafe_on_windows(part) for part in folder.split("/")):
- raise InputError(f"The stored folder {folder!r} has names Windows cannot use.")
-
- root = out_root.resolve()
- if root not in destination.resolve().parents:
- raise InputError(f"The stored folder {folder!r} points outside the output.")
- return destination
-
-
-def _restore_one(
- target: Target,
- out_root: Path,
- force: bool,
- update: bool,
- seven_zip: str | Path | None,
- password: str | None,
- retries: int,
- max_bytes: int | None,
-) -> FileResult:
- volume = target.volume
- entry = target.entry
- try:
- limit = max_bytes
- if entry: # the manifest says how big the file is; never accept more
- limit = entry.size if limit is None else min(limit, entry.size)
-
- folder = entry.folder if entry else ""
- destination_dir = _destination_folder(out_root, folder)
- destination = destination_dir / target.name
- if destination.exists() and not force:
- if not update:
- return FileResult(
- volume,
- Status.SKIPPED,
- f"{target.name} already exists. Use force to overwrite.",
- )
- if entry is None:
- return FileResult(
- volume,
- Status.SKIPPED,
- f"{target.name} exists and there is no manifest to compare with.",
- )
- if _matches(destination, entry):
- return FileResult(volume, Status.SKIPPED, "Already up to date.")
-
- if entry:
- if entry.encrypted and not password:
- return FileResult(volume, Status.FAILED, ENCRYPTED_MESSAGE)
- problems = entry.check_volumes(target.base_dir)
- if problems:
- return FileResult(volume, Status.FAILED, "; ".join(problems))
-
- backend = engine_for_volume(volume, seven_zip)
- out_root.mkdir(parents=True, exist_ok=True)
-
- with tempfile.TemporaryDirectory(dir=out_root, prefix=STAGING_PREFIX) as tmp:
- restored = _retry(
- lambda: backend.unpack(
- volume, Path(tmp), force=True, password=password, max_bytes=limit
- ),
- retries,
- (OSError,),
- )
-
- if entry:
- names = [path.name for path in restored]
- # macOS may hand back the same name in another Unicode form.
- if [_nfc(n) for n in names] != [_nfc(entry.name)]:
- return FileResult(
- volume,
- Status.FAILED,
- f"Archive contains {names}, manifest expects {entry.name!r}.",
- )
- if hash_file(restored[0]) != entry.sha256:
- return FileResult(
- volume,
- Status.FAILED,
- f"Restored {entry.name} does not match its checksum.",
- )
-
- destination_dir.mkdir(parents=True, exist_ok=True)
- final = []
- for path in restored:
- name = entry.name if entry else path.name
- os.replace(path, destination_dir / name)
- final.append(destination_dir / name)
-
- size = sum(path.stat().st_size for path in final)
- checked = "checksum verified" if entry else "checksum not verified"
- message = f"{format_size(size)} restored, {checked}"
- return FileResult(volume, Status.SUCCESS, message, size, restored=final)
- except (BatchVaultError, OSError) as exc:
- return FileResult(volume, Status.FAILED, str(exc))
-
-
-def _nfc(name: str) -> str:
- return unicodedata.normalize("NFC", name)
-
-
-def _matches(path: Path, entry: ManifestEntry) -> bool:
- return path.is_file() and path.stat().st_size == entry.size and (
- hash_file(path) == entry.sha256
- )
-
-
-def verify(
- source: str | Path,
- *,
- against: str | Path | None = None,
- jobs: int = 0,
- seven_zip: str | Path | None = None,
- password: str | None = None,
- public_key: str | Path | None = None,
- progress: Progress | None = None,
-) -> RunResult:
- """Check archives are complete and readable, without restoring anything.
-
- With a manifest, volume sizes and checksums are checked. ``against`` is a
- folder of the original files, compared by SHA-256 (manifest required); each
- original is looked for by name in that folder and then in its subfolders.
- ``public_key`` also requires a valid manifest signature.
- """
- targets = _resolve(source, public_key)
-
- workers, _ = _split_jobs(jobs, len(targets))
- originals = Path(against).expanduser() if against else None
-
- _notify(progress, _plan_event(Path(source), targets))
-
- def work(target: Target) -> FileResult:
- size = target.entry.size if target.entry else 0
- _notify(progress, ProgressEvent("start", target.volume, size))
- outcome = _verify_one(target, originals, seven_zip, password)
- outcome.folder = target.entry.folder if target.entry else ""
- outcome.name = target.name
- _log_result("verify", outcome)
- _notify(progress, ProgressEvent("finish", target.volume, size, outcome.status))
- return outcome
-
- with ThreadPoolExecutor(max_workers=workers) as executor:
- outcomes = list(executor.map(work, targets))
-
- result = RunResult(entries=outcomes)
- if any(target.entry is None for target in targets):
- result.warnings.append(
- "No manifest found for some archives; only their internal integrity "
- "checks were run."
- )
- return result
-
-
-def _find_originals(root: Path, folder: str, name: str) -> list[Path]:
- """The original called ``name``: at its stored folder under ``root``, else
- directly in ``root``, else anywhere below it."""
- for candidate in (_folder_path(root, folder) / name, root / name):
- if candidate.is_file():
- return [candidate]
- return [path for path in root.rglob(name) if path.is_file()]
-
-
-def _verify_one(
- target: Target,
- originals: Path | None,
- seven_zip: str | Path | None,
- password: str | None,
-) -> FileResult:
- volume = target.volume
- entry = target.entry
- try:
- if entry:
- if entry.encrypted and not password:
- return FileResult(volume, Status.FAILED, ENCRYPTED_MESSAGE)
- problems = entry.check_volumes(target.base_dir)
- if problems:
- return FileResult(volume, Status.FAILED, "; ".join(problems))
-
- engine_for_volume(volume, seven_zip).test(volume, password=password)
-
- if originals and entry:
- candidates = _find_originals(originals, entry.folder, entry.name)
- if not candidates:
- message = f"Original not found under {originals}: {entry.name}"
- return FileResult(volume, Status.FAILED, message)
- if not any(hash_file(path) == entry.sha256 for path in candidates):
- message = f"{entry.name} differs from the archive."
- return FileResult(volume, Status.FAILED, message)
-
- if entry:
- message = "volumes, checksums and integrity test OK"
- else:
- message = "integrity test OK (no manifest, so checksums were not checked)"
- if originals and entry:
- message += "; matches the original"
- return FileResult(volume, Status.SUCCESS, message, entry.size if entry else 0)
- except (BatchVaultError, OSError) as exc:
- return FileResult(volume, Status.FAILED, str(exc))
-
-
-def inspect(source: str | Path) -> list[InspectEntry]:
- """Describe archives from their manifest (or file names when there is none)."""
- found = []
- for target in resolve_targets(source):
- entry = target.entry
- if entry is None:
- volumes = existing_volumes(
- target.volume.parent, target.volume.name.rsplit(".", 1)[0]
- )
- found.append(
- InspectEntry(
- name=target.name,
- path=target.volume,
- engine="7zip" if ".7z." in target.volume.name.lower() else "native",
- stored_size=sum(v.stat().st_size for v in volumes),
- volumes=len(volumes),
- )
- )
- continue
-
- paths = [target.base_dir / volume.file for volume in entry.volumes]
- found.append(
- InspectEntry(
- name=entry.name,
- folder=entry.folder,
- path=paths[0],
- engine=entry.engine,
- format=entry.format,
- level=entry.level,
- encrypted=entry.encrypted,
- size=entry.size,
- stored_size=sum(volume.size for volume in entry.volumes),
- volumes=len(paths),
- complete=all(path.is_file() for path in paths),
- )
- )
- return found
-
-
-def list_files(
- folder: str | Path,
- *,
- include: Iterable[str] = (),
- exclude: Iterable[str] = (),
-) -> list[FileEntry]:
- """Files under ``folder`` (recursively), filtered by include/exclude patterns.
-
- Patterns match against the path relative to ``folder``.
- """
- root = Path(folder).expanduser()
- if not root.is_dir():
- raise InputError(f"Folder not found: {root}")
-
- path_filter = PathFilter(include, exclude)
- # Record the folder's parent so that packing the listing keeps the folder's
- # own name (Resources/...), not just what is inside it.
- recorded = root.resolve().parent
- return [
- FileEntry(path.resolve(), path.stat().st_size, recorded)
- for path in sorted(root.rglob("*"))
- if path.is_file() and path_filter.matches(path.relative_to(root).as_posix())
- ]
-
-
-def list_archives(folder: str | Path) -> list[FileEntry]:
- """Archives under ``folder``: from its manifest if present, else by scanning."""
- root = Path(folder).expanduser()
- if not root.is_dir():
- raise InputError(f"Folder not found: {root}")
-
- try:
- targets = resolve_targets(root)
- except InputError:
- return []
- return [
- FileEntry(target.volume.resolve(), target.entry.size if target.entry else None)
- for target in targets
- ]
diff --git a/src/batchvault/archive/__init__.py b/src/batchvault/archive/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/src/batchvault/blocks.py b/src/batchvault/archive/blocks.py
similarity index 83%
rename from src/batchvault/blocks.py
rename to src/batchvault/archive/blocks.py
index 7ff7145..7a96b7a 100644
--- a/src/batchvault/blocks.py
+++ b/src/batchvault/archive/blocks.py
@@ -8,19 +8,13 @@
from __future__ import annotations
-import bz2
-import gzip
-from collections import deque
-from concurrent.futures import ThreadPoolExecutor
+from typing import TYPE_CHECKING
-from batchvault import crypto
-from batchvault.errors import EngineError, EngineNotFoundError
-from batchvault.options import FORMATS, MIB
+from batchvault.core.options import MIB
-try:
- import lzma
-except ImportError: # Python built without liblzma
- lzma = None
+if TYPE_CHECKING:
+ from collections import deque
+ from concurrent.futures import Future
BLOCK_SIZE = 16 * MIB
@@ -31,18 +25,29 @@
def check_format(fmt: str) -> None:
+ from batchvault.core.errors import EngineError, EngineNotFoundError
+ from batchvault.core.options import FORMATS
+
if fmt not in FORMATS:
raise EngineError(f"Unknown format: {fmt}")
- if fmt == "xz" and lzma is None:
- raise EngineNotFoundError(
- "This Python was built without lzma support, so the xz format "
- "is unavailable. Use --format gzip or bzip2."
- )
+ if fmt == "xz":
+ try:
+ import lzma # noqa: F401
+ except ImportError: # Python built without liblzma
+ raise EngineNotFoundError(
+ "This Python was built without lzma support, so the xz format "
+ "is unavailable. Use --format gzip or bzip2."
+ ) from None
def compress_block(fmt: str, level: int, data: bytes) -> bytes:
"""Compress one block as a complete, independent member."""
+ import bz2
+ import gzip
+
if fmt == "xz":
+ import lzma
+
return lzma.compress(data, format=lzma.FORMAT_XZ, preset=level)
if fmt == "gzip":
return gzip.compress(data, compresslevel=level, mtime=0)
@@ -54,8 +59,13 @@ def compress_block(fmt: str, level: int, data: bytes) -> bytes:
def open_decompressed(fmt: str, fileobj):
"""Wrap a file object so reading it yields the uncompressed tar stream."""
+ import bz2
+ import gzip
+
check_format(fmt)
if fmt == "xz":
+ import lzma
+
return lzma.LZMAFile(fileobj, "rb")
if fmt == "gzip":
return gzip.GzipFile(fileobj=fileobj, mode="rb")
@@ -66,6 +76,9 @@ def open_decompressed(fmt: str, fileobj):
def sniff_archive(path) -> tuple[str, bool]:
"""Identify ``(format, encrypted)`` from the first bytes of a volume."""
+ from batchvault.archive import crypto
+ from batchvault.core.errors import EngineError
+
with open(path, "rb") as handle:
head = handle.read(max(_TAR_MAGIC_OFFSET + 5, crypto.HEADER_SIZE))
@@ -108,6 +121,9 @@ def __init__(
cipher=None,
header: bytes = b"",
):
+ from collections import deque
+ from concurrent.futures import ThreadPoolExecutor
+
self._sink = sink
self._fmt = fmt
self._level = level
@@ -116,7 +132,7 @@ def __init__(
self._max_pending = jobs * 2
# With no jobs, blocks are compressed in the calling thread.
self._executor = ThreadPoolExecutor(max_workers=jobs) if jobs else None
- self._pending = deque()
+ self._pending: deque[Future[bytes]] = deque()
self._buffer = bytearray()
self._count = 0
if cipher:
diff --git a/src/batchvault/crypto.py b/src/batchvault/archive/crypto.py
similarity index 96%
rename from src/batchvault/crypto.py
rename to src/batchvault/archive/crypto.py
index 48ae7f7..1c3a628 100644
--- a/src/batchvault/crypto.py
+++ b/src/batchvault/archive/crypto.py
@@ -11,13 +11,9 @@
from __future__ import annotations
-import hashlib
import io
-import os
import struct
-from batchvault.errors import EngineError, EngineNotFoundError
-
MAGIC = b"BVENC1\x00\x00"
FORMAT_IDS = {"xz": 1, "gzip": 2, "bzip2": 3, "store": 4}
_ID_FORMATS = {value: key for key, value in FORMAT_IDS.items()}
@@ -47,6 +43,8 @@
def require_cryptography() -> None:
+ from batchvault.core.errors import EngineNotFoundError
+
try:
import cryptography # noqa: F401
except ImportError as exc:
@@ -54,6 +52,8 @@ def require_cryptography() -> None:
def make_header(fmt: str) -> bytes:
+ import os
+
return (
MAGIC
+ bytes([FORMAT_IDS[fmt], DEFAULT_LOG2_N])
@@ -67,6 +67,8 @@ def is_encrypted_header(data: bytes) -> bool:
def parse_header(data: bytes) -> tuple[str, int, bytes, bytes]:
"""Return ``(format, log2_n, salt, nonce_prefix)`` from an archive header."""
+ from batchvault.core.errors import EngineError
+
if len(data) < HEADER_SIZE or not is_encrypted_header(data):
raise EngineError("Not an encrypted BatchVault archive.")
fmt = _ID_FORMATS.get(data[len(MAGIC)])
@@ -87,6 +89,8 @@ class BlockCipher:
"""Encrypts and decrypts numbered records for one archive."""
def __init__(self, password: str, header: bytes):
+ import hashlib
+
require_cryptography()
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
@@ -120,6 +124,8 @@ def decrypt(self, index: int, sealed: bytes) -> tuple[bytes, bool]:
"""Return ``(data, is_last)``; raise EngineError on a wrong password."""
from cryptography.exceptions import InvalidTag
+ from batchvault.core.errors import EngineError
+
for last in (False, True):
try:
data = self._aes.decrypt(
@@ -157,6 +163,8 @@ def readable(self) -> bool:
return True
def _next_record(self) -> None:
+ from batchvault.core.errors import EngineError
+
head = _read_exact(self._source, _LENGTH.size)
if len(head) < _LENGTH.size:
raise EngineError("Encrypted archive is truncated (no end marker).")
diff --git a/src/batchvault/manifest.py b/src/batchvault/archive/manifest.py
similarity index 90%
rename from src/batchvault/manifest.py
rename to src/batchvault/archive/manifest.py
index 3cf878a..a6f50c0 100644
--- a/src/batchvault/manifest.py
+++ b/src/batchvault/archive/manifest.py
@@ -1,16 +1,10 @@
from __future__ import annotations
-import hashlib
-import json
-import os
-import tempfile
from dataclasses import dataclass, field
-from datetime import UTC, datetime
-from pathlib import Path
+from typing import TYPE_CHECKING
-from batchvault import __version__
-from batchvault.errors import ManifestError
-from batchvault.names import validate_folder
+if TYPE_CHECKING:
+ from pathlib import Path
SCHEMA_VERSION = 1
MANIFEST_NAME = "manifest.json"
@@ -18,6 +12,8 @@
def hash_file(path: Path) -> str:
+ import hashlib
+
digest = hashlib.sha256()
with path.open("rb") as handle:
while chunk := handle.read(_CHUNK):
@@ -26,6 +22,8 @@ def hash_file(path: Path) -> str:
def _require(data: dict, key: str, kind: type, where: str):
+ from batchvault.core.errors import ManifestError
+
value = data.get(key)
# bool is an int subclass; never accept it as a number.
if not isinstance(value, kind) or isinstance(value, bool) and kind is int:
@@ -34,6 +32,9 @@ def _require(data: dict, key: str, kind: type, where: str):
def _folder(value, where: str) -> str:
+ from batchvault.core.errors import ManifestError
+ from batchvault.core.names import validate_folder
+
if not isinstance(value, str):
raise ManifestError(f"{where}: 'folder' must be a string")
try:
@@ -50,6 +51,8 @@ class VolumeInfo:
@classmethod
def from_dict(cls, data, where: str) -> VolumeInfo:
+ from batchvault.core.errors import ManifestError
+
if not isinstance(data, dict):
raise ManifestError(f"{where}: volume entry must be an object")
return cls(
@@ -95,6 +98,8 @@ def to_dict(self) -> dict:
@classmethod
def from_dict(cls, data, index: int) -> ManifestEntry:
+ from batchvault.core.errors import ManifestError
+
where = f"entries[{index}]"
if not isinstance(data, dict):
raise ManifestError(f"{where}: must be an object")
@@ -141,6 +146,10 @@ class Manifest:
created: str = ""
def to_dict(self) -> dict:
+ from datetime import UTC, datetime
+
+ from batchvault import __version__
+
return {
"schema_version": SCHEMA_VERSION,
"created": self.created or datetime.now(UTC).isoformat(timespec="seconds"),
@@ -150,6 +159,8 @@ def to_dict(self) -> dict:
@classmethod
def from_dict(cls, data) -> Manifest:
+ from batchvault.core.errors import ManifestError
+
if not isinstance(data, dict):
raise ManifestError("Manifest must be a JSON object")
@@ -172,6 +183,10 @@ def from_dict(cls, data) -> Manifest:
@classmethod
def from_bytes(cls, raw: bytes, source: Path) -> Manifest:
"""Parse manifest bytes already read from ``source``."""
+ import json
+
+ from batchvault.core.errors import ManifestError
+
try:
data = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
@@ -184,6 +199,8 @@ def from_bytes(cls, raw: bytes, source: Path) -> Manifest:
@classmethod
def load(cls, path: Path) -> Manifest:
+ from batchvault.core.errors import ManifestError
+
try:
raw = path.read_bytes()
except OSError as exc:
@@ -192,6 +209,11 @@ def load(cls, path: Path) -> Manifest:
def save(self, path: Path) -> None:
"""Write atomically so an interrupted run cannot leave a half-written file."""
+ import json
+ import os
+ import tempfile
+ from pathlib import Path
+
path.parent.mkdir(parents=True, exist_ok=True)
text = json.dumps(self.to_dict(), indent=2, ensure_ascii=False) + "\n"
diff --git a/src/batchvault/archive/readme.py b/src/batchvault/archive/readme.py
new file mode 100644
index 0000000..f63367b
--- /dev/null
+++ b/src/batchvault/archive/readme.py
@@ -0,0 +1,191 @@
+"""The ``BATCHVAULT-README.md`` written next to the manifest after ``pack``.
+
+It tells someone who finds the archive years later, with no other context, how it
+was made, how to get BatchVault and exactly how to restore it. It never contains
+a password, key or token: it only says whether one is needed.
+
+The first line is a marker. A file is only ever replaced when it carries the
+marker, so a file of your own with the same name is never overwritten.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from functools import cache
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pathlib import Path
+ from string import Template
+
+ from batchvault.archive.manifest import Manifest
+
+README_NAME = "BATCHVAULT-README.md"
+MARKER = ""
+
+_SAFE = re.compile(r"^[\w.+@%/\\:=,-]+$")
+
+
+@dataclass(frozen=True)
+class ReadmeContext:
+ """What the README says about a run. It deliberately has no secret fields."""
+
+ engine: str
+ format: str
+ level: str # "auto" or "0".."9"
+ volume_size: int
+ encrypted: bool
+ signed: bool
+ update: bool
+ verified: bool
+ delete_source: bool
+ include: tuple[str, ...]
+ exclude: tuple[str, ...]
+ manifest_name: str
+ created: str = ""
+
+
+def _arg(value: str) -> str:
+ """Quote ``value`` for a shell only when it needs it."""
+ import shlex
+
+ if value.startswith("-"):
+ value = "./" + value
+ return value if _SAFE.match(value) else shlex.quote(value)
+
+
+def _yes(flag: bool) -> str:
+ return "yes" if flag else "no"
+
+
+def _patterns(values: tuple[str, ...]) -> str:
+ return ", ".join(f"`{value}`" for value in values) if values else "none"
+
+
+@cache
+def _template(name: str) -> Template:
+ """Load ``templates/.md.tmpl`` shipped next to this module."""
+ from importlib.resources import files
+ from string import Template
+
+ resource = files("batchvault.archive").joinpath("templates", f"{name}.md.tmpl")
+ return Template(resource.read_text(encoding="utf-8"))
+
+
+def _fragment(name: str, **values: str) -> str:
+ """Render a small template without its final newline, to use inside a line."""
+ return _template(name).substitute(values).rstrip("\n")
+
+
+def _block(name: str, **values: str) -> str:
+ """Render a template as its own paragraph, or nothing when ``name`` is empty."""
+ return f"\n{_fragment(name, **values)}\n" if name else ""
+
+
+def render_readme(context: ReadmeContext, manifest: Manifest) -> str:
+ """Return the README text for ``manifest`` created with ``context``."""
+ from datetime import UTC, datetime
+
+ from batchvault import __version__
+ from batchvault.core.options import format_size
+
+ entries = list(manifest.entries.values())
+ volumes = [volume for entry in entries for volume in entry.volumes]
+ stored = sum(volume.size for volume in volumes)
+ original = sum(entry.size for entry in entries)
+ created = context.created or datetime.now(UTC).isoformat(timespec="seconds")
+
+ needs_crypto = context.encrypted or context.signed
+ sevenzip = context.engine == "7zip"
+ version = __version__
+ manifest_name = context.manifest_name
+
+ one_volume = ""
+ if entries and entries[0].volumes:
+ first = _arg(entries[0].volumes[0].file)
+ one_volume = _fragment("one-volume", first_volume=first) + "\n"
+
+ return _template("readme").substitute(
+ marker=MARKER,
+ version=version,
+ created=created,
+ files=len(entries),
+ volumes=len(volumes),
+ original=format_size(original),
+ stored=format_size(stored),
+ manifest_name=manifest_name,
+ manifest_arg=_arg(manifest_name),
+ engine=context.engine,
+ format=context.format,
+ level=context.level,
+ volume_size=format_size(context.volume_size),
+ encrypted=_yes(context.encrypted),
+ signed=_yes(context.signed),
+ include=_patterns(context.include),
+ exclude=_patterns(context.exclude),
+ update=_yes(context.update),
+ verified=_yes(context.verified),
+ delete_source=_yes(context.delete_source),
+ encrypted_note=_block("encrypted" if context.encrypted else ""),
+ signed_note=_block(
+ "signed" if context.signed else "", manifest_name=manifest_name
+ ),
+ extra="[crypto]" if needs_crypto else "",
+ crypto=' with the "crypto" extra' if needs_crypto else "",
+ docker_variant=_fragment("sevenzip-docker", version=version)
+ if sevenzip
+ else "",
+ sevenzip_note=_block("sevenzip" if sevenzip else ""),
+ restore_intro=_fragment(
+ "restore-encrypted" if context.encrypted else "restore"
+ ),
+ sign_flag=" --public-key batchvault.pub" if context.signed else "",
+ pass_flag=" --password-file password.txt" if context.encrypted else "",
+ one_volume=one_volume,
+ )
+
+
+def _ours(path: Path) -> bool:
+ """True if ``path`` is a file BatchVault wrote (its first line is the marker)."""
+ try:
+ with path.open("rb") as handle:
+ first = handle.readline(len(MARKER) + 16)
+ except OSError:
+ return False
+ return first.decode("utf-8", errors="replace").strip() == MARKER
+
+
+def write_readme(path: Path, text: str) -> str | None:
+ """Write the README atomically; return a warning instead of raising.
+
+ Nothing is replaced unless it is a regular file with our marker, so a source
+ file that happens to have this name (stored as a folder), a file of your own
+ or a link is left alone.
+ """
+ import os
+ import tempfile
+ from pathlib import Path
+
+ try:
+ if path.is_symlink() or path.is_dir() or (path.exists() and not _ours(path)):
+ return (
+ f"{path.name} was not written: something else already uses that "
+ "name. Use --no-readme to silence this."
+ )
+
+ path.parent.mkdir(parents=True, exist_ok=True)
+ fd, name = tempfile.mkstemp(
+ dir=path.parent, prefix=path.name + ".", suffix=".tmp"
+ )
+ temp = Path(name)
+ try:
+ with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as handle:
+ handle.write(text)
+ os.replace(temp, path)
+ except BaseException:
+ temp.unlink(missing_ok=True)
+ raise
+ except OSError as exc:
+ return f"Could not write {path.name}: {exc}"
+ return None
diff --git a/src/batchvault/archive/resume.py b/src/batchvault/archive/resume.py
new file mode 100644
index 0000000..a67b839
--- /dev/null
+++ b/src/batchvault/archive/resume.py
@@ -0,0 +1,347 @@
+"""Run journals: the state that lets ``--resume`` continue an interrupted run.
+
+A journal is a JSON Lines file kept in the output folder while a run is in
+progress and removed when the run finishes without failures. The first line is a
+header holding every setting of the run, so ``--resume`` needs no other options.
+Each later line records one file that was finished (and, for ``unpack``, placed
+and verified).
+
+Lines are appended and flushed to disk one at a time, so a crash or power cut
+loses at most the line being written; a torn last line is ignored. Passwords and
+keys are never written.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.archive.manifest import ManifestEntry
+
+PACK_JOURNAL = ".batchvault-pack.jsonl"
+UNPACK_JOURNAL = ".batchvault-unpack.jsonl"
+VERSION = 1
+
+
+def _need(data: dict, key: str, kind: type | tuple[type, ...], where: str):
+ from batchvault.core.errors import ResumeError
+
+ value = data.get(key)
+ if isinstance(value, bool) and kind is not bool:
+ value = None
+ if not isinstance(value, kind):
+ raise ResumeError(f"The journal is damaged: {where} has no valid '{key}'.")
+ return value
+
+
+def _strings(data: dict, key: str, where: str) -> list[str]:
+ from batchvault.core.errors import ResumeError
+
+ values = _need(data, key, list, where)
+ if not all(isinstance(value, str) for value in values):
+ raise ResumeError(f"The journal is damaged: {where} '{key}' must be text.")
+ return values
+
+
+@dataclass(frozen=True)
+class PackSettings:
+ """Everything that shaped a ``pack`` run (no secrets)."""
+
+ sources: list[tuple[str, str]] # (absolute path, stored folder)
+ engine: str # the engine that was used: "native" or "7zip"
+ format: str | None
+ level: int | str # 0-9 or "auto"
+ volume_size: int
+ jobs: int
+ retries: int
+ include: list[str]
+ exclude: list[str]
+ manifest: str
+ update: bool
+ verify: bool
+ delete_source: bool
+ strict: bool
+ readme: bool
+ encrypted: bool
+ signed: bool
+
+ def to_dict(self) -> dict:
+ data = dict(vars(self))
+ data["sources"] = [list(pair) for pair in self.sources]
+ return data
+
+ @classmethod
+ def from_dict(cls, data: dict) -> PackSettings:
+ from batchvault.core.errors import ResumeError
+
+ where = "the settings"
+ raw_sources = _need(data, "sources", list, where)
+ sources = []
+ for pair in raw_sources:
+ if (
+ not isinstance(pair, list)
+ or len(pair) != 2
+ or not all(isinstance(part, str) for part in pair)
+ ):
+ raise ResumeError("The journal is damaged: a source is malformed.")
+ sources.append((pair[0], pair[1]))
+
+ level = data.get("level")
+ if isinstance(level, bool) or not isinstance(level, int | str):
+ raise ResumeError("The journal is damaged: the settings have no 'level'.")
+ file_format = data.get("format")
+ if file_format is not None and not isinstance(file_format, str):
+ raise ResumeError("The journal is damaged: 'format' must be text.")
+
+ return cls(
+ sources=sources,
+ engine=_need(data, "engine", str, where),
+ format=file_format,
+ level=level,
+ volume_size=_need(data, "volume_size", int, where),
+ jobs=_need(data, "jobs", int, where),
+ retries=_need(data, "retries", int, where),
+ include=_strings(data, "include", where),
+ exclude=_strings(data, "exclude", where),
+ manifest=_need(data, "manifest", str, where),
+ update=_need(data, "update", bool, where),
+ verify=_need(data, "verify", bool, where),
+ delete_source=_need(data, "delete_source", bool, where),
+ strict=_need(data, "strict", bool, where),
+ readme=_need(data, "readme", bool, where),
+ encrypted=_need(data, "encrypted", bool, where),
+ signed=_need(data, "signed", bool, where),
+ )
+
+
+@dataclass(frozen=True)
+class UnpackSettings:
+ """Everything that shaped an ``unpack`` run (no secrets)."""
+
+ source: str # absolute path of what was given as SOURCE
+ manifests: dict[str, str] # manifest path -> SHA-256 of its bytes
+ signed: bool # a public key was required
+
+ def to_dict(self) -> dict:
+ return dict(vars(self))
+
+ @classmethod
+ def from_dict(cls, data: dict) -> UnpackSettings:
+ from batchvault.core.errors import ResumeError
+
+ where = "the settings"
+ manifests = _need(data, "manifests", dict, where)
+ if not all(
+ isinstance(key, str) and isinstance(value, str)
+ for key, value in manifests.items()
+ ):
+ raise ResumeError("The journal is damaged: 'manifests' must be text.")
+ return cls(
+ source=_need(data, "source", str, where),
+ manifests=manifests,
+ signed=_need(data, "signed", bool, where),
+ )
+
+
+@dataclass
+class Loaded:
+ """A journal read back from disk."""
+
+ created: str
+ version: str # BatchVault version that wrote it
+ settings: dict
+ records: list[dict] = field(default_factory=list)
+ good_size: int = 0 # bytes up to the end of the last complete line
+
+
+def read(path: Path, command: str) -> Loaded:
+ """Read a journal; raise ``ResumeError`` if there is none or it is unusable.
+
+ A torn last line (the run died while writing it) is ignored; damage anywhere
+ else means the journal cannot be trusted.
+ """
+ import json
+
+ from batchvault.core.errors import ResumeError
+
+ try:
+ raw = path.read_bytes()
+ except FileNotFoundError:
+ raise ResumeError(
+ f"Nothing to resume: {path} does not exist. "
+ "Resume continues a run that was interrupted or had failures; "
+ f"run {command} without --resume to start."
+ ) from None
+ except OSError as exc:
+ raise ResumeError(f"Cannot read the journal {path}: {exc}") from exc
+
+ lines = raw.split(b"\n")
+ complete = lines[:-1] # the text after the last newline was never finished
+ good_size = len(raw) - len(lines[-1])
+ if not complete:
+ raise ResumeError(f"The journal {path} is empty or damaged; cannot resume.")
+
+ parsed = []
+ for number, line in enumerate(complete, 1):
+ try:
+ value = json.loads(line.decode("utf-8"))
+ except (UnicodeDecodeError, json.JSONDecodeError):
+ raise ResumeError(
+ f"The journal {path} is damaged at line {number}; cannot resume."
+ ) from None
+ if not isinstance(value, dict):
+ raise ResumeError(
+ f"The journal {path} is damaged at line {number}; cannot resume."
+ )
+ parsed.append(value)
+
+ header = parsed[0]
+ if header.get("kind") != "header" or header.get("version") != VERSION:
+ raise ResumeError(
+ f"The journal {path} was not written by this version of BatchVault "
+ "(unknown format); cannot resume."
+ )
+ if header.get("command") != command:
+ raise ResumeError(
+ f"The journal {path} belongs to {header.get('command')!r}, not {command!r}."
+ )
+ settings = header.get("settings")
+ if not isinstance(settings, dict):
+ raise ResumeError(f"The journal {path} has no settings; cannot resume.")
+
+ return Loaded(
+ created=str(header.get("created", "")),
+ version=str(header.get("batchvault_version", "")),
+ settings=settings,
+ records=parsed[1:],
+ good_size=good_size,
+ )
+
+
+def load_pack(path: Path) -> tuple[PackSettings, dict[str, ManifestEntry], Loaded]:
+ """The settings and finished files of an interrupted ``pack``."""
+ from batchvault.archive.manifest import ManifestEntry
+ from batchvault.core.errors import ResumeError
+
+ loaded = read(path, "pack")
+ settings = PackSettings.from_dict(loaded.settings)
+
+ finished: dict[str, ManifestEntry] = {}
+ for index, record in enumerate(loaded.records):
+ if record.get("kind") != "file" or not isinstance(record.get("key"), str):
+ raise ResumeError(f"The journal {path} has an unknown record.")
+ try:
+ finished[record["key"]] = ManifestEntry.from_dict(
+ record.get("entry"), index
+ )
+ except Exception as exc: # a malformed entry: the journal is not trustworthy
+ raise ResumeError(f"The journal {path} has a damaged entry: {exc}") from exc
+ return settings, finished, loaded
+
+
+def load_unpack(path: Path) -> tuple[UnpackSettings, dict[str, str], Loaded]:
+ """The settings and placed files (path -> SHA-256) of an interrupted ``unpack``."""
+ from batchvault.core.errors import ResumeError
+
+ loaded = read(path, "unpack")
+ settings = UnpackSettings.from_dict(loaded.settings)
+
+ placed: dict[str, str] = {}
+ for record in loaded.records:
+ name, digest = record.get("path"), record.get("sha256")
+ if record.get("kind") != "file" or not isinstance(name, str):
+ raise ResumeError(f"The journal {path} has an unknown record.")
+ if not isinstance(digest, str):
+ raise ResumeError(f"The journal {path} has a damaged record.")
+ placed[name] = digest
+ return settings, placed, loaded
+
+
+class Journal:
+ """An open journal that finished files are appended to (thread-safe)."""
+
+ def __init__(self, path: Path, handle):
+ import threading
+
+ self.path = path
+ self._handle = handle
+ self._lock = threading.Lock()
+ self.appended = 0 # records written by this run
+
+ @classmethod
+ def create(cls, path: Path, command: str, settings: dict) -> Journal:
+ """Start a new journal, replacing any old one."""
+ import json
+ import os
+ import tempfile
+ from datetime import UTC, datetime
+ from pathlib import Path
+
+ from batchvault import __version__
+
+ header = {
+ "kind": "header",
+ "version": VERSION,
+ "command": command,
+ "batchvault_version": __version__,
+ "created": datetime.now(UTC).isoformat(timespec="seconds"),
+ "settings": settings,
+ }
+ path.parent.mkdir(parents=True, exist_ok=True)
+ # Owner-only, written whole before the name appears (as for the manifest).
+ fd, name = tempfile.mkstemp(
+ dir=path.parent, prefix=path.name + ".", suffix=".tmp"
+ )
+ temp = Path(name)
+ try:
+ with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as handle:
+ handle.write(json.dumps(header, ensure_ascii=False) + "\n")
+ handle.flush()
+ os.fsync(handle.fileno())
+ os.replace(temp, path)
+ except BaseException:
+ temp.unlink(missing_ok=True)
+ raise
+ return cls._open(path)
+
+ @classmethod
+ def reopen(cls, path: Path, good_size: int) -> Journal:
+ """Continue an existing journal, dropping a torn last line first."""
+ import os
+
+ from batchvault.core.errors import ResumeError
+
+ try:
+ os.truncate(path, good_size)
+ except OSError as exc:
+ raise ResumeError(f"Cannot continue the journal {path}: {exc}") from exc
+ return cls._open(path)
+
+ @classmethod
+ def _open(cls, path: Path) -> Journal:
+ handle = path.open("a", encoding="utf-8", newline="\n")
+ return cls(path, handle)
+
+ def append(self, record: dict) -> None:
+ import json
+ import os
+
+ line = json.dumps(record, ensure_ascii=False) + "\n"
+ with self._lock:
+ self._handle.write(line)
+ self._handle.flush()
+ os.fsync(self._handle.fileno())
+ self.appended += 1
+
+ def close(self) -> None:
+ with self._lock:
+ if not self._handle.closed:
+ self._handle.close()
+
+ def remove(self) -> None:
+ """Close and delete the journal (the run finished)."""
+ self.close()
+ self.path.unlink(missing_ok=True)
diff --git a/src/batchvault/signing.py b/src/batchvault/archive/signing.py
similarity index 89%
rename from src/batchvault/signing.py
rename to src/batchvault/archive/signing.py
index 84ae693..aceb09a 100644
--- a/src/batchvault/signing.py
+++ b/src/batchvault/archive/signing.py
@@ -7,12 +7,10 @@
from __future__ import annotations
-import base64
-import os
-from pathlib import Path
+from typing import TYPE_CHECKING
-from batchvault.crypto import require_cryptography
-from batchvault.errors import InputError, SignatureError
+if TYPE_CHECKING:
+ from pathlib import Path
SIGNATURE_SUFFIX = ".sig"
@@ -23,6 +21,8 @@ def signature_path(manifest_path: Path) -> Path:
def _write_private(path: Path, data: bytes) -> None:
# Create the file readable by its owner only, before any key is written.
+ import os
+
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "wb") as handle:
handle.write(data)
@@ -39,6 +39,9 @@ def generate_keypair(
The private key is encrypted with ``password`` when one is given.
"""
+ from batchvault.archive.crypto import require_cryptography
+ from batchvault.core.errors import InputError
+
require_cryptography()
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
@@ -70,6 +73,9 @@ def generate_keypair(
def load_private_key(path: Path, password: str | None = None):
+ from batchvault.archive.crypto import require_cryptography
+ from batchvault.core.errors import SignatureError
+
require_cryptography()
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
@@ -91,6 +97,9 @@ def load_private_key(path: Path, password: str | None = None):
def load_public_key(path: Path):
+ from batchvault.archive.crypto import require_cryptography
+ from batchvault.core.errors import SignatureError
+
require_cryptography()
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
@@ -109,6 +118,8 @@ def load_public_key(path: Path):
def sign_manifest(manifest_path: Path, private_key) -> Path:
"""Sign the manifest with a loaded private key; return the signature file."""
+ import base64
+
signature = private_key.sign(manifest_path.read_bytes())
target = signature_path(manifest_path)
target.write_text(base64.b64encode(signature).decode("ascii") + "\n")
@@ -124,8 +135,12 @@ def verify_manifest_bytes(
signature file sits next to it. Checking the bytes that are then parsed
leaves no gap in which the file could be swapped.
"""
+ import base64
+
from cryptography.exceptions import InvalidSignature
+ from batchvault.core.errors import SignatureError
+
public_key = load_public_key(public_key_path)
sig_file = signature_path(manifest_path)
if not sig_file.is_file():
@@ -145,6 +160,8 @@ def verify_manifest_bytes(
def verify_manifest_signature(manifest_path: Path, public_key_path: Path) -> None:
"""Raise SignatureError unless the manifest has a valid signature."""
+ from batchvault.core.errors import SignatureError
+
try:
data = manifest_path.read_bytes()
except OSError as exc:
diff --git a/src/batchvault/archive/templates/encrypted.md.tmpl b/src/batchvault/archive/templates/encrypted.md.tmpl
new file mode 100644
index 0000000..3b26bfd
--- /dev/null
+++ b/src/batchvault/archive/templates/encrypted.md.tmpl
@@ -0,0 +1 @@
+**This archive is encrypted.** You need the password it was created with. It is not stored here or in the manifest, and without it the files cannot be restored.
diff --git a/src/batchvault/archive/templates/one-volume.md.tmpl b/src/batchvault/archive/templates/one-volume.md.tmpl
new file mode 100644
index 0000000..d7cec71
--- /dev/null
+++ b/src/batchvault/archive/templates/one-volume.md.tmpl
@@ -0,0 +1 @@
+- one volume (any volume finds the rest): `batchvault unpack $first_volume -o restored`
diff --git a/src/batchvault/archive/templates/readme.md.tmpl b/src/batchvault/archive/templates/readme.md.tmpl
new file mode 100644
index 0000000..85407ab
--- /dev/null
+++ b/src/batchvault/archive/templates/readme.md.tmpl
@@ -0,0 +1,74 @@
+$marker
+# About this archive
+
+This folder is a [BatchVault](https://github.com/coldsofttech/BatchVault) archive: your files, compressed and split into volumes, with a manifest that records every volume and its checksum. This file explains how it was made and how to get the files back.
+
+## How this archive was made
+
+- Created with BatchVault $version on $created
+- Files: $files, in $volumes volume(s)
+- Size: $original original, $stored stored
+- Manifest: `$manifest_name` (lists every volume with its SHA-256 checksum)
+
+Settings of the run that wrote this file:
+
+| Setting | Value |
+|---|---|
+| Engine | $engine |
+| Format | $format |
+| Compression level | $level |
+| Volume size | $volume_size |
+| Encrypted | $encrypted |
+| Signed manifest | $signed |
+| Include patterns | $include |
+| Exclude patterns | $exclude |
+| Update mode (`--update`) | $update |
+| Tested after creating | $verified |
+| Originals deleted | $delete_source |
+
+Each file's own engine, format and level are recorded in the manifest; after an `--update` run the files may have been archived at different times and with different settings.
+${encrypted_note}${signed_note}
+## Install BatchVault
+
+BatchVault needs Python 3.11.4 or later. To get exactly the version that made this archive:
+
+```
+pip install "batchvault${extra}==$version"
+```
+
+or with pipx or uv$crypto:
+
+```
+pipx install "batchvault${extra}==$version"
+uv tool install "batchvault${extra}==$version"
+```
+
+A newer version normally reads older archives too. Other ways to run it:
+
+- Docker: `ghcr.io/coldsofttech/batchvault:$version`$docker_variant
+- GitHub Action: `coldsofttech/BatchVault@v1`
+${sevenzip_note}
+## Restore the files
+
+$restore_intro
+
+```
+batchvault unpack $manifest_arg -o restored$sign_flag$pass_flag
+```
+
+The files appear in `restored/`, in the same folders they came from. Every volume and every restored file is checked against its checksum before it is placed there.
+
+To check the archive without restoring it:
+
+```
+batchvault verify $manifest_arg$sign_flag$pass_flag
+```
+
+### Other ways to restore
+
+`unpack` and `verify` accept any of these in place of the manifest:
+
+- a folder: `batchvault unpack . -o restored`
+${one_volume}- a list of volumes or manifests, one path per line: `batchvault unpack volumes.txt -o restored`
+
+Add `--public-key` to check a signature and `--password-file` for an encrypted archive. Run `batchvault unpack --help` for everything else.
diff --git a/src/batchvault/archive/templates/restore-encrypted.md.tmpl b/src/batchvault/archive/templates/restore-encrypted.md.tmpl
new file mode 100644
index 0000000..7bec05b
--- /dev/null
+++ b/src/batchvault/archive/templates/restore-encrypted.md.tmpl
@@ -0,0 +1 @@
+Put the password in a file called `password.txt` (just the password) or set the `BATCHVAULT_PASSWORD` environment variable, then run this from this folder:
diff --git a/src/batchvault/archive/templates/restore.md.tmpl b/src/batchvault/archive/templates/restore.md.tmpl
new file mode 100644
index 0000000..fce4881
--- /dev/null
+++ b/src/batchvault/archive/templates/restore.md.tmpl
@@ -0,0 +1 @@
+Run this from this folder:
diff --git a/src/batchvault/archive/templates/sevenzip-docker.md.tmpl b/src/batchvault/archive/templates/sevenzip-docker.md.tmpl
new file mode 100644
index 0000000..5138ef1
--- /dev/null
+++ b/src/batchvault/archive/templates/sevenzip-docker.md.tmpl
@@ -0,0 +1 @@
+ (use the `-7zip` variant: `:$version-7zip`)
diff --git a/src/batchvault/archive/templates/sevenzip.md.tmpl b/src/batchvault/archive/templates/sevenzip.md.tmpl
new file mode 100644
index 0000000..8ed632b
--- /dev/null
+++ b/src/batchvault/archive/templates/sevenzip.md.tmpl
@@ -0,0 +1 @@
+These volumes are 7z archives: restoring them needs 7-Zip installed (pass its location with `--7z PATH` if it is not on your PATH).
diff --git a/src/batchvault/archive/templates/signed.md.tmpl b/src/batchvault/archive/templates/signed.md.tmpl
new file mode 100644
index 0000000..82df735
--- /dev/null
+++ b/src/batchvault/archive/templates/signed.md.tmpl
@@ -0,0 +1 @@
+**The manifest is signed** (`$manifest_name.sig`). You need the matching public key to check the signature. It is not stored here; ask whoever made the archive.
diff --git a/src/batchvault/volumes.py b/src/batchvault/archive/volumes.py
similarity index 90%
rename from src/batchvault/volumes.py
rename to src/batchvault/archive/volumes.py
index 9a80c8a..daa5a50 100644
--- a/src/batchvault/volumes.py
+++ b/src/batchvault/archive/volumes.py
@@ -1,15 +1,12 @@
from __future__ import annotations
import io
-import os
import re
-import shutil
-import time
-import unicodedata
-from pathlib import Path
+from typing import TYPE_CHECKING
-from batchvault.errors import VolumeError
-from batchvault.security import restrict_permissions
+if TYPE_CHECKING:
+ from pathlib import Path
+ from typing import BinaryIO
VOLUME_RE = re.compile(r"^(?P .+\.(?:7z|bv))\.(?P\d{3,})$", re.IGNORECASE)
@@ -24,6 +21,8 @@ def split_volume_name(name: str) -> tuple[str, int] | None:
def _same_name(a: str, b: str) -> bool:
"""Compare names ignoring case and Unicode form (macOS decomposes accents)."""
+ import unicodedata
+
return (
unicodedata.normalize("NFC", a).casefold()
== unicodedata.normalize("NFC", b).casefold()
@@ -50,12 +49,18 @@ def find_volumes(path: Path) -> list[Path]:
volume or if a volume inside the numbered range is missing. A missing
*trailing* volume cannot be detected without a manifest.
"""
+ from batchvault.core.errors import VolumeError
+
parsed = split_volume_name(path.name)
if parsed is None:
raise VolumeError(f"Not an archive volume: {path}")
volumes = existing_volumes(path.parent, parsed[0])
- numbers = [split_volume_name(volume.name)[1] for volume in volumes]
+ numbers = []
+ for volume in volumes:
+ info = split_volume_name(volume.name)
+ if info:
+ numbers.append(info[1])
missing = sorted(set(range(1, max(numbers) + 1)) - set(numbers))
if missing:
@@ -76,6 +81,11 @@ def cleanup_staging(root: Path, max_age: float = 86400.0) -> int:
Looks through the whole tree under ``root``. Only folders untouched for
``max_age`` seconds are removed, so a run in progress is never disturbed.
"""
+ import os
+ import shutil
+ import time
+ from pathlib import Path
+
if not root.is_dir():
return 0
@@ -101,6 +111,10 @@ def install_volumes(built: list[Path], dest_dir: Path, existing: list[Path]):
Old volumes are removed only here, after the new set was built completely.
"""
+ import os
+
+ from batchvault.core.security import restrict_permissions
+
for old in existing:
old.unlink()
@@ -119,10 +133,11 @@ class VolumeWriter:
"""Write-only file object that rolls over to a new volume every N bytes."""
def __init__(self, directory: Path, base: str, volume_size: int):
+
self._directory = directory
self._base = base
self._volume_size = volume_size
- self._file = None
+ self._file: BinaryIO | None = None
self._room = 0
self.volumes: list[Path] = []
@@ -134,6 +149,7 @@ def write(self, data) -> int:
if self._room == 0:
self._next_volume()
chunk = view[: self._room]
+ assert self._file is not None # opened by _next_volume
self._file.write(chunk)
self._room -= len(chunk)
view = view[len(chunk) :]
diff --git a/src/batchvault/cli.py b/src/batchvault/cli.py
deleted file mode 100644
index 772107c..0000000
--- a/src/batchvault/cli.py
+++ /dev/null
@@ -1,806 +0,0 @@
-"""Command-line interface: a thin wrapper over :mod:`batchvault.api`.
-
-Exit codes: 0 success, 1 some files failed, were skipped or the checks failed,
-2 usage or configuration error, 3 unexpected error, 130 interrupted.
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import logging
-import os
-import sys
-import threading
-from contextlib import contextmanager
-from pathlib import Path
-
-from rich.console import Console
-from rich.markup import escape
-
-from batchvault import __version__, api, notify, signing
-from batchvault.config import (
- defaults_for,
- find_config,
- load_config,
- reject_restricted,
-)
-from batchvault.doctor import passed, run_checks
-from batchvault.engines.registry import ENGINE_CHOICES
-from batchvault.errors import BatchVaultError, ConfigError, InputError
-from batchvault.inputs import ROOT_HEADER, looks_like_archive
-from batchvault.levels import AUTO, level_argument
-from batchvault.logs import LOG_LEVELS, close_logging, setup_logging
-from batchvault.manifest import Manifest
-from batchvault.models import RunResult, Status
-from batchvault.options import FORMATS
-from batchvault.security import redact, redact_arguments
-
-log = logging.getLogger(__name__)
-
-ENV_PASSWORD = "BATCHVAULT_PASSWORD"
-ENV_KEY_PASSWORD = "BATCHVAULT_KEY_PASSWORD"
-ENV_NOTIFY_URL = "BATCHVAULT_NOTIFY_URL"
-ENV_CONFIG = "BATCHVAULT_CONFIG"
-
-
-def _parents() -> dict[str, argparse.ArgumentParser]:
- parents = {name: argparse.ArgumentParser(add_help=False) for name in (
- "common", "jobs", "seven", "patterns", "run", "retry", "public",
- )} # fmt: skip
-
- common = parents["common"]
- common.add_argument(
- "--json", action="store_true", help="Print the result as JSON on stdout."
- )
- common.add_argument(
- "-q", "--quiet", action="store_true", help="No progress or summary output."
- )
- common.add_argument("--log-file", help="Write the log here instead of the default.")
- common.add_argument(
- "--log-level", choices=LOG_LEVELS, default="INFO", help="Log level."
- )
- common.add_argument(
- "--no-log", action="store_true", help="Do not write a log file."
- )
- common.add_argument(
- "--config", metavar="FILE", help="Config file (default: batchvault.toml)."
- )
-
- parents["jobs"].add_argument(
- "-j",
- "--jobs",
- type=int,
- default=0,
- metavar="N",
- help="Worker threads to use; 0 means none, one file at a time (default: 0).",
- )
-
- parents["seven"].add_argument(
- "--7z", dest="seven_zip", metavar="PATH", help="Path to the 7-Zip executable."
- )
-
- patterns = parents["patterns"]
- patterns.add_argument(
- "--include",
- action="append",
- default=[],
- metavar="GLOB",
- help="Only keep matching paths (repeatable; @FILE reads patterns from a file).",
- )
- patterns.add_argument(
- "--exclude",
- action="append",
- default=[],
- metavar="GLOB",
- help="Drop matching paths (repeatable; wins over --include).",
- )
-
- run = parents["run"]
- run.add_argument(
- "--json-events",
- action="store_true",
- help="Print one JSON event per line on stdout while running.",
- )
- run.add_argument(
- "--notify",
- metavar="URL",
- help=f"POST a JSON summary to this webhook (or set {ENV_NOTIFY_URL}).",
- )
- run.add_argument(
- "--notify-on",
- choices=notify.NOTIFY_ON,
- default="failure",
- help="When to notify (default: failure).",
- )
- run.add_argument(
- "--password-file",
- metavar="FILE",
- help=f"Read the archive password from this file (or set {ENV_PASSWORD}).",
- )
-
- parents["retry"].add_argument(
- "--retries",
- type=int,
- default=3,
- metavar="N",
- help="Retry a file this many times after an I/O or engine error (default: 3).",
- )
-
- parents["public"].add_argument(
- "--public-key",
- metavar="FILE",
- help="Require a valid manifest signature made with the matching private key.",
- )
- return parents
-
-
-def _configure(subparser, name: str, config: dict | None) -> None:
- """Apply defaults from the config file to one command."""
- if not config:
- return
- actions = {action.dest: action for action in subparser._actions}
- defaults = defaults_for(config, name, actions)
- if defaults:
- subparser.set_defaults(**defaults)
-
-
-def build_parser(config: dict | None = None) -> argparse.ArgumentParser:
- p = _parents()
- common, jobs, seven, patterns, run, retry, public = (
- p["common"], p["jobs"], p["seven"], p["patterns"], p["run"], p["retry"],
- p["public"],
- ) # fmt: skip
-
- parser = argparse.ArgumentParser(
- prog="batchvault",
- description="Batch archive, compress, split and restore files.",
- )
- parser.add_argument(
- "--version", action="version", version=f"%(prog)s {__version__}"
- )
- commands = parser.add_subparsers(dest="command", required=True, metavar="COMMAND")
-
- pack = commands.add_parser(
- "pack",
- parents=[common, jobs, seven, patterns, run, retry],
- help="Compress files into split volumes.",
- description="Compress files into split volumes and write a manifest.",
- epilog="Example: batchvault pack --list archive_list.txt -o archives",
- )
- pack.add_argument("files", nargs="*", metavar="FILE", help="Files to archive.")
- pack.add_argument(
- "--list",
- metavar="FILE",
- help="Text or JSON list of files to archive ('-' reads standard input).",
- )
- pack.add_argument("-o", "--output", default="archives", help="Output folder.")
- pack.add_argument("--engine", choices=ENGINE_CHOICES, default="auto")
- pack.add_argument(
- "--format",
- choices=FORMATS,
- help="Native engine compression (default: xz). 7-Zip always writes 7z.",
- )
- pack.add_argument(
- "--level",
- type=level_argument,
- choices=[*range(10), AUTO],
- default=AUTO,
- metavar="0-9|auto",
- help="Compression level, or auto to pick one per file type (default: auto).",
- )
- pack.add_argument(
- "--volume-size",
- default="10m",
- metavar="SIZE",
- help="Volume size, e.g. 500k, 10m, 2g (default: 10m).",
- )
- pack.add_argument("--manifest", metavar="PATH", help="Manifest file location.")
- pack.add_argument(
- "--root",
- metavar="DIR",
- help="Advanced: store paths relative to DIR. By default folders are kept "
- "automatically (relative to the folder you listed, or the folder your "
- "files share).",
- )
- pack.add_argument("--force", action="store_true", help="Replace existing archives.")
- pack.add_argument(
- "--update",
- action="store_true",
- help="Skip unchanged files and re-archive changed ones.",
- )
- pack.add_argument(
- "--verify", action="store_true", help="Test each archive after creating it."
- )
- pack.add_argument(
- "--encrypt",
- action="store_true",
- help=f"Encrypt with the password from {ENV_PASSWORD} or --password-file.",
- )
- pack.add_argument(
- "--sign-key",
- metavar="FILE",
- help=f"Sign the manifest with this private key (password: {ENV_KEY_PASSWORD}).",
- )
- pack.add_argument(
- "--delete-source",
- action="store_true",
- help="Delete each original after proving the archive restores identically.",
- )
- pack.add_argument(
- "--yes", action="store_true", help="Do not ask before --delete-source deletes."
- )
- pack.add_argument(
- "--allow-archives",
- action="store_true",
- help="Allow archiving BatchVault/7-Zip volumes and manifests.",
- )
- pack.add_argument(
- "--strict",
- action="store_true",
- help="Treat settings the engine cannot honour as errors.",
- )
- pack.add_argument(
- "--dry-run",
- action="store_true",
- help="Show what would be archived without writing anything.",
- )
- pack.set_defaults(handler=cmd_pack)
- _configure(pack, "pack", config)
-
- unpack = commands.add_parser(
- "unpack",
- parents=[common, jobs, seven, run, retry, public],
- help="Restore files from archives.",
- description="Restore files from a manifest, folder, volume or list file.",
- epilog="Example: batchvault unpack archives -o restored",
- )
- unpack.add_argument(
- "source", metavar="SOURCE", help="Manifest, folder, volume or list file."
- )
- unpack.add_argument("-o", "--output", default="restored", help="Output folder.")
- unpack.add_argument(
- "--force", action="store_true", help="Overwrite existing files."
- )
- unpack.add_argument(
- "--update",
- action="store_true",
- help="Skip files that already match; replace the rest.",
- )
- unpack.add_argument(
- "--max-output-size",
- metavar="SIZE",
- help="Refuse an archive that would expand beyond this size, per file "
- "(e.g. 50g). Use it when restoring archives you do not trust.",
- )
- unpack.set_defaults(handler=cmd_unpack)
- _configure(unpack, "unpack", config)
-
- verify = commands.add_parser(
- "verify",
- parents=[common, jobs, seven, run, public],
- help="Check that archives are complete and intact.",
- description="Check archives without restoring them.",
- )
- verify.add_argument(
- "source", metavar="SOURCE", help="Manifest, folder, volume or list file."
- )
- verify.add_argument(
- "--against",
- metavar="FOLDER",
- help="Also compare with the original files in this folder (and below).",
- )
- verify.set_defaults(handler=cmd_verify)
- _configure(verify, "verify", config)
-
- inspect = commands.add_parser(
- "inspect",
- parents=[common],
- help="Show what is in an archive set.",
- description="Show sizes, ratios, engine, format and completeness.",
- )
- inspect.add_argument(
- "source", metavar="SOURCE", help="Manifest, folder, volume or list file."
- )
- inspect.set_defaults(handler=cmd_inspect)
- _configure(inspect, "inspect", config)
-
- listing = commands.add_parser("list", help="List files or archives.")
- kinds = listing.add_subparsers(dest="kind", required=True, metavar="KIND")
-
- files = kinds.add_parser(
- "files",
- parents=[common, patterns],
- help="List the files in a folder.",
- description="List files (JSON on stdout unless -o is given).",
- )
- files.add_argument("folder")
- files.add_argument("-o", "--output", metavar="FILE", help="Write .json or .txt.")
- files.set_defaults(handler=cmd_list_files)
- _configure(files, "list-files", config)
-
- archives = kinds.add_parser(
- "archives",
- parents=[common],
- help="List the archives in a folder.",
- description="List archives (JSON on stdout unless -o is given).",
- )
- archives.add_argument("folder")
- archives.add_argument("-o", "--output", metavar="FILE", help="Write .json or .txt.")
- archives.set_defaults(handler=cmd_list_archives)
- _configure(archives, "list-archives", config)
-
- keygen = commands.add_parser(
- "keygen",
- parents=[common],
- help="Create a key pair for signing manifests.",
- description=(
- "Create an Ed25519 key pair (needs batchvault[crypto]). The private "
- f"key is encrypted if {ENV_KEY_PASSWORD} is set."
- ),
- )
- keygen.add_argument("--private", default="batchvault.key", metavar="FILE")
- keygen.add_argument("--public", default="batchvault.pub", metavar="FILE")
- keygen.add_argument("--force", action="store_true", help="Replace existing keys.")
- keygen.set_defaults(handler=cmd_keygen)
- _configure(keygen, "keygen", config)
-
- sign = commands.add_parser(
- "sign",
- parents=[common],
- help="Sign a manifest.",
- description="Write a detached signature next to a manifest.",
- )
- sign.add_argument("manifest", metavar="MANIFEST")
- sign.add_argument("--key", required=True, metavar="FILE", help="Private key.")
- sign.set_defaults(handler=cmd_sign)
- _configure(sign, "sign", config)
-
- doctor = commands.add_parser(
- "doctor",
- parents=[common, seven],
- help="Check this machine's setup.",
- description="Check Python modules, 7-Zip and the log folder.",
- )
- doctor.set_defaults(handler=cmd_doctor)
- _configure(doctor, "doctor", config)
-
- return parser
-
-
-def _consoles() -> tuple[Console, Console]:
- return Console(), Console(stderr=True)
-
-
-class EventStream:
- """Progress callback that prints one JSON object per line on stdout."""
-
- def __init__(self):
- self._lock = threading.Lock()
-
- def __call__(self, event) -> None:
- self._write(event.to_dict())
-
- def result(self, result: RunResult) -> None:
- self._write({"event": "result", **result.to_dict()})
-
- def _write(self, data: dict) -> None:
- with self._lock:
- print(json.dumps(data, ensure_ascii=False), flush=True)
-
-
-@contextmanager
-def _progress(args, label: str, err: Console):
- from batchvault import ui
-
- if args.json_events:
- yield EventStream()
- else:
- with ui.RunProgress(err, label, enabled=not args.quiet) as progress:
- yield progress
-
-
-def _check_output_flags(args) -> None:
- if args.json and args.json_events:
- raise InputError("Use either --json or --json-events, not both.")
-
-
-def _read_password(args) -> str | None:
- path = getattr(args, "password_file", None)
- if path:
- try:
- lines = Path(path).expanduser().read_text(encoding="utf-8").splitlines()
- except OSError as exc:
- raise InputError(f"Cannot read password file {path}: {exc}") from exc
- if not lines or not lines[0]:
- raise InputError(f"Password file {path} is empty.")
- return lines[0]
- return os.environ.get(ENV_PASSWORD) or None
-
-
-def _notify_url(args) -> str | None:
- url = args.notify or os.environ.get(ENV_NOTIFY_URL)
- if not url:
- return None
-
- notify.validate_url(url)
- if notify.is_insecure(url):
- _, err = _consoles()
- err.print(
- "[yellow]Warning: the notification URL uses http://, so the summary "
- "(file names and paths) is sent unencrypted. Prefer https://.[/yellow]"
- )
- return url
-
-
-def _finish(
- args, command: str, result: RunResult, title: str | None, progress, url
-) -> int:
- from batchvault import ui
-
- out, err = _consoles()
- if args.json_events:
- progress.result(result)
- elif args.json:
- print(json.dumps(result.to_dict(), indent=2, ensure_ascii=False))
- elif not args.quiet:
- ui.print_summary(out, result, title, command)
-
- if url and notify.should_notify(args.notify_on, result):
- problem = notify.send(url, notify.build_payload(command, result))
- if problem:
- log.warning("Notification failed: %s", redact(problem))
- err.print(
- f"[yellow]Warning: notification failed ({escape(problem)}).[/yellow]"
- )
-
- return 0 if result.status == Status.SUCCESS else 1
-
-
-def _confirm_deletion(args, progress):
- def confirm(paths: list[Path]) -> bool:
- if args.yes:
- return True
- quiet = args.json or args.json_events or args.quiet
- if quiet or not sys.stdin.isatty():
- return False # never delete without an explicit yes
-
- if hasattr(progress, "pause"):
- progress.pause()
- print(f"{len(paths)} original file(s) were archived and verified:")
- for path in paths[:10]:
- print(f" {path}")
- if len(paths) > 10:
- print(f" ... and {len(paths) - 10} more")
- answer = input("Delete them? [y/N] ")
- return answer.strip().lower() in ("y", "yes")
-
- return confirm
-
-
-def _describe_level(level) -> str:
- return "auto (chosen for each file's type)" if level == AUTO else str(level)
-
-
-def _describe_sources(args) -> str:
- parts = []
- if args.files:
- parts.append(f"{len(args.files)} given")
- if args.list:
- parts.append("list: " + ("standard input" if args.list == "-" else args.list))
- return ", ".join(parts) or "-"
-
-
-def cmd_pack(args) -> int:
- from batchvault import ui
-
- _check_output_flags(args)
- if not args.files and not args.list:
- raise InputError("Give at least one FILE or use --list.")
-
- password = None
- if args.encrypt:
- password = _read_password(args)
- if not password:
- raise InputError(
- f"--encrypt needs a password: set {ENV_PASSWORD} "
- "or use --password-file."
- )
- url = _notify_url(args)
-
- out, err = _consoles()
- if not (args.quiet or args.json or args.json_events):
- ui.print_run_info(
- out,
- "BatchVault pack",
- [
- ("Engine", args.engine),
- ("Format", args.format or "xz (native engine)"),
- ("Level", _describe_level(args.level)),
- ("Volume size", args.volume_size),
- ("Jobs", str(args.jobs) if args.jobs else "0 (sequential)"),
- ("Source files", _describe_sources(args)),
- (
- "Folders",
- f"kept, relative to {Path(args.root).resolve()}"
- if args.root
- else "kept",
- ),
- ("Archives folder", str(Path(args.output).resolve())),
- ],
- )
-
- with _progress(args, "Archiving", err) as progress:
- result = api.pack(
- args.files,
- args.output,
- list_file=args.list,
- root=args.root,
- engine=args.engine,
- seven_zip=args.seven_zip,
- format=args.format,
- level=args.level,
- volume_size=args.volume_size,
- jobs=args.jobs,
- include=args.include,
- exclude=args.exclude,
- force=args.force,
- update=args.update,
- verify=args.verify,
- manifest=args.manifest,
- allow_archives=args.allow_archives,
- strict=args.strict,
- dry_run=args.dry_run,
- password=password,
- retries=args.retries,
- sign_key=args.sign_key,
- sign_key_password=os.environ.get(ENV_KEY_PASSWORD) or None,
- delete_source=args.delete_source and not args.dry_run,
- confirm=_confirm_deletion(args, progress),
- progress=progress,
- )
-
- title = None # the default title names the source files and archives
- if args.dry_run:
- title = "Pack plan (dry run): source files -> archives"
- if not (args.quiet or args.json or args.json_events):
- out.print("[bold]Dry run:[/bold] nothing was written.")
- return _finish(args, "pack", result, title, progress, url)
-
-
-def cmd_unpack(args) -> int:
- from batchvault import ui
-
- _check_output_flags(args)
- password = _read_password(args)
- url = _notify_url(args)
-
- out, err = _consoles()
- if not (args.quiet or args.json or args.json_events):
- ui.print_run_info(
- out,
- "BatchVault unpack",
- [
- ("Archives", args.source),
- ("Restore files to", str(Path(args.output).resolve())),
- ("Existing files", "overwrite" if args.force else "skip"),
- ("Jobs", str(args.jobs) if args.jobs else "0 (sequential)"),
- ],
- )
- with _progress(args, "Restoring", err) as progress:
- result = api.unpack(
- args.source,
- args.output,
- force=args.force,
- update=args.update,
- jobs=args.jobs,
- seven_zip=args.seven_zip,
- password=password,
- public_key=args.public_key,
- retries=args.retries,
- max_output_size=args.max_output_size,
- progress=progress,
- )
- return _finish(args, "unpack", result, None, progress, url)
-
-
-def cmd_verify(args) -> int:
- from batchvault import ui
-
- _check_output_flags(args)
- password = _read_password(args)
- url = _notify_url(args)
-
- out, err = _consoles()
- if not (args.quiet or args.json or args.json_events):
- ui.print_run_info(
- out,
- "BatchVault verify",
- [
- ("Archives", args.source),
- ("Original files", args.against or "not compared"),
- ("Jobs", str(args.jobs) if args.jobs else "0 (sequential)"),
- ],
- )
- with _progress(args, "Verifying", err) as progress:
- result = api.verify(
- args.source,
- against=args.against,
- jobs=args.jobs,
- seven_zip=args.seven_zip,
- password=password,
- public_key=args.public_key,
- progress=progress,
- )
- return _finish(args, "verify", result, None, progress, url)
-
-
-def cmd_inspect(args) -> int:
- from batchvault import ui
-
- entries = api.inspect(args.source)
- if args.json:
- print(json.dumps([e.to_dict() for e in entries], indent=2, ensure_ascii=False))
- elif not args.quiet:
- ui.print_inspect(Console(), entries, args.source)
- return 0
-
-
-def cmd_keygen(args) -> int:
- private, public = Path(args.private).expanduser(), Path(args.public).expanduser()
- signing.generate_keypair(
- private,
- public,
- password=os.environ.get(ENV_KEY_PASSWORD) or None,
- overwrite=args.force,
- )
- if args.json:
- print(json.dumps({"private": str(private), "public": str(public)}))
- elif not args.quiet:
- out, _ = _consoles()
- out.print(f"Private key: {escape(str(private))} (keep it secret)")
- out.print(f"Public key: {escape(str(public))} (share it)")
- return 0
-
-
-def cmd_sign(args) -> int:
- manifest_path = Path(args.manifest).expanduser()
- Manifest.load(manifest_path) # refuse to sign something that is not a manifest
- key = signing.load_private_key(
- Path(args.key).expanduser(), os.environ.get(ENV_KEY_PASSWORD) or None
- )
- signature = signing.sign_manifest(manifest_path, key)
- if args.json:
- print(json.dumps({"signature": str(signature)}))
- elif not args.quiet:
- _consoles()[0].print(f"Signature written to {escape(str(signature))}")
- return 0
-
-
-def _write_listing(args, entries, noun: str) -> None:
- _, err = _consoles()
- if not args.output:
- print(json.dumps([e.to_dict() for e in entries], indent=2, ensure_ascii=False))
- return
-
- path = Path(args.output).expanduser()
- if path.suffix.lower() == ".json":
- text = json.dumps([e.to_dict() for e in entries], indent=2, ensure_ascii=False)
- text += "\n"
- else:
- text = "".join(f"{entry.path}\n" for entry in entries)
- roots = {entry.root for entry in entries if entry.root}
- if len(roots) == 1:
- # Remembered by ``pack`` so the listed folder's name is kept.
- text = f"{ROOT_HEADER} {roots.pop()}\n" + text
- path.write_text(text, encoding="utf-8")
-
- if not args.quiet:
- err.print(f"Found {len(entries)} {noun}(s). List saved to: {escape(str(path))}")
-
-
-def cmd_list_files(args) -> int:
- _, err = _consoles()
- entries = api.list_files(args.folder, include=args.include, exclude=args.exclude)
-
- if any(looks_like_archive(entry.path) for entry in entries) and not args.quiet:
- err.print(
- "[yellow]Warning: this folder contains archive volumes or a manifest. "
- "Did you mean `batchvault list archives`?[/yellow]"
- )
- _write_listing(args, entries, "file")
- return 0
-
-
-def cmd_list_archives(args) -> int:
- _, err = _consoles()
- entries = api.list_archives(args.folder)
-
- if not entries and not args.quiet:
- err.print(
- "[yellow]Warning: no archives found. "
- "Did you mean `batchvault list files`?[/yellow]"
- )
- _write_listing(args, entries, "archive")
- return 0
-
-
-def cmd_doctor(args) -> int:
- from batchvault import ui
-
- checks = run_checks(args.seven_zip)
- if args.json:
- print(
- json.dumps(
- [vars(check) for check in checks], indent=2, ensure_ascii=False
- )
- )
- elif not args.quiet:
- ui.print_checks(Console(), checks)
- return 0 if passed(checks) else 1
-
-
-def _use_utf8() -> None:
- # Keep file names readable when output is redirected on Windows.
- for stream in (sys.stdout, sys.stderr):
- try:
- stream.reconfigure(encoding="utf-8")
- except (AttributeError, ValueError):
- pass
-
-
-def _load_config(argv: list[str]) -> dict | None:
- pre = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
- pre.add_argument("--config")
- known, _ = pre.parse_known_args(argv)
-
- explicit = known.config or os.environ.get(ENV_CONFIG)
- if explicit:
- path = Path(explicit).expanduser()
- if not path.is_file():
- raise ConfigError(f"Config file not found: {path}")
- return load_config(path) # named by the user, so trusted
-
- path = find_config()
- if path is None:
- return None
- config = load_config(path)
- reject_restricted(config, path) # found by searching: not fully trusted
- return config
-
-
-def main(argv: list[str] | None = None) -> int:
- _use_utf8()
- _, err = _consoles()
- argv = list(sys.argv[1:] if argv is None else argv)
-
- try:
- parser = build_parser(_load_config(argv))
- except BatchVaultError as exc:
- err.print(f"[bold red]Error:[/bold red] {escape(str(exc))}")
- return 2
-
- args = parser.parse_args(argv)
-
- log_path = None
- if not args.no_log:
- log_path = setup_logging(args.log_level, args.log_file)
- log.info("batchvault %s: %s", __version__, " ".join(redact_arguments(argv)))
-
- try:
- return args.handler(args)
- except BatchVaultError as exc:
- log.error("%s", exc)
- err.print(f"[bold red]Error:[/bold red] {escape(str(exc))}")
- return 2
- except KeyboardInterrupt:
- log.warning("Interrupted")
- err.print("[yellow]Interrupted.[/yellow]")
- return 130
- except Exception as exc:
- log.exception("Unexpected error")
- where = f" See the log: {log_path}" if log_path else ""
- err.print(f"[bold red]Unexpected error:[/bold red] {escape(str(exc))}.{where}")
- return 3
- finally:
- close_logging()
diff --git a/src/batchvault/cli/__init__.py b/src/batchvault/cli/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/src/batchvault/cli/_common.py b/src/batchvault/cli/_common.py
new file mode 100644
index 0000000..ad6a9eb
--- /dev/null
+++ b/src/batchvault/cli/_common.py
@@ -0,0 +1,146 @@
+"""Helpers shared by the command-line commands."""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import threading
+from contextlib import contextmanager
+from pathlib import Path
+
+from rich.console import Console
+from rich.markup import escape
+
+from batchvault.cli import notify
+from batchvault.core.errors import InputError
+from batchvault.core.models import RunResult, Status
+from batchvault.core.security import redact
+
+log = logging.getLogger(__name__)
+
+
+ENV_PASSWORD = "BATCHVAULT_PASSWORD"
+
+
+ENV_KEY_PASSWORD = "BATCHVAULT_KEY_PASSWORD"
+
+
+ENV_NOTIFY_URL = "BATCHVAULT_NOTIFY_URL"
+
+
+def _consoles() -> tuple[Console, Console]:
+ return Console(), Console(stderr=True)
+
+
+class EventStream:
+ """Progress callback that prints one JSON object per line on stdout."""
+
+ def __init__(self):
+ self._lock = threading.Lock()
+
+ def __call__(self, event) -> None:
+ self._write(event.to_dict())
+
+ def result(self, result: RunResult) -> None:
+ self._write({"event": "result", **result.to_dict()})
+
+ def _write(self, data: dict) -> None:
+ with self._lock:
+ print(json.dumps(data, ensure_ascii=False), flush=True)
+
+
+@contextmanager
+def _progress(args, label: str, err: Console):
+ from batchvault.cli import ui
+
+ if args.json_events:
+ yield EventStream()
+ else:
+ with ui.RunProgress(err, label, enabled=not args.quiet) as progress:
+ yield progress
+
+
+def _check_output_flags(args) -> None:
+ if args.json and args.json_events:
+ raise InputError("Use either --json or --json-events, not both.")
+
+
+def _read_password(args) -> str | None:
+ path = getattr(args, "password_file", None)
+ if path:
+ try:
+ lines = Path(path).expanduser().read_text(encoding="utf-8").splitlines()
+ except OSError as exc:
+ raise InputError(f"Cannot read password file {path}: {exc}") from exc
+ if not lines or not lines[0]:
+ raise InputError(f"Password file {path} is empty.")
+ return lines[0]
+ return os.environ.get(ENV_PASSWORD) or None
+
+
+def _notify_url(args) -> str | None:
+ url = args.notify or os.environ.get(ENV_NOTIFY_URL)
+ if not url:
+ return None
+
+ notify.validate_url(url)
+ if notify.is_insecure(url):
+ _, err = _consoles()
+ err.print(
+ "[yellow]Warning: the notification URL uses http://, so the summary "
+ "(file names and paths) is sent unencrypted. Prefer https://.[/yellow]"
+ )
+ return url
+
+
+def _finish(
+ args, command: str, result: RunResult, title: str | None, progress, url
+) -> int:
+ from batchvault.cli import ui
+
+ out, err = _consoles()
+ if args.json_events:
+ progress.result(result)
+ elif args.json:
+ print(json.dumps(result.to_dict(), indent=2, ensure_ascii=False))
+ elif not args.quiet:
+ ui.print_summary(out, result, title, command)
+
+ if url and notify.should_notify(args.notify_on, result):
+ problem = notify.send(url, notify.build_payload(command, result))
+ if problem:
+ log.warning("Notification failed: %s", redact(problem))
+ err.print(
+ f"[yellow]Warning: notification failed ({escape(problem)}).[/yellow]"
+ )
+
+ return 0 if result.status == Status.SUCCESS else 1
+
+
+def _given(args, dest: str, names: tuple[str, ...]) -> bool:
+ """True if the user passed an option: on the command line or with a new value.
+
+ A value from the settings file is the parser's default, so it never counts.
+ """
+ if getattr(args, dest, None) != args.command_parser.get_default(dest):
+ return True
+ for token in args.argv:
+ for name in names:
+ if token == name or token.startswith(name + "="):
+ return True
+ if not name.startswith("--") and token.startswith(name):
+ return True
+ return False
+
+
+def _reject_given(args, fixed: dict[str, tuple[str, ...]], extra: list[str]) -> None:
+ clashing = extra + [
+ names[0] for dest, names in fixed.items() if _given(args, dest, names)
+ ]
+ if clashing:
+ raise InputError(
+ "--resume continues the earlier run with its own settings, so do not "
+ f"also pass: {', '.join(clashing)}. Remove them, or run without "
+ "--resume to start again."
+ )
diff --git a/src/batchvault/cli/cli.py b/src/batchvault/cli/cli.py
new file mode 100644
index 0000000..e9b2970
--- /dev/null
+++ b/src/batchvault/cli/cli.py
@@ -0,0 +1,100 @@
+"""Command-line interface: a thin wrapper over :mod:`batchvault.core.api`."""
+
+from __future__ import annotations
+
+import logging
+
+log = logging.getLogger(__name__)
+
+
+ENV_CONFIG = "BATCHVAULT_CONFIG"
+
+
+def _use_utf8() -> None:
+ # Keep file names readable when output is redirected on Windows.
+ import io
+ import sys
+
+ for stream in (sys.stdout, sys.stderr):
+ if not isinstance(stream, io.TextIOWrapper):
+ continue
+ try:
+ stream.reconfigure(encoding="utf-8")
+ except (AttributeError, ValueError):
+ pass
+
+
+def _load_config(argv: list[str]) -> dict | None:
+ import argparse
+ import os
+ from pathlib import Path
+
+ from batchvault.core.config import find_config, load_config, reject_restricted
+ from batchvault.core.errors import ConfigError
+
+ pre = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
+ pre.add_argument("--config")
+ known, _ = pre.parse_known_args(argv)
+
+ explicit = known.config or os.environ.get(ENV_CONFIG)
+ if explicit:
+ path = Path(explicit).expanduser()
+ if not path.is_file():
+ raise ConfigError(f"Config file not found: {path}")
+ return load_config(path) # named by the user, so trusted
+
+ found = find_config()
+ if found is None:
+ return None
+ config = load_config(found)
+ reject_restricted(config, found) # found by searching: not fully trusted
+ return config
+
+
+def main(argv: list[str] | None = None) -> int:
+ import sys
+
+ from rich.markup import escape
+
+ from batchvault import __version__
+ from batchvault.cli._common import _consoles
+ from batchvault.cli.logs import close_logging, setup_logging
+ from batchvault.cli.parser import build_parser
+ from batchvault.core.errors import BatchVaultError
+ from batchvault.core.security import redact_arguments
+
+ _use_utf8()
+ _, err = _consoles()
+ argv = list(sys.argv[1:] if argv is None else argv)
+
+ try:
+ parser = build_parser(_load_config(argv))
+ except BatchVaultError as exc:
+ err.print(f"[bold red]Error:[/bold red] {escape(str(exc))}")
+ return 2
+
+ args = parser.parse_args(argv)
+ args.argv = argv
+
+ log_path = None
+ if not args.no_log:
+ log_path = setup_logging(args.log_level, args.log_file)
+ log.info("batchvault %s: %s", __version__, " ".join(redact_arguments(argv)))
+
+ try:
+ return args.handler(args)
+ except BatchVaultError as exc:
+ log.error("%s", exc)
+ err.print(f"[bold red]Error:[/bold red] {escape(str(exc))}")
+ return 2
+ except KeyboardInterrupt:
+ log.warning("Interrupted")
+ err.print("[yellow]Interrupted.[/yellow]")
+ return 130
+ except Exception as exc:
+ log.exception("Unexpected error")
+ where = f" See the log: {log_path}" if log_path else ""
+ err.print(f"[bold red]Unexpected error:[/bold red] {escape(str(exc))}.{where}")
+ return 3
+ finally:
+ close_logging()
diff --git a/src/batchvault/cli/commands/__init__.py b/src/batchvault/cli/commands/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/src/batchvault/cli/commands/doctor.py b/src/batchvault/cli/commands/doctor.py
new file mode 100644
index 0000000..9b251c6
--- /dev/null
+++ b/src/batchvault/cli/commands/doctor.py
@@ -0,0 +1,21 @@
+"""The ``doctor`` command."""
+
+from __future__ import annotations
+
+
+def cmd_doctor(args) -> int:
+ import json
+
+ from rich.console import Console
+
+ from batchvault.cli import ui
+ from batchvault.cli.doctor import passed, run_checks
+
+ checks = run_checks(args.seven_zip)
+ if args.json:
+ print(
+ json.dumps([vars(check) for check in checks], indent=2, ensure_ascii=False)
+ )
+ elif not args.quiet:
+ ui.print_checks(Console(), checks)
+ return 0 if passed(checks) else 1
diff --git a/src/batchvault/cli/commands/inspect.py b/src/batchvault/cli/commands/inspect.py
new file mode 100644
index 0000000..61c7ca4
--- /dev/null
+++ b/src/batchvault/cli/commands/inspect.py
@@ -0,0 +1,19 @@
+"""The ``inspect`` command."""
+
+from __future__ import annotations
+
+
+def cmd_inspect(args) -> int:
+ import json
+
+ from rich.console import Console
+
+ from batchvault.cli import ui
+ from batchvault.core import api
+
+ entries = api.inspect(args.source)
+ if args.json:
+ print(json.dumps([e.to_dict() for e in entries], indent=2, ensure_ascii=False))
+ elif not args.quiet:
+ ui.print_inspect(Console(), entries, args.source)
+ return 0
diff --git a/src/batchvault/cli/commands/keys.py b/src/batchvault/cli/commands/keys.py
new file mode 100644
index 0000000..42890ca
--- /dev/null
+++ b/src/batchvault/cli/commands/keys.py
@@ -0,0 +1,53 @@
+"""The ``keygen`` and ``sign`` commands."""
+
+from __future__ import annotations
+
+
+def cmd_keygen(args) -> int:
+ import json
+ import os
+ from pathlib import Path
+
+ from rich.markup import escape
+
+ from batchvault.archive import signing
+ from batchvault.cli._common import ENV_KEY_PASSWORD, _consoles
+
+ private, public = Path(args.private).expanduser(), Path(args.public).expanduser()
+ signing.generate_keypair(
+ private,
+ public,
+ password=os.environ.get(ENV_KEY_PASSWORD) or None,
+ overwrite=args.force,
+ )
+ if args.json:
+ print(json.dumps({"private": str(private), "public": str(public)}))
+ elif not args.quiet:
+ out, _ = _consoles()
+ out.print(f"Private key: {escape(str(private))} (keep it secret)")
+ out.print(f"Public key: {escape(str(public))} (share it)")
+ return 0
+
+
+def cmd_sign(args) -> int:
+ import json
+ import os
+ from pathlib import Path
+
+ from rich.markup import escape
+
+ from batchvault.archive import signing
+ from batchvault.archive.manifest import Manifest
+ from batchvault.cli._common import ENV_KEY_PASSWORD, _consoles
+
+ manifest_path = Path(args.manifest).expanduser()
+ Manifest.load(manifest_path) # refuse to sign something that is not a manifest
+ key = signing.load_private_key(
+ Path(args.key).expanduser(), os.environ.get(ENV_KEY_PASSWORD) or None
+ )
+ signature = signing.sign_manifest(manifest_path, key)
+ if args.json:
+ print(json.dumps({"signature": str(signature)}))
+ elif not args.quiet:
+ _consoles()[0].print(f"Signature written to {escape(str(signature))}")
+ return 0
diff --git a/src/batchvault/cli/commands/listing.py b/src/batchvault/cli/commands/listing.py
new file mode 100644
index 0000000..ffd8fc1
--- /dev/null
+++ b/src/batchvault/cli/commands/listing.py
@@ -0,0 +1,66 @@
+"""The ``list files`` and ``list archives`` commands."""
+
+from __future__ import annotations
+
+
+def _write_listing(args, entries, noun: str) -> None:
+ import json
+ from pathlib import Path
+
+ from rich.markup import escape
+
+ from batchvault.cli._common import _consoles
+ from batchvault.core.inputs import ROOT_HEADER
+
+ _, err = _consoles()
+ if not args.output:
+ print(json.dumps([e.to_dict() for e in entries], indent=2, ensure_ascii=False))
+ return
+
+ path = Path(args.output).expanduser()
+ if path.suffix.lower() == ".json":
+ text = json.dumps([e.to_dict() for e in entries], indent=2, ensure_ascii=False)
+ text += "\n"
+ else:
+ text = "".join(f"{entry.path}\n" for entry in entries)
+ roots = {entry.root for entry in entries if entry.root}
+ if len(roots) == 1:
+ # Remembered by ``pack`` so the listed folder's name is kept.
+ text = f"{ROOT_HEADER} {roots.pop()}\n" + text
+ path.write_text(text, encoding="utf-8")
+
+ if not args.quiet:
+ err.print(f"Found {len(entries)} {noun}(s). List saved to: {escape(str(path))}")
+
+
+def cmd_list_files(args) -> int:
+ from batchvault.cli._common import _consoles
+ from batchvault.core import api
+ from batchvault.core.inputs import looks_like_archive
+
+ _, err = _consoles()
+ entries = api.list_files(args.folder, include=args.include, exclude=args.exclude)
+
+ if any(looks_like_archive(entry.path) for entry in entries) and not args.quiet:
+ err.print(
+ "[yellow]Warning: this folder contains archive volumes or a manifest. "
+ "Did you mean `batchvault list archives`?[/yellow]"
+ )
+ _write_listing(args, entries, "file")
+ return 0
+
+
+def cmd_list_archives(args) -> int:
+ from batchvault.cli._common import _consoles
+ from batchvault.core import api
+
+ _, err = _consoles()
+ entries = api.list_archives(args.folder)
+
+ if not entries and not args.quiet:
+ err.print(
+ "[yellow]Warning: no archives found. "
+ "Did you mean `batchvault list files`?[/yellow]"
+ )
+ _write_listing(args, entries, "archive")
+ return 0
diff --git a/src/batchvault/cli/commands/pack.py b/src/batchvault/cli/commands/pack.py
new file mode 100644
index 0000000..f1ae30f
--- /dev/null
+++ b/src/batchvault/cli/commands/pack.py
@@ -0,0 +1,200 @@
+"""The ``pack`` command."""
+
+from __future__ import annotations
+
+
+def _confirm_deletion(args, progress):
+ import sys
+ from pathlib import Path
+
+ def confirm(paths: list[Path]) -> bool:
+ if args.yes:
+ return True
+ quiet = args.json or args.json_events or args.quiet
+ if quiet or not sys.stdin.isatty():
+ return False # never delete without an explicit yes
+
+ if hasattr(progress, "pause"):
+ progress.pause()
+ print(f"{len(paths)} original file(s) were archived and verified:")
+ for path in paths[:10]:
+ print(f" {path}")
+ if len(paths) > 10:
+ print(f" ... and {len(paths) - 10} more")
+ answer = input("Delete them? [y/N] ")
+ return answer.strip().lower() in ("y", "yes")
+
+ return confirm
+
+
+def _describe_level(level) -> str:
+ from batchvault.core.levels import AUTO
+
+ return "auto (chosen for each file's type)" if level == AUTO else str(level)
+
+
+def _describe_sources(args) -> str:
+ parts = []
+ if args.files:
+ parts.append(f"{len(args.files)} given")
+ if args.list:
+ parts.append("list: " + ("standard input" if args.list == "-" else args.list))
+ return ", ".join(parts) or "-"
+
+
+# Options whose values come from the earlier run when resuming. Passing any of
+# them with --resume is refused rather than ignored or allowed to mix outputs.
+PACK_RESUME_FIXED: dict[str, tuple[str, ...]] = {
+ "list": ("--list",),
+ "engine": ("--engine",),
+ "format": ("--format",),
+ "level": ("--level",),
+ "volume_size": ("--volume-size",),
+ "include": ("--include",),
+ "exclude": ("--exclude",),
+ "root": ("--root",),
+ "manifest": ("--manifest",),
+ "force": ("--force",),
+ "update": ("--update",),
+ "verify": ("--verify",),
+ "encrypt": ("--encrypt",),
+ "delete_source": ("--delete-source",),
+ "allow_archives": ("--allow-archives",),
+ "strict": ("--strict",),
+ "dry_run": ("--dry-run",),
+ "readme": ("--no-readme",),
+}
+
+
+def cmd_pack(args) -> int:
+ import os
+ from pathlib import Path
+
+ from batchvault.archive import resume as journals
+ from batchvault.cli import ui
+ from batchvault.cli._common import (
+ ENV_KEY_PASSWORD,
+ ENV_PASSWORD,
+ _check_output_flags,
+ _consoles,
+ _finish,
+ _given,
+ _notify_url,
+ _progress,
+ _read_password,
+ _reject_given,
+ )
+ from batchvault.core import api
+ from batchvault.core.errors import InputError
+ from batchvault.core.options import format_size
+
+ _check_output_flags(args)
+
+ saved = None
+ if args.resume:
+ _reject_given(args, PACK_RESUME_FIXED, ["FILE"] if args.files else [])
+ saved, _, _ = journals.load_pack(
+ Path(args.output).expanduser().resolve() / journals.PACK_JOURNAL
+ )
+ elif not args.files and not args.list:
+ raise InputError("Give at least one FILE or use --list.")
+
+ jobs, retries = args.jobs, args.retries
+ if saved is not None: # the earlier run's values unless given again
+ jobs = jobs if _given(args, "jobs", ("--jobs", "-j")) else saved.jobs
+ retries = retries if _given(args, "retries", ("--retries",)) else saved.retries
+
+ password = None
+ if args.encrypt or (saved is not None and saved.encrypted):
+ password = _read_password(args)
+ if not password:
+ raise InputError(
+ ("The earlier run was encrypted: " if saved else "--encrypt: ")
+ + f"set {ENV_PASSWORD} or use --password-file."
+ )
+ url = _notify_url(args)
+
+ out, err = _consoles()
+ if not (args.quiet or args.json or args.json_events):
+ if saved is not None:
+ rows = [
+ ("Resuming", "the earlier run in this folder (its settings are kept)"),
+ ("Engine", saved.engine),
+ ("Format", saved.format or "xz (native engine)"),
+ ("Level", _describe_level(saved.level)),
+ ("Volume size", format_size(saved.volume_size)),
+ ("Jobs", str(jobs) if jobs else "0 (sequential)"),
+ ("Source files", f"{len(saved.sources)} from the earlier run"),
+ ("Archives folder", str(Path(args.output).resolve())),
+ ]
+ else:
+ rows = [
+ ("Engine", args.engine),
+ ("Format", args.format or "xz (native engine)"),
+ ("Level", _describe_level(args.level)),
+ ("Volume size", args.volume_size),
+ ("Jobs", str(args.jobs) if args.jobs else "0 (sequential)"),
+ ("Source files", _describe_sources(args)),
+ (
+ "Folders",
+ f"kept, relative to {Path(args.root).resolve()}"
+ if args.root
+ else "kept",
+ ),
+ ("Archives folder", str(Path(args.output).resolve())),
+ ]
+ ui.print_run_info(out, "BatchVault pack", rows)
+
+ if args.resume:
+ with _progress(args, "Archiving", err) as progress:
+ result = api.pack(
+ output=args.output,
+ resume=True,
+ seven_zip=args.seven_zip,
+ jobs=jobs,
+ retries=retries,
+ password=password,
+ sign_key=args.sign_key,
+ sign_key_password=os.environ.get(ENV_KEY_PASSWORD) or None,
+ confirm=_confirm_deletion(args, progress),
+ progress=progress,
+ )
+ return _finish(args, "pack", result, None, progress, url)
+
+ with _progress(args, "Archiving", err) as progress:
+ result = api.pack(
+ args.files,
+ args.output,
+ list_file=args.list,
+ root=args.root,
+ engine=args.engine,
+ seven_zip=args.seven_zip,
+ format=args.format,
+ level=args.level,
+ volume_size=args.volume_size,
+ jobs=args.jobs,
+ include=args.include,
+ exclude=args.exclude,
+ force=args.force,
+ update=args.update,
+ verify=args.verify,
+ manifest=args.manifest,
+ allow_archives=args.allow_archives,
+ strict=args.strict,
+ dry_run=args.dry_run,
+ password=password,
+ retries=args.retries,
+ sign_key=args.sign_key,
+ sign_key_password=os.environ.get(ENV_KEY_PASSWORD) or None,
+ delete_source=args.delete_source and not args.dry_run,
+ readme=args.readme,
+ confirm=_confirm_deletion(args, progress),
+ progress=progress,
+ )
+
+ title = None # the default title names the source files and archives
+ if args.dry_run:
+ title = "Pack plan (dry run): source files -> archives"
+ if not (args.quiet or args.json or args.json_events):
+ out.print("[bold]Dry run:[/bold] nothing was written.")
+ return _finish(args, "pack", result, title, progress, url)
diff --git a/src/batchvault/cli/commands/unpack.py b/src/batchvault/cli/commands/unpack.py
new file mode 100644
index 0000000..59b356a
--- /dev/null
+++ b/src/batchvault/cli/commands/unpack.py
@@ -0,0 +1,67 @@
+"""The ``unpack`` command."""
+
+from __future__ import annotations
+
+UNPACK_RESUME_FIXED: dict[str, tuple[str, ...]] = {
+ "force": ("--force",),
+ "update": ("--update",),
+}
+
+
+def cmd_unpack(args) -> int:
+ from pathlib import Path
+
+ from batchvault.cli import ui
+ from batchvault.cli._common import (
+ _check_output_flags,
+ _consoles,
+ _finish,
+ _notify_url,
+ _progress,
+ _read_password,
+ _reject_given,
+ )
+ from batchvault.core import api
+ from batchvault.core.errors import InputError
+
+ _check_output_flags(args)
+ if args.resume:
+ _reject_given(args, UNPACK_RESUME_FIXED, ["SOURCE"] if args.source else [])
+ elif not args.source:
+ raise InputError("Give SOURCE, or use --resume to continue an earlier run.")
+ password = _read_password(args)
+ url = _notify_url(args)
+
+ out, err = _consoles()
+ if not (args.quiet or args.json or args.json_events):
+ ui.print_run_info(
+ out,
+ "BatchVault unpack",
+ [
+ (
+ "Archives",
+ "from the earlier run in the output folder (resuming)"
+ if args.resume
+ else args.source,
+ ),
+ ("Restore files to", str(Path(args.output).resolve())),
+ ("Existing files", "overwrite" if args.force else "skip"),
+ ("Jobs", str(args.jobs) if args.jobs else "0 (sequential)"),
+ ],
+ )
+ with _progress(args, "Restoring", err) as progress:
+ result = api.unpack(
+ args.source,
+ args.output,
+ force=args.force,
+ update=args.update,
+ jobs=args.jobs,
+ seven_zip=args.seven_zip,
+ password=password,
+ public_key=args.public_key,
+ retries=args.retries,
+ max_output_size=args.max_output_size,
+ resume=args.resume,
+ progress=progress,
+ )
+ return _finish(args, "unpack", result, None, progress, url)
diff --git a/src/batchvault/cli/commands/verify.py b/src/batchvault/cli/commands/verify.py
new file mode 100644
index 0000000..12dd56c
--- /dev/null
+++ b/src/batchvault/cli/commands/verify.py
@@ -0,0 +1,43 @@
+"""The ``verify`` command."""
+
+from __future__ import annotations
+
+
+def cmd_verify(args) -> int:
+ from batchvault.cli import ui
+ from batchvault.cli._common import (
+ _check_output_flags,
+ _consoles,
+ _finish,
+ _notify_url,
+ _progress,
+ _read_password,
+ )
+ from batchvault.core import api
+
+ _check_output_flags(args)
+ password = _read_password(args)
+ url = _notify_url(args)
+
+ out, err = _consoles()
+ if not (args.quiet or args.json or args.json_events):
+ ui.print_run_info(
+ out,
+ "BatchVault verify",
+ [
+ ("Archives", args.source),
+ ("Original files", args.against or "not compared"),
+ ("Jobs", str(args.jobs) if args.jobs else "0 (sequential)"),
+ ],
+ )
+ with _progress(args, "Verifying", err) as progress:
+ result = api.verify(
+ args.source,
+ against=args.against,
+ jobs=args.jobs,
+ seven_zip=args.seven_zip,
+ password=password,
+ public_key=args.public_key,
+ progress=progress,
+ )
+ return _finish(args, "verify", result, None, progress, url)
diff --git a/src/batchvault/doctor.py b/src/batchvault/cli/doctor.py
similarity index 88%
rename from src/batchvault/doctor.py
rename to src/batchvault/cli/doctor.py
index bbb75ab..db4377c 100644
--- a/src/batchvault/doctor.py
+++ b/src/batchvault/cli/doctor.py
@@ -2,16 +2,8 @@
from __future__ import annotations
-import os
-import platform
-import sys
-import tempfile
from dataclasses import dataclass
-from batchvault.engines.sevenzip import find_7zip
-from batchvault.errors import EngineNotFoundError
-from batchvault.logs import default_log_dir
-
@dataclass(frozen=True)
class Check:
@@ -30,6 +22,10 @@ def _module(name: str, label: str, required: bool = False) -> Check:
def _log_dir_check() -> Check:
+ import tempfile
+
+ from batchvault.cli.logs import default_log_dir
+
folder = default_log_dir()
try:
folder.mkdir(parents=True, exist_ok=True)
@@ -41,6 +37,13 @@ def _log_dir_check() -> Check:
def run_checks(seven_zip: str | None = None) -> list[Check]:
+ import os
+ import platform
+ import sys
+
+ from batchvault.core.errors import EngineNotFoundError
+ from batchvault.engines.sevenzip import find_7zip
+
checks = [
Check(
"Python",
diff --git a/src/batchvault/logs.py b/src/batchvault/cli/logs.py
similarity index 86%
rename from src/batchvault/logs.py
rename to src/batchvault/cli/logs.py
index 267804b..2c4337d 100644
--- a/src/batchvault/logs.py
+++ b/src/batchvault/cli/logs.py
@@ -1,10 +1,10 @@
+from __future__ import annotations
+
import logging
-import os
-import sys
-from datetime import datetime
-from pathlib import Path
+from typing import TYPE_CHECKING
-from batchvault.security import redact, restrict_permissions
+if TYPE_CHECKING:
+ from pathlib import Path
LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR")
DEFAULT_LEVEL = "INFO"
@@ -17,11 +17,17 @@ class RedactingFormatter(logging.Formatter):
"""Hides credentials embedded in URLs, including in tracebacks."""
def format(self, record):
+ from batchvault.core.security import redact
+
return redact(super().format(record))
def default_log_dir() -> Path:
"""The conventional per-user log folder for the current OS."""
+ import os
+ import sys
+ from pathlib import Path
+
override = os.environ.get(ENV_LOG_DIR)
if override:
return Path(override).expanduser()
@@ -50,6 +56,12 @@ def prune_logs(directory: Path, keep: int = KEEP_LOGS) -> None:
def setup_logging(level=DEFAULT_LEVEL, log_file=None):
"""Log to a file for this run. Returns its path, or None if it can't be opened."""
+ import os
+ from datetime import datetime
+ from pathlib import Path
+
+ from batchvault.core.security import restrict_permissions
+
logger = logging.getLogger(LOGGER_NAME)
logger.setLevel(getattr(logging, level.upper()))
close_logging()
diff --git a/src/batchvault/notify.py b/src/batchvault/cli/notify.py
similarity index 84%
rename from src/batchvault/notify.py
rename to src/batchvault/cli/notify.py
index 9c63523..1b9612a 100644
--- a/src/batchvault/notify.py
+++ b/src/batchvault/cli/notify.py
@@ -2,20 +2,18 @@
from __future__ import annotations
-import json
-import platform
-import urllib.error
-import urllib.request
-from urllib.parse import urlparse
+from typing import TYPE_CHECKING
-from batchvault.errors import OptionsError
-from batchvault.models import RunResult, Status
+if TYPE_CHECKING:
+ from batchvault.core.models import RunResult
NOTIFY_ON = ("failure", "success", "always")
TIMEOUT_SECONDS = 10
def should_notify(when: str, result: RunResult) -> bool:
+ from batchvault.core.models import Status
+
if when == "always":
return True
ok = result.status == Status.SUCCESS
@@ -23,6 +21,10 @@ def should_notify(when: str, result: RunResult) -> bool:
def validate_url(url: str) -> str:
+ from urllib.parse import urlparse
+
+ from batchvault.core.errors import OptionsError
+
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
raise OptionsError("The notification URL must start with http:// or https://")
@@ -31,12 +33,18 @@ def validate_url(url: str) -> str:
def is_insecure(url: str) -> bool:
"""True for plain http:// to anything other than this machine."""
+ from urllib.parse import urlparse
+
parsed = urlparse(url)
local = parsed.hostname in ("localhost", "127.0.0.1", "::1")
return parsed.scheme == "http" and not local
def build_payload(command: str, result: RunResult) -> dict:
+ import platform
+
+ from batchvault.core.models import Status
+
counts = ", ".join(
f"{result.count(status)} {status.value}"
for status in (Status.SUCCESS, Status.SKIPPED, Status.FAILED)
@@ -58,6 +66,9 @@ def build_payload(command: str, result: RunResult) -> dict:
def send(url: str, payload: dict) -> str | None:
"""POST the payload as JSON. Returns an error message, or None on success."""
+ import json
+ import urllib.request
+
request = urllib.request.Request(
validate_url(url),
data=json.dumps(payload).encode("utf-8"),
diff --git a/src/batchvault/cli/parser.py b/src/batchvault/cli/parser.py
new file mode 100644
index 0000000..1a832db
--- /dev/null
+++ b/src/batchvault/cli/parser.py
@@ -0,0 +1,376 @@
+"""The argparse definition of every command and option."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ import argparse
+
+
+def _parents() -> dict[str, argparse.ArgumentParser]:
+ import argparse
+
+ from batchvault.cli import notify
+ from batchvault.cli._common import ENV_NOTIFY_URL, ENV_PASSWORD
+ from batchvault.cli.logs import LOG_LEVELS
+
+ parents = {name: argparse.ArgumentParser(add_help=False) for name in (
+ "common", "jobs", "seven", "patterns", "run", "retry", "public",
+ )} # fmt: skip
+
+ common = parents["common"]
+ common.add_argument(
+ "--json", action="store_true", help="Print the result as JSON on stdout."
+ )
+ common.add_argument(
+ "-q", "--quiet", action="store_true", help="No progress or summary output."
+ )
+ common.add_argument("--log-file", help="Write the log here instead of the default.")
+ common.add_argument(
+ "--log-level", choices=LOG_LEVELS, default="INFO", help="Log level."
+ )
+ common.add_argument(
+ "--no-log", action="store_true", help="Do not write a log file."
+ )
+ common.add_argument(
+ "--config", metavar="FILE", help="Config file (default: batchvault.toml)."
+ )
+
+ parents["jobs"].add_argument(
+ "-j",
+ "--jobs",
+ type=int,
+ default=0,
+ metavar="N",
+ help="Worker threads to use; 0 means none, one file at a time (default: 0).",
+ )
+
+ parents["seven"].add_argument(
+ "--7z", dest="seven_zip", metavar="PATH", help="Path to the 7-Zip executable."
+ )
+
+ patterns = parents["patterns"]
+ patterns.add_argument(
+ "--include",
+ action="append",
+ default=[],
+ metavar="GLOB",
+ help="Only keep matching paths (repeatable; @FILE reads patterns from a file).",
+ )
+ patterns.add_argument(
+ "--exclude",
+ action="append",
+ default=[],
+ metavar="GLOB",
+ help="Drop matching paths (repeatable; wins over --include).",
+ )
+
+ run = parents["run"]
+ run.add_argument(
+ "--json-events",
+ action="store_true",
+ help="Print one JSON event per line on stdout while running.",
+ )
+ run.add_argument(
+ "--notify",
+ metavar="URL",
+ help=f"POST a JSON summary to this webhook (or set {ENV_NOTIFY_URL}).",
+ )
+ run.add_argument(
+ "--notify-on",
+ choices=notify.NOTIFY_ON,
+ default="failure",
+ help="When to notify (default: failure).",
+ )
+ run.add_argument(
+ "--password-file",
+ metavar="FILE",
+ help=f"Read the archive password from this file (or set {ENV_PASSWORD}).",
+ )
+
+ parents["retry"].add_argument(
+ "--retries",
+ type=int,
+ default=3,
+ metavar="N",
+ help="Retry a file this many times after an I/O or engine error (default: 3).",
+ )
+
+ parents["public"].add_argument(
+ "--public-key",
+ metavar="FILE",
+ help="Require a valid manifest signature made with the matching private key.",
+ )
+ return parents
+
+
+def _configure(subparser, name: str, config: dict | None) -> None:
+ """Apply defaults from the config file to one command."""
+ from batchvault.core.config import defaults_for
+
+ if not config:
+ return
+ actions = {action.dest: action for action in subparser._actions}
+ defaults = defaults_for(config, name, actions)
+ if defaults:
+ subparser.set_defaults(**defaults)
+
+
+def build_parser(config: dict | None = None) -> argparse.ArgumentParser:
+ import argparse
+
+ from batchvault import __version__
+ from batchvault.cli._common import ENV_KEY_PASSWORD, ENV_PASSWORD
+ from batchvault.cli.commands.doctor import cmd_doctor
+ from batchvault.cli.commands.inspect import cmd_inspect
+ from batchvault.cli.commands.keys import cmd_keygen, cmd_sign
+ from batchvault.cli.commands.listing import cmd_list_archives, cmd_list_files
+ from batchvault.cli.commands.pack import cmd_pack
+ from batchvault.cli.commands.unpack import cmd_unpack
+ from batchvault.cli.commands.verify import cmd_verify
+ from batchvault.core.levels import AUTO, level_argument
+ from batchvault.core.options import FORMATS
+ from batchvault.engines.registry import ENGINE_CHOICES
+
+ p = _parents()
+ common, jobs, seven, patterns, run, retry, public = (
+ p["common"], p["jobs"], p["seven"], p["patterns"], p["run"], p["retry"],
+ p["public"],
+ ) # fmt: skip
+
+ parser = argparse.ArgumentParser(
+ prog="batchvault",
+ description="Batch archive, compress, split and restore files.",
+ )
+ parser.add_argument(
+ "--version", action="version", version=f"%(prog)s {__version__}"
+ )
+ commands = parser.add_subparsers(dest="command", required=True, metavar="COMMAND")
+
+ pack = commands.add_parser(
+ "pack",
+ parents=[common, jobs, seven, patterns, run, retry],
+ help="Compress files into split volumes.",
+ description="Compress files into split volumes and write a manifest.",
+ epilog="Example: batchvault pack --list archive_list.txt -o archives",
+ )
+ pack.add_argument("files", nargs="*", metavar="FILE", help="Files to archive.")
+ pack.add_argument(
+ "--list",
+ metavar="FILE",
+ help="Text or JSON list of files to archive ('-' reads standard input).",
+ )
+ pack.add_argument("-o", "--output", default="archives", help="Output folder.")
+ pack.add_argument("--engine", choices=ENGINE_CHOICES, default="auto")
+ pack.add_argument(
+ "--format",
+ choices=FORMATS,
+ help="Native engine compression (default: xz). 7-Zip always writes 7z.",
+ )
+ pack.add_argument(
+ "--level",
+ type=level_argument,
+ choices=[*range(10), AUTO],
+ default=AUTO,
+ metavar="0-9|auto",
+ help="Compression level, or auto to pick one per file type (default: auto).",
+ )
+ pack.add_argument(
+ "--volume-size",
+ default="10m",
+ metavar="SIZE",
+ help="Volume size, e.g. 500k, 10m, 2g (default: 10m).",
+ )
+ pack.add_argument("--manifest", metavar="PATH", help="Manifest file location.")
+ pack.add_argument(
+ "--root",
+ metavar="DIR",
+ help="Advanced: store paths relative to DIR. By default folders are kept "
+ "automatically (relative to the folder you listed, or the folder your "
+ "files share).",
+ )
+ pack.add_argument("--force", action="store_true", help="Replace existing archives.")
+ pack.add_argument(
+ "--update",
+ action="store_true",
+ help="Skip unchanged files and re-archive changed ones.",
+ )
+ pack.add_argument(
+ "--verify", action="store_true", help="Test each archive after creating it."
+ )
+ pack.add_argument(
+ "--encrypt",
+ action="store_true",
+ help=f"Encrypt with the password from {ENV_PASSWORD} or --password-file.",
+ )
+ pack.add_argument(
+ "--sign-key",
+ metavar="FILE",
+ help=f"Sign the manifest with this private key (password: {ENV_KEY_PASSWORD}).",
+ )
+ pack.add_argument(
+ "--no-readme",
+ dest="readme",
+ action="store_false",
+ help="Do not write BATCHVAULT-README.md next to the manifest.",
+ )
+ pack.add_argument(
+ "--delete-source",
+ action="store_true",
+ help="Delete each original after proving the archive restores identically.",
+ )
+ pack.add_argument(
+ "--yes", action="store_true", help="Do not ask before --delete-source deletes."
+ )
+ pack.add_argument(
+ "--allow-archives",
+ action="store_true",
+ help="Allow archiving BatchVault/7-Zip volumes and manifests.",
+ )
+ pack.add_argument(
+ "--strict",
+ action="store_true",
+ help="Treat settings the engine cannot honour as errors.",
+ )
+ pack.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="Show what would be archived without writing anything.",
+ )
+ pack.add_argument(
+ "--resume",
+ action="store_true",
+ help="Continue an interrupted run in the output folder. Every setting "
+ "comes from the earlier run, so give only -o (and a password or key).",
+ )
+ pack.set_defaults(handler=cmd_pack, command_parser=pack)
+ _configure(pack, "pack", config)
+
+ unpack = commands.add_parser(
+ "unpack",
+ parents=[common, jobs, seven, run, retry, public],
+ help="Restore files from archives.",
+ description="Restore files from a manifest, folder, volume or list file.",
+ epilog="Example: batchvault unpack archives -o restored",
+ )
+ unpack.add_argument(
+ "source",
+ nargs="?",
+ metavar="SOURCE",
+ help="Manifest, folder, volume or list file (not needed with --resume).",
+ )
+ unpack.add_argument("-o", "--output", default="restored", help="Output folder.")
+ unpack.add_argument(
+ "--force", action="store_true", help="Overwrite existing files."
+ )
+ unpack.add_argument(
+ "--update",
+ action="store_true",
+ help="Skip files that already match; replace the rest.",
+ )
+ unpack.add_argument(
+ "--max-output-size",
+ metavar="SIZE",
+ help="Refuse an archive that would expand beyond this size, per file "
+ "(e.g. 50g). Use it when restoring archives you do not trust.",
+ )
+ unpack.add_argument(
+ "--resume",
+ action="store_true",
+ help="Continue an interrupted run into the output folder, using the "
+ "source of the earlier run (give only -o).",
+ )
+ unpack.set_defaults(handler=cmd_unpack, command_parser=unpack)
+ _configure(unpack, "unpack", config)
+
+ verify = commands.add_parser(
+ "verify",
+ parents=[common, jobs, seven, run, public],
+ help="Check that archives are complete and intact.",
+ description="Check archives without restoring them.",
+ )
+ verify.add_argument(
+ "source", metavar="SOURCE", help="Manifest, folder, volume or list file."
+ )
+ verify.add_argument(
+ "--against",
+ metavar="FOLDER",
+ help="Also compare with the original files in this folder (and below).",
+ )
+ verify.set_defaults(handler=cmd_verify)
+ _configure(verify, "verify", config)
+
+ inspect = commands.add_parser(
+ "inspect",
+ parents=[common],
+ help="Show what is in an archive set.",
+ description="Show sizes, ratios, engine, format and completeness.",
+ )
+ inspect.add_argument(
+ "source", metavar="SOURCE", help="Manifest, folder, volume or list file."
+ )
+ inspect.set_defaults(handler=cmd_inspect)
+ _configure(inspect, "inspect", config)
+
+ listing = commands.add_parser("list", help="List files or archives.")
+ kinds = listing.add_subparsers(dest="kind", required=True, metavar="KIND")
+
+ files = kinds.add_parser(
+ "files",
+ parents=[common, patterns],
+ help="List the files in a folder.",
+ description="List files (JSON on stdout unless -o is given).",
+ )
+ files.add_argument("folder")
+ files.add_argument("-o", "--output", metavar="FILE", help="Write .json or .txt.")
+ files.set_defaults(handler=cmd_list_files)
+ _configure(files, "list-files", config)
+
+ archives = kinds.add_parser(
+ "archives",
+ parents=[common],
+ help="List the archives in a folder.",
+ description="List archives (JSON on stdout unless -o is given).",
+ )
+ archives.add_argument("folder")
+ archives.add_argument("-o", "--output", metavar="FILE", help="Write .json or .txt.")
+ archives.set_defaults(handler=cmd_list_archives)
+ _configure(archives, "list-archives", config)
+
+ keygen = commands.add_parser(
+ "keygen",
+ parents=[common],
+ help="Create a key pair for signing manifests.",
+ description=(
+ "Create an Ed25519 key pair (needs batchvault[crypto]). The private "
+ f"key is encrypted if {ENV_KEY_PASSWORD} is set."
+ ),
+ )
+ keygen.add_argument("--private", default="batchvault.key", metavar="FILE")
+ keygen.add_argument("--public", default="batchvault.pub", metavar="FILE")
+ keygen.add_argument("--force", action="store_true", help="Replace existing keys.")
+ keygen.set_defaults(handler=cmd_keygen)
+ _configure(keygen, "keygen", config)
+
+ sign = commands.add_parser(
+ "sign",
+ parents=[common],
+ help="Sign a manifest.",
+ description="Write a detached signature next to a manifest.",
+ )
+ sign.add_argument("manifest", metavar="MANIFEST")
+ sign.add_argument("--key", required=True, metavar="FILE", help="Private key.")
+ sign.set_defaults(handler=cmd_sign)
+ _configure(sign, "sign", config)
+
+ doctor = commands.add_parser(
+ "doctor",
+ parents=[common, seven],
+ help="Check this machine's setup.",
+ description="Check Python modules, 7-Zip and the log folder.",
+ )
+ doctor.set_defaults(handler=cmd_doctor)
+ _configure(doctor, "doctor", config)
+
+ return parser
diff --git a/src/batchvault/ui.py b/src/batchvault/cli/ui.py
similarity index 82%
rename from src/batchvault/ui.py
rename to src/batchvault/cli/ui.py
index 154e715..faef16a 100644
--- a/src/batchvault/ui.py
+++ b/src/batchvault/cli/ui.py
@@ -2,28 +2,21 @@
from __future__ import annotations
-import threading
-from pathlib import Path
-
-from rich.console import Console
-from rich.markup import escape
-from rich.panel import Panel
-from rich.progress import (
- BarColumn,
- Progress,
- ProgressColumn,
- SpinnerColumn,
- TaskProgressColumn,
- TextColumn,
- TimeElapsedColumn,
-)
-from rich.table import Table
-from rich.text import Text
-
-from batchvault.doctor import Check
-from batchvault.eta import EtaEstimator, format_duration
-from batchvault.models import InspectEntry, ProgressEvent, RunResult, Status
-from batchvault.options import format_size
+from typing import TYPE_CHECKING
+
+from rich.progress import ProgressColumn
+
+from batchvault.core.models import Status
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from rich.console import Console
+ from rich.progress import TaskID
+ from rich.text import Text
+
+ from batchvault.cli.doctor import Check
+ from batchvault.core.models import InspectEntry, ProgressEvent, RunResult
STATUS_STYLE = {
Status.SUCCESS: ("OK", "green"),
@@ -37,6 +30,10 @@ class EtaColumn(ProgressColumn):
"""Shows the run's own ETA for tasks created with an ``estimator`` field."""
def render(self, task):
+ from rich.text import Text
+
+ from batchvault.core.eta import format_duration
+
estimator = task.fields.get("estimator")
if estimator is None:
return Text("")
@@ -48,6 +45,8 @@ def render(self, task):
def status_text(status: Status) -> Text:
+ from rich.text import Text
+
label, style = STATUS_STYLE[status]
return Text(label, style=style)
@@ -60,13 +59,26 @@ class RunProgress:
"""
def __init__(self, console: Console, label: str, enabled: bool = True):
+ import threading
+
+ from rich.progress import (
+ BarColumn,
+ Progress,
+ SpinnerColumn,
+ TaskProgressColumn,
+ TextColumn,
+ TimeElapsedColumn,
+ )
+
+ from batchvault.core.eta import EtaEstimator
+
self._enabled = enabled
self._label = label
self._lock = threading.Lock()
self._eta = EtaEstimator()
self._by_bytes = False
- self._overall = None
- self._tasks: dict[Path, int] = {}
+ self._overall: TaskID | None = None
+ self._tasks: dict[Path, TaskID] = {}
self._progress = Progress(
SpinnerColumn(),
TextColumn("[bold cyan]{task.description}"),
@@ -92,6 +104,8 @@ def pause(self) -> None:
self._progress.stop()
def __call__(self, event: ProgressEvent) -> None:
+ from rich.markup import escape
+
if not self._enabled:
return
@@ -110,21 +124,29 @@ def __call__(self, event: ProgressEvent) -> None:
self._finish(event)
def _finish(self, event: ProgressEvent) -> None:
+ from rich.text import Text
+
task = self._tasks.pop(event.source, None)
if task is not None:
self._progress.remove_task(task)
if self._overall is not None:
- self._progress.advance(
- self._overall, event.size if self._by_bytes else 1
- )
+ self._progress.advance(self._overall, event.size if self._by_bytes else 1)
+ if event.resumed and self._by_bytes:
+ self._eta.skip(event.size)
- label, style = STATUS_STYLE.get(event.status, ("", ""))
+ label, style = (
+ STATUS_STYLE.get(event.status, ("", "")) if event.status else ("", "")
+ )
self._progress.console.print(
Text.assemble((f"{label:<8}", style), event.source.name)
)
def print_run_info(console: Console, title: str, rows: list[tuple[str, str]]) -> None:
+ from rich.panel import Panel
+ from rich.table import Table
+ from rich.text import Text
+
info = Table.grid(padding=(0, 2))
info.add_column(style="bold cyan")
info.add_column()
@@ -150,6 +172,8 @@ def _shown(folder: str, name: str) -> str:
def _detail_text(entry) -> Text:
+ from rich.text import Text
+
style = "dim" if entry.status == Status.SUCCESS else ""
text = Text(entry.message, style=style)
for warning in entry.warnings:
@@ -161,6 +185,10 @@ def print_summary(
console: Console, result: RunResult, title: str | None = None, kind: str = "pack"
) -> None:
"""Print one row per file. ``kind`` is pack, unpack or verify."""
+ from rich.markup import escape
+ from rich.table import Table
+ from rich.text import Text
+
table = Table(title=title or SUMMARY_TITLES.get(kind, ""), show_lines=False)
# The same columns as `inspect`: the original file, its archive, the outcome
@@ -196,11 +224,18 @@ def print_summary(
if status != Status.PARTIAL and result.count(status)
)
console.print(f"[bold]Summary:[/bold] {escape(counts or 'nothing to do')}")
+ if result.resume:
+ console.print(
+ f"[bold]Resumed:[/bold] {result.reused} reused from the earlier run, "
+ f"{result.redone} redone."
+ )
if result.manifest and any(e.status == Status.SUCCESS for e in result.entries):
console.print(f"Manifest: {escape(str(result.manifest))}")
def _archive_status(entry: InspectEntry) -> Text:
+ from rich.text import Text
+
if entry.complete is True:
return Text("OK", style="green")
if entry.complete is False:
@@ -209,6 +244,8 @@ def _archive_status(entry: InspectEntry) -> Text:
def _inspect_details(entry: InspectEntry) -> Text:
+ from rich.text import Text
+
notes = []
if entry.encrypted:
notes.append(("encrypted", "yellow"))
@@ -229,6 +266,12 @@ def print_inspect(
console: Console, entries: list[InspectEntry], source: str | None = None
) -> None:
"""Describe archives with the same layout and colours as the result tables."""
+ from rich.markup import escape
+ from rich.table import Table
+ from rich.text import Text
+
+ from batchvault.core.options import format_size
+
original = sum(e.size or 0 for e in entries)
stored = sum(e.stored_size or 0 for e in entries)
missing = sum(1 for e in entries if e.complete is False)
@@ -284,6 +327,9 @@ def print_inspect(
def print_checks(console: Console, checks: list[Check]) -> None:
+ from rich.table import Table
+ from rich.text import Text
+
table = Table(title="BatchVault doctor")
table.add_column("Check")
table.add_column("Result")
diff --git a/src/batchvault/core/__init__.py b/src/batchvault/core/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/src/batchvault/core/api/__init__.py b/src/batchvault/core/api/__init__.py
new file mode 100644
index 0000000..c9e1e4d
--- /dev/null
+++ b/src/batchvault/core/api/__init__.py
@@ -0,0 +1,49 @@
+"""Public Python API. The command-line interface is a thin wrapper over this.
+
+Each operation lives in its own module and is loaded on first use, so importing
+one of them does not pull in the others.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from batchvault.core.api.listing import inspect, list_archives, list_files
+ from batchvault.core.api.packing.run import pack
+ from batchvault.core.api.unpacking.run import unpack
+ from batchvault.core.api.verifying import verify
+
+__all__ = [
+ "inspect",
+ "list_archives",
+ "list_files",
+ "pack",
+ "unpack",
+ "verify",
+]
+
+# Public name to the module that defines it.
+_LAZY = {
+ "inspect": "batchvault.core.api.listing",
+ "list_archives": "batchvault.core.api.listing",
+ "list_files": "batchvault.core.api.listing",
+ "pack": "batchvault.core.api.packing.run",
+ "unpack": "batchvault.core.api.unpacking.run",
+ "verify": "batchvault.core.api.verifying",
+}
+
+
+def __getattr__(name: str) -> object:
+ module = _LAZY.get(name)
+ if module is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ from importlib import import_module
+
+ value = getattr(import_module(module), name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(__all__)
diff --git a/src/batchvault/core/api/_common.py b/src/batchvault/core/api/_common.py
new file mode 100644
index 0000000..2e3ae83
--- /dev/null
+++ b/src/batchvault/core/api/_common.py
@@ -0,0 +1,185 @@
+"""Helpers shared by more than one operation of the public API."""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+from typing import TYPE_CHECKING
+
+from batchvault.core.models import ProgressEvent
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.core.locate import Target
+ from batchvault.core.models import FileResult
+
+log = logging.getLogger(__name__)
+
+
+Progress = Callable[[ProgressEvent], None]
+
+
+RETRY_BASE_DELAY = 1.0
+
+
+RETRY_MAX_DELAY = 30.0
+
+
+ENCRYPTED_MESSAGE = "Archive is encrypted; a password is required."
+
+
+def _log_result(action: str, outcome: FileResult) -> None:
+ from batchvault.core.models import Status
+
+ level = logging.WARNING if outcome.status == Status.FAILED else logging.INFO
+ log.log(
+ level, "%s %s: %s %s", action, outcome.source, outcome.status, outcome.message
+ )
+
+
+def _split_jobs(jobs: int | None, files: int) -> tuple[int, int]:
+ """Split one thread budget into (files in parallel, threads per file).
+
+ A budget of 0 means no worker threads: one file at a time, compressed in
+ the calling thread.
+ """
+ from batchvault.core.errors import InputError
+
+ total = jobs or 0
+ if total < 0:
+ raise InputError("jobs cannot be negative")
+ if total == 0:
+ return 1, 0
+ workers = max(1, min(files, total))
+ return workers, max(1, total // workers)
+
+
+def _notify(progress: Progress | None, event: ProgressEvent) -> None:
+ if progress:
+ progress(event)
+
+
+def _retry(call, retries: int, retry_on: tuple[type[Exception], ...]):
+ """Run ``call()``, retrying transient failures with exponential backoff."""
+ import time
+
+ attempt = 0
+ while True:
+ try:
+ return call()
+ except FileNotFoundError:
+ raise
+ except retry_on as exc:
+ if attempt >= retries:
+ raise
+ delay = min(RETRY_BASE_DELAY * 2**attempt, RETRY_MAX_DELAY)
+ log.warning(
+ "Attempt %d failed (%s); retrying in %.1fs", attempt + 1, exc, delay
+ )
+ time.sleep(delay)
+ attempt += 1
+
+
+class _Stopped(Exception):
+ """A queued file was not started because the run was stopped."""
+
+
+def _run_all(workers: int, work: Callable, items: list) -> list:
+ """Run ``work`` on every item in order; after an error or Ctrl+C, start no more.
+
+ Files already running finish, so each is either complete or absent. The
+ first error is raised.
+ """
+ import threading
+ from concurrent.futures import ThreadPoolExecutor
+
+ stop = threading.Event()
+
+ def guarded(item):
+ if stop.is_set():
+ raise _Stopped
+ try:
+ return work(item)
+ except BaseException:
+ stop.set()
+ raise
+
+ executor = ThreadPoolExecutor(max_workers=workers)
+ try:
+ results = list(executor.map(guarded, items))
+ except BaseException:
+ stop.set()
+ executor.shutdown(wait=True, cancel_futures=True)
+ raise
+ executor.shutdown()
+ return results
+
+
+def _reject_resume_conflicts(function, given: dict) -> None:
+ """Refuse settings passed together with ``resume`` (they come from the journal)."""
+ import inspect as inspect_module
+ from pathlib import Path
+
+ from batchvault.core.errors import OptionsError
+
+ parameters = inspect_module.signature(function).parameters
+ clashing = []
+ for name, value in given.items():
+ default = parameters[name].default
+ if value is None or isinstance(value, bool | int | str | Path):
+ differs = value != default
+ else:
+ differs = list(value) != list(default)
+ if differs:
+ clashing.append(name)
+ if clashing:
+ raise OptionsError(
+ "resume takes every setting from the earlier run, so do not also pass: "
+ + ", ".join(clashing)
+ + "."
+ )
+
+
+def _folder_path(base: Path, folder: str) -> Path:
+ """``base`` joined with a stored forward-slash folder (``base`` if empty)."""
+ return base.joinpath(*folder.split("/")) if folder else base
+
+
+def _resolve(source: str | Path, public_key: str | Path | None) -> list[Target]:
+ """Find the archives for ``source``.
+
+ With ``public_key``, each manifest's signature is checked on the very bytes
+ that are then parsed, so the file cannot be swapped in between.
+ """
+ from pathlib import Path
+
+ from batchvault.archive import signing
+ from batchvault.archive.manifest import Manifest
+ from batchvault.core.errors import ManifestError, SignatureError
+ from batchvault.core.locate import resolve_targets
+
+ if not public_key:
+ return resolve_targets(source)
+
+ key = Path(public_key)
+
+ def load_signed(path: Path) -> Manifest:
+ try:
+ raw = path.read_bytes()
+ except OSError as exc:
+ raise ManifestError(f"Cannot read manifest {path}: {exc}") from exc
+ signing.verify_manifest_bytes(raw, path, key)
+ return Manifest.from_bytes(raw, path)
+
+ targets = resolve_targets(source, load_signed)
+ if any(target.manifest is None for target in targets):
+ raise SignatureError("Signature check needs a manifest, but none was found.")
+ return targets
+
+
+def _plan_event(where: Path, targets: list[Target]) -> ProgressEvent:
+ # Sizes are only known from a manifest; without one the UI counts files.
+ known = all(target.entry for target in targets)
+ total = sum(target.entry.size for target in targets if target.entry)
+ return ProgressEvent("plan", where, total if known else 0, count=len(targets))
diff --git a/src/batchvault/core/api/listing.py b/src/batchvault/core/api/listing.py
new file mode 100644
index 0000000..db014ae
--- /dev/null
+++ b/src/batchvault/core/api/listing.py
@@ -0,0 +1,108 @@
+"""Describe archives: ``inspect``, ``list_files`` and ``list_archives``."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from collections.abc import Iterable
+ from pathlib import Path
+
+ from batchvault.core.models import FileEntry, InspectEntry
+
+
+def inspect(source: str | Path) -> list[InspectEntry]:
+ """Describe archives from their manifest (or file names when there is none)."""
+ from batchvault.archive.volumes import existing_volumes
+ from batchvault.core.locate import resolve_targets
+ from batchvault.core.models import InspectEntry
+
+ found = []
+ for target in resolve_targets(source):
+ entry = target.entry
+ if entry is None:
+ volumes = existing_volumes(
+ target.volume.parent, target.volume.name.rsplit(".", 1)[0]
+ )
+ found.append(
+ InspectEntry(
+ name=target.name,
+ path=target.volume,
+ engine="7zip" if ".7z." in target.volume.name.lower() else "native",
+ stored_size=sum(v.stat().st_size for v in volumes),
+ volumes=len(volumes),
+ )
+ )
+ continue
+
+ base_dir = target.base_dir or target.volume.parent
+ paths = [base_dir / volume.file for volume in entry.volumes]
+ found.append(
+ InspectEntry(
+ name=entry.name,
+ folder=entry.folder,
+ path=paths[0],
+ engine=entry.engine,
+ format=entry.format,
+ level=entry.level,
+ encrypted=entry.encrypted,
+ size=entry.size,
+ stored_size=sum(volume.size for volume in entry.volumes),
+ volumes=len(paths),
+ complete=all(path.is_file() for path in paths),
+ )
+ )
+ return found
+
+
+def list_files(
+ folder: str | Path,
+ *,
+ include: Iterable[str] = (),
+ exclude: Iterable[str] = (),
+) -> list[FileEntry]:
+ """Files under ``folder`` (recursively), filtered by include/exclude patterns.
+
+ Patterns match against the path relative to ``folder``.
+ """
+ from pathlib import Path
+
+ from batchvault.core.errors import InputError
+ from batchvault.core.filters import PathFilter
+ from batchvault.core.models import FileEntry
+
+ root = Path(folder).expanduser()
+ if not root.is_dir():
+ raise InputError(f"Folder not found: {root}")
+
+ path_filter = PathFilter(include, exclude)
+ # Record the folder's parent so that packing the listing keeps the folder's
+ # own name (Resources/...), not just what is inside it.
+ recorded = root.resolve().parent
+ return [
+ FileEntry(path.resolve(), path.stat().st_size, recorded)
+ for path in sorted(root.rglob("*"))
+ if path.is_file() and path_filter.matches(path.relative_to(root).as_posix())
+ ]
+
+
+def list_archives(folder: str | Path) -> list[FileEntry]:
+ """Archives under ``folder``: from its manifest if present, else by scanning."""
+ from pathlib import Path
+
+ from batchvault.core.errors import InputError
+ from batchvault.core.locate import resolve_targets
+ from batchvault.core.models import FileEntry
+
+ root = Path(folder).expanduser()
+ if not root.is_dir():
+ raise InputError(f"Folder not found: {root}")
+
+ try:
+ targets = resolve_targets(root)
+ except InputError:
+ return []
+ return [
+ FileEntry(target.volume.resolve(), target.entry.size if target.entry else None)
+ for target in targets
+ ]
diff --git a/src/batchvault/core/api/packing/__init__.py b/src/batchvault/core/api/packing/__init__.py
new file mode 100644
index 0000000..b763140
--- /dev/null
+++ b/src/batchvault/core/api/packing/__init__.py
@@ -0,0 +1,23 @@
+"""The ``pack`` operation of the public API, loaded on first use."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from batchvault.core.api.packing.run import pack
+
+__all__ = ["pack"]
+
+
+def __getattr__(name: str) -> object:
+ if name != "pack":
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ from batchvault.core.api.packing.run import pack
+
+ globals()[name] = pack
+ return pack
+
+
+def __dir__() -> list[str]:
+ return sorted(__all__)
diff --git a/src/batchvault/core/api/packing/finish.py b/src/batchvault/core/api/packing/finish.py
new file mode 100644
index 0000000..c5d47c2
--- /dev/null
+++ b/src/batchvault/core/api/packing/finish.py
@@ -0,0 +1,114 @@
+"""What happens after the files are packed: README, signature, source deletion."""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.archive.manifest import Manifest, ManifestEntry
+ from batchvault.core.api.packing.plan import Confirm
+ from batchvault.core.models import FileResult, RunResult
+ from batchvault.core.options import PackOptions
+ from batchvault.engines.base import Engine
+
+log = logging.getLogger(__name__)
+
+
+def _write_readme(
+ manifest_path: Path,
+ manifest: Manifest,
+ backend: Engine,
+ options: PackOptions,
+ auto_level: bool,
+ include: tuple[str, ...],
+ exclude: tuple[str, ...],
+ update: bool,
+ verified: bool,
+ delete_source: bool,
+ encrypted: bool,
+ signed: bool,
+ result: RunResult,
+) -> None:
+ from batchvault.archive.readme import (
+ README_NAME,
+ ReadmeContext,
+ render_readme,
+ write_readme,
+ )
+ from batchvault.core.api.packing.plan import _format_name
+
+ context = ReadmeContext(
+ engine=backend.name,
+ format=_format_name(backend, options),
+ level="auto" if auto_level else str(options.level),
+ volume_size=options.volume_size,
+ encrypted=encrypted,
+ signed=signed,
+ update=update,
+ verified=verified,
+ delete_source=delete_source,
+ include=include,
+ exclude=exclude,
+ manifest_name=manifest_path.name,
+ )
+ warning = write_readme(
+ manifest_path.with_name(README_NAME), render_readme(context, manifest)
+ )
+ if warning:
+ result.warnings.append(warning)
+
+
+def _sign_or_clear(manifest_path: Path, private_key, result: RunResult) -> None:
+ from batchvault.archive import signing
+
+ if private_key is not None:
+ signing.sign_manifest(manifest_path, private_key)
+ return
+
+ stale = signing.signature_path(manifest_path)
+ if stale.exists():
+ stale.unlink()
+ result.warnings.append(
+ "The manifest changed, so its old signature was removed. "
+ "Sign it again with a private key."
+ )
+
+
+def _delete_sources(
+ backend: Engine,
+ outcomes: list[tuple[FileResult, ManifestEntry | None]],
+ password: str | None,
+ confirm: Confirm | None,
+ result: RunResult,
+) -> None:
+ from batchvault.core.api.packing.single import _roundtrip_problem
+ from batchvault.core.models import Status
+
+ verified = []
+ for outcome, entry in outcomes:
+ if outcome.status != Status.SUCCESS or entry is None:
+ continue
+ problem = _roundtrip_problem(backend, outcome, entry, password)
+ if problem:
+ outcome.warnings.append(f"Original kept: {problem}.")
+ log.warning("Original kept for %s: %s", outcome.source, problem)
+ else:
+ verified.append(outcome)
+
+ if not verified:
+ return
+ if confirm is not None and not confirm([o.source for o in verified]):
+ result.warnings.append("Original files were kept (deletion not confirmed).")
+ return
+
+ for outcome in verified:
+ try:
+ outcome.source.unlink()
+ except OSError as exc:
+ outcome.warnings.append(f"Could not delete the original: {exc}")
+ else:
+ outcome.message += "; original deleted"
+ log.info("Deleted original %s", outcome.source)
diff --git a/src/batchvault/core/api/packing/plan.py b/src/batchvault/core/api/packing/plan.py
new file mode 100644
index 0000000..9389b57
--- /dev/null
+++ b/src/batchvault/core/api/packing/plan.py
@@ -0,0 +1,169 @@
+"""Decide what to pack and whether an existing archive can be reused."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ import os
+
+ from batchvault.archive.manifest import Manifest, ManifestEntry
+ from batchvault.core.levels import LevelChoice
+ from batchvault.core.models import RunResult
+ from batchvault.core.options import PackOptions
+ from batchvault.engines.base import Engine
+
+
+Confirm = Callable[[list[Path]], bool]
+
+
+def _relative(path: Path, base: Path) -> str:
+ import os
+
+ try:
+ return Path(os.path.relpath(path, base)).as_posix()
+ except ValueError: # different drive on Windows
+ return str(path)
+
+
+def _format_name(backend: Engine, options: PackOptions) -> str:
+ return "7z" if backend.name == "7zip" else options.format
+
+
+def _up_to_date(
+ entry: ManifestEntry,
+ stat: os.stat_result,
+ backend: Engine,
+ options: PackOptions,
+ manifest_dir: Path,
+) -> bool:
+ """True if an archive already matches the file and the requested settings."""
+ if not entry.mtime_ns or entry.mtime_ns != stat.st_mtime_ns:
+ return False
+ if entry.size != stat.st_size:
+ return False
+ if (
+ entry.engine != backend.name
+ or entry.format != _format_name(backend, options)
+ or entry.level != options.level
+ or entry.volume_size != options.volume_size
+ or entry.encrypted != bool(options.password)
+ ):
+ return False
+
+ for volume in entry.volumes:
+ path = manifest_dir / volume.file
+ if not path.is_file() or path.stat().st_size != volume.size:
+ return False
+ return True
+
+
+def _check_resume_password(
+ backend: Engine,
+ journaled: dict[str, ManifestEntry],
+ manifest_dir: Path,
+ password: str,
+) -> None:
+ """Refuse a password that does not open the archives made by the earlier run.
+
+ Reusing files made with another password would leave an archive that cannot
+ be restored with a single password. The smallest intact encrypted archive
+ is tested, which keeps the check cheap.
+ """
+ from batchvault.core.errors import EngineError, ResumeError
+
+ for entry in sorted(journaled.values(), key=lambda item: item.size):
+ if not entry.encrypted or entry.check_volumes(manifest_dir):
+ continue
+ try:
+ backend.test(manifest_dir / entry.volumes[0].file, password=password)
+ except EngineError as exc:
+ raise ResumeError(
+ "The password does not match the archives made by the earlier "
+ f"run ({exc})."
+ ) from exc
+ return
+
+
+def _file_options(
+ options: PackOptions, source: Path, auto_level: bool
+) -> tuple[PackOptions, LevelChoice | None]:
+ """The options for one file; with ``--level auto`` the level suits its type."""
+ import dataclasses
+
+ from batchvault.core.levels import choose_level
+
+ if not auto_level or options.format == "store": # store ignores the level
+ return options, None
+ choice = choose_level(source)
+ return dataclasses.replace(options, level=choice.level), choice
+
+
+def _plan_pack(
+ backend: Engine,
+ sources: list[Path],
+ folders: dict[Path, str],
+ out_root: Path,
+ manifest_dir: Path,
+ options: PackOptions,
+ warnings: list[str],
+ known: Manifest | None,
+ auto_level: bool = False,
+) -> RunResult:
+ from batchvault.archive.volumes import existing_volumes
+ from batchvault.core.api._common import _folder_path
+ from batchvault.core.models import FileResult, RunResult, Status
+
+ result = RunResult(warnings=warnings, engine=backend.name)
+ suffix = backend.capabilities.archive_suffix
+
+ for source in sources:
+ stat = source.stat()
+ folder = folders[source]
+ existing = existing_volumes(
+ _folder_path(out_root, folder) / source.name, f"{source.name}{suffix}"
+ )
+ key = f"{folder}/{source.name}" if folder else source.name
+ entry = known.entries.get(key) if known else None
+ file_options, choice = _file_options(options, source, auto_level)
+
+ if entry and _up_to_date(entry, stat, backend, file_options, manifest_dir):
+ outcome = FileResult(
+ source, Status.SKIPPED, "Would skip: up to date.", stat.st_size
+ )
+ elif existing and not (file_options.force or known is not None):
+ outcome = FileResult(
+ source,
+ Status.SKIPPED,
+ "Would skip: archive already exists.",
+ stat.st_size,
+ )
+ else:
+ message = "Would update" if existing else "Would archive"
+ if choice:
+ message += f" at level {choice.level} ({choice.kind})"
+ outcome = FileResult(source, Status.SUCCESS, message + ".", stat.st_size)
+ outcome.folder = folder
+ outcome.level = file_options.level
+ outcome.name = source.name
+ result.entries.append(outcome)
+ return result
+
+
+def _pack_summary(
+ size: int, stored: int, volumes: int, verified: bool, choice: LevelChoice | None
+) -> str:
+ """e.g. ``3 volumes, 12.3 MiB -> 4.1 MiB (33%), level 6 (text), verified``."""
+ from batchvault.core.options import format_size
+
+ sizes = f"{format_size(size)} -> {format_size(stored)}"
+ if size:
+ sizes += f" ({stored / size:.0%})"
+ parts = [f"{volumes} volume{'' if volumes == 1 else 's'}", sizes]
+ if choice:
+ parts.append(f"level {choice.level} ({choice.kind})")
+ if verified:
+ parts.append("verified")
+ return ", ".join(parts)
diff --git a/src/batchvault/core/api/packing/run.py b/src/batchvault/core/api/packing/run.py
new file mode 100644
index 0000000..c645b13
--- /dev/null
+++ b/src/batchvault/core/api/packing/run.py
@@ -0,0 +1,303 @@
+"""The ``pack`` operation: plan the run, pack each file, write the manifest."""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+from batchvault.core.levels import AUTO
+
+if TYPE_CHECKING:
+ from collections.abc import Iterable
+ from pathlib import Path
+
+ from batchvault.archive.manifest import ManifestEntry
+ from batchvault.core.api._common import Progress
+ from batchvault.core.api.packing.plan import Confirm
+ from batchvault.core.models import FileResult, RunResult
+
+log = logging.getLogger(__name__)
+
+
+def pack(
+ files: Iterable[str | Path] = (),
+ output: str | Path = "archives",
+ *,
+ list_file: str | Path | None = None,
+ root: str | Path | None = None,
+ engine: str = "auto",
+ seven_zip: str | Path | None = None,
+ format: str | None = None,
+ level: int | str = AUTO,
+ volume_size: str | int = "10m",
+ jobs: int = 0,
+ include: Iterable[str] = (),
+ exclude: Iterable[str] = (),
+ force: bool = False,
+ update: bool = False,
+ verify: bool = False,
+ manifest: str | Path | None = None,
+ allow_archives: bool = False,
+ strict: bool = False,
+ dry_run: bool = False,
+ password: str | None = None,
+ retries: int = 3,
+ sign_key: str | Path | None = None,
+ sign_key_password: str | None = None,
+ delete_source: bool = False,
+ readme: bool = True,
+ resume: bool = False,
+ confirm: Confirm | None = None,
+ progress: Progress | None = None,
+) -> RunResult:
+ """Archive files into split volumes and record them in a manifest.
+
+ Each file gets its own folder ``//`` holding its volumes.
+ Folders are kept: a file in a subfolder is stored under
+ ``///`` and ``unpack`` puts it back there. The
+ folder is relative to the one ``list files`` recorded (so the listed folder's
+ name is kept), else to the deepest folder all the files share; ``root``
+ overrides this with a folder of your choice.
+
+ Existing archives are skipped unless ``force`` is set; with ``update`` they
+ are skipped only if the file and settings are unchanged, and re-archived
+ otherwise. ``jobs`` is the total thread budget; ``progress`` is called from
+ worker threads. With ``strict``, settings the engine cannot honour raise an
+ error instead of a warning. With ``dry_run`` nothing is written.
+
+ ``password`` encrypts the archives (the native engine needs
+ ``batchvault[crypto]``). ``retries`` retries a file that failed on an I/O or
+ engine error. ``sign_key`` signs the manifest with an Ed25519 private key.
+ ``delete_source`` deletes each original after a full restore-and-compare
+ proves the archive is good (and ``confirm``, if given, agrees).
+
+ Unless ``readme`` is false, a ``BATCHVAULT-README.md`` is written next to the
+ manifest: how the archive was made, how to install BatchVault and how to
+ restore it. It never holds a password or key. An existing file of that name
+ is only replaced if BatchVault wrote it.
+
+ While it runs, ``pack`` keeps a journal (``.batchvault-pack.jsonl``) in the
+ output folder, recording every finished file; it is removed when the run ends
+ without failures. With ``resume`` the run continues from that journal: every
+ setting (files, engine, format, level, volume size, patterns, ...) is taken
+ from the earlier run, so passing one of them as well is an error, and so is
+ having no journal. Files that are finished, unchanged and whose volumes still
+ match their checksums are reused; the rest are done again. ``password`` and
+ ``sign_key`` are never stored, so give them again if the earlier run used
+ them. ``jobs``, ``retries``, ``seven_zip``, ``confirm`` and ``progress`` may
+ be passed freely.
+ """
+ from pathlib import Path
+
+ from batchvault.archive import resume as journals
+ from batchvault.archive import signing
+ from batchvault.archive.crypto import require_cryptography
+ from batchvault.archive.manifest import MANIFEST_NAME, Manifest
+ from batchvault.archive.volumes import cleanup_staging
+ from batchvault.core.api._common import (
+ _notify,
+ _reject_resume_conflicts,
+ _run_all,
+ _split_jobs,
+ )
+ from batchvault.core.api.packing.finish import (
+ _delete_sources,
+ _sign_or_clear,
+ _write_readme,
+ )
+ from batchvault.core.api.packing.plan import _check_resume_password, _plan_pack
+ from batchvault.core.api.packing.single import _pack_one
+ from batchvault.core.errors import OptionsError, ResumeError
+ from batchvault.core.filters import PathFilter
+ from batchvault.core.inputs import gather_sources
+ from batchvault.core.levels import parse_level
+ from batchvault.core.models import FileResult, ProgressEvent, RunResult, Status
+ from batchvault.core.options import PackOptions, parse_size
+ from batchvault.engines.registry import get_engine, option_warnings
+
+ if retries < 0:
+ raise OptionsError("retries cannot be negative")
+ files, include, exclude = list(files), tuple(include), tuple(exclude)
+ out_root = Path(output).expanduser().resolve()
+
+ saved = None
+ journaled: dict[str, ManifestEntry] = {}
+ loaded = None
+ if resume:
+ _reject_resume_conflicts(
+ pack,
+ dict(
+ files=files, list_file=list_file, root=root, engine=engine,
+ format=format, level=level, volume_size=volume_size,
+ include=include, exclude=exclude, force=force, update=update,
+ verify=verify, manifest=manifest, allow_archives=allow_archives,
+ strict=strict, dry_run=dry_run, delete_source=delete_source,
+ readme=readme,
+ ),
+ ) # fmt: skip
+ saved, journaled, loaded = journals.load_pack(out_root / journals.PACK_JOURNAL)
+ engine, format, level = saved.engine, saved.format, saved.level
+ volume_size, include, exclude = (
+ saved.volume_size,
+ tuple(saved.include),
+ tuple(saved.exclude),
+ )
+ manifest, update, verify = saved.manifest, saved.update, saved.verify
+ delete_source, strict, readme = saved.delete_source, saved.strict, saved.readme
+ if saved.encrypted and not password:
+ raise ResumeError(
+ "The earlier run was encrypted, so the password is required to "
+ "resume (it is never stored)."
+ )
+ if password and not saved.encrypted:
+ raise ResumeError(
+ "The earlier run was not encrypted; do not pass a password, or "
+ "the archive would mix encrypted and plain files."
+ )
+ if saved.signed and not sign_key:
+ raise ResumeError(
+ "The earlier run signed the manifest, so the private key is "
+ "required to sign it again (it is never stored)."
+ )
+ if sign_key and not saved.signed:
+ raise ResumeError(
+ "The earlier run did not sign the manifest; do not pass a private key."
+ )
+
+ level = parse_level(level)
+ auto_level = level == AUTO # each file gets the level that suits its type
+
+ if saved is not None:
+ sourced = [(Path(path), folder) for path, folder in saved.sources]
+ else:
+ root_path = Path(root).expanduser().resolve() if root is not None else None
+ sourced = gather_sources(
+ files,
+ list_file,
+ path_filter=PathFilter(include, exclude),
+ allow_archives=allow_archives,
+ root=root_path,
+ )
+ sources = [path for path, _ in sourced]
+ folders = dict(sourced)
+ backend = get_engine(engine, seven_zip)
+ warnings = option_warnings(backend, format, level)
+ if strict and warnings:
+ raise OptionsError(" ".join(warnings))
+ if password and backend.name == "native":
+ require_cryptography()
+
+ private_key = None
+ if sign_key and not dry_run:
+ private_key = signing.load_private_key(Path(sign_key), sign_key_password)
+
+ workers, per_file_jobs = _split_jobs(jobs, len(sources))
+ options = PackOptions(
+ level=level if isinstance(level, int) else 5,
+ volume_size=parse_size(volume_size),
+ jobs=per_file_jobs,
+ force=force,
+ format=format or "xz",
+ password=password,
+ )
+
+ manifest_path = (
+ Path(manifest).expanduser().resolve() if manifest else out_root / MANIFEST_NAME
+ )
+ # Load first so a damaged manifest stops the run before any work is done.
+ known = Manifest.load(manifest_path) if manifest_path.exists() else Manifest()
+ if dry_run:
+ return _plan_pack(
+ backend, sources, folders, out_root, manifest_path.parent, options,
+ warnings, known if update else None, auto_level,
+ ) # fmt: skip
+
+ cleanup_staging(out_root)
+ out_root.mkdir(parents=True, exist_ok=True)
+
+ journal_path = out_root / journals.PACK_JOURNAL
+ missing: list[tuple[FileResult, ManifestEntry | None]] = []
+ if resume and loaded is not None:
+ if password:
+ _check_resume_password(backend, journaled, manifest_path.parent, password)
+ present = []
+ for source in sources: # a source that vanished cannot be archived again
+ if source.is_file():
+ present.append(source)
+ continue
+ failure = FileResult(source, Status.FAILED, "Source file not found.")
+ failure.folder, failure.name = folders[source], source.name
+ missing.append((failure, None))
+ sources = present
+ journal = journals.Journal.reopen(journal_path, loaded.good_size)
+ else:
+ journal = journals.Journal.create(
+ journal_path,
+ "pack",
+ journals.PackSettings(
+ sources=[(str(path), folder) for path, folder in sourced],
+ engine=backend.name, format=format, level=level,
+ volume_size=options.volume_size, jobs=jobs, retries=retries,
+ include=list(include), exclude=list(exclude),
+ manifest=str(manifest_path), update=update, verify=verify,
+ delete_source=delete_source, strict=strict, readme=readme,
+ encrypted=bool(password), signed=bool(sign_key),
+ ).to_dict(),
+ ) # fmt: skip
+
+ log.info(
+ "%s %d file(s) with %s into %s",
+ "Resuming" if resume else "Packing",
+ len(sources),
+ backend.name,
+ out_root,
+ )
+ total = sum(source.stat().st_size for source in sources)
+ _notify(progress, ProgressEvent("plan", out_root, total, count=len(sources)))
+
+ def work(source: Path) -> tuple[FileResult, ManifestEntry | None]:
+ folder = folders[source]
+ key = f"{folder}/{source.name}" if folder else source.name
+ outcome, entry = _pack_one(
+ backend, source, folder, out_root, manifest_path.parent,
+ options, verify, progress, known if update else None, retries,
+ auto_level, journaled.get(key), resume,
+ ) # fmt: skip
+ if entry is not None and not outcome.resumed:
+ journal.append({"kind": "file", "key": key, "entry": entry.to_dict()})
+ return outcome, entry
+
+ try:
+ outcomes = _run_all(workers, work, sources)
+ except BaseException:
+ journal.close() # keep it: it is what --resume continues from
+ raise
+ outcomes += missing
+
+ result = RunResult(
+ warnings=warnings, manifest=manifest_path, engine=backend.name, resume=resume
+ )
+ for file_result, entry in outcomes:
+ result.entries.append(file_result)
+ if entry:
+ known.entries[entry.path] = entry
+
+ if any(entry for _, entry in outcomes):
+ known.save(manifest_path)
+ _sign_or_clear(manifest_path, private_key, result)
+ if readme:
+ _write_readme(
+ manifest_path, known, backend, options, auto_level, include, exclude,
+ update, verify, delete_source, bool(password), private_key is not None,
+ result,
+ ) # fmt: skip
+
+ if delete_source:
+ _delete_sources(backend, outcomes, password, confirm, result)
+
+ # Keep the journal while something failed so --resume can retry just that.
+ if any(file_result.status == Status.FAILED for file_result, _ in outcomes):
+ journal.close()
+ else:
+ journal.remove()
+ return result
diff --git a/src/batchvault/core/api/packing/single.py b/src/batchvault/core/api/packing/single.py
new file mode 100644
index 0000000..553b4f9
--- /dev/null
+++ b/src/batchvault/core/api/packing/single.py
@@ -0,0 +1,190 @@
+"""Pack one file into volumes and check the result."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ import os
+ from pathlib import Path
+
+ from batchvault.archive.manifest import Manifest, ManifestEntry
+ from batchvault.core.api._common import Progress
+ from batchvault.core.models import FileResult
+ from batchvault.core.options import PackOptions
+ from batchvault.engines.base import Engine
+
+
+def _pack_one(
+ backend: Engine,
+ source: Path,
+ folder: str,
+ out_root: Path,
+ manifest_dir: Path,
+ options: PackOptions,
+ verify: bool,
+ progress: Progress | None,
+ known: Manifest | None,
+ retries: int,
+ auto_level: bool = False,
+ journaled: ManifestEntry | None = None,
+ resume: bool = False,
+) -> tuple[FileResult, ManifestEntry | None]:
+ import dataclasses
+
+ from batchvault.archive.manifest import ManifestEntry, VolumeInfo, hash_file
+ from batchvault.core.api._common import _folder_path, _log_result, _notify, _retry
+ from batchvault.core.api.packing.plan import (
+ _file_options,
+ _format_name,
+ _pack_summary,
+ _relative,
+ _up_to_date,
+ )
+ from batchvault.core.errors import BatchVaultError, EngineError, OutputExistsError
+ from batchvault.core.models import FileResult, ProgressEvent, Status
+ from batchvault.core.options import format_size
+
+ stat = source.stat()
+ size = stat.st_size
+ _notify(progress, ProgressEvent("start", source, size))
+ options, choice = _file_options(options, source, auto_level)
+
+ entry = None
+ try:
+ key = f"{folder}/{source.name}" if folder else source.name
+ previous = known.entries.get(key) if known is not None else None
+ redone = ""
+ if journaled is not None:
+ redone = _reuse_problem(journaled, stat, manifest_dir)
+
+ if journaled is not None and not redone:
+ # An earlier run finished this file and nothing has changed since.
+ entry = journaled
+ volumes = [manifest_dir / volume.file for volume in journaled.volumes]
+ stored = sum(volume.size for volume in journaled.volumes)
+ outcome = FileResult(
+ source,
+ Status.SUCCESS,
+ f"Resumed: already archived, {len(volumes)} "
+ f"volume{'' if len(volumes) == 1 else 's'}, "
+ f"{format_size(stored)}, checksums verified",
+ size,
+ volumes,
+ archive=volumes[0],
+ stored_size=stored,
+ level=journaled.level,
+ resumed=True,
+ )
+ elif previous and _up_to_date(previous, stat, backend, options, manifest_dir):
+ outcome = FileResult(source, Status.SKIPPED, "Up to date.", size)
+ else:
+ # In update mode a changed file replaces its old archive; a resumed
+ # run replaces what an earlier run left (its own output).
+ file_options = options
+ if known is not None or resume:
+ file_options = dataclasses.replace(options, force=True)
+
+ destination = _folder_path(out_root, folder) / source.name
+ volumes = _retry(
+ lambda: backend.pack(source, destination, file_options),
+ retries,
+ (OSError, EngineError),
+ )
+ entry = ManifestEntry(
+ name=source.name,
+ folder=folder,
+ size=size,
+ sha256=hash_file(source),
+ engine=backend.name,
+ format=_format_name(backend, options),
+ level=options.level,
+ volume_size=options.volume_size,
+ volumes=[
+ VolumeInfo(
+ _relative(volume, manifest_dir),
+ volume.stat().st_size,
+ hash_file(volume),
+ )
+ for volume in volumes
+ ],
+ encrypted=bool(options.password),
+ mtime_ns=stat.st_mtime_ns,
+ )
+ if verify:
+ backend.test(volumes[0], password=options.password)
+ stored = sum(volume.size for volume in entry.volumes)
+ outcome = FileResult(
+ source,
+ Status.SUCCESS,
+ _pack_summary(size, stored, len(volumes), verify, choice),
+ size,
+ volumes,
+ archive=volumes[0],
+ stored_size=stored,
+ level=options.level,
+ )
+ if redone:
+ outcome.redone = redone
+ outcome.message += f"; redone: {redone}"
+ except OutputExistsError as exc:
+ outcome = FileResult(source, Status.SKIPPED, str(exc), size)
+ except (BatchVaultError, OSError) as exc:
+ outcome, entry = FileResult(source, Status.FAILED, str(exc), size), None
+
+ outcome.folder = folder
+ outcome.name = source.name
+ _log_result("pack", outcome)
+ _notify(
+ progress,
+ ProgressEvent("finish", source, size, outcome.status, resumed=outcome.resumed),
+ )
+ return outcome, entry
+
+
+def _reuse_problem(
+ entry: ManifestEntry, stat: os.stat_result, manifest_dir: Path
+) -> str:
+ """Why a file an earlier run finished cannot be reused; empty if it can.
+
+ The volumes are checked against their checksums, never trusted by name or
+ size alone: a power cut can leave files that exist but are cut short.
+ """
+ if stat.st_size != entry.size or not entry.mtime_ns:
+ return "the source file changed"
+ if stat.st_mtime_ns != entry.mtime_ns:
+ return "the source file changed"
+ problems = entry.check_volumes(manifest_dir)
+ return problems[0] if problems else ""
+
+
+def _roundtrip_problem(
+ backend: Engine, outcome: FileResult, entry: ManifestEntry, password: str | None
+) -> str | None:
+ """Restore the archive and compare it with the original; None if identical."""
+ import tempfile
+ from pathlib import Path
+
+ from batchvault.archive.manifest import hash_file
+ from batchvault.archive.volumes import STAGING_PREFIX
+ from batchvault.core.errors import BatchVaultError
+
+ try:
+ parent = outcome.volumes[0].parent
+ with tempfile.TemporaryDirectory(dir=parent, prefix=STAGING_PREFIX) as tmp:
+ restored = backend.unpack(
+ outcome.volumes[0],
+ Path(tmp),
+ force=True,
+ password=password,
+ max_bytes=entry.size,
+ )
+ if [path.name for path in restored] != [entry.name]:
+ return "the archive does not contain the expected file"
+ if hash_file(restored[0]) != entry.sha256:
+ return "the restored file differs from the original"
+ if hash_file(outcome.source) != entry.sha256:
+ return "the original changed while it was being archived"
+ except (BatchVaultError, OSError) as exc:
+ return str(exc)
+ return None
diff --git a/src/batchvault/core/api/unpacking/__init__.py b/src/batchvault/core/api/unpacking/__init__.py
new file mode 100644
index 0000000..bb39dcb
--- /dev/null
+++ b/src/batchvault/core/api/unpacking/__init__.py
@@ -0,0 +1,23 @@
+"""The ``unpack`` operation of the public API, loaded on first use."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from batchvault.core.api.unpacking.run import unpack
+
+__all__ = ["unpack"]
+
+
+def __getattr__(name: str) -> object:
+ if name != "unpack":
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ from batchvault.core.api.unpacking.run import unpack
+
+ globals()[name] = unpack
+ return unpack
+
+
+def __dir__() -> list[str]:
+ return sorted(__all__)
diff --git a/src/batchvault/core/api/unpacking/restore.py b/src/batchvault/core/api/unpacking/restore.py
new file mode 100644
index 0000000..f8f9b60
--- /dev/null
+++ b/src/batchvault/core/api/unpacking/restore.py
@@ -0,0 +1,186 @@
+"""Restore one file from its volumes and place it in the destination."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.archive import resume as journals
+ from batchvault.archive.manifest import ManifestEntry
+ from batchvault.core.locate import Target
+ from batchvault.core.models import FileResult
+
+
+def _destination_folder(out_root: Path, folder: str) -> Path:
+ """Where files stored under ``folder`` are restored, kept inside ``out_root``."""
+ import os
+
+ from batchvault.core.api._common import _folder_path
+ from batchvault.core.errors import InputError
+ from batchvault.core.names import unsafe_on_windows
+
+ destination = _folder_path(out_root, folder)
+ if not folder:
+ return destination
+
+ if os.name == "nt" and any(unsafe_on_windows(part) for part in folder.split("/")):
+ raise InputError(f"The stored folder {folder!r} has names Windows cannot use.")
+
+ root = out_root.resolve()
+ if root not in destination.resolve().parents:
+ raise InputError(f"The stored folder {folder!r} points outside the output.")
+ return destination
+
+
+def _restore_one(
+ target: Target,
+ out_root: Path,
+ force: bool,
+ update: bool,
+ seven_zip: str | Path | None,
+ password: str | None,
+ retries: int,
+ max_bytes: int | None,
+ resume: bool = False,
+ placed: dict[str, str] | None = None,
+ journal: journals.Journal | None = None,
+) -> FileResult:
+ import os
+ import tempfile
+ from pathlib import Path
+
+ from batchvault.archive.manifest import hash_file
+ from batchvault.archive.volumes import STAGING_PREFIX
+ from batchvault.core.api._common import ENCRYPTED_MESSAGE, _retry
+ from batchvault.core.errors import BatchVaultError
+ from batchvault.core.models import FileResult, Status
+ from batchvault.core.options import format_size
+ from batchvault.engines.registry import engine_for_volume
+
+ volume = target.volume
+ entry = target.entry
+ try:
+ limit = max_bytes
+ if entry: # the manifest says how big the file is; never accept more
+ limit = entry.size if limit is None else min(limit, entry.size)
+
+ folder = entry.folder if entry else ""
+ destination_dir = _destination_folder(out_root, folder)
+ destination = destination_dir / target.name
+ redone = ""
+ if resume and entry is not None and destination.exists():
+ if _matches(destination, entry):
+ return FileResult(
+ volume,
+ Status.SUCCESS,
+ f"Resumed: already restored, {format_size(entry.size)}, "
+ "checksum verified",
+ entry.size,
+ restored=[destination],
+ resumed=True,
+ )
+ if destination.relative_to(out_root).as_posix() not in (placed or {}):
+ return FileResult(
+ volume,
+ Status.SKIPPED,
+ f"{target.name} exists and differs from the archive, and an "
+ "earlier run did not write it, so it was left alone. "
+ "Use force to overwrite.",
+ )
+ redone = "the restored file no longer matches its checksum"
+ elif destination.exists() and not force:
+ if not update:
+ return FileResult(
+ volume,
+ Status.SKIPPED,
+ f"{target.name} already exists. Use force to overwrite.",
+ )
+ if entry is None:
+ return FileResult(
+ volume,
+ Status.SKIPPED,
+ f"{target.name} exists and there is no manifest to compare with.",
+ )
+ if _matches(destination, entry):
+ return FileResult(volume, Status.SKIPPED, "Already up to date.")
+
+ if entry:
+ if entry.encrypted and not password:
+ return FileResult(volume, Status.FAILED, ENCRYPTED_MESSAGE)
+ problems = entry.check_volumes(target.base_dir or volume.parent)
+ if problems:
+ return FileResult(volume, Status.FAILED, "; ".join(problems))
+
+ backend = engine_for_volume(volume, seven_zip)
+ out_root.mkdir(parents=True, exist_ok=True)
+
+ with tempfile.TemporaryDirectory(dir=out_root, prefix=STAGING_PREFIX) as tmp:
+ restored = _retry(
+ lambda: backend.unpack(
+ volume, Path(tmp), force=True, password=password, max_bytes=limit
+ ),
+ retries,
+ (OSError,),
+ )
+
+ if entry:
+ names = [path.name for path in restored]
+ # macOS may hand back the same name in another Unicode form.
+ if [_nfc(n) for n in names] != [_nfc(entry.name)]:
+ return FileResult(
+ volume,
+ Status.FAILED,
+ f"Archive contains {names}, manifest expects {entry.name!r}.",
+ )
+ if hash_file(restored[0]) != entry.sha256:
+ return FileResult(
+ volume,
+ Status.FAILED,
+ f"Restored {entry.name} does not match its checksum.",
+ )
+
+ destination_dir.mkdir(parents=True, exist_ok=True)
+ final = []
+ for path in restored:
+ name = entry.name if entry else path.name
+ os.replace(path, destination_dir / name)
+ final.append(destination_dir / name)
+
+ if journal is not None and entry is not None:
+ journal.append(
+ {
+ "kind": "file",
+ "path": final[0].relative_to(out_root).as_posix(),
+ "sha256": entry.sha256,
+ "size": entry.size,
+ }
+ )
+
+ size = sum(path.stat().st_size for path in final)
+ checked = "checksum verified" if entry else "checksum not verified"
+ message = f"{format_size(size)} restored, {checked}"
+ if redone:
+ message += f"; redone: {redone}"
+ return FileResult(
+ volume, Status.SUCCESS, message, size, restored=final, redone=redone
+ )
+ except (BatchVaultError, OSError) as exc:
+ return FileResult(volume, Status.FAILED, str(exc))
+
+
+def _nfc(name: str) -> str:
+ import unicodedata
+
+ return unicodedata.normalize("NFC", name)
+
+
+def _matches(path: Path, entry: ManifestEntry) -> bool:
+ from batchvault.archive.manifest import hash_file
+
+ return (
+ path.is_file()
+ and path.stat().st_size == entry.size
+ and (hash_file(path) == entry.sha256)
+ )
diff --git a/src/batchvault/core/api/unpacking/run.py b/src/batchvault/core/api/unpacking/run.py
new file mode 100644
index 0000000..28b8dca
--- /dev/null
+++ b/src/batchvault/core/api/unpacking/run.py
@@ -0,0 +1,184 @@
+"""The ``unpack`` operation: plan the run and restore each file."""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.core.api._common import Progress
+ from batchvault.core.models import RunResult
+
+log = logging.getLogger(__name__)
+
+
+def unpack(
+ source: str | Path | None = None,
+ output: str | Path = "restored",
+ *,
+ force: bool = False,
+ update: bool = False,
+ jobs: int = 0,
+ seven_zip: str | Path | None = None,
+ password: str | None = None,
+ public_key: str | Path | None = None,
+ retries: int = 3,
+ max_output_size: str | int | None = None,
+ resume: bool = False,
+ progress: Progress | None = None,
+) -> RunResult:
+ """Restore files from a manifest, a folder, any volume, or a list of volumes.
+
+ With a manifest, volumes are checked first and the restored file is checked
+ against its recorded SHA-256 before it is moved into place. Existing files
+ are skipped unless ``force`` is set; with ``update`` they are skipped only
+ when they already match the archive, and replaced otherwise. ``public_key``
+ requires a valid manifest signature before anything is restored.
+
+ ``max_output_size`` refuses an archive that would expand beyond that many
+ bytes (a decompression bomb) before anything is written. With a manifest an
+ archive may never expand beyond the size recorded for its file.
+
+ While it runs, ``unpack`` keeps a journal (``.batchvault-unpack.jsonl``) in
+ the output folder, recording the files it placed; it is removed when the run
+ ends without failures. With ``resume`` (and no ``source``) the run continues
+ from that journal, using the source of the earlier run: a file that is
+ already there and matches its checksum is kept, a missing one is restored,
+ and one this run wrote earlier but that no longer matches is restored again.
+ A file that differs and was not written by this run is never touched.
+ ``force`` and ``update`` cannot be combined with ``resume``; ``public_key``
+ must be given again if the earlier run required a signature.
+ """
+ from pathlib import Path
+
+ from batchvault.archive import resume as journals
+ from batchvault.archive.manifest import hash_file
+ from batchvault.archive.volumes import cleanup_staging
+ from batchvault.core.api._common import (
+ _log_result,
+ _notify,
+ _plan_event,
+ _reject_resume_conflicts,
+ _resolve,
+ _run_all,
+ _split_jobs,
+ )
+ from batchvault.core.api.unpacking.restore import _restore_one
+ from batchvault.core.errors import OptionsError, ResumeError
+ from batchvault.core.locate import Target
+ from batchvault.core.models import FileResult, ProgressEvent, RunResult, Status
+ from batchvault.core.options import parse_size
+
+ if retries < 0:
+ raise OptionsError("retries cannot be negative")
+ max_bytes = parse_size(max_output_size) if max_output_size is not None else None
+ out_root = Path(output).expanduser().resolve()
+
+ saved = None
+ placed: dict[str, str] = {}
+ loaded = None
+ if resume:
+ _reject_resume_conflicts(
+ unpack, dict(source=source, force=force, update=update)
+ )
+ saved, placed, loaded = journals.load_unpack(out_root / journals.UNPACK_JOURNAL)
+ if saved.signed and not public_key:
+ raise ResumeError(
+ "The earlier run required a valid signature, so the public key "
+ "is required to resume."
+ )
+ source = saved.source
+ elif source is None:
+ raise OptionsError("source is required unless resume is set.")
+
+ targets = _resolve(source, public_key)
+
+ manifests = {
+ str(path): hash_file(path)
+ for path in {target.manifest for target in targets if target.manifest}
+ }
+ if resume and saved is not None:
+ if any(target.entry is None for target in targets):
+ raise ResumeError("Resuming needs a manifest to check the files against.")
+ if manifests != saved.manifests:
+ raise ResumeError(
+ "The archive's manifest is not the one the earlier run used "
+ "(it was replaced or changed), so the output cannot be resumed. "
+ "Run unpack without --resume to start again."
+ )
+
+ cleanup_staging(out_root)
+ workers, _ = _split_jobs(jobs, len(targets))
+
+ journal = None
+ created_folder = not out_root.exists()
+ if manifests:
+ out_root.mkdir(parents=True, exist_ok=True)
+ journal_path = out_root / journals.UNPACK_JOURNAL
+ if resume and loaded is not None:
+ journal = journals.Journal.reopen(journal_path, loaded.good_size)
+ else:
+ journal = journals.Journal.create(
+ journal_path,
+ "unpack",
+ journals.UnpackSettings(
+ source=str(Path(str(source)).expanduser().resolve()),
+ manifests=manifests,
+ signed=bool(public_key),
+ ).to_dict(),
+ )
+
+ log.info(
+ "%s %d archive(s) into %s",
+ "Resuming" if resume else "Unpacking",
+ len(targets),
+ out_root,
+ )
+ _notify(progress, _plan_event(out_root, targets))
+
+ def work(target: Target) -> FileResult:
+ size = target.entry.size if target.entry else 0
+ _notify(progress, ProgressEvent("start", target.volume, size))
+ outcome = _restore_one(
+ target, out_root, force, update, seven_zip, password, retries, max_bytes,
+ resume, placed, journal,
+ ) # fmt: skip
+ outcome.folder = target.entry.folder if target.entry else ""
+ outcome.name = target.name
+ _log_result("unpack", outcome)
+ _notify(
+ progress,
+ ProgressEvent(
+ "finish", target.volume, size, outcome.status, resumed=outcome.resumed
+ ),
+ )
+ return outcome
+
+ try:
+ outcomes = _run_all(workers, work, targets)
+ except BaseException:
+ if journal:
+ journal.close() # keep it: it is what --resume continues from
+ raise
+
+ # Keep the journal while something failed so --resume can retry just that.
+ if journal:
+ failed = any(outcome.status == Status.FAILED for outcome in outcomes)
+ if failed and (journal.appended or resume):
+ journal.close()
+ else: # finished, or failed before restoring anything: nothing to resume
+ journal.remove()
+ if created_folder:
+ try:
+ out_root.rmdir() # only if it is empty
+ except OSError:
+ pass
+
+ result = RunResult(entries=outcomes, resume=resume)
+ if any(target.entry is None for target in targets):
+ result.warnings.append(
+ "No manifest found for some archives; their checksums were not verified."
+ )
+ return result
diff --git a/src/batchvault/core/api/verifying.py b/src/batchvault/core/api/verifying.py
new file mode 100644
index 0000000..dea9757
--- /dev/null
+++ b/src/batchvault/core/api/verifying.py
@@ -0,0 +1,125 @@
+"""Check archives against their manifests: the ``verify`` operation."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.core.api._common import Progress
+ from batchvault.core.locate import Target
+ from batchvault.core.models import FileResult, RunResult
+
+
+def verify(
+ source: str | Path,
+ *,
+ against: str | Path | None = None,
+ jobs: int = 0,
+ seven_zip: str | Path | None = None,
+ password: str | None = None,
+ public_key: str | Path | None = None,
+ progress: Progress | None = None,
+) -> RunResult:
+ """Check archives are complete and readable, without restoring anything.
+
+ With a manifest, volume sizes and checksums are checked. ``against`` is a
+ folder of the original files, compared by SHA-256 (manifest required); each
+ original is looked for by name in that folder and then in its subfolders.
+ ``public_key`` also requires a valid manifest signature.
+ """
+ from concurrent.futures import ThreadPoolExecutor
+ from pathlib import Path
+
+ from batchvault.core.api._common import (
+ _log_result,
+ _notify,
+ _plan_event,
+ _resolve,
+ _split_jobs,
+ )
+ from batchvault.core.models import ProgressEvent, RunResult
+
+ targets = _resolve(source, public_key)
+
+ workers, _ = _split_jobs(jobs, len(targets))
+ originals = Path(against).expanduser() if against else None
+
+ _notify(progress, _plan_event(Path(source), targets))
+
+ def work(target: Target) -> FileResult:
+ size = target.entry.size if target.entry else 0
+ _notify(progress, ProgressEvent("start", target.volume, size))
+ outcome = _verify_one(target, originals, seven_zip, password)
+ outcome.folder = target.entry.folder if target.entry else ""
+ outcome.name = target.name
+ _log_result("verify", outcome)
+ _notify(progress, ProgressEvent("finish", target.volume, size, outcome.status))
+ return outcome
+
+ with ThreadPoolExecutor(max_workers=workers) as executor:
+ outcomes = list(executor.map(work, targets))
+
+ result = RunResult(entries=outcomes)
+ if any(target.entry is None for target in targets):
+ result.warnings.append(
+ "No manifest found for some archives; only their internal integrity "
+ "checks were run."
+ )
+ return result
+
+
+def _find_originals(root: Path, folder: str, name: str) -> list[Path]:
+ """The original called ``name``: at its stored folder under ``root``, else
+ directly in ``root``, else anywhere below it."""
+ from batchvault.core.api._common import _folder_path
+
+ for candidate in (_folder_path(root, folder) / name, root / name):
+ if candidate.is_file():
+ return [candidate]
+ return [path for path in root.rglob(name) if path.is_file()]
+
+
+def _verify_one(
+ target: Target,
+ originals: Path | None,
+ seven_zip: str | Path | None,
+ password: str | None,
+) -> FileResult:
+ from batchvault.archive.manifest import hash_file
+ from batchvault.core.api._common import ENCRYPTED_MESSAGE
+ from batchvault.core.errors import BatchVaultError
+ from batchvault.core.models import FileResult, Status
+ from batchvault.engines.registry import engine_for_volume
+
+ volume = target.volume
+ entry = target.entry
+ try:
+ if entry:
+ if entry.encrypted and not password:
+ return FileResult(volume, Status.FAILED, ENCRYPTED_MESSAGE)
+ problems = entry.check_volumes(target.base_dir or volume.parent)
+ if problems:
+ return FileResult(volume, Status.FAILED, "; ".join(problems))
+
+ engine_for_volume(volume, seven_zip).test(volume, password=password)
+
+ if originals and entry:
+ candidates = _find_originals(originals, entry.folder, entry.name)
+ if not candidates:
+ message = f"Original not found under {originals}: {entry.name}"
+ return FileResult(volume, Status.FAILED, message)
+ if not any(hash_file(path) == entry.sha256 for path in candidates):
+ message = f"{entry.name} differs from the archive."
+ return FileResult(volume, Status.FAILED, message)
+
+ if entry:
+ message = "volumes, checksums and integrity test OK"
+ else:
+ message = "integrity test OK (no manifest, so checksums were not checked)"
+ if originals and entry:
+ message += "; matches the original"
+ return FileResult(volume, Status.SUCCESS, message, entry.size if entry else 0)
+ except (BatchVaultError, OSError) as exc:
+ return FileResult(volume, Status.FAILED, str(exc))
diff --git a/src/batchvault/config.py b/src/batchvault/core/config.py
similarity index 97%
rename from src/batchvault/config.py
rename to src/batchvault/core/config.py
index f638a94..4d685b7 100644
--- a/src/batchvault/config.py
+++ b/src/batchvault/core/config.py
@@ -27,7 +27,7 @@
import tomllib
from pathlib import Path
-from batchvault.errors import ConfigError
+from batchvault.core.errors import ConfigError
CONFIG_FILE = "batchvault.toml"
PYPROJECT = "pyproject.toml"
@@ -152,8 +152,7 @@ def defaults_for(config: dict, command: str, actions: dict) -> dict:
dest = mapping.get(key)
if dest is None:
raise ConfigError(
- f"Unknown key '{key}' in [{section}]. "
- f"Known: {', '.join(mapping)}."
+ f"Unknown key '{key}' in [{section}]. Known: {', '.join(mapping)}."
)
if dest in actions:
_check_value(dest, value, actions[dest], f"[{section}]")
diff --git a/src/batchvault/core/data/levels.toml b/src/batchvault/core/data/levels.toml
new file mode 100644
index 0000000..f37f93f
--- /dev/null
+++ b/src/batchvault/core/data/levels.toml
@@ -0,0 +1,64 @@
+# Compression level chosen for each kind of file by ``--level auto``.
+#
+# A file's signature (its first bytes) is checked first; an already compressed
+# signature always gets the "incompressible" group's level, which must stay. Otherwise
+# the extension is looked up in the groups below, in order, and the first match wins. Level 0 means
+# "do not try"; levels run from 0 to 9. Extensions are lower case, without the dot.
+
+# Level for a file that matches no group.
+default_level = 5
+default_kind = "unknown type"
+
+# Formats that are already compressed (or encrypted): nothing left to squeeze.
+[incompressible]
+kind = "already compressed"
+level = 0
+extensions = [
+ "zip", "7z", "rar", "gz", "tgz", "xz", "txz", "bz2", "tbz2", "zst", "lz4", "lzma", "z",
+ "cab", "jar", "war", "ear", "apk", "ipa", "docx", "xlsx", "pptx", "odt", "ods", "odp",
+ "epub", "jpg", "jpeg", "png", "gif", "webp", "heic", "heif", "avif", "cr2", "nef", "arw",
+ "dng", "orf", "rw2", "mp4", "m4v", "mov", "mkv", "webm", "avi", "wmv", "flv", "mpg",
+ "mpeg", "m2ts", "3gp", "mp3", "aac", "m4a", "ogg", "oga", "opus", "flac", "wma", "wv",
+]
+
+# Plain text and similar: very repetitive, so a strong level pays off.
+[text]
+kind = "text"
+level = 6
+extensions = [
+ "txt", "log", "csv", "tsv", "json", "jsonl", "ndjson", "xml", "html", "htm", "xhtml",
+ "css", "scss", "sass", "less", "js", "mjs", "jsx", "tsx", "md", "markdown", "rst", "yaml",
+ "yml", "toml", "ini", "cfg", "conf", "properties", "sql", "py", "pyi", "java", "kt", "c",
+ "h", "cc", "cpp", "hpp", "cs", "go", "rs", "rb", "php", "swift", "sh", "bash", "zsh",
+ "bat", "cmd", "ps1", "tex", "bib", "srt", "vtt", "ass", "svg", "rtf", "eml", "ics", "vcf",
+ "gpx", "kml", "diff", "patch", "gradle", "tf", "ipynb", "r", "pl", "lua", "dart", "scala",
+ "vue",
+]
+
+# Uncompressed pictures and sound.
+[media]
+kind = "uncompressed media"
+level = 6
+extensions = [
+ "wav", "aiff", "aif", "bmp", "tif", "tiff", "ppm", "pgm", "pbm", "tga", "dds", "pcm",
+ "psd",
+]
+
+# Programs, disk images, databases and containers: compressible, but the
+# strongest levels gain little for the memory they need.
+[binary]
+kind = "binary data"
+level = 5
+extensions = [
+ "exe", "dll", "so", "dylib", "o", "a", "lib", "class", "pyc", "wasm", "iso", "img", "vhd",
+ "vhdx", "vmdk", "qcow2", "vdi", "db", "sqlite", "sqlite3", "mdb", "bak", "dump", "pak",
+ "tar",
+]
+
+# PDF documents.
+[pdf]
+kind = "pdf"
+level = 3
+extensions = [
+ "pdf",
+]
diff --git a/src/batchvault/errors.py b/src/batchvault/core/errors.py
similarity index 89%
rename from src/batchvault/errors.py
rename to src/batchvault/core/errors.py
index 8880235..810887b 100644
--- a/src/batchvault/errors.py
+++ b/src/batchvault/core/errors.py
@@ -34,5 +34,9 @@ class ManifestError(BatchVaultError):
"""A manifest is missing, malformed or fails validation."""
+class ResumeError(BatchVaultError):
+ """A run cannot be resumed: no journal, or it is damaged or does not match."""
+
+
class VolumeError(BatchVaultError):
"""Archive volumes are missing, incomplete or corrupt."""
diff --git a/src/batchvault/eta.py b/src/batchvault/core/eta.py
similarity index 80%
rename from src/batchvault/eta.py
rename to src/batchvault/core/eta.py
index a1410d0..1418ab6 100644
--- a/src/batchvault/eta.py
+++ b/src/batchvault/core/eta.py
@@ -21,13 +21,20 @@ class EtaEstimator:
def __init__(self, clock=time.monotonic):
self.clock = clock
self.started_at = None
+ self.skipped = 0 # bytes finished before this run (a resumed run)
def start(self):
if self.started_at is None:
self.started_at = self.clock()
+ def skip(self, size):
+ """Count work an earlier run did, so it does not inflate the speed."""
+ self.skipped += size
+
def estimate(self, done_bytes, total_bytes):
"""Seconds remaining, or None while there is not enough data yet."""
+ done_bytes -= self.skipped
+ total_bytes -= self.skipped
if self.started_at is None or done_bytes <= 0:
return None
diff --git a/src/batchvault/filters.py b/src/batchvault/core/filters.py
similarity index 95%
rename from src/batchvault/filters.py
rename to src/batchvault/core/filters.py
index 8555fb9..05822b8 100644
--- a/src/batchvault/filters.py
+++ b/src/batchvault/core/filters.py
@@ -5,12 +5,12 @@
import pathspec
-from batchvault.errors import InputError
+from batchvault.core.errors import InputError
def load_patterns(values: Iterable[str]) -> list[str]:
"""Expand ``@file`` references into the patterns listed in that file."""
- patterns = []
+ patterns: list[str] = []
for value in values:
if value.startswith("@"):
path = Path(value[1:]).expanduser()
diff --git a/src/batchvault/inputs.py b/src/batchvault/core/inputs.py
similarity index 96%
rename from src/batchvault/inputs.py
rename to src/batchvault/core/inputs.py
index 02bbd4c..1ee0610 100644
--- a/src/batchvault/inputs.py
+++ b/src/batchvault/core/inputs.py
@@ -6,10 +6,11 @@
from collections.abc import Iterable
from pathlib import Path
-from batchvault.errors import InputError
-from batchvault.filters import PathFilter
-from batchvault.manifest import MANIFEST_NAME
-from batchvault.volumes import split_volume_name
+from batchvault.archive.manifest import MANIFEST_NAME
+from batchvault.archive.readme import README_NAME
+from batchvault.archive.volumes import split_volume_name
+from batchvault.core.errors import InputError
+from batchvault.core.filters import PathFilter
# A text list written by ``list files`` starts with this comment. It records the
# folder the files were listed from, so ``pack`` keeps that folder's name.
@@ -134,7 +135,10 @@ def assign_folders(
def looks_like_archive(path: Path) -> bool:
- return split_volume_name(path.name) is not None or path.name == MANIFEST_NAME
+ return split_volume_name(path.name) is not None or path.name in (
+ MANIFEST_NAME,
+ README_NAME,
+ )
def gather_sources(
diff --git a/src/batchvault/levels.py b/src/batchvault/core/levels.py
similarity index 61%
rename from src/batchvault/levels.py
rename to src/batchvault/core/levels.py
index dab2af6..1a109c1 100644
--- a/src/batchvault/levels.py
+++ b/src/batchvault/core/levels.py
@@ -9,18 +9,12 @@
import argparse
from dataclasses import dataclass
+from functools import cache
from pathlib import Path
-from batchvault.errors import OptionsError
+from batchvault.core.errors import OptionsError
AUTO = "auto"
-DEFAULT_LEVEL = 5
-
-TEXT_LEVEL = 6
-MEDIA_LEVEL = 6
-BINARY_LEVEL = 5
-PDF_LEVEL = 3
-INCOMPRESSIBLE_LEVEL = 0
@dataclass(frozen=True)
@@ -60,48 +54,6 @@ def level_argument(text: str) -> int | str:
) from None
-def _extensions(text: str) -> frozenset[str]:
- return frozenset(text.split())
-
-
-# Formats that are already compressed (or encrypted): nothing left to squeeze.
-INCOMPRESSIBLE = _extensions(
- """
- zip 7z rar gz tgz xz txz bz2 tbz2 zst lz4 lzma z cab jar war ear apk ipa
- docx xlsx pptx odt ods odp epub
- jpg jpeg png gif webp heic heif avif cr2 nef arw dng orf rw2
- mp4 m4v mov mkv webm avi wmv flv mpg mpeg m2ts 3gp
- mp3 aac m4a ogg oga opus flac wma wv
- """
-)
-
-# Plain text and similar: very repetitive, so a strong level pays off.
-TEXT = _extensions(
- """
- txt log csv tsv json jsonl ndjson xml html htm xhtml css scss sass less
- js mjs jsx tsx md markdown rst yaml yml toml ini cfg conf properties sql
- py pyi java kt c h cc cpp hpp cs go rs rb php swift sh bash zsh bat cmd ps1
- tex bib srt vtt ass svg rtf eml ics vcf gpx kml diff patch gradle tf
- ipynb r pl lua dart scala vue
- """
-)
-
-# Uncompressed pictures and sound.
-UNCOMPRESSED_MEDIA = _extensions(
- "wav aiff aif bmp tif tiff ppm pgm pbm tga dds pcm psd"
-)
-
-# Programs, disk images, databases and containers: compressible, but the
-# strongest levels gain little for the memory they need.
-BINARY = _extensions(
- """
- exe dll so dylib o a lib class pyc wasm iso img vhd vhdx vmdk qcow2 vdi
- db sqlite sqlite3 mdb bak dump pak tar
- """
-)
-
-PDF = _extensions("pdf")
-
# (offset, bytes) at the start of a file that mark an already compressed format.
_SIGNATURES = (
(0, b"PK\x03\x04"), # zip, docx, jar, apk, epub, ...
@@ -125,6 +77,55 @@ def _extensions(text: str) -> frozenset[str]:
)
+@dataclass(frozen=True)
+class _Group:
+ kind: str
+ level: int
+ extensions: frozenset[str]
+
+
+@dataclass(frozen=True)
+class _Rules:
+ default_level: int
+ default_kind: str
+ groups: dict[str, _Group] # in file order; the first group is "incompressible"
+
+
+@cache
+def _rules() -> _Rules:
+ """Read ``data/levels.toml``, which says which level suits which file type."""
+ import tomllib
+ from importlib.resources import files
+
+ resource = files("batchvault.core").joinpath("data", "levels.toml")
+ data = tomllib.loads(resource.read_text(encoding="utf-8"))
+ default_level = data.pop("default_level")
+ default_kind = data.pop("default_kind")
+ groups = {
+ name: _Group(
+ kind=group["kind"],
+ level=group["level"],
+ extensions=frozenset(group["extensions"]),
+ )
+ for name, group in data.items()
+ }
+ for level in (default_level, *[group.level for group in groups.values()]):
+ if isinstance(level, bool) or not isinstance(level, int) or not 0 <= level <= 9:
+ raise ValueError(f"levels.toml has a level outside 0-9: {level!r}")
+ return _Rules(default_level, default_kind, groups)
+
+
+def __getattr__(name: str) -> int:
+ """``DEFAULT_LEVEL`` and ``_LEVEL`` (``TEXT_LEVEL``), read on first use."""
+ rules = _rules()
+ if name == "DEFAULT_LEVEL":
+ return rules.default_level
+ for group_name, group in rules.groups.items():
+ if name == f"{group_name.upper()}_LEVEL":
+ return group.level
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
def _has_compressed_signature(head: bytes) -> bool:
if any(head[at : at + len(mark)] == mark for at, mark in _SIGNATURES):
return True
@@ -152,18 +153,13 @@ def choose_level(path: Path) -> LevelChoice:
except OSError:
head = b""
+ rules = _rules()
+ incompressible = rules.groups["incompressible"]
if _has_compressed_signature(head):
- return LevelChoice(INCOMPRESSIBLE_LEVEL, "already compressed")
+ return LevelChoice(incompressible.level, incompressible.kind)
extension = path.suffix.lower().lstrip(".")
- if extension in INCOMPRESSIBLE:
- return LevelChoice(INCOMPRESSIBLE_LEVEL, "already compressed")
- if extension in TEXT:
- return LevelChoice(TEXT_LEVEL, "text")
- if extension in UNCOMPRESSED_MEDIA:
- return LevelChoice(MEDIA_LEVEL, "uncompressed media")
- if extension in BINARY:
- return LevelChoice(BINARY_LEVEL, "binary data")
- if extension in PDF:
- return LevelChoice(PDF_LEVEL, "pdf")
- return LevelChoice(DEFAULT_LEVEL, "unknown type")
+ for group in rules.groups.values():
+ if extension in group.extensions:
+ return LevelChoice(group.level, group.kind)
+ return LevelChoice(rules.default_level, rules.default_kind)
diff --git a/src/batchvault/locate.py b/src/batchvault/core/locate.py
similarity index 90%
rename from src/batchvault/locate.py
rename to src/batchvault/core/locate.py
index 430829b..612366d 100644
--- a/src/batchvault/locate.py
+++ b/src/batchvault/core/locate.py
@@ -6,10 +6,10 @@
from dataclasses import dataclass
from pathlib import Path
-from batchvault.errors import InputError, ManifestError
-from batchvault.inputs import read_list_entries
-from batchvault.manifest import MANIFEST_NAME, Manifest, ManifestEntry
-from batchvault.volumes import split_volume_name
+from batchvault.archive.manifest import MANIFEST_NAME, Manifest, ManifestEntry
+from batchvault.archive.volumes import split_volume_name
+from batchvault.core.errors import InputError, ManifestError
+from batchvault.core.inputs import read_list_entries
@dataclass
@@ -25,7 +25,8 @@ class Target:
def name(self) -> str:
if self.entry:
return self.entry.name
- base = split_volume_name(self.volume.name)[0]
+ parsed = split_volume_name(self.volume.name)
+ base = parsed[0] if parsed else self.volume.name
return base.rsplit(".", 1)[0]
@@ -85,9 +86,7 @@ def _volume_target(volume: Path, load: Loader = Manifest.load) -> Target:
return Target(volume)
-def resolve_targets(
- source: str | Path, load: Loader = Manifest.load
-) -> list[Target]:
+def resolve_targets(source: str | Path, load: Loader = Manifest.load) -> list[Target]:
"""Archives referred to by a manifest, folder, volume file or list file.
Every manifest is read through ``load``, so a caller can verify a manifest's
diff --git a/src/batchvault/models.py b/src/batchvault/core/models.py
similarity index 84%
rename from src/batchvault/models.py
rename to src/batchvault/core/models.py
index f8c2e60..bf79fd2 100644
--- a/src/batchvault/models.py
+++ b/src/batchvault/core/models.py
@@ -3,6 +3,7 @@
from dataclasses import dataclass, field
from enum import StrEnum
from pathlib import Path
+from typing import Any
class Status(StrEnum):
@@ -33,6 +34,8 @@ class FileResult:
folder: str = "" # folder of the file under the root ("" = flat)
name: str = "" # the original file's name (pack, unpack and verify)
level: int | None = None # compression level used (pack)
+ resumed: bool = False # finished by an earlier run and reused (--resume)
+ redone: str = "" # why a file from an earlier run was done again (--resume)
def to_dict(self) -> dict:
return {
@@ -48,6 +51,8 @@ def to_dict(self) -> dict:
"stored_size": self.stored_size,
"restored": [str(path) for path in self.restored],
"warnings": self.warnings,
+ "resumed": self.resumed,
+ "redone": self.redone or None,
}
@@ -59,6 +64,7 @@ class RunResult:
manifest: Path | None = None
warnings: list[str] = field(default_factory=list)
engine: str = ""
+ resume: bool = False # the run continued an earlier one (--resume)
@property
def status(self) -> Status:
@@ -74,14 +80,27 @@ def status(self) -> Status:
def count(self, status: Status) -> int:
return sum(1 for entry in self.entries if entry.status == status)
+ @property
+ def reused(self) -> int:
+ """Files an earlier run had finished, used again without redoing them."""
+ return sum(1 for entry in self.entries if entry.resumed)
+
+ @property
+ def redone(self) -> int:
+ """Files from an earlier run that had to be done again."""
+ return sum(1 for entry in self.entries if entry.redone)
+
def to_dict(self) -> dict:
- return {
+ data: dict[str, Any] = {
"status": self.status.value,
"engine": self.engine,
"manifest": str(self.manifest) if self.manifest else None,
"warnings": self.warnings,
"entries": [entry.to_dict() for entry in self.entries],
}
+ if self.resume:
+ data["resume"] = {"reused": self.reused, "redone": self.redone}
+ return data
@dataclass(frozen=True)
@@ -114,6 +133,7 @@ class ProgressEvent:
size: int = 0
status: Status | None = None
count: int = 0
+ resumed: bool = False # a ``finish`` for a file reused from an earlier run
def to_dict(self) -> dict:
return {
@@ -122,6 +142,7 @@ def to_dict(self) -> dict:
"size": self.size,
"status": self.status.value if self.status else None,
"count": self.count,
+ "resumed": self.resumed,
}
diff --git a/src/batchvault/names.py b/src/batchvault/core/names.py
similarity index 100%
rename from src/batchvault/names.py
rename to src/batchvault/core/names.py
diff --git a/src/batchvault/options.py b/src/batchvault/core/options.py
similarity index 97%
rename from src/batchvault/options.py
rename to src/batchvault/core/options.py
index aa27e34..30535fb 100644
--- a/src/batchvault/options.py
+++ b/src/batchvault/core/options.py
@@ -3,7 +3,7 @@
import re
from dataclasses import dataclass, field
-from batchvault.errors import OptionsError
+from batchvault.core.errors import OptionsError
KIB = 1024
MIB = 1024 * KIB
diff --git a/src/batchvault/security.py b/src/batchvault/core/security.py
similarity index 100%
rename from src/batchvault/security.py
rename to src/batchvault/core/security.py
diff --git a/src/batchvault/engines/base.py b/src/batchvault/engines/base.py
index cf82cb4..ede6636 100644
--- a/src/batchvault/engines/base.py
+++ b/src/batchvault/engines/base.py
@@ -2,9 +2,12 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
-from pathlib import Path
+from typing import TYPE_CHECKING
-from batchvault.options import PackOptions
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.core.options import PackOptions
@dataclass(frozen=True)
diff --git a/src/batchvault/engines/native.py b/src/batchvault/engines/native.py
index 3f88d0f..6f2ea05 100644
--- a/src/batchvault/engines/native.py
+++ b/src/batchvault/engines/native.py
@@ -1,40 +1,17 @@
from __future__ import annotations
-import os
-import tarfile
-import tempfile
-import zlib
-from pathlib import Path, PurePosixPath
-
-from batchvault import crypto
-from batchvault.blocks import (
- BLOCK_SIZE,
- BlockWriter,
- check_format,
- open_decompressed,
- sniff_archive,
-)
+from typing import TYPE_CHECKING
+
+from batchvault.archive.blocks import BLOCK_SIZE
+from batchvault.core.options import MIB
from batchvault.engines.base import Capabilities, Engine
-from batchvault.errors import EngineError, OutputExistsError
-from batchvault.names import unsafe_on_windows as _unsafe_on_windows
-from batchvault.options import MIB, PackOptions
-from batchvault.volumes import (
- VolumeReader,
- VolumeWriter,
- existing_volumes,
- find_volumes,
- install_volumes,
-)
-
-try:
- import lzma
-
- _LZMA_ERRORS: tuple[type[Exception], ...] = (lzma.LZMAError,)
-except ImportError:
- _LZMA_ERRORS = ()
-
-# Errors raised while reading damaged, truncated or unreadable archive data.
-_READ_ERRORS = (tarfile.TarError, EOFError, zlib.error, OSError, *_LZMA_ERRORS)
+
+if TYPE_CHECKING:
+ import tarfile
+ from pathlib import Path
+ from typing import Any
+
+ from batchvault.core.options import PackOptions
_STREAM_BUFFER = MIB
_CHUNK = MIB
@@ -42,6 +19,20 @@
_MAX_TRAILING = 4 * MIB
+def _read_errors() -> tuple[type[BaseException], ...]:
+ """Errors raised while reading damaged, truncated or unreadable archive data."""
+ import tarfile
+ import zlib
+
+ try:
+ import lzma
+
+ lzma_errors: tuple[type[Exception], ...] = (lzma.LZMAError,)
+ except ImportError:
+ lzma_errors = ()
+ return (tarfile.TarError, EOFError, zlib.error, OSError, *lzma_errors)
+
+
def _normalize(info: tarfile.TarInfo) -> tarfile.TarInfo:
"""Drop ownership so archives do not depend on the user who made them."""
info.uid = info.gid = 0
@@ -56,6 +47,8 @@ def __init__(self, limit: int | None):
self.remaining = limit
def charge(self, member: tarfile.TarInfo) -> None:
+ from batchvault.core.errors import EngineError
+
if self.remaining is None:
return
if member.size > self.remaining:
@@ -68,6 +61,12 @@ def charge(self, member: tarfile.TarInfo) -> None:
def _check_member(member: tarfile.TarInfo, windows: bool | None = None) -> None:
"""Accept only a plain, single-part regular file name that is safe to write."""
+ import os
+ from pathlib import PurePosixPath
+
+ from batchvault.core.errors import EngineError
+ from batchvault.core.names import unsafe_on_windows as _unsafe_on_windows
+
if windows is None:
windows = os.name == "nt"
@@ -98,6 +97,16 @@ def is_available(self) -> bool:
return True
def pack(self, source: Path, dest_dir: Path, options: PackOptions) -> list[Path]:
+ from pathlib import Path
+
+ from batchvault.archive.blocks import BlockWriter, check_format
+ from batchvault.archive.volumes import (
+ VolumeWriter,
+ existing_volumes,
+ install_volumes,
+ )
+ from batchvault.core.errors import OutputExistsError
+
if not source.is_file():
raise FileNotFoundError(f"Source file not found: {source}")
check_format(options.format)
@@ -109,6 +118,11 @@ def pack(self, source: Path, dest_dir: Path, options: PackOptions) -> list[Path]
f"Archive volumes already exist in {dest_dir}. Use --force to replace."
)
+ import tarfile
+ import tempfile
+
+ from batchvault.archive import crypto
+
header = cipher = None
if options.password:
header = crypto.make_header(options.format)
@@ -147,6 +161,12 @@ def unpack(
password: str | None = None,
max_bytes: int | None = None,
) -> list[Path]:
+ import os
+ from pathlib import Path
+
+ from batchvault.archive.volumes import find_volumes
+ from batchvault.core.errors import EngineError, OutputExistsError
+
volumes = find_volumes(volume)
output_dir.mkdir(parents=True, exist_ok=True)
budget = _Budget(max_bytes)
@@ -155,6 +175,8 @@ def handle(member, tar) -> None:
budget.charge(member) # refuse a decompression bomb before writing
self._extract(member, tar, staging)
+ import tempfile
+
with tempfile.TemporaryDirectory(dir=output_dir, prefix=".batchvault-") as tmp:
staging = Path(tmp)
self._read(volumes, handle, password)
@@ -183,12 +205,16 @@ def handle(member, tar) -> None:
return restored
def test(self, volume: Path, *, password: str | None = None) -> None:
+ from batchvault.archive.volumes import find_volumes
+
self._read(find_volumes(volume), self._drain_member, password)
@staticmethod
def _extract(member, tar, staging: Path) -> None:
# Belt and braces next to _check_member and the "data" filter: whatever
# the name looks like, the file must end up inside the staging folder.
+ from batchvault.core.errors import EngineError
+
root = staging.resolve()
if root not in (root / member.name).resolve().parents:
raise EngineError(f"Unsafe entry in archive: {member.name!r}")
@@ -203,16 +229,25 @@ def _drain_member(member, tar) -> None:
@staticmethod
def _read(volumes: list[Path], handle_member, password: str | None) -> None:
"""Stream the archive and call ``handle_member(member, tar)`` for each file."""
+ from batchvault.archive.blocks import open_decompressed, sniff_archive
+ from batchvault.archive.volumes import VolumeReader
+ from batchvault.core.errors import EngineError
+
fmt, encrypted = sniff_archive(volumes[0])
if encrypted and not password:
raise EngineError(
f"{volumes[0].name} is encrypted. A password is required."
)
+ import tarfile
+
+ from batchvault.archive import crypto
+
try:
with VolumeReader(volumes) as reader:
- source = reader
+ source: Any = reader
if encrypted:
+ assert password is not None # checked above
fmt, source = crypto.open_decrypted(reader, password)
with open_decompressed(fmt, source) as stream:
@@ -231,7 +266,7 @@ def _read(volumes: list[Path], handle_member, password: str | None) -> None:
f"{volumes[0].name} has unexpected data after the "
"end of the archive."
)
- except _READ_ERRORS as exc:
+ except _read_errors() as exc:
raise EngineError(
f"Archive is corrupt or incomplete ({volumes[0].name}): {exc}"
) from exc
diff --git a/src/batchvault/engines/registry.py b/src/batchvault/engines/registry.py
index e270332..f787a54 100644
--- a/src/batchvault/engines/registry.py
+++ b/src/batchvault/engines/registry.py
@@ -1,12 +1,11 @@
from __future__ import annotations
-from pathlib import Path
+from typing import TYPE_CHECKING
-from batchvault.engines.base import Engine
-from batchvault.engines.native import NativeEngine
-from batchvault.engines.sevenzip import SevenZipEngine, find_7zip
-from batchvault.errors import InputError
-from batchvault.volumes import split_volume_name
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from batchvault.engines.base import Engine
ENGINE_CHOICES = ("auto", "native", "7zip")
@@ -17,19 +16,31 @@ def get_engine(name: str = "auto", seven_zip: str | Path | None = None) -> Engin
``auto`` uses 7-Zip when it is found (or when a path to it was given) and
falls back to the native engine.
"""
+ from batchvault.core.errors import InputError
+
if name == "native":
+ from batchvault.engines.native import NativeEngine
+
return NativeEngine()
if name in ("7zip", "7z"):
+ from batchvault.engines.sevenzip import SevenZipEngine, find_7zip
+
find_7zip(seven_zip) # raises EngineNotFoundError with install help
return SevenZipEngine(seven_zip)
if name == "auto":
+ from batchvault.engines.sevenzip import SevenZipEngine, find_7zip
+
if seven_zip:
find_7zip(seven_zip)
return SevenZipEngine(seven_zip)
sevenzip = SevenZipEngine()
- return sevenzip if sevenzip.is_available() else NativeEngine()
+ if sevenzip.is_available():
+ return sevenzip
+ from batchvault.engines.native import NativeEngine
+
+ return NativeEngine()
choices = ", ".join(ENGINE_CHOICES)
raise InputError(f"Unknown engine {name!r}. Choose from: {choices}.")
@@ -37,17 +48,24 @@ def get_engine(name: str = "auto", seven_zip: str | Path | None = None) -> Engin
def engine_for_volume(volume: Path, seven_zip: str | Path | None = None) -> Engine:
"""Pick the engine that can read a volume, from its file name."""
+ from batchvault.archive.volumes import split_volume_name
+ from batchvault.core.errors import InputError
+
parsed = split_volume_name(volume.name)
if parsed is None:
raise InputError(f"Not an archive volume: {volume}")
if parsed[0].lower().endswith(".7z"):
+ from batchvault.engines.sevenzip import SevenZipEngine, find_7zip
+
find_7zip(seven_zip)
return SevenZipEngine(seven_zip)
+ from batchvault.engines.native import NativeEngine
+
return NativeEngine()
def option_warnings(
- engine: Engine, format_given: str | None, level: int
+ engine: Engine, format_given: str | None, level: int | str
) -> list[str]:
"""Settings the chosen engine cannot honour exactly."""
warnings = []
diff --git a/src/batchvault/engines/sevenzip.py b/src/batchvault/engines/sevenzip.py
index cca8e48..40e9418 100644
--- a/src/batchvault/engines/sevenzip.py
+++ b/src/batchvault/engines/sevenzip.py
@@ -1,15 +1,13 @@
from __future__ import annotations
-import os
-import subprocess
-import sys
-import tempfile
from pathlib import Path
+from typing import TYPE_CHECKING
from batchvault.engines.base import Capabilities, Engine
-from batchvault.errors import EngineError, EngineNotFoundError, OutputExistsError
-from batchvault.options import PackOptions
-from batchvault.volumes import existing_volumes, find_volumes, install_volumes
+
+if TYPE_CHECKING:
+ from batchvault.core.errors import EngineError
+ from batchvault.core.options import PackOptions
ENV_VAR = "BATCHVAULT_7Z"
COMMANDS = ("7zz", "7z", "7za", "7zr")
@@ -28,6 +26,9 @@ def _search_path(command: str) -> Path | None:
that first) and ignores empty and relative ``PATH`` entries, so a program
planted in the folder you happen to be working in is never run.
"""
+ import os
+ import sys
+
cwd = Path.cwd().resolve()
if sys.platform == "win32":
extensions = os.environ.get("PATHEXT", ".COM;.EXE;.BAT;.CMD").split(";")
@@ -58,6 +59,11 @@ def find_7zip(explicit: str | Path | None = None) -> Path:
Order: explicit path, ``BATCHVAULT_7Z``, ``PATH`` (never the current folder),
then well-known Windows install folders.
"""
+ import os
+ import sys
+
+ from batchvault.core.errors import EngineNotFoundError
+
configured = explicit or os.environ.get(ENV_VAR)
if configured:
path = Path(configured).expanduser()
@@ -132,6 +138,8 @@ def build_test_command(
def folder_size(folder: Path) -> int:
"""Total size of the files below ``folder`` (files that vanish are ignored)."""
+ import os
+
total = 0
for root, _, names in os.walk(folder):
for name in names:
@@ -159,6 +167,8 @@ def executable(self) -> Path:
return self._executable
def is_available(self) -> bool:
+ from batchvault.core.errors import EngineNotFoundError
+
try:
find_7zip(self._explicit)
except EngineNotFoundError:
@@ -167,11 +177,15 @@ def is_available(self) -> bool:
@staticmethod
def _failure(code: int, output: str) -> EngineError:
+ from batchvault.core.errors import EngineError
+
tail = "\n".join(output.strip().splitlines()[-OUTPUT_TAIL_LINES:])
return EngineError(f"7-Zip failed (exit code {code}).\n{tail}".rstrip())
def _run(self, command: list[str]) -> str:
"""Run 7-Zip; return its standard output, raise EngineError on failure."""
+ import subprocess
+
result = subprocess.run(
command,
capture_output=True,
@@ -192,11 +206,16 @@ def _run_limited(self, command: list[str], watch: Path, max_bytes: int) -> str:
is still small instead of filling the disk. Output goes to a temporary
file so a full pipe can never stall the program.
"""
+ from batchvault.core.errors import EngineError
+
too_big = EngineError(
f"The archive would expand beyond the allowed size ({max_bytes} bytes); "
"extraction was stopped."
)
+ import subprocess
+ import tempfile
+
with tempfile.TemporaryFile() as output:
process = subprocess.Popen(
command,
@@ -227,6 +246,9 @@ def _run_limited(self, command: list[str], watch: Path, max_bytes: int) -> str:
return text
def pack(self, source: Path, dest_dir: Path, options: PackOptions) -> list[Path]:
+ from batchvault.archive.volumes import existing_volumes, install_volumes
+ from batchvault.core.errors import EngineError, OutputExistsError
+
if not source.is_file():
raise FileNotFoundError(f"Source file not found: {source}")
@@ -240,6 +262,8 @@ def pack(self, source: Path, dest_dir: Path, options: PackOptions) -> list[Path]
dest_dir.mkdir(parents=True, exist_ok=True)
# Build in a temporary folder so a failed run never damages what exists.
+ import tempfile
+
with tempfile.TemporaryDirectory(dir=dest_dir, prefix=".batchvault-") as tmp:
staging = Path(tmp)
self._run(
@@ -261,9 +285,16 @@ def unpack(
password: str | None = None,
max_bytes: int | None = None,
) -> list[Path]:
+ import os
+
+ from batchvault.archive.volumes import find_volumes
+ from batchvault.core.errors import EngineError, OutputExistsError
+
first = find_volumes(volume)[0]
output_dir.mkdir(parents=True, exist_ok=True)
+ import tempfile
+
with tempfile.TemporaryDirectory(dir=output_dir, prefix=".batchvault-") as tmp:
staging = Path(tmp)
command = build_unpack_command(self.executable, first, staging, password)
@@ -299,5 +330,7 @@ def unpack(
return restored
def test(self, volume: Path, *, password: str | None = None) -> None:
+ from batchvault.archive.volumes import find_volumes
+
first = find_volumes(volume)[0]
self._run(build_test_command(self.executable, first, password))
diff --git a/sweepito.toml b/sweepito.toml
new file mode 100644
index 0000000..50daafc
--- /dev/null
+++ b/sweepito.toml
@@ -0,0 +1,13 @@
+[defaults]
+profiles = ["python"]
+risk = "safe"
+
+[clean]
+# Wheel and sdist output from `uv build` (also written by the package end-to-end test); it is
+# rebuilt on demand. Build output is opt-in because it is slow to rebuild for some projects.
+enable = ["python.build", "python.dist"]
+# Exclusions always win over cleanup rules. Keep the uv environment between commits.
+exclude = [".venv/"]
+
+[hook]
+quiet = true
diff --git a/tests/conftest.py b/tests/conftest.py
index 97501c0..934d46c 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -14,6 +14,6 @@ def fast_key_derivation(monkeypatch):
The cost is stored in each archive's header, so archives still open; one test
checks that the real default is strong.
"""
- from batchvault import crypto
+ from batchvault.archive import crypto
monkeypatch.setattr(crypto, "DEFAULT_LOG2_N", 14)
diff --git a/tests/e2e/test_package_e2e.py b/tests/e2e/test_package_e2e.py
index b36b461..816ae91 100644
--- a/tests/e2e/test_package_e2e.py
+++ b/tests/e2e/test_package_e2e.py
@@ -110,8 +110,18 @@ def test_wheel_contents(wheel):
names = set(archive.namelist())
assert "batchvault/py.typed" in names
- for module in ("api", "cli", "ui", "engines/native", "engines/sevenzip"):
+ for module in (
+ "core/api/__init__",
+ "core/api/packing/run",
+ "cli/cli",
+ "cli/ui",
+ "engines/native",
+ "engines/sevenzip",
+ ):
assert f"batchvault/{module}.py" in names
+ # Data files read at run time must ship in the wheel.
+ for data in ("core/data/levels.toml", "archive/templates/readme.md.tmpl"):
+ assert f"batchvault/{data}" in names
assert not [n for n in names if n.startswith(("tests/", ".temp/", "docs/"))]
@@ -220,7 +230,7 @@ def test_crypto_extra_enables_encryption_and_signing(crypto_venv, tmp_path):
"import sys, json\n"
"from pathlib import Path\n"
"import batchvault as bv\n"
- "from batchvault import signing\n"
+ "from batchvault.archive import signing\n"
"work = Path(sys.argv[1])\n"
"source = work / 'notes.txt'\n"
"source.write_text('hello ' * 1000)\n"
diff --git a/tests/integration/native/test_native_api.py b/tests/integration/native/test_native_api.py
new file mode 100644
index 0000000..bbbc276
--- /dev/null
+++ b/tests/integration/native/test_native_api.py
@@ -0,0 +1,138 @@
+"""End-to-end runs through the public API and CLI with the native engine."""
+
+import json
+import os
+
+import pytest
+
+import batchvault as bv
+from batchvault.cli.cli import main
+
+pytestmark = pytest.mark.integration
+
+
+@pytest.fixture
+def media(tmp_path):
+ folder = tmp_path / "media"
+ folder.mkdir()
+ (folder / "données 1.bin").write_bytes(os.urandom(20_000))
+ (folder / "notes.txt").write_text("hello " * 2000, encoding="utf-8")
+ return folder
+
+
+def pack(media, tmp_path, **kwargs):
+ kwargs.setdefault("volume_size", "4k")
+ return bv.pack(
+ [media / "données 1.bin", media / "notes.txt"],
+ tmp_path / "archives",
+ engine="native",
+ **kwargs,
+ )
+
+
+def test_pack_unpack_verify(tmp_path, media):
+ result = pack(media, tmp_path, jobs=2)
+ assert result.status == bv.Status.SUCCESS
+ assert result.engine == "native"
+
+ manifest = json.loads(result.manifest.read_text(encoding="utf-8"))
+ assert {e["engine"] for e in manifest["entries"]} == {"native"}
+ assert {e["format"] for e in manifest["entries"]} == {"xz"}
+
+ assert bv.verify(tmp_path / "archives", against=media).status == bv.Status.SUCCESS
+
+ restored = bv.unpack(tmp_path / "archives", tmp_path / "out")
+ assert restored.status == bv.Status.SUCCESS
+ for name in ("données 1.bin", "notes.txt"):
+ assert (tmp_path / "out" / name).read_bytes() == (media / name).read_bytes()
+
+
+def test_unpack_from_any_volume_without_manifest(tmp_path, media):
+ pack(media, tmp_path)
+ archives = tmp_path / "archives"
+ (archives / "manifest.json").unlink()
+
+ volume = sorted((archives / "données 1.bin").glob("*.bv.*"))[2]
+ result = bv.unpack(volume, tmp_path / "out")
+ assert result.status == bv.Status.SUCCESS
+ assert result.warnings
+ assert (tmp_path / "out" / "données 1.bin").read_bytes() == (
+ media / "données 1.bin"
+ ).read_bytes()
+
+
+def test_missing_last_volume_is_caught(tmp_path, media):
+ pack(media, tmp_path)
+ folder = tmp_path / "archives" / "données 1.bin"
+ sorted(folder.glob("*.bv.*"))[-1].unlink()
+
+ assert bv.verify(tmp_path / "archives").count(bv.Status.FAILED) == 1
+
+
+def test_update_only_repacks_changed_files(tmp_path, media):
+ pack(media, tmp_path)
+ assert pack(media, tmp_path, update=True).count(bv.Status.SKIPPED) == 2
+
+ (media / "notes.txt").write_text("changed " * 3000, encoding="utf-8")
+ later = (media / "notes.txt").stat().st_mtime + 30
+ os.utime(media / "notes.txt", (later, later))
+
+ result = pack(media, tmp_path, update=True)
+ assert {e.source.name: e.status for e in result.entries} == {
+ "données 1.bin": bv.Status.SKIPPED,
+ "notes.txt": bv.Status.SUCCESS,
+ }
+
+
+def test_encrypted_roundtrip(tmp_path, media):
+ pytest.importorskip("cryptography")
+ pack(media, tmp_path, password="s3cret", verify=True)
+ archives = tmp_path / "archives"
+
+ assert all(e.encrypted for e in bv.inspect(archives))
+ assert bv.verify(archives).count(bv.Status.FAILED) == 2 # no password
+ assert bv.verify(archives, password="wrong").count(bv.Status.FAILED) == 2
+ assert bv.verify(archives, password="s3cret").status == bv.Status.SUCCESS
+
+ bv.unpack(archives, tmp_path / "out", password="s3cret")
+ assert (tmp_path / "out" / "notes.txt").read_bytes() == (
+ media / "notes.txt"
+ ).read_bytes()
+
+
+def test_delete_source_after_roundtrip(tmp_path, media):
+ original = (media / "notes.txt").read_bytes()
+ bv.pack([media / "notes.txt"], tmp_path / "a", engine="native", delete_source=True)
+ assert not (media / "notes.txt").exists()
+
+ bv.unpack(tmp_path / "a", tmp_path / "out")
+ assert (tmp_path / "out" / "notes.txt").read_bytes() == original
+
+
+def test_cli_with_native(tmp_path, media, capsys):
+ code = main([
+ "pack", str(media / "notes.txt"), "-o", str(tmp_path / "a"),
+ "--engine", "native", "--volume-size", "4k", "--json", "-q", "--no-log",
+ ]) # fmt: skip
+ assert code == 0
+ assert json.loads(capsys.readouterr().out)["engine"] == "native"
+
+ assert main(["unpack", str(tmp_path / "a"), "-o", str(tmp_path / "o"), "-q",
+ "--no-log"]) == 0 # fmt: skip
+
+
+def test_folder_tree_is_kept_with_native(tmp_path):
+ tree = tmp_path / "tree"
+ (tree / "Resources" / "intro").mkdir(parents=True)
+ first = tree / "Resources" / "intro" / "notes.txt"
+ second = tree / "Resources" / "notes.txt" # same name, other folder
+ first.write_text("intro " * 500)
+ second.write_text("resources " * 500)
+
+ result = bv.pack([first, second], tmp_path / "a", engine="native", root=tree)
+ assert result.status == bv.Status.SUCCESS
+ assert (tmp_path / "a/Resources/intro/notes.txt/notes.txt.bv.001").is_file()
+
+ bv.unpack(tmp_path / "a", tmp_path / "out")
+ assert (tmp_path / "out/Resources/intro/notes.txt").read_text() == first.read_text()
+ assert (tmp_path / "out/Resources/notes.txt").read_text() == second.read_text()
diff --git a/tests/integration/native/test_native_roundtrip.py b/tests/integration/native/test_native_roundtrip.py
new file mode 100644
index 0000000..2389a6c
--- /dev/null
+++ b/tests/integration/native/test_native_roundtrip.py
@@ -0,0 +1,83 @@
+import os
+import unicodedata
+
+import pytest
+
+from batchvault.core.errors import EngineError, OutputExistsError
+from batchvault.core.options import PackOptions
+from batchvault.engines.native import NativeEngine
+
+pytestmark = pytest.mark.integration
+
+engine = NativeEngine()
+
+
+@pytest.fixture
+def source(tmp_path):
+ path = tmp_path / "données 1.bin"
+ path.write_bytes(os.urandom(20_000))
+ return path
+
+
+@pytest.mark.parametrize("fmt", ["xz", "gzip", "bzip2"])
+def test_roundtrip_with_split_volumes(tmp_path, source, fmt):
+ archives = tmp_path / "archives"
+ volumes = engine.pack(
+ source, archives, PackOptions(volume_size=4096, jobs=2, format=fmt)
+ )
+
+ assert len(volumes) > 1
+ # macOS may report accents in decomposed form; the name is the same.
+ assert unicodedata.normalize("NFC", volumes[0].name) == "données 1.bin.bv.001"
+
+ engine.test(volumes[0])
+
+ restored = engine.unpack(volumes[2], tmp_path / "restored")
+ assert [unicodedata.normalize("NFC", p.name) for p in restored] == [
+ unicodedata.normalize("NFC", source.name)
+ ]
+ assert restored[0].read_bytes() == source.read_bytes()
+
+
+def test_large_file_across_many_volumes(tmp_path):
+ path = tmp_path / "big.bin"
+ path.write_bytes(os.urandom(3_000_000))
+
+ volumes = engine.pack(
+ path, tmp_path / "archives", PackOptions(volume_size=256 * 1024, jobs=2)
+ )
+ assert len(volumes) > 10
+
+ restored = engine.unpack(volumes[-1], tmp_path / "restored")
+ assert restored[0].read_bytes() == path.read_bytes()
+
+
+def test_existing_archive_is_not_replaced_without_force(tmp_path, source):
+ archives = tmp_path / "archives"
+ engine.pack(source, archives, PackOptions())
+
+ with pytest.raises(OutputExistsError):
+ engine.pack(source, archives, PackOptions())
+
+ engine.pack(source, archives, PackOptions(force=True))
+
+
+def test_unpack_does_not_overwrite_without_force(tmp_path, source):
+ volumes = engine.pack(source, tmp_path / "archives", PackOptions())
+ output = tmp_path / "restored"
+ engine.unpack(volumes[0], output)
+
+ with pytest.raises(OutputExistsError):
+ engine.unpack(volumes[0], output)
+
+ engine.unpack(volumes[0], output, force=True)
+
+
+def test_corrupt_volume_fails_test(tmp_path, source):
+ volumes = engine.pack(source, tmp_path / "archives", PackOptions())
+ data = bytearray(volumes[0].read_bytes())
+ data[len(data) // 2] ^= 0xFF
+ volumes[0].write_bytes(bytes(data))
+
+ with pytest.raises(EngineError):
+ engine.test(volumes[0])
diff --git a/tests/integration/test_sevenzip_api.py b/tests/integration/sevenzip/test_sevenzip_api.py
similarity index 97%
rename from tests/integration/test_sevenzip_api.py
rename to tests/integration/sevenzip/test_sevenzip_api.py
index 8292b1d..f5d1ddc 100644
--- a/tests/integration/test_sevenzip_api.py
+++ b/tests/integration/sevenzip/test_sevenzip_api.py
@@ -6,7 +6,7 @@
import pytest
import batchvault as bv
-from batchvault.cli import main
+from batchvault.cli.cli import main
from batchvault.engines.sevenzip import SevenZipEngine
pytestmark = pytest.mark.integration
@@ -105,9 +105,7 @@ def test_encrypted_roundtrip(tmp_path, media):
def test_delete_source_after_roundtrip(tmp_path, media):
original = (media / "notes.txt").read_bytes()
- bv.pack(
- [media / "notes.txt"], tmp_path / "a", engine="7zip", delete_source=True
- )
+ bv.pack([media / "notes.txt"], tmp_path / "a", engine="7zip", delete_source=True)
assert not (media / "notes.txt").exists()
bv.unpack(tmp_path / "a", tmp_path / "out")
diff --git a/tests/integration/test_sevenzip_roundtrip.py b/tests/integration/sevenzip/test_sevenzip_roundtrip.py
similarity index 94%
rename from tests/integration/test_sevenzip_roundtrip.py
rename to tests/integration/sevenzip/test_sevenzip_roundtrip.py
index bcdd3d6..44886d5 100644
--- a/tests/integration/test_sevenzip_roundtrip.py
+++ b/tests/integration/sevenzip/test_sevenzip_roundtrip.py
@@ -3,9 +3,9 @@
import pytest
+from batchvault.core.errors import EngineError, OutputExistsError
+from batchvault.core.options import PackOptions
from batchvault.engines.sevenzip import SevenZipEngine
-from batchvault.errors import EngineError, OutputExistsError
-from batchvault.options import PackOptions
pytestmark = pytest.mark.integration
diff --git a/tests/test_crypto.py b/tests/unit/archive/test_crypto.py
similarity index 93%
rename from tests/test_crypto.py
rename to tests/unit/archive/test_crypto.py
index 65cd742..8a9d14a 100644
--- a/tests/test_crypto.py
+++ b/tests/unit/archive/test_crypto.py
@@ -4,9 +4,9 @@
pytest.importorskip("cryptography")
+from batchvault.core.errors import EngineError # noqa: E402
+from batchvault.core.options import PackOptions # noqa: E402
from batchvault.engines.native import NativeEngine # noqa: E402
-from batchvault.errors import EngineError # noqa: E402
-from batchvault.options import PackOptions # noqa: E402
engine = NativeEngine(block_size=4096)
PASSWORD = "correct horse"
@@ -89,7 +89,7 @@ def test_dropping_whole_records_is_detected(tmp_path, source):
def test_real_default_is_a_strong_setting(monkeypatch):
- from batchvault import crypto
+ from batchvault.archive import crypto
monkeypatch.undo() # drop the cheap setting the test suite uses
assert crypto.DEFAULT_LOG2_N >= 17 # at least 128 MiB of scrypt memory
@@ -97,7 +97,7 @@ def test_real_default_is_a_strong_setting(monkeypatch):
def test_archives_open_whatever_cost_they_were_made_with(tmp_path, source, monkeypatch):
- from batchvault import crypto
+ from batchvault.archive import crypto
monkeypatch.setattr(crypto, "DEFAULT_LOG2_N", 14)
volumes = pack(source, tmp_path)
@@ -109,7 +109,7 @@ def test_archives_open_whatever_cost_they_were_made_with(tmp_path, source, monke
@pytest.mark.parametrize("log2_n", [0, 13, 19, 30, 255])
def test_header_asking_for_absurd_memory_is_refused(tmp_path, source, log2_n):
- from batchvault import crypto
+ from batchvault.archive import crypto
volumes = pack(source, tmp_path)
data = bytearray(volumes[0].read_bytes())
@@ -121,7 +121,7 @@ def test_header_asking_for_absurd_memory_is_refused(tmp_path, source, log2_n):
def test_header_cost_cannot_be_lowered_without_detection(tmp_path, source):
- from batchvault import crypto
+ from batchvault.archive import crypto
volumes = pack(source, tmp_path)
data = bytearray(volumes[0].read_bytes())
@@ -135,7 +135,7 @@ def test_header_cost_cannot_be_lowered_without_detection(tmp_path, source):
def test_an_absurd_record_length_is_refused_without_allocating(tmp_path):
import io
- from batchvault import crypto
+ from batchvault.archive import crypto
header = crypto.make_header("store")
stream = io.BytesIO(header + (0xFFFFFFFF).to_bytes(4, "big") + b"x" * 64)
diff --git a/tests/test_manifest.py b/tests/unit/archive/test_manifest.py
similarity index 96%
rename from tests/test_manifest.py
rename to tests/unit/archive/test_manifest.py
index 5a2fd65..e485d6d 100644
--- a/tests/test_manifest.py
+++ b/tests/unit/archive/test_manifest.py
@@ -2,13 +2,13 @@
import pytest
-from batchvault.errors import ManifestError
-from batchvault.manifest import (
+from batchvault.archive.manifest import (
Manifest,
ManifestEntry,
VolumeInfo,
hash_file,
)
+from batchvault.core.errors import ManifestError
def make_entry(name="a.txt", **overrides):
diff --git a/tests/unit/archive/test_readme.py b/tests/unit/archive/test_readme.py
new file mode 100644
index 0000000..764eab0
--- /dev/null
+++ b/tests/unit/archive/test_readme.py
@@ -0,0 +1,223 @@
+import os
+import re
+import shlex
+import shutil
+
+import pytest
+
+import batchvault as bv
+from batchvault.archive import signing
+from batchvault.archive.readme import MARKER, README_NAME
+from batchvault.cli.cli import main
+
+try:
+ import cryptography # noqa: F401
+
+ HAS_CRYPTO = True
+except ImportError:
+ HAS_CRYPTO = False
+
+needs_crypto = pytest.mark.skipif(not HAS_CRYPTO, reason="cryptography not installed")
+
+PASSWORD = "s3cret-pass-phrase"
+
+
+@pytest.fixture
+def media(tmp_path):
+ folder = tmp_path / "media"
+ folder.mkdir()
+ (folder / "notes.txt").write_text("hello " * 300, encoding="utf-8")
+ (folder / "data.bin").write_bytes(os.urandom(3000))
+ return folder
+
+
+def pack(media, tmp_path, **options):
+ return bv.pack(
+ [media / "notes.txt", media / "data.bin"],
+ tmp_path / "a",
+ engine="native",
+ volume_size="1k",
+ **options,
+ )
+
+
+def readme_of(tmp_path):
+ return (tmp_path / "a" / README_NAME).read_text(encoding="utf-8")
+
+
+def commands_in(text, verb):
+ """The ``batchvault ...`` lines in the README's code blocks."""
+ return [
+ line
+ for block in re.findall(r"```\n(.*?)```", text, re.S)
+ for line in block.splitlines()
+ if line.startswith(f"batchvault {verb} ")
+ ]
+
+
+def run_readme_command(command, folder, monkeypatch):
+ monkeypatch.chdir(folder)
+ return main([*shlex.split(command)[1:], "--no-log", "-q"])
+
+
+def test_plain_archive_readme(tmp_path, media):
+ result = pack(media, tmp_path, include=["*.txt", "*.bin"], exclude=["*.tmp"])
+ assert result.warnings == []
+
+ text = readme_of(tmp_path)
+ assert text.splitlines()[0] == MARKER
+ assert f"BatchVault {bv.__version__}" in text
+ assert "Files: 2" in text
+ assert "`manifest.json`" in text
+ assert "| Encrypted | no |" in text and "| Signed manifest | no |" in text
+ assert "| Volume size | 1.0 KiB |" in text
+ assert "`*.txt`, `*.bin`" in text and "`*.tmp`" in text
+ assert f'pip install "batchvault=={bv.__version__}"' in text
+ assert "pipx" in text and "uv tool install" in text and "3.11.4" in text
+ assert "ghcr.io/coldsofttech/batchvault" in text and "BatchVault@v1" in text
+ assert "--password-file" not in text.split("## Restore")[1].split("###")[0]
+ assert "--public-key" not in text.split("## Restore")[1].split("###")[0]
+
+
+def test_readme_commands_work_as_written(tmp_path, media, monkeypatch):
+ pack(media, tmp_path)
+ text = readme_of(tmp_path)
+ folder = tmp_path / "a"
+
+ assert run_readme_command(commands_in(text, "verify")[0], folder, monkeypatch) == 0
+
+ folder_command = re.search(r"`(batchvault unpack \. -o restored)`", text)
+ assert folder_command, text
+ unpacks = [*commands_in(text, "unpack"), folder_command.group(1)]
+ assert len(unpacks) == 2
+ for command in unpacks:
+ assert run_readme_command(command, folder, monkeypatch) == 0
+ assert (folder / "restored" / "notes.txt").read_text() == "hello " * 300
+ assert (folder / "restored" / "data.bin").read_bytes() == (
+ media / "data.bin"
+ ).read_bytes()
+ shutil.rmtree(folder / "restored")
+
+
+def test_single_volume_command_in_readme(tmp_path, media, monkeypatch):
+ pack(media, tmp_path)
+ text = readme_of(tmp_path)
+ command = re.search(r"`(batchvault unpack \S+\.001 -o restored)`", text)
+ assert command, text
+
+ folder = tmp_path / "a"
+ assert run_readme_command(command.group(1), folder, monkeypatch) == 0
+ assert any((folder / "restored").iterdir())
+
+
+@needs_crypto
+def test_encrypted_readme_has_no_secret(tmp_path, media, monkeypatch):
+ pack(media, tmp_path, password=PASSWORD)
+ text = readme_of(tmp_path)
+
+ assert PASSWORD not in text
+ assert "| Encrypted | yes |" in text
+ assert "This archive is encrypted" in text
+ assert "batchvault[crypto]==" in text
+ assert "BATCHVAULT_PASSWORD" in text
+
+ (tmp_path / "a" / "password.txt").write_text(PASSWORD, encoding="utf-8")
+ folder = tmp_path / "a"
+ for verb in ("verify", "unpack"):
+ command = commands_in(text, verb)[0]
+ assert "--password-file password.txt" in command
+ assert run_readme_command(command, folder, monkeypatch) == 0
+ assert (folder / "restored" / "notes.txt").exists()
+
+
+@needs_crypto
+def test_signed_readme(tmp_path, media, monkeypatch):
+ private, public = tmp_path / "k.key", tmp_path / "k.pub"
+ signing.generate_keypair(private, public)
+ pack(media, tmp_path, sign_key=private)
+ text = readme_of(tmp_path)
+
+ assert "| Signed manifest | yes |" in text
+ assert "manifest.json.sig" in text
+ assert "BEGIN" not in text # no key material
+ assert str(private) not in text
+
+ folder = tmp_path / "a"
+ (folder / "batchvault.pub").write_bytes(public.read_bytes())
+ for verb in ("verify", "unpack"):
+ command = commands_in(text, verb)[0]
+ assert "--public-key batchvault.pub" in command
+ assert run_readme_command(command, folder, monkeypatch) == 0
+
+
+def test_no_readme_option(tmp_path, media):
+ pack(media, tmp_path, readme=False)
+ assert not (tmp_path / "a" / README_NAME).exists()
+
+
+def test_no_readme_cli_flag(tmp_path, media):
+ args = ["pack", str(media / "notes.txt"), "-o", str(tmp_path / "a"),
+ "--engine", "native", "-q", "--no-log"] # fmt: skip
+ assert main([*args, "--no-readme"]) == 0
+ assert not (tmp_path / "a" / README_NAME).exists()
+ assert main([*args, "--force"]) == 0
+ assert (tmp_path / "a" / README_NAME).exists()
+
+
+def test_dry_run_writes_nothing(tmp_path, media):
+ pack(media, tmp_path, dry_run=True)
+ assert not (tmp_path / "a").exists()
+
+
+def test_readme_is_not_an_archive_volume(tmp_path, media):
+ pack(media, tmp_path)
+ assert bv.verify(tmp_path / "a").status == bv.Status.SUCCESS
+ assert {e.name for e in bv.inspect(tmp_path / "a")} == {"notes.txt", "data.bin"}
+
+
+def test_update_refreshes_readme_and_keeps_working(tmp_path, media):
+ pack(media, tmp_path)
+ first = readme_of(tmp_path)
+
+ again = pack(media, tmp_path, update=True)
+ assert again.warnings == []
+ assert "Files: 2" in readme_of(tmp_path)
+
+ (media / "notes.txt").write_text("changed " * 200, encoding="utf-8")
+ pack(media, tmp_path, update=True)
+ assert "Update mode (`--update`) | yes" in readme_of(tmp_path)
+ assert readme_of(tmp_path) != first
+ assert bv.verify(tmp_path / "a").status == bv.Status.SUCCESS
+
+
+def test_own_readme_is_never_overwritten(tmp_path, media):
+ out = tmp_path / "a"
+ out.mkdir()
+ (out / README_NAME).write_text("my own notes\n", encoding="utf-8")
+
+ result = pack(media, tmp_path)
+ assert (out / README_NAME).read_text() == "my own notes\n"
+ assert any(README_NAME in w for w in result.warnings)
+
+
+def test_source_with_the_readme_name_is_left_alone(tmp_path, media):
+ (media / README_NAME).write_text("a source file\n", encoding="utf-8")
+ result = bv.pack(
+ [media / README_NAME],
+ tmp_path / "a",
+ engine="native",
+ allow_archives=True,
+ )
+ assert result.status == bv.Status.SUCCESS
+ assert (tmp_path / "a" / README_NAME).is_dir() # the archive of that source
+ assert any(README_NAME in w for w in result.warnings)
+
+ restored = tmp_path / "r"
+ bv.unpack(tmp_path / "a", restored)
+ assert (restored / README_NAME).read_text() == "a source file\n"
+
+
+def test_packing_the_archive_folder_skips_the_readme(tmp_path, media):
+ pack(media, tmp_path)
+ with pytest.raises(bv.InputError):
+ bv.pack([tmp_path / "a" / README_NAME], tmp_path / "b", engine="native")
diff --git a/tests/unit/archive/test_resume.py b/tests/unit/archive/test_resume.py
new file mode 100644
index 0000000..e40d71f
--- /dev/null
+++ b/tests/unit/archive/test_resume.py
@@ -0,0 +1,589 @@
+import json
+import os
+from pathlib import Path
+
+import pytest
+
+import batchvault as bv
+from batchvault.archive import signing
+from batchvault.archive.resume import PACK_JOURNAL, UNPACK_JOURNAL
+from batchvault.cli.cli import main
+
+try:
+ import cryptography # noqa: F401
+
+ HAS_CRYPTO = True
+except ImportError:
+ HAS_CRYPTO = False
+
+needs_crypto = pytest.mark.skipif(not HAS_CRYPTO, reason="cryptography not installed")
+
+PASSWORD = "correct horse battery staple"
+
+
+@pytest.fixture
+def media(tmp_path):
+ folder = tmp_path / "media"
+ folder.mkdir()
+ for name in ("a.bin", "b.bin", "c.bin"):
+ (folder / name).write_bytes(os.urandom(6000))
+ return folder
+
+
+def sources(media):
+ return [media / "a.bin", media / "b.bin", media / "c.bin"]
+
+
+def interrupt_after(count):
+ """A progress callback that raises Ctrl+C after ``count`` files finished."""
+ seen = []
+
+ def progress(event):
+ if event.kind == "finish":
+ seen.append(event)
+ if len(seen) == count:
+ raise KeyboardInterrupt
+
+ return progress
+
+
+def pack_args(media, tmp_path, **extra):
+ options = dict(engine="native", volume_size="2k", format="xz", level=5)
+ options.update(extra)
+ return sources(media), tmp_path / "a", options
+
+
+def interrupted_pack(media, tmp_path, after=2, **extra):
+ files, output, options = pack_args(media, tmp_path, **extra)
+ with pytest.raises(KeyboardInterrupt):
+ bv.pack(files, output, progress=interrupt_after(after), **options)
+ return output
+
+
+def journal_lines(path):
+ return [json.loads(line) for line in path.read_text().splitlines()]
+
+
+def first_volume(entry_dir: Path) -> Path:
+ return sorted(entry_dir.glob("*.001"))[0]
+
+
+# ---------------------------------------------------------------- pack
+
+
+def test_interrupted_pack_leaves_a_journal(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+
+ lines = journal_lines(output / PACK_JOURNAL)
+ assert lines[0]["kind"] == "header"
+ assert lines[0]["settings"]["volume_size"] == 2048
+ assert [line["key"] for line in lines[1:]] == ["a.bin"]
+ assert not (output / "manifest.json").exists()
+
+
+def test_journal_never_holds_a_secret(tmp_path, media):
+ pytest.importorskip("cryptography")
+ files, output, options = pack_args(media, tmp_path)
+ with pytest.raises(KeyboardInterrupt):
+ bv.pack(
+ files, output, password=PASSWORD, progress=interrupt_after(2), **options
+ )
+ text = (output / PACK_JOURNAL).read_text()
+ assert PASSWORD not in text
+ assert json.loads(text.splitlines()[0])["settings"]["encrypted"] is True
+
+
+def test_resume_finishes_the_run_and_cleans_up(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+
+ result = bv.pack(output=output, resume=True)
+
+ assert result.status == bv.Status.SUCCESS and result.resume
+ assert [e.name for e in result.entries] == ["a.bin", "b.bin", "c.bin"]
+ assert result.reused == 1 and result.redone == 0
+ assert result.entries[0].resumed and not result.entries[1].resumed
+ assert "Resumed" in result.entries[0].message
+ assert not (output / PACK_JOURNAL).exists()
+ assert (output / "BATCHVAULT-README.md").exists()
+ assert bv.verify(output).status == bv.Status.SUCCESS
+
+ restored = tmp_path / "r"
+ bv.unpack(output, restored)
+ for path in sources(media):
+ assert (restored / path.name).read_bytes() == path.read_bytes()
+
+
+def test_resumed_archive_matches_an_uninterrupted_one(tmp_path, media):
+ files, _, options = pack_args(media, tmp_path)
+ bv.pack(files, tmp_path / "clean", **options)
+
+ output = interrupted_pack(media, tmp_path)
+ bv.pack(output=output, resume=True)
+
+ def volumes(folder):
+ manifest = json.loads((folder / "manifest.json").read_text())
+ return {
+ e["name"]: (e["sha256"], [(v["size"], v["sha256"]) for v in e["volumes"]])
+ for e in manifest["entries"]
+ }
+
+ assert volumes(output) == volumes(tmp_path / "clean")
+
+
+def test_resume_without_a_journal_is_an_error(tmp_path):
+ with pytest.raises(bv.ResumeError, match="Nothing to resume"):
+ bv.pack(output=tmp_path / "empty", resume=True)
+
+
+def test_completed_run_leaves_no_journal(tmp_path, media):
+ files, output, options = pack_args(media, tmp_path)
+ bv.pack(files, output, **options)
+ assert not (output / PACK_JOURNAL).exists()
+ with pytest.raises(bv.ResumeError):
+ bv.pack(output=output, resume=True)
+
+
+def test_resume_refuses_settings_given_again(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+
+ with pytest.raises(bv.OptionsError, match="volume_size"):
+ bv.pack(output=output, resume=True, volume_size="1k")
+ with pytest.raises(bv.OptionsError, match="files"):
+ bv.pack(sources(media), output, resume=True)
+ with pytest.raises(bv.OptionsError, match="dry_run"):
+ bv.pack(output=output, resume=True, dry_run=True)
+ # nothing was changed by the refusals
+ assert bv.pack(output=output, resume=True).status == bv.Status.SUCCESS
+
+
+@pytest.mark.parametrize(
+ "damage, reason",
+ [
+ ("flip", "fails its checksum"),
+ ("truncate", "wrong size"),
+ ("delete", "missing volume"),
+ ],
+)
+def test_damaged_volume_is_redone_never_trusted(tmp_path, media, damage, reason):
+ output = interrupted_pack(media, tmp_path)
+ volume = first_volume(output / "a.bin")
+ data = volume.read_bytes()
+ if damage == "flip":
+ volume.write_bytes(bytes([data[0] ^ 0xFF]) + data[1:])
+ elif damage == "truncate":
+ volume.write_bytes(data[: len(data) // 2])
+ else:
+ volume.unlink()
+
+ result = bv.pack(output=output, resume=True)
+
+ assert result.status == bv.Status.SUCCESS
+ redone = result.entries[0]
+ assert redone.redone and reason in redone.redone and not redone.resumed
+ assert "redone" in redone.message
+ assert result.redone == 1 and result.reused == 0
+ assert bv.verify(output).status == bv.Status.SUCCESS
+
+
+def test_changed_source_is_redone(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ (media / "a.bin").write_bytes(os.urandom(5000))
+
+ result = bv.pack(output=output, resume=True)
+
+ assert result.entries[0].redone == "the source file changed"
+ restored = tmp_path / "r"
+ bv.unpack(output, restored)
+ assert (restored / "a.bin").read_bytes() == (media / "a.bin").read_bytes()
+
+
+def test_torn_last_journal_line_is_ignored(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ with (output / PACK_JOURNAL).open("a") as handle:
+ handle.write('{"kind": "file", "key": "b.bin", "ent') # died mid-write
+
+ result = bv.pack(output=output, resume=True)
+
+ assert result.status == bv.Status.SUCCESS and result.reused == 1
+ assert bv.verify(output).status == bv.Status.SUCCESS
+
+
+def test_damaged_journal_is_refused(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ path = output / PACK_JOURNAL
+ lines = path.read_text().splitlines()
+ path.write_text("\n".join([lines[0], "not json", lines[1]]) + "\n")
+
+ with pytest.raises(bv.ResumeError, match="damaged"):
+ bv.pack(output=output, resume=True)
+
+ path.write_text("")
+ with pytest.raises(bv.ResumeError):
+ bv.pack(output=output, resume=True)
+
+
+def test_journal_from_unpack_is_not_used_for_pack(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ (output / PACK_JOURNAL).rename(output / UNPACK_JOURNAL)
+ with pytest.raises(bv.ResumeError):
+ bv.pack(output=output, resume=True)
+
+
+def test_vanished_source_fails_and_keeps_the_journal(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ (media / "c.bin").unlink()
+
+ result = bv.pack(output=output, resume=True)
+
+ assert result.status == bv.Status.PARTIAL
+ failed = [e for e in result.entries if e.status == bv.Status.FAILED]
+ assert [e.name for e in failed] == ["c.bin"]
+ assert (output / PACK_JOURNAL).exists() # a later --resume can retry
+
+ (media / "c.bin").write_bytes(os.urandom(6000))
+ again = bv.pack(output=output, resume=True)
+ assert again.status == bv.Status.SUCCESS
+ assert not (output / PACK_JOURNAL).exists()
+
+
+def test_progress_counts_reused_work(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ events = []
+
+ bv.pack(output=output, resume=True, progress=events.append)
+
+ plan = events[0]
+ assert plan.kind == "plan" and plan.count == 3 and plan.size == 18000
+ finished = {e.source.name: e for e in events if e.kind == "finish"}
+ assert finished["a.bin"].resumed is True
+ assert finished["b.bin"].resumed is False
+
+
+def test_resume_with_delete_source_deletes_only_after_a_full_check(tmp_path, media):
+ output = interrupted_pack(media, tmp_path, delete_source=True)
+ originals = {p.name: p.read_bytes() for p in sources(media)}
+
+ result = bv.pack(output=output, resume=True, confirm=lambda paths: True)
+
+ assert result.status == bv.Status.SUCCESS
+ assert not any(p.exists() for p in sources(media))
+ restored = tmp_path / "r"
+ bv.unpack(output, restored)
+ assert {p.name: p.read_bytes() for p in restored.iterdir()} == originals
+
+
+def test_resume_declined_deletion_keeps_originals(tmp_path, media):
+ output = interrupted_pack(media, tmp_path, delete_source=True)
+
+ bv.pack(output=output, resume=True, confirm=lambda paths: False)
+
+ assert all(p.exists() for p in sources(media))
+
+
+@needs_crypto
+def test_resume_encrypted_needs_the_password(tmp_path, media):
+ files, output, options = pack_args(media, tmp_path)
+ with pytest.raises(KeyboardInterrupt):
+ bv.pack(
+ files, output, password=PASSWORD, progress=interrupt_after(2), **options
+ )
+
+ with pytest.raises(bv.ResumeError, match="password is required"):
+ bv.pack(output=output, resume=True)
+ with pytest.raises(bv.ResumeError, match="does not match"):
+ bv.pack(output=output, resume=True, password="wrong")
+
+ result = bv.pack(output=output, resume=True, password=PASSWORD)
+ assert result.status == bv.Status.SUCCESS and result.reused == 1
+ assert bv.verify(output, password=PASSWORD).status == bv.Status.SUCCESS
+ restored = tmp_path / "r"
+ bv.unpack(output, restored, password=PASSWORD)
+ for path in sources(media):
+ assert (restored / path.name).read_bytes() == path.read_bytes()
+
+
+@needs_crypto
+def test_resume_rejects_a_password_for_a_plain_run(tmp_path, media):
+ output = interrupted_pack(media, tmp_path)
+ with pytest.raises(bv.ResumeError, match="not encrypted"):
+ bv.pack(output=output, resume=True, password=PASSWORD)
+
+
+@needs_crypto
+def test_resume_signed_needs_the_key(tmp_path, media):
+ private, public = tmp_path / "k.key", tmp_path / "k.pub"
+ signing.generate_keypair(private, public)
+ output = interrupted_pack(media, tmp_path, sign_key=private)
+
+ with pytest.raises(bv.ResumeError, match="private key is required"):
+ bv.pack(output=output, resume=True)
+
+ result = bv.pack(output=output, resume=True, sign_key=private)
+ assert result.status == bv.Status.SUCCESS
+ assert (output / "manifest.json.sig").exists()
+ assert bv.verify(output, public_key=public).status == bv.Status.SUCCESS
+
+
+@needs_crypto
+def test_resume_rejects_a_key_for_an_unsigned_run(tmp_path, media):
+ private, public = tmp_path / "k.key", tmp_path / "k.pub"
+ signing.generate_keypair(private, public)
+ output = interrupted_pack(media, tmp_path)
+ with pytest.raises(bv.ResumeError, match="did not sign"):
+ bv.pack(output=output, resume=True, sign_key=private)
+
+
+# -------------------------------------------------------------- unpack
+
+
+@pytest.fixture
+def archive(tmp_path, media):
+ files, output, options = pack_args(media, tmp_path)
+ bv.pack(files, output, **options)
+ return output
+
+
+def interrupted_unpack(archive, tmp_path, after=2, **extra):
+ restored = tmp_path / "restored"
+ with pytest.raises(KeyboardInterrupt):
+ bv.unpack(archive, restored, progress=interrupt_after(after), **extra)
+ return restored
+
+
+def test_interrupted_unpack_leaves_a_journal(tmp_path, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+
+ lines = journal_lines(restored / UNPACK_JOURNAL)
+ assert lines[0]["command"] == "unpack"
+ assert [line["path"] for line in lines[1:]] == ["a.bin", "b.bin"]
+ assert not (restored / "c.bin").exists()
+
+
+def test_unpack_resume_finishes_and_cleans_up(tmp_path, media, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+
+ result = bv.unpack(output=restored, resume=True)
+
+ assert result.status == bv.Status.SUCCESS and result.resume
+ assert result.reused == 2 and result.redone == 0
+ assert [e.resumed for e in result.entries] == [True, True, False]
+ assert not (restored / UNPACK_JOURNAL).exists()
+ for path in sources(media):
+ assert (restored / path.name).read_bytes() == path.read_bytes()
+
+
+def test_unpack_resume_without_a_journal_is_an_error(tmp_path):
+ with pytest.raises(bv.ResumeError, match="Nothing to resume"):
+ bv.unpack(output=tmp_path / "nowhere", resume=True)
+
+
+def test_completed_unpack_leaves_no_journal(tmp_path, archive):
+ restored = tmp_path / "restored"
+ bv.unpack(archive, restored)
+ assert not (restored / UNPACK_JOURNAL).exists()
+
+
+def test_unpack_resume_refuses_source_force_and_update(tmp_path, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+ for extra in ({"source": archive}, {"force": True}, {"update": True}):
+ with pytest.raises(bv.OptionsError, match="resume takes every setting"):
+ bv.unpack(output=restored, resume=True, **extra)
+
+
+def test_unpack_requires_a_source_without_resume(tmp_path):
+ with pytest.raises(bv.OptionsError, match="source is required"):
+ bv.unpack(output=tmp_path / "x")
+
+
+def test_unpack_redoes_a_file_it_wrote_that_no_longer_matches(tmp_path, media, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+ (restored / "a.bin").write_bytes(b"cut short") # ours (journaled), now bad
+
+ result = bv.unpack(output=restored, resume=True)
+
+ assert result.status == bv.Status.SUCCESS
+ assert result.entries[0].redone and not result.entries[0].resumed
+ assert result.redone == 1 and result.reused == 1
+ assert (restored / "a.bin").read_bytes() == (media / "a.bin").read_bytes()
+
+
+def test_unpack_never_touches_unrelated_files(tmp_path, media, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+ unrelated = restored / "notes.txt"
+ unrelated.write_text("mine")
+ lookalike = restored / "c.bin" # same name as a file still to restore
+ lookalike.write_bytes(b"someone else's file")
+
+ result = bv.unpack(output=restored, resume=True)
+
+ assert unrelated.read_text() == "mine"
+ assert lookalike.read_bytes() == b"someone else's file"
+ skipped = [e for e in result.entries if e.status == bv.Status.SKIPPED]
+ assert [e.name for e in skipped] == ["c.bin"]
+ assert "left alone" in skipped[0].message
+ assert result.status == bv.Status.PARTIAL
+
+
+def test_unpack_resume_refuses_a_changed_manifest(tmp_path, media, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+ files, _, options = pack_args(media, tmp_path)
+ (media / "a.bin").write_bytes(os.urandom(100))
+ bv.pack(files, archive, force=True, **options)
+
+ with pytest.raises(bv.ResumeError, match="manifest"):
+ bv.unpack(output=restored, resume=True)
+
+
+def test_unpack_resume_keeps_the_journal_while_something_failed(
+ tmp_path, media, archive
+):
+ restored = interrupted_unpack(archive, tmp_path)
+ volume = first_volume(archive / "c.bin")
+ volume.write_bytes(b"x" + volume.read_bytes()[1:]) # corrupt what is left
+
+ result = bv.unpack(output=restored, resume=True)
+
+ assert result.status == bv.Status.PARTIAL
+ assert (restored / UNPACK_JOURNAL).exists()
+ assert not (restored / "c.bin").exists() # nothing unverified was placed
+
+
+@needs_crypto
+def test_unpack_resume_encrypted_and_signed(tmp_path, media):
+ private, public = tmp_path / "k.key", tmp_path / "k.pub"
+ signing.generate_keypair(private, public)
+ files, output, options = pack_args(media, tmp_path)
+ bv.pack(files, output, password=PASSWORD, sign_key=private, **options)
+
+ restored = tmp_path / "restored"
+ with pytest.raises(KeyboardInterrupt):
+ bv.unpack(
+ output, restored, password=PASSWORD, public_key=public,
+ progress=interrupt_after(2),
+ ) # fmt: skip
+
+ with pytest.raises(bv.ResumeError, match="public key is required"):
+ bv.unpack(output=restored, resume=True, password=PASSWORD)
+
+ result = bv.unpack(
+ output=restored, resume=True, password=PASSWORD, public_key=public
+ )
+ assert result.status == bv.Status.SUCCESS and result.reused == 2
+ for path in sources(media):
+ assert (restored / path.name).read_bytes() == path.read_bytes()
+
+
+# ----------------------------------------------------------------- CLI
+
+
+def run(*args):
+ return main([*args, "--no-log", "-q"])
+
+
+def test_cli_pack_resume_needs_only_the_output(tmp_path, media, capsys):
+ output = interrupted_pack(media, tmp_path)
+
+ assert main(["pack", "--resume", "-o", str(output), "--no-log", "--json"]) == 0
+ result = json.loads(capsys.readouterr().out)
+ assert result["resume"] == {"reused": 1, "redone": 0}
+ assert result["entries"][0]["resumed"] is True
+ assert not (output / PACK_JOURNAL).exists()
+
+
+def test_cli_pack_resume_refuses_settings(tmp_path, media, capsys):
+ output = interrupted_pack(media, tmp_path)
+
+ code = run("pack", "--resume", "-o", str(output), "--level", "5")
+ assert code == 2
+ assert "--level" in capsys.readouterr().err
+ assert run("pack", "--resume", "-o", str(output), "--volume-size=1k") == 2
+ assert run("pack", str(media / "a.bin"), "--resume", "-o", str(output)) == 2
+ assert (output / PACK_JOURNAL).exists() # untouched
+
+
+def test_cli_pack_resume_without_a_journal(tmp_path, capsys):
+ assert run("pack", "--resume", "-o", str(tmp_path / "none")) == 2
+ assert "Nothing to resume" in capsys.readouterr().err
+
+
+def test_cli_resume_settings_file_values_do_not_count_as_given(
+ tmp_path, media, monkeypatch
+):
+ output = interrupted_pack(media, tmp_path)
+ (tmp_path / "batchvault.toml").write_text(
+ '[pack]\nvolume-size = "7k"\nlevel = 9\n', encoding="utf-8"
+ )
+ monkeypatch.chdir(tmp_path)
+
+ assert run("pack", "--resume", "-o", str(output)) == 0
+ manifest = json.loads((output / "manifest.json").read_text())
+ assert {e["volume_size"] for e in manifest["entries"]} == {2048} # not 7k
+
+
+@needs_crypto
+def test_cli_pack_resume_encrypted(tmp_path, media, monkeypatch):
+ files, output, options = pack_args(media, tmp_path)
+ with pytest.raises(KeyboardInterrupt):
+ bv.pack(
+ files, output, password=PASSWORD, progress=interrupt_after(2), **options
+ )
+
+ monkeypatch.delenv("BATCHVAULT_PASSWORD", raising=False)
+ assert run("pack", "--resume", "-o", str(output)) == 2
+
+ monkeypatch.setenv("BATCHVAULT_PASSWORD", PASSWORD)
+ assert run("pack", "--resume", "-o", str(output)) == 0
+ assert bv.verify(output, password=PASSWORD).status == bv.Status.SUCCESS
+
+
+def test_cli_unpack_resume(tmp_path, media, archive, capsys):
+ restored = interrupted_unpack(archive, tmp_path)
+
+ assert main(["unpack", "--resume", "-o", str(restored), "--no-log", "--json"]) == 0
+ result = json.loads(capsys.readouterr().out)
+ assert result["resume"] == {"reused": 2, "redone": 0}
+
+
+def test_cli_unpack_resume_refuses_settings(tmp_path, archive):
+ restored = interrupted_unpack(archive, tmp_path)
+ assert run("unpack", str(archive), "--resume", "-o", str(restored)) == 2
+ assert run("unpack", "--resume", "--force", "-o", str(restored)) == 2
+ assert run("unpack", "--resume", "-o", str(tmp_path / "none")) == 2
+
+
+def test_cli_unpack_needs_a_source_without_resume(tmp_path):
+ assert run("unpack", "-o", str(tmp_path / "x")) == 2
+
+
+def test_config_resume_key_is_accepted(tmp_path, media, monkeypatch):
+ output = interrupted_pack(media, tmp_path)
+ (tmp_path / "batchvault.toml").write_text("[pack]\nresume = true\n")
+ monkeypatch.chdir(tmp_path)
+ assert run("pack", "-o", str(output)) == 0
+
+
+# ------------------------------------------------------------------ ETA
+
+
+def test_eta_does_not_count_reused_work_as_speed():
+ from batchvault.core.eta import EtaEstimator
+
+ now = [0.0]
+ eta = EtaEstimator(clock=lambda: now[0])
+ eta.start()
+ eta.skip(1000) # an earlier run already did this much
+ now[0] = 10.0
+
+ # 100 of the remaining 1000 bytes took 10 s -> 90 s left, not a tiny number
+ assert eta.estimate(done_bytes=1100, total_bytes=2000) == pytest.approx(90.0)
+
+
+def test_failed_unpack_into_a_new_folder_leaves_nothing_behind(tmp_path, archive):
+ for volume in archive.glob("*/*.001"):
+ volume.write_bytes(b"x" + volume.read_bytes()[1:]) # every archive is bad
+ restored = tmp_path / "new-folder"
+
+ result = bv.unpack(archive, restored)
+
+ assert result.status == bv.Status.FAILED
+ assert not restored.exists()
diff --git a/tests/test_signing.py b/tests/unit/archive/test_signing.py
similarity index 97%
rename from tests/test_signing.py
rename to tests/unit/archive/test_signing.py
index 5c97ab0..dcde6bc 100644
--- a/tests/test_signing.py
+++ b/tests/unit/archive/test_signing.py
@@ -3,8 +3,8 @@
pytest.importorskip("cryptography")
import batchvault as bv # noqa: E402
-from batchvault import signing # noqa: E402
-from batchvault.errors import InputError, SignatureError # noqa: E402
+from batchvault.archive import signing # noqa: E402
+from batchvault.core.errors import InputError, SignatureError # noqa: E402
@pytest.fixture
diff --git a/tests/test_volume_io.py b/tests/unit/archive/test_volume_io.py
similarity index 91%
rename from tests/test_volume_io.py
rename to tests/unit/archive/test_volume_io.py
index ae60834..7133259 100644
--- a/tests/test_volume_io.py
+++ b/tests/unit/archive/test_volume_io.py
@@ -1,7 +1,7 @@
import os
-from batchvault.blocks import BlockWriter, compress_block
-from batchvault.volumes import VolumeReader, VolumeWriter
+from batchvault.archive.blocks import BlockWriter, compress_block
+from batchvault.archive.volumes import VolumeReader, VolumeWriter
def test_writer_rolls_over_and_reader_joins(tmp_path):
diff --git a/tests/test_volumes.py b/tests/unit/archive/test_volumes.py
similarity index 92%
rename from tests/test_volumes.py
rename to tests/unit/archive/test_volumes.py
index 61d51d4..f807807 100644
--- a/tests/test_volumes.py
+++ b/tests/unit/archive/test_volumes.py
@@ -1,7 +1,7 @@
import pytest
-from batchvault.errors import VolumeError
-from batchvault.volumes import existing_volumes, find_volumes, split_volume_name
+from batchvault.archive.volumes import existing_volumes, find_volumes, split_volume_name
+from batchvault.core.errors import VolumeError
def make_volumes(directory, base, numbers):
diff --git a/tests/test_cli.py b/tests/unit/cli/test_cli.py
similarity index 99%
rename from tests/test_cli.py
rename to tests/unit/cli/test_cli.py
index 868b7a3..1a16af7 100644
--- a/tests/test_cli.py
+++ b/tests/unit/cli/test_cli.py
@@ -4,7 +4,7 @@
import pytest
from batchvault import __version__
-from batchvault.cli import main
+from batchvault.cli.cli import main
@pytest.fixture
diff --git a/tests/test_cli_features.py b/tests/unit/cli/test_cli_features.py
similarity index 99%
rename from tests/test_cli_features.py
rename to tests/unit/cli/test_cli_features.py
index 01b84d8..7a674ce 100644
--- a/tests/test_cli_features.py
+++ b/tests/unit/cli/test_cli_features.py
@@ -5,7 +5,7 @@
import pytest
-from batchvault.cli import main
+from batchvault.cli.cli import main
try:
import cryptography # noqa: F401
diff --git a/tests/test_logs.py b/tests/unit/cli/test_logs.py
similarity index 88%
rename from tests/test_logs.py
rename to tests/unit/cli/test_logs.py
index 965445c..a6b0ba6 100644
--- a/tests/test_logs.py
+++ b/tests/unit/cli/test_logs.py
@@ -1,7 +1,7 @@
import os
-from batchvault.logs import default_log_dir, prune_logs
-from batchvault.security import redact
+from batchvault.cli.logs import default_log_dir, prune_logs
+from batchvault.core.security import redact
def test_redact_hides_url_credentials():
diff --git a/tests/test_notify.py b/tests/unit/cli/test_notify.py
similarity index 92%
rename from tests/test_notify.py
rename to tests/unit/cli/test_notify.py
index 872fa97..20fd191 100644
--- a/tests/test_notify.py
+++ b/tests/unit/cli/test_notify.py
@@ -4,9 +4,9 @@
import pytest
-from batchvault import notify
-from batchvault.errors import OptionsError
-from batchvault.models import FileResult, RunResult, Status
+from batchvault.cli import notify
+from batchvault.core.errors import OptionsError
+from batchvault.core.models import FileResult, RunResult, Status
def make_result(status=Status.SUCCESS):
diff --git a/tests/test_summary.py b/tests/unit/cli/test_summary.py
similarity index 98%
rename from tests/test_summary.py
rename to tests/unit/cli/test_summary.py
index 3c0d6e4..21ab32e 100644
--- a/tests/test_summary.py
+++ b/tests/unit/cli/test_summary.py
@@ -8,9 +8,9 @@
from rich.console import Console
import batchvault as bv
-from batchvault import ui
-from batchvault.models import FileResult, RunResult, Status
-from batchvault.options import format_size
+from batchvault.cli import ui
+from batchvault.core.models import FileResult, RunResult, Status
+from batchvault.core.options import format_size
@pytest.fixture
diff --git a/tests/test_api.py b/tests/unit/core/api/test_api.py
similarity index 97%
rename from tests/test_api.py
rename to tests/unit/core/api/test_api.py
index 3647d8f..e39e4cb 100644
--- a/tests/test_api.py
+++ b/tests/unit/core/api/test_api.py
@@ -5,7 +5,7 @@
import batchvault as bv
from batchvault import Status
-from batchvault.errors import InputError, ManifestError
+from batchvault.core.errors import InputError, ManifestError
@pytest.fixture
@@ -90,9 +90,9 @@ def test_unpack_sources(tmp_path, media, how):
result = bv.unpack(source, tmp_path / "restored")
assert result.status == Status.SUCCESS
- assert (tmp_path / "restored" / "notes.txt").read_text(
- encoding="utf-8"
- ) == (media / "notes.txt").read_text(encoding="utf-8")
+ assert (tmp_path / "restored" / "notes.txt").read_text(encoding="utf-8") == (
+ media / "notes.txt"
+ ).read_text(encoding="utf-8")
assert not [p for p in (tmp_path / "restored").iterdir() if p.name.startswith(".")]
diff --git a/tests/test_features.py b/tests/unit/core/api/test_features.py
similarity index 96%
rename from tests/test_features.py
rename to tests/unit/core/api/test_features.py
index 1f9fd55..2acdb2e 100644
--- a/tests/test_features.py
+++ b/tests/unit/core/api/test_features.py
@@ -5,10 +5,11 @@
import pytest
import batchvault as bv
-from batchvault import api
+from batchvault.archive.volumes import cleanup_staging
+from batchvault.core.api import _common
+from batchvault.core.api.packing import single
+from batchvault.core.errors import OptionsError
from batchvault.engines.native import NativeEngine
-from batchvault.errors import OptionsError
-from batchvault.volumes import cleanup_staging
try:
import cryptography # noqa: F401
@@ -105,7 +106,7 @@ def test_unpack_update(tmp_path, media):
@pytest.fixture
def flaky(monkeypatch):
- monkeypatch.setattr(api, "RETRY_BASE_DELAY", 0)
+ monkeypatch.setattr(_common, "RETRY_BASE_DELAY", 0)
original = NativeEngine.pack
state = {"calls": 0, "fail_times": 2, "error": OSError}
@@ -205,7 +206,7 @@ def test_declined_confirmation_keeps_originals(tmp_path, media):
def test_failed_roundtrip_keeps_original(tmp_path, media, monkeypatch):
- monkeypatch.setattr(api, "_roundtrip_problem", lambda *args: "it does not match")
+ monkeypatch.setattr(single, "_roundtrip_problem", lambda *args: "it does not match")
result = bv.pack(
[media / "notes.txt"], tmp_path / "a", engine="native", delete_source=True
)
diff --git a/tests/test_folders.py b/tests/unit/core/api/test_folders.py
similarity index 98%
rename from tests/test_folders.py
rename to tests/unit/core/api/test_folders.py
index 9e4a6ee..67b9edd 100644
--- a/tests/test_folders.py
+++ b/tests/unit/core/api/test_folders.py
@@ -9,10 +9,10 @@
from rich.console import Console
import batchvault as bv
-from batchvault import ui
-from batchvault.cli import main
-from batchvault.errors import InputError, ManifestError
-from batchvault.volumes import cleanup_staging
+from batchvault.archive.volumes import cleanup_staging
+from batchvault.cli import ui
+from batchvault.cli.cli import main
+from batchvault.core.errors import InputError, ManifestError
FILES = {
"Resources/intro/lesson1.bin": os.urandom(5000),
diff --git a/tests/test_config.py b/tests/unit/core/test_config.py
similarity index 96%
rename from tests/test_config.py
rename to tests/unit/core/test_config.py
index af42247..d66469a 100644
--- a/tests/test_config.py
+++ b/tests/unit/core/test_config.py
@@ -2,9 +2,9 @@
import pytest
-from batchvault.cli import main
-from batchvault.config import find_config, load_config
-from batchvault.errors import ConfigError
+from batchvault.cli.cli import main
+from batchvault.core.config import find_config, load_config
+from batchvault.core.errors import ConfigError
@pytest.fixture
@@ -131,7 +131,7 @@ def test_error_explains_how_to_trust_the_file(tmp_path, monkeypatch, capsys):
def test_pyproject_section_is_restricted_too(tmp_path, monkeypatch):
(tmp_path / "pyproject.toml").write_text(
- '[tool.batchvault.pack]\ndelete-source = true\n'
+ "[tool.batchvault.pack]\ndelete-source = true\n"
)
monkeypatch.chdir(tmp_path)
assert run("doctor", "-q") == 2
diff --git a/tests/test_eta.py b/tests/unit/core/test_eta.py
similarity index 88%
rename from tests/test_eta.py
rename to tests/unit/core/test_eta.py
index 97e0f86..d2794db 100644
--- a/tests/test_eta.py
+++ b/tests/unit/core/test_eta.py
@@ -1,4 +1,4 @@
-from batchvault.eta import MIN_SECONDS, EtaEstimator, format_duration
+from batchvault.core.eta import MIN_SECONDS, EtaEstimator, format_duration
class FakeClock:
diff --git a/tests/test_filters.py b/tests/unit/core/test_filters.py
similarity index 93%
rename from tests/test_filters.py
rename to tests/unit/core/test_filters.py
index 0eca12f..ed370be 100644
--- a/tests/test_filters.py
+++ b/tests/unit/core/test_filters.py
@@ -1,4 +1,4 @@
-from batchvault.filters import PathFilter, load_patterns
+from batchvault.core.filters import PathFilter, load_patterns
def test_no_patterns_matches_everything():
diff --git a/tests/test_hardening.py b/tests/unit/core/test_hardening.py
similarity index 97%
rename from tests/test_hardening.py
rename to tests/unit/core/test_hardening.py
index 073435f..2edb87e 100644
--- a/tests/test_hardening.py
+++ b/tests/unit/core/test_hardening.py
@@ -12,7 +12,10 @@
import pytest
import batchvault as bv
-from batchvault.cli import main
+from batchvault.cli.cli import main
+from batchvault.core.errors import EngineError
+from batchvault.core.options import MIB, PackOptions
+from batchvault.core.security import redact_arguments
from batchvault.engines.native import NativeEngine
from batchvault.engines.sevenzip import (
SevenZipEngine,
@@ -21,11 +24,8 @@
build_unpack_command,
folder_size,
)
-from batchvault.errors import EngineError
-from batchvault.options import MIB, PackOptions
-from batchvault.security import redact_arguments
-REPO = Path(__file__).resolve().parents[1]
+REPO = Path(__file__).resolve().parents[3]
engine = NativeEngine(block_size=4096)
@@ -300,7 +300,7 @@ def test_a_path_that_looks_like_an_option_is_just_a_path(tmp_path, media):
def test_cleanup_ignores_linked_folders(tmp_path):
import time
- from batchvault.volumes import cleanup_staging
+ from batchvault.archive.volumes import cleanup_staging
elsewhere = tmp_path / "elsewhere"
victim = elsewhere / ".batchvault-keep"
diff --git a/tests/test_inputs.py b/tests/unit/core/test_inputs.py
similarity index 91%
rename from tests/test_inputs.py
rename to tests/unit/core/test_inputs.py
index 3a011d3..d9ea700 100644
--- a/tests/test_inputs.py
+++ b/tests/unit/core/test_inputs.py
@@ -3,9 +3,9 @@
import pytest
-from batchvault.errors import InputError
-from batchvault.filters import PathFilter
-from batchvault.inputs import collect_sources, gather_sources, read_list_entries
+from batchvault.core.errors import InputError
+from batchvault.core.filters import PathFilter
+from batchvault.core.inputs import collect_sources, gather_sources, read_list_entries
@pytest.fixture
@@ -64,10 +64,12 @@ def test_colliding_places_are_still_refused(tmp_path):
# Two lists recorded different roots, so both files would be stored as x/same.txt.
listing = tmp_path / "list.json"
listing.write_text(
- json.dumps([
- {"path": str(tmp_path / "a/x/same.txt"), "root": str(tmp_path / "a")},
- {"path": str(tmp_path / "b/x/same.txt"), "root": str(tmp_path / "b")},
- ]),
+ json.dumps(
+ [
+ {"path": str(tmp_path / "a/x/same.txt"), "root": str(tmp_path / "a")},
+ {"path": str(tmp_path / "b/x/same.txt"), "root": str(tmp_path / "b")},
+ ]
+ ),
encoding="utf-8",
)
with pytest.raises(InputError, match="same archive folder"):
diff --git a/tests/test_levels.py b/tests/unit/core/test_levels.py
similarity index 93%
rename from tests/test_levels.py
rename to tests/unit/core/test_levels.py
index 4278ff3..16562f9 100644
--- a/tests/test_levels.py
+++ b/tests/unit/core/test_levels.py
@@ -6,9 +6,9 @@
import pytest
import batchvault as bv
-from batchvault.cli import main
-from batchvault.errors import OptionsError
-from batchvault.levels import (
+from batchvault.cli.cli import main
+from batchvault.core.errors import OptionsError
+from batchvault.core.levels import (
AUTO,
BINARY_LEVEL,
DEFAULT_LEVEL,
@@ -33,8 +33,24 @@ def make(tmp_path, name, content=b"plain bytes\n" * 100):
@pytest.mark.parametrize(
"name",
- ["a.mp4", "a.MKV", "a.jpg", "a.png", "a.zip", "a.7z", "a.gz", "a.docx", "a.mp3",
- "a.flac", "a.webp", "a.heic", "a.tar.gz", "a.xlsx", "a.cr2", "a.webm"],
+ [
+ "a.mp4",
+ "a.MKV",
+ "a.jpg",
+ "a.png",
+ "a.zip",
+ "a.7z",
+ "a.gz",
+ "a.docx",
+ "a.mp3",
+ "a.flac",
+ "a.webp",
+ "a.heic",
+ "a.tar.gz",
+ "a.xlsx",
+ "a.cr2",
+ "a.webm",
+ ],
)
def test_already_compressed_files_get_level_zero(tmp_path, name):
choice = choose_level(make(tmp_path, name))
@@ -42,8 +58,20 @@ def test_already_compressed_files_get_level_zero(tmp_path, name):
@pytest.mark.parametrize(
- "name", ["a.txt", "a.LOG", "a.csv", "a.json", "a.xml", "a.py", "a.sql", "a.md",
- "a.html", "a.yaml", "a.svg"],
+ "name",
+ [
+ "a.txt",
+ "a.LOG",
+ "a.csv",
+ "a.json",
+ "a.xml",
+ "a.py",
+ "a.sql",
+ "a.md",
+ "a.html",
+ "a.yaml",
+ "a.svg",
+ ],
)
def test_text_gets_a_strong_level(tmp_path, name):
choice = choose_level(make(tmp_path, name))
@@ -261,6 +289,6 @@ def test_config_can_set_auto_and_rejects_nonsense(tmp_path, mixed):
assert entry["entries"][0]["level"] == 0
for bad in ('"fast"', "12", "-1"):
- config.write_text(f'[pack]\nlevel = {bad}\n')
+ config.write_text(f"[pack]\nlevel = {bad}\n")
assert run("pack", source, "-o", str(tmp_path / "b"), "-q",
"--config", str(config)) == 2 # fmt: skip
diff --git a/tests/test_options.py b/tests/unit/core/test_options.py
similarity index 87%
rename from tests/test_options.py
rename to tests/unit/core/test_options.py
index 838d346..299e30a 100644
--- a/tests/test_options.py
+++ b/tests/unit/core/test_options.py
@@ -1,7 +1,7 @@
import pytest
-from batchvault.errors import OptionsError
-from batchvault.options import MIB, PackOptions, parse_size
+from batchvault.core.errors import OptionsError
+from batchvault.core.options import MIB, PackOptions, parse_size
@pytest.mark.parametrize(
diff --git a/tests/test_native.py b/tests/unit/engines/test_native.py
similarity index 91%
rename from tests/test_native.py
rename to tests/unit/engines/test_native.py
index 1941370..1127fed 100644
--- a/tests/test_native.py
+++ b/tests/unit/engines/test_native.py
@@ -4,10 +4,10 @@
import pytest
-from batchvault.blocks import sniff_format
+from batchvault.archive.blocks import sniff_format
+from batchvault.core.errors import EngineError, OutputExistsError, VolumeError
+from batchvault.core.options import PackOptions
from batchvault.engines.native import NativeEngine, _check_member
-from batchvault.errors import EngineError, OutputExistsError, VolumeError
-from batchvault.options import PackOptions
# Small blocks and volumes so every test spans several of each.
engine = NativeEngine(block_size=4096)
@@ -162,8 +162,22 @@ def _member(name, kind=tarfile.REGTYPE):
@pytest.mark.parametrize(
"name",
- ["C:foo", "file.txt:stream", "NUL", "con.txt", "Aux", "COM1", "lpt9.log",
- "name.", "name ", 'a"b', "a= '3.15'",
+ "python_full_version < '3.15'",
+]
+
+[[package]]
+name = "ast-serialize"
+version = "0.13.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/21/d5/5dc1ee8a80c3b20a0b63c61b43ba6bc53517faecd1def47bcd752d22268e/ast_serialize-0.13.1.tar.gz", hash = "sha256:401cc84e8888d4d76f2c8d296fa006ed1a5ff993b6aff90dd777c03973efd025", size = 955137, upload-time = "2026-10-17T00:04:17.183Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f3/a8/fbd136b9f70c59e75f59a8e974c84102976f1201828b4f886480ba956936/ast_serialize-0.13.1-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:3e129d44b0cffbea24a7f40117ce49e30993c3d119e584d4817ebd6f87246456", size = 897306, upload-time = "2026-10-17T00:02:52.479Z" },
+ { url = "https://files.pythonhosted.org/packages/50/3f/2ad51e9823a790e7a70fd781e9595ab3abeace27f3aefcaa1939457efd72/ast_serialize-0.13.1-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:523d45bab16ddaf1399c04c4fe81bbf180f81e33c2f7bcbe76be56b0f7311ab4", size = 1236082, upload-time = "2026-10-17T00:02:54.82Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/5c/8e959db5cc2237f9d4c560a8e2253c98f9746883fb4774651e4cda458901/ast_serialize-0.13.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:638333fb8ae72c62753f9e794fda20290883ddc65b0d85e26e08f8bb227567ad", size = 1221242, upload-time = "2026-10-17T00:02:56.395Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/05/ab1c56c3b7adcac88764bba609502d6b4989e449b2dd60eb478e1d20e369/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cc34c670747d85d4232834665e247009413eaebe47c69eaf60a5c439aaecaed9", size = 1282105, upload-time = "2026-10-17T00:02:58.275Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/f4/41981fcd2ddffe0760e5326be9dcbd2eda9478a8430a637e5123bd8fcb26/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8c8ebd419811c435812d242fb2a173e57dfede198c9ac9a0a584666ebc9b9331", size = 1287905, upload-time = "2026-10-17T00:03:00.068Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/43/b783addc5681153707121991b7b47892f7e9c8ba58a0d03b71a3b023ba47/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8899c038b1c15ae8b98e379057fb3ef398ceb0b21d1737a8156dd5a0923b7e51", size = 1541806, upload-time = "2026-10-17T00:03:01.503Z" },
+ { url = "https://files.pythonhosted.org/packages/47/10/969db095038ee15270f65ad20d1129c5e4da605bf7e93b70ae640f1c8a9f/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e14c4de98f965647a74e6eaf37e20207124e45451ccd36e84a6f6ebab45e0372", size = 1309946, upload-time = "2026-10-17T00:03:02.971Z" },
+ { url = "https://files.pythonhosted.org/packages/06/dd/7c623d2fb0b868948952d1787e629a43b866f7d4caefd5445e20aefacec4/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7f3e8a3eff69af640db985360fef35073fb46956dfc3e5c614cf00920d2746dd", size = 1302773, upload-time = "2026-10-17T00:03:04.378Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/c2/4d49469110a19a44fc6269f4fd59679bcf84df068f185f89b73bee4da19b/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:75d7750fe98f6f35a96c43aba8d98a2e623e22c778a0c17ab9ede92eb10f5d82", size = 1299579, upload-time = "2026-10-17T00:03:05.88Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/9a/2d381f1f262dda15affc8ad643fba1619eb07e7d9f8bd904deffef183006/ast_serialize-0.13.1-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:43e8ddd63e80fc280c2f790157386f27f9b9f6ccb9c4f8e803b863cbfb085769", size = 1355969, upload-time = "2026-10-17T00:03:07.321Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/9c/2266794e6c25fa9ce51ce04a9bf4b7e330e3ac9351ba3fa14e487678acdb/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9221e09496fb942901993c1812ee87950cc0ecbb514c7c0f1f5fa6d78f701ed8", size = 1460227, upload-time = "2026-10-17T00:03:08.733Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/d6/14deb28e5d8f328f626a0778ea2cf98fe908f2bbcccd804cb6f449ee1782/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:7a5c38bdb93669c596f94fb7204dd6ebe4fadb19058ac9a2faf2948c2a175f96", size = 1562691, upload-time = "2026-10-17T00:03:10.25Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/19/7a28c0e5ba54f0d3e054f847a2a549b748cb179ab433321a8dc1b39e398d/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:acf4e9f055d75058ab01536da330648746f2f82b7697a85230547f2d5ebe895c", size = 1558669, upload-time = "2026-10-17T00:03:11.799Z" },
+ { url = "https://files.pythonhosted.org/packages/18/9a/f9e2690eab58217f1d395a964a78efa7eaf813a1e38be8dfd9a8d45d0840/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:87427913c325adc811e81e7c9a757b061f47ef1e29114ac479aa0953a2687a98", size = 1665071, upload-time = "2026-10-17T00:03:13.284Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/f3/ef5b159a0d7161b404f05e7ec21cf7735ee7edc10c000cd2cbbc7bb6b69a/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:ad88921f95b92bc144d9ad059a0e16f8c1ae69f5239f2d3c6536fd2774f73dfe", size = 1473364, upload-time = "2026-10-17T00:03:14.563Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/87/6e8ac4a865124a12cc4f915bf1bcfac3bba426cd5f58d3712a873ee61012/ast_serialize-0.13.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f8b17e2921e56621bcb46792bd3e0b82ca3b85de78d5936e7b8174ba1758b0aa", size = 1500566, upload-time = "2026-10-17T00:03:15.917Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/1e/29155f1621433008013ad5c940a03d41a676be0680c41527040472d37583/ast_serialize-0.13.1-cp314-cp314t-win32.whl", hash = "sha256:2f9a9ce8b0994f757b941f7f7fe384a6666b4162b941ebc5c248081baea18e47", size = 1128674, upload-time = "2026-10-17T00:03:17.481Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/64/2506da65143ea138de5822e9173e0b72320a2f70e23bd723f044033175f1/ast_serialize-0.13.1-cp314-cp314t-win_amd64.whl", hash = "sha256:5d99910e6b9a9b9360bcca2ae99aee36cc912ce058df2a57d1c33099b6cb2593", size = 1163306, upload-time = "2026-10-17T00:03:19.046Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/8f/c7cb22638899b0cf15908809c270560e46395e65604bc59e4b623ab9f405/ast_serialize-0.13.1-cp314-cp314t-win_arm64.whl", hash = "sha256:a789cbe14d0a0e9ec9b32a456e5abc81ad8799479c3cfd88a29e3442262aebe6", size = 1133775, upload-time = "2026-10-17T00:03:20.377Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/e2/b0c140fe50fa39dd249e382f7733af6f816c90021fe7959a0b026f86237f/ast_serialize-0.13.1-cp315-abi3.abi3t-macosx_10_12_x86_64.whl", hash = "sha256:23d768369b89d551077dc173da6892fad00443338417523b6732f4e8ba4a7ed5", size = 1236160, upload-time = "2026-10-17T00:03:21.945Z" },
+ { url = "https://files.pythonhosted.org/packages/23/c4/a8a2ca21e3943555b48221f8fba546877a7ad39efaf6505506524967d7c4/ast_serialize-0.13.1-cp315-abi3.abi3t-macosx_11_0_arm64.whl", hash = "sha256:68f8829d114dab6c9db4a8bbb8d0e342958faee1b5e15a29022678df639960f8", size = 1222214, upload-time = "2026-10-17T00:03:23.462Z" },
+ { url = "https://files.pythonhosted.org/packages/15/19/2f58f2ba64e4799b39de85c140aef09a03aa6ea53be75dd292a9d9b9d8ae/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e385b17d5bde892a6184003a9aff3518387908b73f8f5ea539313c07c8c0ede", size = 1282979, upload-time = "2026-10-17T00:03:24.695Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/af/ca6087fe84210c1ccd43e3182917bb70649fa1e69745a90c1d59ba08fea8/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:02686b281ec9b8e9614f6f0db27e12e4cac515f336c5d05b678bc9eb73ff2af5", size = 1288596, upload-time = "2026-10-17T00:03:26.04Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/2c/4ef964c763329a0347125e745ec0451ba4fbaceec9efe2ac729b7cd55f17/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a8ab2c2585b2e22e48cd4aff2136fce4eab4ced569b85f150ef9a5a5d6f34d49", size = 1545540, upload-time = "2026-10-17T00:03:27.52Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/85/aa961dbb75813cfa94de0ce9a16a9510c226b95f604c405421e727f44bb0/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c652d9f5d52d8b81eff3ff2c95c9dd0f281de45f6ad5d3fa95527550c396454a", size = 1309374, upload-time = "2026-10-17T00:03:29.091Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/e6/f2b67c327d9f9533e41791804403ccc2dc457821195a2e51f0b587a90b90/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:121e1d08d58b75c15491cc3003609cbad71ac2b0016043c36d30a6b2032b6605", size = 1303466, upload-time = "2026-10-17T00:03:30.451Z" },
+ { url = "https://files.pythonhosted.org/packages/73/1d/33f18b81d1bb1f5d682943ea92fda8130a6593627f12f6201668d40824fd/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_31_riscv64.whl", hash = "sha256:30cf89b6a6844542d116c758725d13de5b97b9410a51ee577715f761270d0c1c", size = 1300576, upload-time = "2026-10-17T00:03:32.102Z" },
+ { url = "https://files.pythonhosted.org/packages/25/cb/64914144925f1a4abdd9217e717b122255c38fc5ebe4342f92052f3bacbc/ast_serialize-0.13.1-cp315-abi3.abi3t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:732aaae9e498c973fc63fd7f308009b57c4afa6ff59d0b674d894e57d0bca01a", size = 1355960, upload-time = "2026-10-17T00:03:33.676Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/80/bd29379fab2bce19adc0bf9177002b11b23dfe8d8fecf409998b458b4624/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_aarch64.whl", hash = "sha256:d9c1a920b2aa0c1e016bed7721689717034ca0dba64ac3a3ff243b003d4e8dd8", size = 1459779, upload-time = "2026-10-17T00:03:35.09Z" },
+ { url = "https://files.pythonhosted.org/packages/83/89/012dbe733de05ffdf8e40ca0f9fa5d417cb16ec93a74463c57c001914198/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_armv7l.whl", hash = "sha256:847cad4c22953823f5e76d0b126c6f9da670cfa45b1df41b9324e22189d77611", size = 1563399, upload-time = "2026-10-17T00:03:37.285Z" },
+ { url = "https://files.pythonhosted.org/packages/69/37/ee4bcb0358bee38744eb0f18560446572d095fe3bf6b63d6f739d08a839c/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_i686.whl", hash = "sha256:aca07de3ca8e3d9f2b716c3c31f4f384fe4bd2d03e09912417cff7a3e08036f7", size = 1558808, upload-time = "2026-10-17T00:03:38.711Z" },
+ { url = "https://files.pythonhosted.org/packages/54/1c/ff44c5355251d61ef22934c4430bb31a61de4781d25a992154aac9b8c82a/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_ppc64le.whl", hash = "sha256:5db12d843d931b9d1f1519ce66b9cbfe72d8563b75c1204d255322254d1964dd", size = 1666324, upload-time = "2026-10-17T00:03:40.376Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/bb/6da4bfc8aa3403e6447607131bf3463acaf954d990c6c8b6787c1902ddfe/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_riscv64.whl", hash = "sha256:fb8021f4dc3296de18737e3b8b34994411da725374d74a0b9eb32a738f2a23d0", size = 1474089, upload-time = "2026-10-17T00:03:41.715Z" },
+ { url = "https://files.pythonhosted.org/packages/21/97/897e33bef2eefd880e6350e375cc66bc5bcd00a139be480fd655f5b3f095/ast_serialize-0.13.1-cp315-abi3.abi3t-musllinux_1_2_x86_64.whl", hash = "sha256:0d764913374ba21df8cd618045b371ed104ce55c6276de7c59e554a4f5ba26fd", size = 1501542, upload-time = "2026-10-17T00:03:43.08Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/4c/821211ac83870c27e4412381b7c7838336456b40575bf840e4680b2018d9/ast_serialize-0.13.1-cp315-abi3.abi3t-win32.whl", hash = "sha256:2bad302ae2427454ef9dab71bd5c3ad67188860c39d36daee38f72031edb39c4", size = 1126118, upload-time = "2026-10-17T00:03:44.616Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/86/2b2d5e6c4fa352874a80618e742ffd32d249dee6c119f4233b51afc6b8f4/ast_serialize-0.13.1-cp315-abi3.abi3t-win_amd64.whl", hash = "sha256:032b09d86c785c3984e975378b7861f2a68ac44e22ba7b07b93a536b2524eeb8", size = 1163766, upload-time = "2026-10-17T00:03:45.986Z" },
+ { url = "https://files.pythonhosted.org/packages/77/a3/d610cfcdabaa44fd62a54ca1fc57a7dba120643798b089a14914973562de/ast_serialize-0.13.1-cp315-abi3.abi3t-win_arm64.whl", hash = "sha256:929eda0c419e622ab49df3a4c45f81c57dcc10874bbad55d0470a4a03324dac3", size = 1134043, upload-time = "2026-10-17T00:03:48.453Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/89/0e30f9eddf7d44ccf9d25979030754fd0457af61fa4279acb8c293a04f97/ast_serialize-0.13.1-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:bf53a3ba1cbe45399fc1e4ccd684b8f2a11cca450f89706b79122f478534c6f5", size = 897546, upload-time = "2026-10-17T00:03:50.093Z" },
+ { url = "https://files.pythonhosted.org/packages/81/92/d705be88ff452b9ba0088b554a0cbd308fd01879da092b33fd2ad7d1aa60/ast_serialize-0.13.1-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:4bcf1d5f4eefecaa325a34ecf079b2741189e7d0821523f92cb343a389a67489", size = 1242416, upload-time = "2026-10-17T00:03:51.521Z" },
+ { url = "https://files.pythonhosted.org/packages/60/c2/98370845a7f35cb8a25782e78284b48e01bf3128339881d48d613f304f8f/ast_serialize-0.13.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1b158f1c9e3a0d386913924c57995d6df78224868a4d0f215bdd41587c97b34a", size = 1229729, upload-time = "2026-10-17T00:03:52.825Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/8b/2d7ffed76addd9d3fcf37fb171400942735c8912264c763561a85407f6a8/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77a6201b69ce42e919f2fd9a559138d5410b5d8afff7c3a6b1f500b0481ff7d8", size = 1292255, upload-time = "2026-10-17T00:03:54.148Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/2a/bb9e0e719d89cc950fadfcfcc9cb4bc7640419c4228c852f031ab8abf440/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6eb0be64c9b8f1ed26637eef345abe75ea90351c92c85a8aec7838b3af971011", size = 1298501, upload-time = "2026-10-17T00:03:55.559Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/32/13ce07dce48226f5bd9cfd8f708ec101a8be05aa430cda4918ce31ce8c82/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7f2427bddfca208e84de700044fadf9aef2769a70bae740a1bdabb89ee9e1117", size = 1553758, upload-time = "2026-10-17T00:03:56.952Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/fb/9b51d186db0c20ca28cbaaefd50f03b573c89327b9a2627820cbce22196b/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:88df960e67184a71874e5b2ee014f87d1254a0dbc1f903c7bd7b1ab7733ecaef", size = 1318258, upload-time = "2026-10-17T00:03:58.418Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/fa/da311559dd3dae9af5825cf21660775a929b970f72e3786af3f2cd667944/ast_serialize-0.13.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9f99786e0d62082a5b121b3437f3167562133711df22128de1e9886f57f0b226", size = 1313260, upload-time = "2026-10-17T00:04:00.194Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/11/97a4b9e5b4776d72f47fcd9a09a497bbf68ec8c1ae074307a2c51e410d85/ast_serialize-0.13.1-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:a01b515b25a03184eff4acef25b981ae007916379682e908fc8785b342a0bf3e", size = 1309992, upload-time = "2026-10-17T00:04:01.575Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/c4/62f85121a80cd451ef04f550cf0f7ea0ad9ff46cc4e9a5b9ae439b6527b1/ast_serialize-0.13.1-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:29422d7bcd9e7f39411571c36cec45ce8a7a9b3ad4b2e7aa64497c15a9e397f6", size = 1362715, upload-time = "2026-10-17T00:04:02.881Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/fa/c5f4a545f72a7681e3295bab6e3600ee0862ec4391bed961fe9f735b67e0/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:6ffb0f548a7dc126067fdfb7b2325ab5dfcdeecc29faf5e0f0461c96d6ebfa3d", size = 1469931, upload-time = "2026-10-17T00:04:04.181Z" },
+ { url = "https://files.pythonhosted.org/packages/38/a9/65aed4f8fa7a001a7cfdc03e68e844517ab041ba6fb614b7d78a1d6ac056/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:2876444a5f87a7c76f371af3c07c2fda563a08bc8af52568df812e8aba4bf568", size = 1570945, upload-time = "2026-10-17T00:04:05.778Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/8c/44b2da991cede79bfbf52089f33a5f0168945466b3d9cd38b5dcf08e7130/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:885a62ec6481ce58ab24c6357360d7a82023b62dd7e5e7ceccf59e9662a3272b", size = 1569978, upload-time = "2026-10-17T00:04:07.347Z" },
+ { url = "https://files.pythonhosted.org/packages/46/e4/109b2a5beca700237767d5add920c2f8e1ba9bd5799b0faf143d0e9cf4ef/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:2015fd0165e463aac3b1aadc6a1fa1e5820c14ca950821f4fe02f8a3f6a388d6", size = 1676082, upload-time = "2026-10-17T00:04:08.7Z" },
+ { url = "https://files.pythonhosted.org/packages/73/e0/7df70f6dbb11b5762da5416318f08316a1c51348e0d69ec79ca5f98f836a/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:eb41e1906502ac80d24ff343f45008552a3f5b5158607959fe6a07dbfc6eb037", size = 1483207, upload-time = "2026-10-17T00:04:10.162Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/bc/523aa1c08bd69b5654621beb3dc1a9035715a4a63762fb8d219d6fb3eb81/ast_serialize-0.13.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:ca9acc67f796aa2554fa3037dccc022a4b08b38b2ad153edfac9c9bb7cb1f3c6", size = 1512140, upload-time = "2026-10-17T00:04:11.589Z" },
+ { url = "https://files.pythonhosted.org/packages/66/83/6607060659998f00afdea0d8375c1e6e2b7d515ab905090d9a5a02f48851/ast_serialize-0.13.1-cp39-abi3-win32.whl", hash = "sha256:4f98da562e5901d69c7dd30885594c5438f859edf354e9ac7517d188fb026c3a", size = 1131889, upload-time = "2026-10-17T00:04:12.953Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/6f/bafd6490fe7b355627fabe95f4417e3d02da93102c8552550eb9a0774e1e/ast_serialize-0.13.1-cp39-abi3-win_amd64.whl", hash = "sha256:ddf7cbc9c8abf70102c4a088efe154bb86bfb825b0a10e556d2b15972cf99bd7", size = 1171153, upload-time = "2026-10-17T00:04:14.362Z" },
+ { url = "https://files.pythonhosted.org/packages/11/25/4235f4b375a67a37ac595bf478a5407a744eaf7b64113c1c3a5432b8d954/ast_serialize-0.13.1-cp39-abi3-win_arm64.whl", hash = "sha256:48f11d96c9ec71119612f8e79a8c55c12f2b136716d68347433c561d27a25f43", size = 1139035, upload-time = "2026-10-17T00:04:15.767Z" },
+]
[[package]]
name = "batchvault"
-version = "0.1.0"
+version = "0.1.1"
source = { editable = "." }
dependencies = [
{ name = "pathspec" },
@@ -19,6 +87,7 @@ crypto = [
[package.dev-dependencies]
dev = [
{ name = "cryptography" },
+ { name = "mypy" },
{ name = "pre-commit" },
{ name = "pytest" },
{ name = "pytest-xdist" },
@@ -36,6 +105,7 @@ provides-extras = ["crypto"]
[package.metadata.requires-dev]
dev = [
{ name = "cryptography", specifier = ">=42" },
+ { name = "mypy", specifier = ">=1.10" },
{ name = "pre-commit", specifier = ">=3" },
{ name = "pytest", specifier = ">=8" },
{ name = "pytest-xdist", specifier = ">=3" },
@@ -272,6 +342,122 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/56/43/4ca9e49d27a1fcf6bece6f6aec0ea46bb9112489b93d4b688fb415457bdb/iniconfig-2.3.1-py3-none-any.whl", hash = "sha256:9121e2c1fdb355232495be3194c8dfe87ccc2d5dee45947b78e68f499790d7a7", size = 7552, upload-time = "2026-10-06T22:48:36.959Z" },
]
+[[package]]
+name = "librt"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/04/f5/9dc696772d241814bacac7880bac32f2930b5a6ebc1f85317b83161a011c/librt-0.16.0.tar.gz", hash = "sha256:ac38d6d8d66bf3d744148dbbc0b8e193e195a51e364ed55e224631f5721891fc", size = 219839, upload-time = "2026-09-29T00:55:32.891Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ba/0e/b04fd2d76b6a78a02286938251b2c119fb36ddece92c76cb4886ef7d44b8/librt-0.16.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:fe4372c52d4849096c6cc1cda2817d293ec51440c890474ed59ef38d46556f18", size = 148058, upload-time = "2026-09-29T00:44:40.903Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/fc/e1ed24aadb32d38603c0c07f579d739eb4d90e64e90d525b024e4a999b4d/librt-0.16.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c72c5295a84bd249526da9bdca38f2e176d15c31c13bb0063c5053f4ca023421", size = 153073, upload-time = "2026-09-29T00:44:42.165Z" },
+ { url = "https://files.pythonhosted.org/packages/be/38/133ab655409420e0c34a9a9836a7f4e5ae75fba758d7fe177099130d58ba/librt-0.16.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:be56ba9c884143495b517f23fe794ae367d58cd89ea0fdd6d437e3c024a87f9f", size = 493052, upload-time = "2026-09-29T00:44:44.043Z" },
+ { url = "https://files.pythonhosted.org/packages/94/42/aaa663bc9421f5bf7bbbe9d3316d7219749698ec7688ac3b519770a89c04/librt-0.16.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:ef46c1a29ffb8c72e882e22618ec618778eacd0578fb22c6e7cf9c11d15f357b", size = 485506, upload-time = "2026-09-29T00:44:45.479Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/59/83a6eb5087ba8b45b083859e2085256963e06d4c2531944cdd962670bfb5/librt-0.16.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d3c94211ee0c4f8d649ec06b7c115c0ec4eadb873a0e3154ca15cef3f814b071", size = 498708, upload-time = "2026-09-29T00:44:46.916Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/e4/15a8d62b2bc12935080ec7277e2e0b9b233047781eb784a51d3e25af669a/librt-0.16.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:94aed6a8308818b91677957d1bd03188869cd7aeb23c5dba7912a6c0402f7602", size = 515981, upload-time = "2026-09-29T00:44:48.316Z" },
+ { url = "https://files.pythonhosted.org/packages/46/4f/287cc282ac5dd815204c0779e79401bb0a568c3a1335fb319bf0838fb566/librt-0.16.0-cp311-cp311-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:349c0bcb87ebd07481b6ff781e25cdc699723dbe2212e57dabb27f7a13b7b87d", size = 508670, upload-time = "2026-09-29T00:44:49.726Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/5b/ab7292add898ade0d2b5b75d32a923aae315861671aa5c39f5491309f7ae/librt-0.16.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:001bfd59a7d45b17e3e75f2a8c6405280b35e7b84471792778e718c4f368950e", size = 530393, upload-time = "2026-09-29T00:44:51.356Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/9e/8e6bcf027d4bcbc16a4e864b3d158a1d3e5f55ffa53af0238020cad89271/librt-0.16.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:28e038895b998d7a0c7798922ce8a1dc157675df5cf1c9ef0aca809ed804b7a1", size = 534302, upload-time = "2026-09-29T00:44:52.893Z" },
+ { url = "https://files.pythonhosted.org/packages/96/a4/8c5ab4f3b3a8e905ff8305d76976a2e666f52fe049edd398aed2402beb36/librt-0.16.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:0dbe4096a7ecc00fa835d24510ad8545a4efef738dac96e0e63516783ccde905", size = 536150, upload-time = "2026-09-29T00:44:54.421Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/5e/7c4bc267e3cd4b8bd288de9f62d61fff203829c9cc649ea2aa12e9218cf0/librt-0.16.0-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5cd5b092441053364af968ea12084692cb9d4a22f3ce9524e377880bf028761e", size = 514316, upload-time = "2026-09-29T00:44:55.998Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/75/c8bc6b50f43c73819334233f5a3259c81216742cef56d2140c3dd377c3f0/librt-0.16.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3ddeb3c9dedb461bb457c6c7d9aa7fbf35329da313d1a7543d00c8d0f3473c96", size = 557731, upload-time = "2026-09-29T00:44:57.581Z" },
+ { url = "https://files.pythonhosted.org/packages/93/fb/e241f91bc8de6c7ac1c6a20ea6dd381bc52312b44d711ffcd97140db1e57/librt-0.16.0-cp311-cp311-win32.whl", hash = "sha256:e05108e0849966f53a8d2d3112a7af881d0efaa479bc735bba91108f9f2350a7", size = 104922, upload-time = "2026-09-29T00:44:59.078Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/38/c892383d2bb1a6a80b2ec8ede84970a265a792c56614a06282bf3008517c/librt-0.16.0-cp311-cp311-win_amd64.whl", hash = "sha256:5f49cff01bd608ef7d97104cb035c75455e79c2d70bf4a506cf773338ac1860d", size = 125093, upload-time = "2026-09-29T00:49:01.621Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/56/8bbde1ed0b21480163617648f638fdaec9374cf5e9c846ebcee34ec02a42/librt-0.16.0-cp311-cp311-win_arm64.whl", hash = "sha256:1d28ae980ae2218f9c5b95d191e947296f918c9bf0b400d467a9430275bbe678", size = 115910, upload-time = "2026-09-29T00:52:40.131Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/76/bbdaeb87b7c47b5c7343e90222b9bfa4e4a8a83f647e02933ad0129225b1/librt-0.16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fe52bf4641069e7978a14253b036cb9002def1926317e710f2e249f8a8c47742", size = 151028, upload-time = "2026-09-29T00:52:41.508Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/0c/ab8ed3dab0085931aec4a792c7eaac8dc6c5ff4691fda3a5360d9d8a9cd2/librt-0.16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5bcc2c4726ced915b00de0c9856a4eeabfb3fddb93e10e0b8f735b7709358b6d", size = 155283, upload-time = "2026-09-29T00:52:42.873Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/36/494e79d460c80c1f030661e8287c9eca5e1ad652dc2b2180b6cd42abce0a/librt-0.16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ff7baa55f8e7c69851419e50a666015d02a74198716fd45c0125a2112e0a389f", size = 503185, upload-time = "2026-09-29T00:52:44.638Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/34/a8464038dd9db6e4381fa2b6eb73dc9a50888d77102c4c139304ac35cddc/librt-0.16.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:b95d5d92ab83d39e760a52091bb1baba664f3a2351e39b1e16801e5747c2f0e9", size = 496738, upload-time = "2026-09-29T00:52:46.441Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/53/e0e5e334ef0c6ed27039d323819368b9ef6712be87d55ee2bf9799398afd/librt-0.16.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b6d085d70bce51d43c5c7c36d63490770180d8779e71c49305c87b4213918de7", size = 513818, upload-time = "2026-09-29T00:52:48.068Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/f7/7ce72cbf19d0addd05090b339152fd0548a02562c2866a603e6e3b3da2df/librt-0.16.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:36e53948e99bbe3ffea257124cfcae1cfb01831555c9a9c903c9f9a72db7fd07", size = 531952, upload-time = "2026-09-29T00:52:49.816Z" },
+ { url = "https://files.pythonhosted.org/packages/83/22/0b1bcb6a8e723c8b4fd60dfc8ae8ec6461c54073fbc8685efeb8d900d407/librt-0.16.0-cp312-cp312-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:54d11f726aae9df5a6ffbbf0a03a52449bbac84a53ef03669cb41cdfd4ae41bf", size = 524465, upload-time = "2026-09-29T00:52:51.354Z" },
+ { url = "https://files.pythonhosted.org/packages/64/2e/e9c23b8b9df1813da1be205deca9606beb7ddd033972246cd426d05374a0/librt-0.16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4323193ac0cd025f85af531df8ba91bf24d1973b401697347a6282e8fd3fcf5e", size = 543191, upload-time = "2026-09-29T00:52:53.284Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/e5/6a8b21b342c03ed7e230fa3afbfd2edc58e6e87ef1f0d11fa2b9a748c252/librt-0.16.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:e42f8e098b9c5396fefa05fb1cc7e33b0e08fc51da106b5de4a45fd22aac6743", size = 546979, upload-time = "2026-09-29T00:52:54.932Z" },
+ { url = "https://files.pythonhosted.org/packages/84/9e/b5129023eced1be01e01c22757f53be551d463b1bb7264f787927404c1d6/librt-0.16.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:39ec1d5a14e37baf1450a6cabf03fe552340808bf1ad9d71824ab90117716459", size = 555317, upload-time = "2026-09-29T00:52:56.869Z" },
+ { url = "https://files.pythonhosted.org/packages/71/89/28bba5938c725fe91f06bf93f7fa6c6b150229df53a87b454b0d5c2a796e/librt-0.16.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d1aabe3925cbb4a08d15b7b20ba4011b53019da0c4173a25155139b7b1baed65", size = 535846, upload-time = "2026-09-29T00:52:58.475Z" },
+ { url = "https://files.pythonhosted.org/packages/22/92/63773026614f888c5d4e370e395ce42ca604b89f70b3acdfedbf94851b80/librt-0.16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:300c3ffdc459f4a779a8411ecb188e3ac0b1ff3a3a7b099642555dedae06c69b", size = 573700, upload-time = "2026-09-29T00:53:00.127Z" },
+ { url = "https://files.pythonhosted.org/packages/66/8f/347d4677eefb57cd9f8e01d95a1dcee9a91b4e44e664173c32ff6f3752e5/librt-0.16.0-cp312-cp312-win32.whl", hash = "sha256:c17194318e4c0c0348b36f36c2ec7534436fe0a4c15582403162a4f08c80797a", size = 106158, upload-time = "2026-09-29T00:53:02.031Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/2c/5193dc81127cd5ddfad031391b046bf32dda219b38463ab872407ca30646/librt-0.16.0-cp312-cp312-win_amd64.whl", hash = "sha256:25a58a19ea8d83b68209f04912df765e9260635ef77646542ed4b4abe6bc7940", size = 126176, upload-time = "2026-09-29T00:53:03.445Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/3d/9668a400c8dd81d162eba38b33fa49fa6205f1a64493570a13fbb815c3ee/librt-0.16.0-cp312-cp312-win_arm64.whl", hash = "sha256:f7be7cf555bc30ec12622e9447299cc4a9b8ff307548b634794353db0c2065dc", size = 116572, upload-time = "2026-09-29T00:53:04.815Z" },
+ { url = "https://files.pythonhosted.org/packages/46/cd/ae5e0e9dba45d1399aa04a5395bcc0bead40d9fa06dc903634a7b4d7473d/librt-0.16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c5e6144e68b577f157519f2ba88ca20e3ed61c29b00e5cdfa76cd2d45acf059a", size = 151036, upload-time = "2026-09-29T00:53:06.284Z" },
+ { url = "https://files.pythonhosted.org/packages/41/5a/48a16e323c5f9447a94cce7b59babf60fa04e62c3365ecf060c77ed8b320/librt-0.16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:33f41443a1f4e1f099331b3d8120e409fbff84b9760bc1cc9ea496f37ddaa5cc", size = 155211, upload-time = "2026-09-29T00:53:07.72Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/29/0f59299eb4251a409b2e690ad4b7d9f8a676db829d7817ec961f32b44f7e/librt-0.16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7e510b7770bee609617a3374a96548eb114cae048023e3f049ee449e7ff2db32", size = 502587, upload-time = "2026-09-29T00:53:09.191Z" },
+ { url = "https://files.pythonhosted.org/packages/de/ba/d6fb4ef8d1537c396079d72289f16be7cd35a366e5065c51253fea2760b6/librt-0.16.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:efc49c462d4516b8a58b00b490078fa64689fd1fe66970cc190131d7afb8027e", size = 496146, upload-time = "2026-09-29T00:53:11.016Z" },
+ { url = "https://files.pythonhosted.org/packages/52/fc/8c50dd4d7cc97c0ee8f252c8a3104980f234391cf1519b554e8b9de08b60/librt-0.16.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:92caf82ebef5e12d21c72242b70d1e92536f1711cf2a727a4c276de4b4469087", size = 513336, upload-time = "2026-09-29T00:53:12.655Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/59/16c409c56f708eda2db9a0553662845d45e3871c77d70a240dae3f3bdc56/librt-0.16.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:17bac7f7a16b328fff77e440287693eb017abde913595b5827ebccbc21ecd8a6", size = 531689, upload-time = "2026-09-29T00:53:14.39Z" },
+ { url = "https://files.pythonhosted.org/packages/88/82/d34772a6c29d1446dcca6e64d74062efd508523ab351aa16625a4689d5cc/librt-0.16.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5b976054553670829985ed767feb78fb6bcede0175327c4844dd5c281c1be659", size = 524537, upload-time = "2026-09-29T00:53:16.064Z" },
+ { url = "https://files.pythonhosted.org/packages/77/8f/24c5631313746131ccee53bc91fdc8374f9cf25e0082a1fee9c93bb98acc/librt-0.16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:0058f9d68721094105917254c72ac0569117bb7b13b9769cf45d26d89f9d21cd", size = 543231, upload-time = "2026-09-29T00:53:17.939Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/cb/5f8e0d41dbd8b499c2265e939c31acc9ba59845565bf99539ad1c06aebcf/librt-0.16.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:30b7beaf3f4487b7d8adef1f158b49067cb4d5a19fa7a3bf31a4e7a820e435c5", size = 546499, upload-time = "2026-09-29T00:53:19.666Z" },
+ { url = "https://files.pythonhosted.org/packages/be/61/063052de441d1385f59cea4223f184bf9e5d125de1ae3239b490aa1e486e/librt-0.16.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:468df902df016a06eb0e40b0747dc8d14e47d7a38b18b63b1fb167d85cb94d63", size = 555261, upload-time = "2026-09-29T00:53:21.292Z" },
+ { url = "https://files.pythonhosted.org/packages/14/11/a2ada0529372268d6401afa9d457a095b68cd7753532b6f7f33049a19b43/librt-0.16.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:aea7b1f2b125dad5de85f049136651bff256c883c65e6b9209b2da0a1ac3cdef", size = 536029, upload-time = "2026-09-29T00:53:23.07Z" },
+ { url = "https://files.pythonhosted.org/packages/23/9d/5bb6d38853382986dca702fc7e06c8256d30f5fa0676d882773b744610dc/librt-0.16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a8afb6557920860b7a3a596eb804cf37e09e7cf8a803db2478c202acc72d8c2e", size = 573748, upload-time = "2026-09-29T00:53:24.696Z" },
+ { url = "https://files.pythonhosted.org/packages/99/f6/0025cde35ff7f607684dc775a2b2d732cce2561cec050a6ee1fb2e1fc6fe/librt-0.16.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:77c7a2b4fe2c1369e0d5aa1cade26740a7b14be32fbc9a5535d617d20065c39d", size = 79861, upload-time = "2026-09-29T00:53:26.089Z" },
+ { url = "https://files.pythonhosted.org/packages/93/93/303b8592909bd583f83f02818ffbea3f7647ca1e22b1cd5465d04b8145fe/librt-0.16.0-cp313-cp313-win32.whl", hash = "sha256:02d89c813d5ff74b17df72d3a34819d132cd168e56b81bf755b809bd9e46b8c4", size = 106236, upload-time = "2026-09-29T00:53:27.43Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/24/80bbb463c60ed18e29cb26aba386ddb76609580e4e7160710e550587018c/librt-0.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:14ed6ebe3e4f85f326d7920011ad30ff49ed9334e62cf88caef9ba973d9e3a92", size = 126180, upload-time = "2026-09-29T00:53:28.7Z" },
+ { url = "https://files.pythonhosted.org/packages/43/80/b1a6fbdd7da825cdd55c71aa81eb6cfa82c513c360774152eacb50b4a771/librt-0.16.0-cp313-cp313-win_arm64.whl", hash = "sha256:83d4041a3d9b2fd053a8a4e1f22878b3e5833e2712956382d5c048d791454e91", size = 116557, upload-time = "2026-09-29T00:53:30.012Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/93/9e0cf7da129a93c3dc7f45bc3cd4a660f2aaa995aa8a6f95c2583ef41239/librt-0.16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:931a0bb0fcac88f263e269e46eb30ba8e21402cd3c62ca40cb97034c0693fab1", size = 149844, upload-time = "2026-09-29T00:53:31.391Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/26/8a90d2a8f2b2e471bb486b7aec117b8ae622715ed6c39853aec48ea20073/librt-0.16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1bc17e54e5305f8d40b7ca203671ff5a9e59c1d0f8ea0f625dcca53a3984de11", size = 154111, upload-time = "2026-09-29T00:53:32.718Z" },
+ { url = "https://files.pythonhosted.org/packages/35/ce/67abb46258da4d3e42ff5b141db6f38c59357bef84c7979f183b22f924f1/librt-0.16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:877698bf6bca5721d8be345f2fe09778e40ecadea8b58c73075f2b1a53666bf2", size = 494234, upload-time = "2026-09-29T00:53:34.466Z" },
+ { url = "https://files.pythonhosted.org/packages/12/f9/ea7162414a16f8f1bbd3b493ad6d22b926c5471916e078350bdbca8c4e5f/librt-0.16.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:5981c011b306781ce561e18e14230a14524a3d8109b97553666c942c18f31a96", size = 491173, upload-time = "2026-09-29T00:53:36.124Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/09/9b3e869060dd33f9989b80ba4fea306f6db8cecefcdfe6d7346ef0603f65/librt-0.16.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:afced3dfc17cd805ecf7a3d77996a71cf5f2c75aa66eb0c21a9930f4fc992f86", size = 505547, upload-time = "2026-09-29T00:53:37.686Z" },
+ { url = "https://files.pythonhosted.org/packages/50/07/79007d2165f649ea93e08c0962d1d62c70af9cde77965255095bf9d96f9a/librt-0.16.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ca8052401c55d7511dda6760719fda7618067e83535d7d0010096d216c34b667", size = 523057, upload-time = "2026-09-29T00:53:39.301Z" },
+ { url = "https://files.pythonhosted.org/packages/61/0c/8fbaff66d0ba376d8864653f5acce1569bae27e89648671f26f7eca67ab8/librt-0.16.0-cp314-cp314-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1e511762a074005bb0aa569166779834e75e438370226930d0ce1866d4b6a33b", size = 515158, upload-time = "2026-09-29T00:53:41.012Z" },
+ { url = "https://files.pythonhosted.org/packages/df/2e/23ff0dece76f07a4124413a57682efa0bbeb5765ac0122bc0955513f82ca/librt-0.16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f1e8591bd8a5a628cd7f07954c6a1592359a878bf032957a8e9057a41d644311", size = 534183, upload-time = "2026-09-29T00:53:42.451Z" },
+ { url = "https://files.pythonhosted.org/packages/26/c4/e11dea21d9a29486eba78887380374189d472734fa32cc50cb37ca44d3d0/librt-0.16.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4aaefb4ba6c07e1aeebb2795c8958148f1d6f9af3b555b53d23d766edb6d67a", size = 540663, upload-time = "2026-09-29T00:53:44.272Z" },
+ { url = "https://files.pythonhosted.org/packages/44/75/e873ae158a8b7f5359be33e7fd6c1fbe02d9a78a3e89a77de6b0e837f476/librt-0.16.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:d92db7a0f6aee44f1baee94750457e8d2d1c6ccea41842de6268d34e8dc7eddd", size = 545908, upload-time = "2026-09-29T00:53:45.812Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/36/8939d3f6a93e11bd9592e6fe28d2b44f1c2dc4bed6e22e72359a91e18ffb/librt-0.16.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:378dfaffb38e59c24a87cde5713cd865d51ff7383fa12947f3907f306ea1ca55", size = 523302, upload-time = "2026-09-29T00:53:47.596Z" },
+ { url = "https://files.pythonhosted.org/packages/71/14/35309f44a077f0f42ade0e2e7cd88c0cea760c661af205c01ea90d3c0e1f/librt-0.16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:3e0c39bdc85370422e8b637be76eb1fd07d30967551b03e62267dd156f553152", size = 565911, upload-time = "2026-09-29T00:53:49.278Z" },
+ { url = "https://files.pythonhosted.org/packages/78/0c/df6255b94967f3159ebc46f08d8e783c12da6ee269ddb74b2efed63c640c/librt-0.16.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:1b384b90ab79a7bc30b566895809a636e0666f21f3cf12b54823d025b7e83839", size = 79119, upload-time = "2026-09-29T00:53:50.938Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/44/d30d5a5461378c9d33f36736c6791c3b4c4ba4b1ffe0da7350aedcb2c9a0/librt-0.16.0-cp314-cp314-win32.whl", hash = "sha256:52327da75a94012e7f932f913d20d3876bed3c102be00e6c3e8600ff7bdd58a7", size = 100140, upload-time = "2026-09-29T00:53:52.205Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/98/769712f356a1e897df3581bb0c3100375d054a000de26099360ea65b5111/librt-0.16.0-cp314-cp314-win_amd64.whl", hash = "sha256:3f0b8114c44b2ac06ff5dacd08e07e8e807ff4f46083f2a1602685122559be41", size = 120464, upload-time = "2026-09-29T00:53:53.495Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/d3/ae2abccc8bdc8b063c1613a77686e17b74e9d3d60cfe6f12c63fa2821b9f/librt-0.16.0-cp314-cp314-win_arm64.whl", hash = "sha256:8caf96a4ef8fb27d0ac0d1ad8337d26a240acd4a02fe4345d0a8f264753e8f99", size = 110889, upload-time = "2026-09-29T00:53:54.817Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/26/0737d4be058dd6376eade7dd8b380d4b869b6394cee929393a5a431c45bb/librt-0.16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:953107e2f68d0f3512c48f898b0dbf0ce5cc52bba0f318d847c985dc555ee4cc", size = 159838, upload-time = "2026-09-29T00:53:56.22Z" },
+ { url = "https://files.pythonhosted.org/packages/01/96/9bc96531d7c620e9949af904470f02e3fa8f35129ab8e8f281c51eaa3788/librt-0.16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ad37d5b9abd49c9a655dcda7ea52a8a752884062ef1ee71ae17c2f2a0f81fe6a", size = 162018, upload-time = "2026-09-29T00:53:57.532Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/fa/b0289dcb186eb3f97221ba00da5f4bd3ba7fa5e99752d48aa8d336615334/librt-0.16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c4aa329c17bd1aaea4f6e89335d8ccd494b3a5830b6654462273e50e11023f0", size = 703983, upload-time = "2026-09-29T00:53:59.024Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/ab/05ebbbde7530fc5eeb58fbd1581522a9f64f132fa703c4c595eed6e14760/librt-0.16.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:0ead24d2562a49473dddd9efef8581f020007eb0054389c3ee3ffad38b1ca4c9", size = 683847, upload-time = "2026-09-29T00:54:00.671Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/75/f52aeecd4dbadbddf80725ba7de126d8bd5d0eb66247eae17a81ed90dd4d/librt-0.16.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:02118f56a9c36ddd07dfd9b919d9ecc117ba20a90987d56aa4c429fa34509188", size = 698045, upload-time = "2026-09-29T00:54:02.247Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/55/fa277a835cd6eb42380591ceb85f48c5b4d2b2d7e2cb9869e1e17d24d237/librt-0.16.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4e29522c62e28595ff7e324c6834ade51127707f0e255b18d1c1cf03d39c1048", size = 724640, upload-time = "2026-09-29T00:54:04.078Z" },
+ { url = "https://files.pythonhosted.org/packages/25/e4/2cf64354f3fde8ebd591b3b48f96510bee24ac5f7d1e567adbbe210abdbd/librt-0.16.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3ff4b2367926b69c6215635902cccb04048e73094e9862900d27cb2c6bbff143", size = 731748, upload-time = "2026-09-29T00:54:05.741Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/c9/c180af3e94e01aa529fa93d7733fec2abc47f222345400cf21d5481d5f8a/librt-0.16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c6f1b27bf1632a7e016af9f145f82be95e1edd7721a646505c21059257cb5a04", size = 754960, upload-time = "2026-09-29T00:54:07.38Z" },
+ { url = "https://files.pythonhosted.org/packages/95/d6/01073aa78c58f356b10d9c57b3fe9abb143df338142bcd316df417b89db0/librt-0.16.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:5696d7f52e7b37217cb3a8f92c744fe835942602fdd4c1a8bc4741d3bfdce15e", size = 747000, upload-time = "2026-09-29T00:54:09.06Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/d8/1de3783908658d697a8cfc00582f61299ffba7796d7260c112e4700b1109/librt-0.16.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:6072e92dd876ff6ceeb6cf371e35e51f479349837391341f479b08df4564242b", size = 749569, upload-time = "2026-09-29T00:54:10.702Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/32/e817f66c96d6caa8bb8435ff93c4c220624d59efc506be59ab98fcd01d0c/librt-0.16.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:39ca4f2f2fe05de8e63493da592d84311adabe5bef52b193851981da9816b302", size = 729483, upload-time = "2026-09-29T00:54:12.25Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/c9/23992ccd2b9d22798fdd0f61353183a47414e651da782ad83eb680f50833/librt-0.16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f9807485a908f00355820f18e91e045ffdcdc5adb68aaec40a1e2b88c5f7bba1", size = 776584, upload-time = "2026-09-29T00:54:13.837Z" },
+ { url = "https://files.pythonhosted.org/packages/67/3b/e8af957f08e6e2e8e566b099e43d2748418aa565df6ee1d9d3331209fdfe/librt-0.16.0-cp314-cp314t-win32.whl", hash = "sha256:94be5cb7bca4df6201f4183e9e4fa2086c655283d20b38cd84500a69057575a7", size = 104402, upload-time = "2026-09-29T00:54:15.568Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/1c/946e6443d7cd32347a086043395e421e52fd603c9163d2ec970ceab8eed6/librt-0.16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d46ca272b251d033dd4527b0dec5f261a28a52bd5fa0f99c117b0a1f8588cc2d", size = 126227, upload-time = "2026-09-29T00:54:17.165Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/a3/bc4f9959d3c62bcbf9e5fd3470a8bbd8bf33224ed4c25d7fba173201b8ac/librt-0.16.0-cp314-cp314t-win_arm64.whl", hash = "sha256:b9d6d4b14e92d876f8026b54c20c445f36425214c1081dc76f74e40db386b82b", size = 116493, upload-time = "2026-09-29T00:54:18.567Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/4b/10fdb42dfab4c1533e1570e686b18e86ff4328b406361b39fb3016667638/librt-0.16.0-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:6fe436af2eaf630474f491af5d032cbe45f93fcff5c3b9fe4ab194a7255b20ff", size = 149829, upload-time = "2026-09-29T00:54:19.933Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/30/a90ca13f1d3d91af1680000a4038536907018fbd51764767287a05d28b8b/librt-0.16.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:8ff5d26c529336be9bd7ae04483235d77778ee7d6444a95353102b542601ce81", size = 154522, upload-time = "2026-09-29T00:54:21.306Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/dd/bcf364eacfa070bb1fc88d503111ae7177914197ba1fa717c748926e7930/librt-0.16.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:909d8e3c1faee44cb762b1c519ff8613dcc5ceae5c99987a00917b5a31fd1d6a", size = 497593, upload-time = "2026-09-29T00:54:22.798Z" },
+ { url = "https://files.pythonhosted.org/packages/18/c1/2c4e81e347bdabfe8346bfc6dc37ae5e154d61c88e30848ec0606025a5e0/librt-0.16.0-cp315-cp315-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:6d4a64283ee61824b5790de882bc68e2d9d7a5143537cb7a966f7354f71646d4", size = 480459, upload-time = "2026-09-29T00:54:24.364Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/d7/fef2a3cb8400701be496f6e459876f650b3451f407e30cb243de571ae615/librt-0.16.0-cp315-cp315-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5810ba811297fdf37a1531a57667cb8ace0842013ca8606bf9eb7c24cf4be154", size = 507814, upload-time = "2026-09-29T00:54:25.997Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/6f/53762927a32e9dc9eb1d1c1f3528da281290c86d96929a8af652f671266e/librt-0.16.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e56aaf8c167548dc8e5d6f3bd0f48dcdd299a23c73be3f744aab79d99e9c7f5d", size = 525114, upload-time = "2026-09-29T00:54:27.747Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/67/0b9d031f303c4e8c691a9a8ef9d272f13b530c11cc563b819df621b6a348/librt-0.16.0-cp315-cp315-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8f36c58e33b304b525c6c9c5076399c6ebf1109e17b9051a05a407b091b9215b", size = 520465, upload-time = "2026-09-29T00:54:29.331Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/d5/2056a3a85864e882eb17a203a10ddb26fa748bc9718ec67e79059ab46cae/librt-0.16.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:242e00b3d4fa37c3d3c1ca5f5c9adb7d909ddb1eac9c41f2787320d00caa0af2", size = 537257, upload-time = "2026-09-29T00:54:30.915Z" },
+ { url = "https://files.pythonhosted.org/packages/54/57/e0d79790c163cbc0909e209a6f62e30bb713b647f905a176cdc64848fd4d/librt-0.16.0-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:0253721561787b8df8443eb347b7a6461015354e5bdd37ee38a41fef220d2bb0", size = 527442, upload-time = "2026-09-29T00:54:32.478Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/50/1c0c95aba7af51f4752ea34fbf2eb79b36e7cae3a72536248e8c735de735/librt-0.16.0-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:3e483a8d69ede8067db70c0e83007423b6925de6fd53afed01d66160f2e9398c", size = 548139, upload-time = "2026-09-29T00:54:34.183Z" },
+ { url = "https://files.pythonhosted.org/packages/98/91/a8a43dd5138d4f55f88846b8f0c85454a4fad69952cebfcff9948f831290/librt-0.16.0-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:69ba927445cfaaffb4081003ef5224c55a5c2ab67ef956f416ef744916e44121", size = 529708, upload-time = "2026-09-29T00:54:35.761Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/41/d5226881ab2b7c20d9d587b37bdd4a0ec8775a96d00ca87ac9f385587db4/librt-0.16.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:d6a365f2ab45a984d0e00eee0dd17f599ceab8cadab6ea07b6111c8132fc0e42", size = 567781, upload-time = "2026-09-29T00:54:37.409Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/bf/2345ba57a626e8c78c4ddcc724636a8df5a593fe46c1ee76bbf477e32b0d/librt-0.16.0-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:f01f3805f2dae4781c0c34b440e31740d082950bdaf89a6f601ad589a28af57a", size = 79177, upload-time = "2026-09-29T00:54:38.792Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/40/99e77936cc9207f629b077bf7cbc5e2f0827cddd7c29792f04ef881bd2c3/librt-0.16.0-cp315-cp315-win32.whl", hash = "sha256:b0e3e721c75d2e79a76d4422c79d7ba705fe1bbafec907037fe7a657a480a0e3", size = 100127, upload-time = "2026-09-29T00:54:40.251Z" },
+ { url = "https://files.pythonhosted.org/packages/56/1e/801fe26bc622061b9dfd010e166d94142cb774b733217e6d98d1c0cf2638/librt-0.16.0-cp315-cp315-win_amd64.whl", hash = "sha256:bc02954b1295de798bbdb0b4e2d8a28c2117de8b5c73dcbeb27dc32572dfb971", size = 120480, upload-time = "2026-09-29T00:54:41.722Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/a9/d533983055bd36e112627384c2c038845d1df882540b8bcefb566475640b/librt-0.16.0-cp315-cp315-win_arm64.whl", hash = "sha256:c5db585d43449a5f54303d4b2774e45e1babd975cfe1630a3d708c0b80c3e560", size = 110915, upload-time = "2026-09-29T00:54:43.052Z" },
+ { url = "https://files.pythonhosted.org/packages/55/fe/d62238fa9c653b0e0613290349467cb65711b5800e8e474f45e942a0ad95/librt-0.16.0-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:f06c689cb14afd9b612727553a5ec5a40febf113ca41c4413a2b0b334285884b", size = 159714, upload-time = "2026-09-29T00:54:44.497Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/ac/f31efe7818700be72ba4f9af8a80fa67c39808c8d26dacc55dc1f6f35172/librt-0.16.0-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:13b4e8aba90b0b1c82474e9844aa9ffe7ad3faa484350e1da64cb8188d903134", size = 162086, upload-time = "2026-09-29T00:54:45.968Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/16/4d7487bf86a9d7e8e18f37ba5538789233ea693637379b75701bd35ec9de/librt-0.16.0-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a269c46ae327d8e6f8c1f85f7516cb52c0fa48127565a1105a4f4a05ff2a0b4", size = 712727, upload-time = "2026-09-29T00:54:47.486Z" },
+ { url = "https://files.pythonhosted.org/packages/77/74/50cd550ccc1a517b9ed62347625c01ddf8c4afad66490312677c011458f6/librt-0.16.0-cp315-cp315t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:a33e0dae1f8592146a4764d54ce842b278732d21a84e17c3bbe6b1bc158a2248", size = 681106, upload-time = "2026-09-29T00:54:49.126Z" },
+ { url = "https://files.pythonhosted.org/packages/df/5d/7293f712975ee6fdd2251411fd9ef1c62bc99c7b83ecbe62dbc999b1fab7/librt-0.16.0-cp315-cp315t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:47ada6ea32636492c61aa8ad27ae3b9404bfe7a97e3ba946d1984236cc741da0", size = 705921, upload-time = "2026-09-29T00:54:50.905Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f7/8aab946f11d59d1bece9ffc65d994b200c789a8ca5a95c17e17e609f912a/librt-0.16.0-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c43bd6e642d8a248c114327f98dd25ac5a7cb5aa168ef02f0559b91874df16b8", size = 731786, upload-time = "2026-09-29T00:54:52.584Z" },
+ { url = "https://files.pythonhosted.org/packages/43/76/1c42ab31e7cb8384ebf6d3af607213c495222474f4940443ae7639ab7685/librt-0.16.0-cp315-cp315t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c3d1bb7841a816ace6449bb26d3f9560dbfa20e71c568d23f0f62bf1e68f50b1", size = 744964, upload-time = "2026-09-29T00:54:54.217Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/2e/4b19982d933d2dfced671e840b219e4c1cd3f507df6e6dabb51bbd4e3850/librt-0.16.0-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:3931f7a3db322e7f44e02a280e3949326ce9579ad388ee8d691dc7c76da9fb70", size = 765305, upload-time = "2026-09-29T00:54:55.815Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/1b/e872583de2dcb3ac7746e7a2321aeeb168274f3952a87dc66e05ddb29faf/librt-0.16.0-cp315-cp315t-musllinux_1_2_i686.whl", hash = "sha256:f4462528b6000afe8f16907b5c7c2553abf1df005ba5140e6eb394541c3624c3", size = 744959, upload-time = "2026-09-29T00:54:57.532Z" },
+ { url = "https://files.pythonhosted.org/packages/10/de/a18c6bcfb297af2674233e90b3c3661c3a0af0f1434a0c966d2ef708a835/librt-0.16.0-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:80039ba9b6a7d5f1a0175a4cca6bbefead87bd854c80abad1cb30afe47a830db", size = 758138, upload-time = "2026-09-29T00:54:59.484Z" },
+ { url = "https://files.pythonhosted.org/packages/22/1c/0df1d732539c297bb1a093e1fe3204d2faf76a9022cf4e1d1c2fce059790/librt-0.16.0-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:7a1d272724b581bb6bc769dfdafed6da2ecc9886ba2450311de55a4ac2e1e9cd", size = 744268, upload-time = "2026-09-29T00:55:01.179Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/f7/ccaf31331f20c91a5bd9bd48ffc3f743f9c81bb5720cc7b2a720ca204f0a/librt-0.16.0-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:fbe4fb8c5445f7496d7f7f6bb0807875d09d47e6771ffa175fb2df2895fb86ba", size = 785783, upload-time = "2026-09-29T00:55:02.938Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/ea/7421d9e6db894cd6cd788b604b3394a163ded6b942052b6f96c23ccf08e1/librt-0.16.0-cp315-cp315t-win32.whl", hash = "sha256:375bfe6b572a8f6cfc398709356046173bf27e64c4c5edaf5f7062f051fb4bf9", size = 104355, upload-time = "2026-09-29T00:55:04.533Z" },
+ { url = "https://files.pythonhosted.org/packages/85/6d/7c31a506eb847bc58aeb2402d58e17605e821f68ab5df5e66fe274a6df41/librt-0.16.0-cp315-cp315t-win_amd64.whl", hash = "sha256:bd3150023d3dc2bc70f3784e59ffa1140d56ddba3d8125b3d6f9f85221279bfc", size = 126134, upload-time = "2026-09-29T00:55:06.017Z" },
+ { url = "https://files.pythonhosted.org/packages/36/69/7a5d10ac409c4da0355e054a14371871da9b5557fcc42772cd00181c6cce/librt-0.16.0-cp315-cp315t-win_arm64.whl", hash = "sha256:8ceafb70f2a4f0826f11031942e59c0728fd98da112dc346d4352bde1e486866", size = 116391, upload-time = "2026-09-29T00:55:07.484Z" },
+]
+
[[package]]
name = "markdown-it-py"
version = "4.2.0"
@@ -293,6 +479,75 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
]
+[[package]]
+name = "mypy"
+version = "2.4.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ast-serialize" },
+ { name = "librt", marker = "platform_python_implementation != 'PyPy'" },
+ { name = "mypy-extensions" },
+ { name = "pathspec" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4a/3c/78ecece71ee4b5445389a4ccd11627cfb9c69e4de554d11060171f376305/mypy-2.4.1.tar.gz", hash = "sha256:63b590965691e540a335fa74dadf31a7069142f6b72e234689bbd244edf55623", size = 4066728, upload-time = "2026-10-28T19:23:13.286Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4b/8b/2bc7f489528db504221ad7eff6b3f83979ee57d066103c72a85d09046a30/mypy-2.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:79a1e79199e980fb41f219cb3d4746d25c64b0b623840d3ed57df1cdc13a6502", size = 14321495, upload-time = "2026-10-28T19:21:16.612Z" },
+ { url = "https://files.pythonhosted.org/packages/18/9a/b2f037b4982dce33645296e4394b031ade60f697f468761fa3ea8e2f8ece/mypy-2.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e18d0da3898b3bcd16192f15a206e7b8b8dd0f4a0b3e51aa5bd5c2afe749e2c", size = 14467468, upload-time = "2026-10-28T19:22:10.629Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/5a/379f9bfa18bf3f8d43668fbd9ae4f2922dd027a559c461ca9d41cde1d32e/mypy-2.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b69a15fc2e75858d6b15e91ed3a0dcff1a533265183b64734138791fe0f49aa5", size = 15421496, upload-time = "2026-10-28T19:22:48.005Z" },
+ { url = "https://files.pythonhosted.org/packages/83/50/3bde18a2824ec296432cebe01752ccd0ab8a2ad784abe4390040d019b3bf/mypy-2.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:70971cc0bcb1d3f03d2eb68fa430a6481ccc03558dc001d9534d7e5fa274c392", size = 15670425, upload-time = "2026-10-28T19:21:57.026Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/36/8b722b88c325d317354de7d10d8fb9d4dc0f958433e873a91fc4ed21a06e/mypy-2.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:6f7225951a277dd0d6add900731d487174b7d24ac133c09d670ed63b4d8cb90d", size = 11300972, upload-time = "2026-10-28T19:21:48.724Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/d6/fb2bab1245658d68af8abfe776362a780c621fe0f364f2c7b53039a945ee/mypy-2.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:be4c03f33c55500191df2ed4a9b5e7edbdd1fff067fe361a5a9b1b5d3262da6e", size = 10873349, upload-time = "2026-10-28T19:21:32.104Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/1a/0be9dc6af5e0e04128f18a5e36b4ca44bfb0085857d37aa6a7de1ac2b0cb/mypy-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d63737426a048a27705a4eedb828669f4bc0873612a1b46b0b9fe892b2ebaaaf", size = 14381321, upload-time = "2026-10-28T19:21:54.223Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/a6/7907e36711a8a8c703f961f076fab61793383694140bd1fd87589f3f68d4/mypy-2.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7344869cad05f073b8cae2826774fa0161e3eb2f5c685b76813574008d77d750", size = 14466373, upload-time = "2026-10-28T19:22:25.211Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/b8/7313212b29de570c25c9859d7c8e4ef2251a6d34338a1e20031d94297721/mypy-2.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a20ff6cd2af7a670c64f089bfa5fbcebe8ff3bf896c5e6ac7c9862c7960e4494", size = 15553666, upload-time = "2026-10-28T19:22:36.511Z" },
+ { url = "https://files.pythonhosted.org/packages/23/8f/92966b6082ff49da9142e1e527eb9af4359a9a61632a1cfc313da96b37fe/mypy-2.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9ed1195502cca32be5ef756b1693477f4991db2f2e34eaa90cc4ad6bea4f9531", size = 15802641, upload-time = "2026-10-28T19:21:13.421Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/b4/874f6fd54b005020ad63e2cf7a59b178d16a771d478849b29b842cb0cf14/mypy-2.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:4c194c5d878130fca1ae662dfb1feb6274afbab0b40032c80147377e0ea803d5", size = 11383067, upload-time = "2026-10-28T19:22:03.516Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/41/5ae8e6528c2faa96e4a921a999d2c7ac7c565bee184b4282bf62ffa76b5d/mypy-2.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:71eda06e5bae51976ce869fda429159c9cec678bd02f25bb197f0265c8f991be", size = 10946882, upload-time = "2026-10-28T19:21:39.51Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/65/cd19bc06b67d2df4828220245fd98569ba40fac00c60ba48bef1817d8ac3/mypy-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3f1613bfe6726169a5680ec3dad5fe1f6d96285c12aded51ad432eadedd39d45", size = 14214179, upload-time = "2026-10-28T19:22:27.49Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/db/4d54ca4f49a0113d0c956980272aefdc6eec5458f7a72b350785b9ef2249/mypy-2.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:680ba5d9f0e3fe0c746b3d14df15f2a718e4bc308a1145616a1ea691ce7c354b", size = 14376217, upload-time = "2026-10-28T19:21:08.019Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/45/089b4724b85f0f408941be5298f817ee89181c1ce031a73d60ebb150c2c7/mypy-2.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fb70621535c22fd4b1a0450d37ed1d6c5eb39656a1cebfb189084c12de54ba7", size = 15467623, upload-time = "2026-10-28T19:21:21.555Z" },
+ { url = "https://files.pythonhosted.org/packages/27/4c/15edbc64b0b1412bfe94c74192b3b56554ab642b85a31bb06cef6313101f/mypy-2.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4af5ad558033bf71a44a1be149ed61633522824f84c1b6bc2445d3fe2d2a5296", size = 15729867, upload-time = "2026-10-28T19:21:37.557Z" },
+ { url = "https://files.pythonhosted.org/packages/76/5b/7d160fb685994b65b0ca86d352f3afb5424f96a9e6914719f7b861fb6701/mypy-2.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:acdd82cb3a2eb10720e87706a128d75785d005a8c47e36a4973221524f41fc56", size = 11390802, upload-time = "2026-10-28T19:22:18.081Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/ea/453d27f8d531f57c5f14aa73094eaa48e3fb77f680bdede2045cb64fa58b/mypy-2.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:133886ef1a7a8cf673df55d865e41580c6c46aa9b8e31e1325dfe315c038f745", size = 10950427, upload-time = "2026-10-28T19:22:59.003Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/75/282d43d6fea7be984ac63e75ffd8eee146ba2f3b077bb1ca5bf3f973e0c4/mypy-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c45485061ba08f700acc12cfc8b177e6997e6bc353605babfc0abb081e079456", size = 14233251, upload-time = "2026-10-28T19:22:44.977Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/a8/9021263cc7e761451c7d8131bdfab3c0a5c2d67aa2d694670019e1ab86a7/mypy-2.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:727890a32f1858c09549eb55a8b234fe990d460450b28f03e8213ef925f1247c", size = 14394953, upload-time = "2026-10-28T19:21:29.492Z" },
+ { url = "https://files.pythonhosted.org/packages/16/10/4192a5cea09ebbc41c534f57bd0d8f0fb42a8e810632cda6e37f072c59fd/mypy-2.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:beb254fec368c4e7f7f82a5f310c6470b893ec88a90b66426da971c6826b0453", size = 15442824, upload-time = "2026-10-28T19:22:42.068Z" },
+ { url = "https://files.pythonhosted.org/packages/99/e6/d3b0a7474aa4f696594a47a218b6796d9c4346fc81ed288904278cddd9bd/mypy-2.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ec416d5d2ae689e681f5727e328723f18d8b5dc8c903a146b7851dc5727aad8a", size = 15717227, upload-time = "2026-10-28T19:22:54.026Z" },
+ { url = "https://files.pythonhosted.org/packages/48/6a/2ec530dacd4f77ca50d0fa05b854098f2d75e5d18c587d2612b44f10d68c/mypy-2.4.1-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:0e95ad0bec03ced9a73b42514208abd24c789e67f0e1eba476e276dd7a2dea3a", size = 7898637, upload-time = "2026-10-28T19:21:41.709Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/56/de9d7bad12e3868d5f1f4279aed20b1b568b2c5e166f2d3a05b211dca0b1/mypy-2.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:51f742d4154824837ca6e92aa81f00eec222700a3c337bd258e5562a3880d75a", size = 11576656, upload-time = "2026-10-28T19:22:13.232Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/12/4fa9883b9b59e5c96ff62845ad019dc47786031c45d67eaccf452cffffd8/mypy-2.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:f44dd4c5ab22df21206dd926c98381ebdbf5bfd696df37dd38ded2d23143fcb6", size = 11174403, upload-time = "2026-10-28T19:23:01.346Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/f2/827e356fc4f74c60b8da42663336e035b49d948a3c11224b0fc7543086b1/mypy-2.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2875280687e64740e6ddcef43bd4c5852ad45ff34549740f6aea3f0373870b93", size = 15713427, upload-time = "2026-10-28T19:21:26.757Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/fa/e58a4fcae1d238189eac758036c0921992a86e3e6209190efd96d61fc5d4/mypy-2.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ea0714d2bccaf6d010c970e1d3ba2c9c2add79d2030aad2ef03167d788948276", size = 15938481, upload-time = "2026-10-28T19:22:34.371Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/54/a20f881801b88900923f2be1541a4b438ad0f0883e03836641fdc3d7adf4/mypy-2.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:63c6ff0e7ea546943001a91639c9f0c64b60d239268fa5d29e4a6c87f82c2373", size = 17243006, upload-time = "2026-10-28T19:22:39.425Z" },
+ { url = "https://files.pythonhosted.org/packages/de/1c/ff84a6133a6e89c390bff6b25d9bad6598e7891b205f4b30b496906d636d/mypy-2.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a5b5da107dc2c93fb0a22f7a45dbaea5dc131afc7056b5468d78e819f9a138e6", size = 17557728, upload-time = "2026-10-28T19:21:43.607Z" },
+ { url = "https://files.pythonhosted.org/packages/15/4a/effa47d39b8fadf679c08e7c78559e523f9769a42a75662d2d15b2b9cdd6/mypy-2.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:c3a4d0c47053cc90035dd3a2c321112b593ad42935d0cbfd747288e1564fa8ca", size = 12370826, upload-time = "2026-10-28T19:22:29.779Z" },
+ { url = "https://files.pythonhosted.org/packages/46/9f/b214ececb1a97fbb61ab3dfd48ae3cffd92ae2c148af4c9dc94c6c56772b/mypy-2.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:0c5bf267645351887b72903fd2f5fbddcf85234e0e6c4ad94e2fcfb4f0c3f971", size = 11851083, upload-time = "2026-10-28T19:23:11.288Z" },
+ { url = "https://files.pythonhosted.org/packages/57/5b/e44a2edb0b25a9f1429de770e1d0238af23302671dd40c77ac5efeb0c559/mypy-2.4.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:db599d03e7d60d15f2e0a11665c18bc52a88699fcab990fdf8cecd5be8ac801c", size = 14243144, upload-time = "2026-10-28T19:21:10.975Z" },
+ { url = "https://files.pythonhosted.org/packages/59/d4/5c99dc3a28e11d2f0fcbb02df63d2d3e574dedc0f9f69d50901df305a7a6/mypy-2.4.1-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c6c7210e40dd423af7d65cbe428b5906d11dc60659337bdae4e5fdf02ce32ed7", size = 14409316, upload-time = "2026-10-28T19:22:15.515Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/c9/3108b38f4c30eabd80194f5d9771a87e27196c5e0ddb5bf4b99c6ed1e97e/mypy-2.4.1-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2d4a51db92059eecc2c33b2a5194b28585ebfd917292c015fe87a77dbff6cd2", size = 15445573, upload-time = "2026-10-28T19:22:05.635Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/12/84a53eceaf11c46dfd7931fb6b8902d01fb33787d3a991e81c461458150b/mypy-2.4.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:01b7718552d017f88d1151f3cc9d6556d778a90bd58060a2c3d87d070f9099fb", size = 15732989, upload-time = "2026-10-28T19:23:06.512Z" },
+ { url = "https://files.pythonhosted.org/packages/65/ba/bf017dc2abdf83b2f74e29c43834b7af67dc1f1b20202f02dae60b4781a0/mypy-2.4.1-cp315-cp315-pyemscripten_2026_5_wasm32.whl", hash = "sha256:3467b1d13000812ab85dc2a3834b9379ec77798ca1bcabe434e23fb21fead692", size = 7898932, upload-time = "2026-10-28T19:21:59.366Z" },
+ { url = "https://files.pythonhosted.org/packages/76/05/0c22b1f0390120a8592d31209cb6b9f932b6a0b426fe643e550de15139e3/mypy-2.4.1-cp315-cp315-win_amd64.whl", hash = "sha256:080cf64a60b90a73435542feecf4171904cc5710658165b40c6869115d4aa100", size = 11577577, upload-time = "2026-10-28T19:22:01.302Z" },
+ { url = "https://files.pythonhosted.org/packages/73/fb/ac01e173629697ddac2f11e51efd10347d673f9a7311b03f4aba710ebccd/mypy-2.4.1-cp315-cp315-win_arm64.whl", hash = "sha256:f9118a3f50de264b614861bb11814833a742138ac3c8e2e49a860b6977fb2704", size = 11173540, upload-time = "2026-10-28T19:21:46.444Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/e4/b9b9c8eb09e70927ee65b1456cfed35710b6e4ff0b07f364ceaf2202307c/mypy-2.4.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:9d7dafd56179d296651df9e1844b91c9517c379398f017a2a435d077a28943c1", size = 15706623, upload-time = "2026-10-28T19:22:20.528Z" },
+ { url = "https://files.pythonhosted.org/packages/26/74/1ceb1ca884add0565e618d013dd24333039100dc0fcc42661561f2e1cae5/mypy-2.4.1-cp315-cp315t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d923cec43316f0e6736b8298fd00abeaaf348a091c20e7296b3aebceecb63eca", size = 15944095, upload-time = "2026-10-28T19:21:04.779Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/68/27bcf9b4efa22733b6050485ae37845b94c9b8851abe4fc60632d905c32d/mypy-2.4.1-cp315-cp315t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6e081b466e88d4343adffe11aff0f02df34b5b9070b4b700053a0b7294c0be1", size = 17223065, upload-time = "2026-10-28T19:23:03.724Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/a3/ed5724b317bf585c29f3c8be2792e97d45925d65a9c8af40ceeb267072e6/mypy-2.4.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:70196470ac300f444a302a738541101893c2c57804078b33690286a7899149e9", size = 17560589, upload-time = "2026-10-28T19:21:34.539Z" },
+ { url = "https://files.pythonhosted.org/packages/42/2f/005185e6ab4752fd6cc2d1a6d1421fe21979770afe2490c00c555f34a913/mypy-2.4.1-cp315-cp315t-win_amd64.whl", hash = "sha256:de95648e0a6d46fa9929b956ed2c87c5dc9933d4d59945e6390e9d753c74e7c1", size = 12353324, upload-time = "2026-10-28T19:22:51.481Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/d6/20420cc9c780087139662534d128c2f3274361f8bb7e9cf0d5495a846ed5/mypy-2.4.1-cp315-cp315t-win_arm64.whl", hash = "sha256:ef8d5be90032261197ac388d945d8ff4f476c95880798a1911760af64ddfe71f", size = 11845599, upload-time = "2026-10-28T19:22:23.136Z" },
+ { url = "https://files.pythonhosted.org/packages/21/5c/6e8010f8f2c4ad817364592d4b3883633417269b913fc9d5174fdeb7e565/mypy-2.4.1-py3-none-any.whl", hash = "sha256:1784648d6ec8c2b5465e3f26241b845a0b3cb4e56c993c64abe16989e7698999", size = 2801286, upload-time = "2026-10-28T19:21:19.516Z" },
+]
+
+[[package]]
+name = "mypy-extensions"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" },
+]
+
[[package]]
name = "nodeenv"
version = "1.11.0"
@@ -506,6 +761,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/99/d5/4ad4ed31413c301430918e70d3d38ae6f097af0766cb9ab8c9d502b2a4d6/ruff-0.17.3-py3-none-win_arm64.whl", hash = "sha256:81fce74bcd6be0e9da7c67c7553fd0f1f12964a2052cd728ef917b937a5ca1d9", size = 9854332, upload-time = "2026-10-27T00:34:03.847Z" },
]
+[[package]]
+name = "typing-extensions"
+version = "4.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" },
+]
+
[[package]]
name = "virtualenv"
version = "21.14.13"