Qsafe post-quantum encryption, signed manifests, streaming AES v2, manifest hardening#14
Open
SP1R4 wants to merge 14 commits into
Open
Qsafe post-quantum encryption, signed manifests, streaming AES v2, manifest hardening#14SP1R4 wants to merge 14 commits into
SP1R4 wants to merge 14 commits into
Conversation
Five changes that close concrete weaknesses in the backup pipeline:
- Replace paramiko.WarningPolicy with RejectPolicy. Unknown SSH hosts are
refused with an actionable known_hosts hint instead of silently accepted.
New config: [SSH] known_hosts. Shared factory in src/ssh_client.py keeps
sync.py and restore.py from drifting apart.
- Encryption format v1 with a 4-byte magic ("BHE1") and a 1-byte KDF id.
Magic + KDF bytes are bound into the AES-GCM associated_data so flipping
the KDF byte invalidates the tag (prevents downgrade). Legacy .enc files
(no magic, salt+nonce+ct) auto-detected and decryptable for backward
compatibility.
- Argon2id alongside PBKDF2 as a passphrase KDF. Opt-in via
[ENCRYPTION] kdf = argon2id and `pip install 'backup-handler[argon2]'`.
Per-file KDF id stored in the header — switching the default is safe.
- Atomic .enc writes: write to .tmp, fsync, os.replace, then unlink the
plaintext. A crash mid-write no longer leaves a half-written .enc
alongside the original.
- assert_config_safe_for_hooks() refuses to run shell hooks when the
config file is group- or world-writable. Hooks run with shell=True so
the config is the trust boundary; bypass with BACKUP_HANDLER_TRUST_CONFIG=1.
Config schema bumped to v4. New tests cover the v1 header, downgrade
attempts, legacy decrypt, atomic-write cleanup, and the hook-safety check.
All 141 tests pass.
…anup - _copy_single_file: hash source and destination once each instead of three times (verify_backup() rehashed, then we rehashed again for the manifest). Halves disk I/O on every local copy. - update_last_backup_time / update_last_full_backup_time now write the JSON via tmp + fsync + os.replace, so a concurrent reader or power-cut never sees a half-written timestamp file. - Manifest loading enforces backup_manifest_YYYYMMDD_HHMMSS.json exactly. Old code did a string prefix-strip on names matching the glob, which would silently include foreign files or truncated names. - s3_sync: sweep multipart uploads older than 24h under our prefix at the start of every run. boto3's TransferManager aborts on failures it sees, but SIGKILL/OOM kills bypass that path and leave billable parts. - show_status reads total_bytes from the latest manifest when present; rglob().stat() walks were blocking --status for tens of seconds on multi-TB trees. New tests cover the manifest regex (malformed names ignored) and basic BackupManifest save/load.
Move everything under src/backup_handler/ so the wheel installs as a
clean ``backup_handler`` package instead of a ``src/`` directory at the
site-packages root. External submodules (banner, bot, email attachments)
move in alongside the core modules.
Split the 1,314-line cli.py into focused units:
- cli.py entrypoint + argv routing (~170 lines)
- dispatch.py early-exit handlers (--install, --status, --verify,
--snapshot, --restore-snapshot, --snapshot-diff,
--restore)
- orchestrator.py backup_operation + scheduled_operation + helpers
- status.py the --status dashboard
- lock.py single-instance PID lock
- _paths.py single source of truth for filesystem locations
Knock-on changes:
- pyproject.toml: packages = ["src/backup_handler"], version path moved,
scripts entry points at backup_handler.cli:main, lint/format/bandit
excludes pruned of the merged directories.
- Dockerfile: drop the stale ``COPY main.py``.
- tests/: ``from src.X`` -> ``from backup_handler.X``;
conftest.py puts src/ on sys.path.
- email_nots/email.py renamed to backup_handler/email_attachments.py
(the email module name shadowed the stdlib ``email`` package).
PR 6 will replace the hard-coded PROJECT_ROOT paths in _paths.py with
XDG-aware lookups so installed wheels stop pointing into /tmp.
All 132 tests still pass.
- tests/test_e2e_backup.py: full -> verify -> restore roundtrip against a temp tree, encrypted and unencrypted variants. Catches the cross-module bugs (checksum mismatch, header drift, manifest path encoding, decrypt flow) that unit tests miss. The single most valuable test in the suite. - tests/test_sync.py: _copy_single_file copies, records manifest, handles symlinks, records failure on unwritable destinations; sync respects exclude patterns. - tests/test_s3_sync.py: stale-multipart sweep aborts old uploads, leaves fresh ones alone, swallows list/abort errors so a flaky AWS API doesn't abort the backup. - pyproject [tool.coverage.report].fail_under = 25. Below current 26.4% so CI fails when coverage regresses; raise as coverage grows. - pyproject [[tool.mypy.overrides]] adds a strict block for the leaf modules (_paths, encryption, lock, manifest, ssh_client, status) that already have full type hints. A second CI step gates on this strict check (the broader mypy run stays advisory). - compression.py: pyminizip is now a lazy optional import. Plain compression works without it; zip_pw raises a clear error if the C extension is missing. Unblocks PR 5's pyzipper swap.
…blocks pyminizip is a C extension stuck on ZipCrypto (weak, brute-forced in seconds with known plaintext) and stopped building cleanly on Python 3.13+. Swap it for pyzipper — pure Python, WinZip AES-256, drops a build dep: - compression.py: _write_aes_encrypted_zip() writes pyzipper archives with ZIP_DEFLATED + WZ_AES at 256 bits. Arcnames are relative to src_dir so the archive mirrors the source tree. - tests/test_compression.py covers the happy path and the wrong-password rejection. Dependency layout fixed: - requirements.in lists only the direct deps (10 lines, with floors). - requirements.txt is now pip-compile output: every transitive annotated with what brought it in. Old requirements.txt mixed direct and transitive without provenance; bumping anything was a guessing game. - pyproject floors bumped to current stable: cryptography 43->45, paramiko 3.4->3.5, boto3 1.28->1.34, pyTelegramBotAPI 4.22->4.27. CI: - pip-audit no longer continue-on-error — known CVEs in cryptography or paramiko block the merge. Backup tools must not ship with unpatched TLS/crypto libraries. 12 of the 13 open Dependabot PRs on the repo target transitives that this change removes from requirements.txt. They will close automatically.
- _paths.py: deterministic search order for config/data/lock locations. $BACKUP_HANDLER_CONFIG_DIR -> $XDG_CONFIG_HOME/backup-handler -> /etc/backup-handler -> <package_root>/config. Same shape for data (XDG_DATA_HOME -> /var/lib/backup-handler), preserving existing package-root installs first so nobody loses their Logs/ on upgrade. Lock follows XDG_RUNTIME_DIR -> /var/run -> data dir. - cli.py: banner only prints under sys.stdout.isatty(). Cron firings no longer log 30 lines of ANSI escapes per invocation. - dispatch.handle_restore: refuse when --from-dir resolves to the same path as --to-dir or contains it. The old behavior happily overwrote the backup tree mid-read. Skipped for ssh://, s3://, user@host: sources since they cannot collide with a local destination. - dispatch.py: drop the duplicate stderr prints — logger already has a StreamHandler() (stderr) attached, so logger.error() reaches the user terminal without the second print(). - New tests cover the XDG search order, lock fallback, the remote-path detector, and the self-overwriting-restore refusal.
- README:
- New "Why this vs. restic / borg / duplicity?" section near the top —
states what this tool is for and what it isn't, so evaluators can
self-select instead of scrolling 1,200 lines.
- Preflight, --install, system snapshot, and restore drill sections
reduced to one-paragraph teasers that link to the corresponding
docs/ files. README is 1,035 lines now, down from 1,198.
- Python badge corrected to 3.10+ (matches pyproject).
- Compression feature row references pyzipper / AES-256, not pyminizip.
- docs/: new directory with the extracted deep dives —
preflight.md, install.md, snapshots.md, restore-drill.md. These cover
the long-form material that was crowding out the quickstart.
- CHANGELOG: comprehensive [Unreleased] entry covering the security,
correctness, refactor, deps, and UX changes from the six prior PRs in
this series. Grouped per Keep a Changelog (Security / Changed / Added).
- CONTRIBUTING: dev workflow now references the post-refactor module
paths and the new strict-mypy leaf-module check; release process
points at the new __version__ location.
Integrates the Qsafe project (X25519 + ML-KEM-1024 + AES-256-GCM hybrid, NIST FIPS 203 Level 5) as an encryption-at-rest backend and manifest signer. Encryption ([ENCRYPTION] backend = qsafe): - Backups encrypt to recipient *public* keys (qsafe_recipients, comma-separated for multi-recipient escrow) — the backup host never holds the decryption secret; qsafe_secret_key is needed only for restore/verify and can live off-host. - decrypt_file dispatches by magic bytes (QSAFE00x vs BHE1 vs legacy), so AES and Qsafe .enc files coexist and old backups keep restoring. - Engine: libqsafe Python bindings, falling back to the qsafe CLI (passphrase via QSAFE_PASSPHRASE env, never argv). Signed manifests ([ENCRYPTION] qsafe_sign_key / qsafe_sign_pub): - Each backup_manifest_*.json gets a detached ML-DSA-87 signature at backup time; works with either encryption backend. - --verify marks a manifest with an invalid signature corrupted; point-in-time --restore aborts on one. Missing signatures only warn. - Manifest .sig files are excluded from encryption so they stay verifiable without decryption keys. Config schema bumped to 5. qsafe_backend.py and manifest.py are strict-mypy gated (pyproject overrides + CI blocking list). Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
…t runbook - --verify now authenticates Qsafe .enc files in place via the AEAD tag: no plaintext is ever written to disk during verification. AES files keep the decrypt-to-temp + size-check path. - Qsafe readiness preflight in backup_operation: aborts with exit 2 (critical alert + status sentinel) before any files are copied when the engine (bindings/CLI) or configured key files are missing. Previously encryption failed after the copy, leaving a plaintext backup on disk. - RUNBOOK section 6: key inventory, key ceremony, Shamir escrow custody, rotation rules (retain old secret keys until retention expires), and escrow-key drill requirements. Restore-drill doc cross-references it. Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
…safe CI Four hardening/performance improvements: Streaming AES format v2 (magic BHE2, default for new .enc files): chunked AES-256-GCM, per-chunk counter nonce with a final-chunk flag, full header bound as AAD. Constant-memory encrypt/decrypt (300 MB file roundtrips in ~34 MB RSS vs whole-file-in-RAM), and truncation, extension, reordering, or KDF-downgrade all fail authentication. v1 and legacy files remain decryptable. Exact file resolution (manifest rel_path): every record_copy call site (local full/incremental/differential, SSH, S3, DB dump) records the backup-relative path. Restore and verify resolve entries exactly; previously restore took rglob(filename)[0], so two same-named files in different subdirectories could silently restore the wrong content. Old manifests fall back to the filename heuristics. Ciphertext binding (manifest enc_checksum): after encryption, the SHA-256 of each .enc file is recorded and the manifest is signed after that pass. --verify now proves encrypted-backup integrity with no keys, and detects swapped valid ciphertexts (AEAD alone cannot). Point-in-time restore additionally compares restored plaintext against the manifest checksum and fails on mismatch instead of restoring altered content. CI qsafe-integration job: builds liboqs (cached) + the Qsafe CLI from source on ubuntu, so the roundtrip/tamper/signature tests actually run on every PR instead of taking their skip path. Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
The PR lineage included seven commits that had never run through CI; this makes every job green: - sync.py: import verify_backup — incremental and differential backups raised NameError at the post-copy verification step (real bug, masked by untested paths). - Readiness check: validate config completeness (recipients, key files) before engine availability, so missing-key errors are actionable on hosts without qsafe — also makes the readiness tests engine-agnostic. - test extra: add argon2-cffi so Argon2id KDF paths run in CI and the AAD-downgrade test fails on the tag, not on a missing import. - Strict mypy: annotate logger params (lock, status, ssh_client, encryption), coerce argon2/json returns, os.walk str() — plus types-tqdm in the dev extra. - Ruff/black: fix the remaining 48 findings — banner trailing whitespace, unused imports/variables, undefined main in orchestrator __main__, contextlib.suppress, N999/S103/S108 per-file-ignores with rationale, black-compatible noqa placement. - Bandit B105 / ruff S105: mark the YOUR_APP_PASSWORD template placeholder comparison as not-a-credential. - .gitleaks.toml: allowlist qsafe CLI --key-file/--pub-file doc examples that trip the generic-api-key rule (scanned historically, so an inline fix could not work). Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
…p-audit, private Qsafe clone
- scripts/simulate_unmounted_disk.py still imported the pre-src-layout
`src.preflight` module — broken since the refactor. Point it at
backup_handler.preflight (and satisfy import sorting).
- gitleaks: the flagged "secret" was documentation prose ("key ceremony,
escrow/Shamir"), not the CLI example — and it lives in a historical
commit, so rewording cannot fix it. Allowlist the project's own
markdown docs; code/config/history remain fully scanned. Verified
clean with gitleaks 8.24.3 over all 46 commits.
- pip-audit: --strict treats skipped editable dists as fatal, and the
job installed with -e. Install non-editable so the project itself is
audited, and drop the now-contradictory --skip-editable.
- qsafe-integration: SP1R4/Qsafe is a private repo, so the anonymous
clone exits 128. Use the QSAFE_REPO_TOKEN secret when present, with
an anonymous fallback should the repo go public.
Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
…transitives - tests/test_config.py: black 26 hugs multiline-string args; reformat so the CI formatter (installed at latest) agrees. - pip-audit: audit requirements.txt instead of the installed environment. The project itself is not on PyPI, so --strict flagged it unauditable regardless of editable/non-editable install. - Regenerate requirements.txt with pip-compile --upgrade: clears 18 known vulnerabilities across urllib3 (2.2.2), cryptography (45.0.7), idna, jaraco-context, pynacl, and requests. pip-audit --strict is now clean against the lockfile. Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
gitleaks-action lists PR commits through the GitHub API; that worked implicitly while the repo was public, but on a private repo it fails with "Resource not accessible by integration" unless the job token has pull-requests: read. Claude-Session: https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates the Qsafe project as a post-quantum encryption backend and hardens the manifest/verify/restore pipeline. Three commits:
1. Qsafe encryption backend + ML-DSA-87 signed manifests
[ENCRYPTION] backend = qsafe: backups encrypt to recipient public keys (X25519 + ML-KEM-1024 + AES-256-GCM, NIST FIPS 203 Level 5). The backup host never holds the decryption secret;qsafe_secret_keyis needed only for restore/verify. Multi-recipient escrow via comma-separatedqsafe_recipients.decrypt_filedispatches by magic bytes (QSAFE00x/BHE2/BHE1/ legacy), so all formats coexist and old backups keep restoring.qsafe_sign_key/qsafe_sign_pub): detached ML-DSA-87 signature per manifest; invalid signatures mark the manifest corrupted in--verifyand abort point-in-time restores.2. In-place verification, fail-fast preflight, key-management runbook
--verifyauthenticates Qsafe files in place via the AEAD tag — no plaintext ever written during verification.3. Streaming AES v2 + manifest hardening
BHE2, default): chunked AES-256-GCM, per-chunk counter nonce with final-chunk flag, header bound as AAD. Constant memory (300 MB file roundtrips in ~34 MB RSS); truncation/extension/reorder/KDF-downgrade all fail authentication.rel_pathin manifests: restore/verify resolve entries exactly instead ofrglob(name)[0]— same-named files can no longer restore the wrong content.enc_checksumin manifests: ciphertext SHA-256 recorded post-encryption (signing runs after).--verifyproves encrypted-backup integrity with no keys, and detects swapped valid ciphertexts, which AEAD alone cannot. Restore also compares restored plaintext against the manifest checksum and fails on mismatch.qsafe-integrationjob: builds liboqs (cached) + the Qsafe CLI from source so the roundtrip/tamper/signature tests actually run on every PR.Test plan
qsafeCLI: backup → keyless verify → tampered-manifest abort → swapped-ciphertext detection → escrow-key restore, all exercised.qsafe_backend.py,manifest.py; ruff/format clean on all changed files.https://claude.ai/code/session_01WDbZuLNLRv2W3rdsS3wX4j