Skip to content

hash/defrag: support the blake3 algorithm - #202

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:blake3
Jul 27, 2026
Merged

hash/defrag: support the blake3 algorithm#202
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
ThomasWaldmann:blake3

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Adds "blake3" as an alternative to the hashlib algorithms for Store.hash /
backend.hash and for the content-hash target names computed by defrag.

blake3 is not part of hashlib, so it comes from the optional blake3 package:

pip install 'borgstore[blake3]'

Implementation

  • new borgstore.utils.hashing module: new(algorithm, data=b"") and
    file_digest(fileobj, algorithm) - hashlib for everything else, the blake3
    package for "blake3".
    If the package is not installed, this raises the usual
    ValueError: Unsupported hash algorithm: blake3 (pip install 'borgstore[blake3]'),
    so callers do not need any new error handling.
  • _base.BackendBase.hash and .defrag use it, which covers rclone, s3 and the
    REST server side.
  • posixfs keeps using the streaming file_digest path (no full read into memory)
    for blake3, too.
  • sftp skips its server-side check-file attempt for blake3: the extension only
    defines md5/sha*, and a failed attempt would permanently set
    check_file_supported = False, disabling server-side sha256 hashing for the
    rest of the connection.
  • the REST protocol needs no change (the algorithm is already a query parameter),
    but the blake3 package must be installed on the server for server-side
    hash/defrag computations.

Nesting is unaffected: a blake3 hexdigest is 64 hex chars, same as sha256.

Tests

New tests/test_hashing.py plus blake3 cases for backend hash, Store hash,
Store defrag, REST server hash and REST backend defrag. All are skipif-guarded,
so the test suite passes with and without the blake3 package installed
(verified both ways). tox/CI now install the blake3 extra.

blake3 is not part of hashlib, so it comes from the optional "blake3"
package: pip install 'borgstore[blake3]'. The new borgstore.utils.hashing
module wraps hashlib and adds blake3 for hashing purposes, so all the
backends (and the REST server) support it in hash and defrag.

For backends computing hashes remotely, the package needs to be installed
on the server side. sftp always computes blake3 client-side, because the
SFTP "check-file" extension does not support blake3 (and asking for it
would make us give up on check-file for all later calls).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ThomasWaldmann
ThomasWaldmann merged commit 0d52c50 into borgbackup:master Jul 27, 2026
8 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the blake3 branch July 27, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant