crypto: misc. improvements to code and docs

docs/changes.rst

@@ -191,6 +191,7 @@ Fixes:
 
 Other changes:
 
+- crypto: start a new session after encrypting 2TiB with one aes256-ocb session key, #6501
 - removed some global options (they were difficult to use and spammed the help output):
 
   - --remote-path -> BORG_REMOTE_PATH

docs/internals/data-structures.rst

@@ -768,8 +768,9 @@ AEAD modes
 For new repositories, borg only uses modern AEAD ciphers: AES-OCB or CHACHA20-POLY1305.
 
 For each borg invocation, a new sessionkey is derived from the borg key material
-and the 48bit IV starts from 0 again (both ciphers internally add a 32bit counter
-to our IV, so we'll just count up by 1 per chunk).
+and the 48bit IV starts from 0 again. The cipher blocks of a chunk do not consume
+IVs here (CHACHA20-POLY1305 counts them in its internal 32bit block counter, AES-OCB
+derives the per-block offsets from the IV), so we just count up by 1 per chunk.
 
 The encryption layout is best seen at the bottom of this diagram:
 
@@ -779,11 +780,58 @@ The encryption layout is best seen at the bottom of this diagram:
 
 No special IV/counter management is needed here due to the use of session keys.
 
-A 48 bit IV is way more than needed: If you only backed up 4kiB chunks (2^12B),
-the IV would "limit" the data encrypted in one session to 2^(12+48)B == 2.3 exabytes,
-meaning you would run against other limitations (RAM, storage, time) way before that.
-In practice, chunks are usually bigger, for big files even much bigger, giving an
-even higher limit.
+The 48 bit IV limits the number of **messages** (chunks and metadata objects) that we
+encrypt with one session key to 2^48 - borg refuses to encrypt more rather than reusing
+an IV. That is way more than needed: even if you only backed up 4kiB chunks (2^12B),
+2^48 messages would be 2^(12+48)B == 1.2 exabytes of input data, meaning you would run
+against other limitations (RAM, storage, time) way before that.
+
+How much **data** we may encrypt with one session key is a different question, which is
+not answered by the IV size, but by the security bounds of the ciphers, see below.
+
+.. _aead_usage_limits:
+
+AEAD usage limits
+~~~~~~~~~~~~~~~~~
+
+The relevant quantities are the number of encrypted messages (q), the amount of data
+encrypted with one key and the number of forgery attempts (v, decryptions of tampered
+data that borg refuses). ``p`` is the attacker's success probability we still consider
+acceptable. See issue #6501 for the details and for the computations.
+
+- **Number of messages** (both ciphers): limited to 2^48 per session key by the IV size,
+  see above. This is never the binding limit for either cipher.
+- **Data volume** (AES-OCB): the attacker's advantage grows with the square of the amount
+  of data encrypted using one key: about ``6 * sigma^2 / 2^128``, ``sigma`` being the
+  number of 128bit cipher blocks, including the authenticated header. RFC 7253 derives
+  from this bound that one key should encrypt at most 2^48 blocks (4PiB), which
+  corresponds to p == 2^-32. borg aims higher and starts a new session after 2^37 blocks
+  (2TiB), which corresponds to p == 2^-51 per session key.
+
+  CHACHA20-POLY1305 does not have such a limit at all: its confidentiality bound does not
+  depend on the amount of data encrypted.
+- **Forgery attempts** (CHACHA20-POLY1305): ``v <= p * 2^103 / (L' + 1)``, ``L'`` being
+  the message length (payload plus authenticated header) in 128bit blocks. For borg's
+  biggest messages, that is about 2^33 forgery attempts at p == 2^-50, so an attacker
+  would have to make borg read more than 100PiB of tampered data. Note that this is
+  counted over **all** session keys, so - unlike the data volume limit - it can not be
+  improved by starting more sessions.
+
+  For AES-OCB, the corresponding limit is much higher (its 128bit authentication tag
+  gives a term in the order of ``v * L / 2^128``), so the CHACHA20-POLY1305 limit is the
+  one to look at.
+
+We do **not** count or enforce the forgery attempts limit, we just document it here:
+a failed decryption means we got tampered or corrupted data and borg refuses it, usually
+aborting the whole command (``borg check`` and archive listing keep going, but only to
+report the damage). Getting anywhere near the limit computed above would require feeding
+borg a lot more tampered data than any real repository will ever hold.
+
+Starting a new session just means computing a new random session id and deriving a new
+session key from it (and counting the IV from 0 again). That is cheap and it does not
+need any special handling when reading, because the session id is part of every chunk
+header. Because the advantages of the individual session keys just add up, frequent
+session key changes also keep the total advantage low over the lifetime of a borg key.
 
 Legacy modes
 ~~~~~~~~~~~~

docs/internals/security.rst

@@ -184,6 +184,9 @@ Notable:
 - More modern and often faster AEAD ciphers instead of self-assembled stuff.
 - Due to the usage of session keys, which just start at 0 per session, IVs (nonces)
   do not need long-term special care here as they did for the legacy encryption modes.
+- The session key is also changed within a borg invocation if we encrypted a lot of data
+  with it (AES-OCB only, see :ref:`aead_usage_limits`). Reading is not affected by this,
+  because the session id is part of every message.
 - The id is now also input into the authentication tag computation.
   This strongly associates the id with the written data (== associates the key with
   the value). When later reading the data for some id, authentication will only

src/borg/crypto/key.py

@@ -3,8 +3,9 @@
 import os
 import textwrap
 from hashlib import sha256
+from math import ceil
 from pathlib import Path
-from typing import Literal, ClassVar
+from typing import Literal, ClassVar, Optional
 from collections.abc import Callable
 
 from ..logger import create_logger
@@ -1092,6 +1093,14 @@ class AEADKeyBase(KeyBase):
 
     MAX_IV = 2**48 - 1
 
+    # Maximum amount of data (in 128bit cipher blocks, counting plaintext **and** AAD) that we
+    # encrypt using one session key. When this is exceeded, we just start a new session (see
+    # new_session): that is cheap (one sha256) and fully transparent, because the sessionID
+    # travels in every chunk header, so old chunks stay decryptable.
+    #
+    # None means "no limit needed for this ciphersuite", see the subclasses for the reasoning.
+    MAX_SESSION_BLOCKS: Optional[int] = None
+
     # default storage; an individual key's actual storage is tracked per-instance in self.storage.
     STORAGE = KeyBlobStorage.REPO
     # an AEAD key may be stored as a keyfile or inside the repository (see borg key change-location).
@@ -1121,6 +1130,14 @@ def encrypt(self, id, data, aad=b""):
         # aad is additional authenticated data: authenticated together with data, but not encrypted
         # or returned.
         reserved = b"\0"
+        if self.MAX_SESSION_BLOCKS is not None:
+            # blocks the cipher will process for this message: payload and AAD are separate strings
+            # and thus rounded up separately, +2 covers the per-message cipher setup.
+            header_len = 1 + 1 + 6 + 24  # see Layout
+            blocks = ceil(len(data) / 16) + ceil((header_len + len(aad) + len(id)) / 16) + 2
+            if self.session_blocks + blocks > self.MAX_SESSION_BLOCKS:
+                self.new_session()  # do not encrypt more than that with the same session key
+            self.session_blocks += blocks
         iv = self.cipher.next_iv()
         if iv > self.MAX_IV:  # see the data-structures docs about why the IV range is enough
             raise IntegrityError("IV overflow, should never happen.")
@@ -1130,6 +1147,8 @@ def encrypt(self, id, data, aad=b""):
 
     def decrypt(self, id, data, aad=b""):
         # to decrypt existing data, we need to get a cipher configured for the sessionid and iv from header
+        # note: we deliberately do not count the failed decryptions (forgery attempts) here, although
+        # there is a limit for them also - see the "AEAD usage limits" docs and #6501 about why.
         self.assert_type(data[0], id)
         iv_48bit = data[2:8]
         sessionid = bytes(data[8:32])
@@ -1180,21 +1199,43 @@ def _get_cipher(self, sessionid, iv):
 
     def init_ciphers(self, manifest_data=None, iv=0):
         # in every new session we start with a fresh sessionid and at iv == 0, manifest_data and iv params are ignored
+        self.new_session()
+
+    def new_session(self):
+        """start a new session: fresh random sessionid, fresh session key, iv counting from 0 again"""
         self.sessionid = os.urandom(24)
         self.cipher = self._get_cipher(self.sessionid, iv=0)
+        self.session_blocks = 0  # cipher blocks encrypted using the current session key
 
 
 # Each of these is one unified key class per crypto suite. A key of this class may be stored either as
 # a keyfile or inside the repository (repokey) - that is a per-key storage property (self.storage), not
 # a class distinction. The class is selected from the manifest's key-type byte (see identify_key), which
 # only encodes the crypto suite (there is exactly one type byte per suite now).
 
+# AES-OCB has a birthday-type bound: an attacker's advantage in distinguishing the ciphertexts from
+# random is about 6 * sigma^2 / 2^128, sigma being the number of 128bit cipher blocks encrypted using
+# one key (Krovetz/Rogaway OCB3, Theorem 1; RFC 7253 states this as s^2 / 2^128 and derives its "use
+# a key for at most 2^48 blocks (4PiB)" rule of thumb from it - that is an advantage of 2^-32).
+# We aim higher and use 2^37 blocks (2TiB) per session key, giving an advantage of about 2^-51, which
+# is in line with the target probability used in the examples of draft-irtf-cfrg-aead-limits.
+# Rolling the session key this often is practically free and it is also what helps in the multi-key
+# setting: the advantages of the individual session keys just add up, so the quantity that matters
+# over the lifetime of the borg key is sum(sigma_i^2), not (sum sigma_i)^2, see #6501.
+AES_OCB_MAX_SESSION_BLOCKS = 2**37  # 2TiB
+
+# chacha20-poly1305 does not need such a limit: its confidentiality bound does not depend on the
+# amount of data encrypted at all (draft-irtf-cfrg-aead-limits 6.3.1: CA <= 0) and its integrity bound
+# only limits the number of **forgery attempts** (failed decryptions), which is counted over all keys
+# (7.2.1) and thus can not be improved by using more session keys anyway.
+
 
 class AESOCBKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
     TYPE = KeyType.AESOCB
     TYPES_ACCEPTABLE = {TYPE}
     ENC_NAME = "aes256-ocb"  # IDHASH_NAME = "sha256" via ID_HMAC_SHA_256 mix-in
     CIPHERSUITE = AES256_OCB
+    MAX_SESSION_BLOCKS = AES_OCB_MAX_SESSION_BLOCKS
 
 
 class CHPOKey(ID_HMAC_SHA_256, AEADKeyBase, FlexiKey):
@@ -1209,6 +1250,7 @@ class Blake3AESOCBKey(ID_BLAKE3_256, AEADKeyBase, FlexiKey):
     TYPES_ACCEPTABLE = {TYPE}
     ENC_NAME = "aes256-ocb"  # IDHASH_NAME = "blake3" via ID_BLAKE3_256 mix-in
     CIPHERSUITE = AES256_OCB
+    MAX_SESSION_BLOCKS = AES_OCB_MAX_SESSION_BLOCKS
 
 
 class Blake3CHPOKey(ID_BLAKE3_256, AEADKeyBase, FlexiKey):

src/borg/crypto/low_level.pyi

@@ -117,16 +117,6 @@ class CHACHA20_POLY1305(_AEAD_BASE):
     def requirements_check(cls) -> None: ...
     def __init__(self, key: bytes, iv: int | bytes | None = None, header_len: int = 0, aad_offset: int = 0) -> None: ...
 
-class AES:
-    """A thin wrapper around the OpenSSL EVP cipher API - for legacy code, like key file encryption."""
-
-    def __init__(self, enc_key: bytes, iv: int | bytes | None = None) -> None: ...
-    def encrypt(self, data: bytes, iv: int | bytes | None = None) -> bytes: ...
-    def decrypt(self, data: bytes) -> bytes: ...
-    def block_count(self, length: int) -> int: ...
-    def set_iv(self, iv: int | bytes) -> None: ...
-    def next_iv(self) -> int: ...
-
 class CSPRNG:
     """
     Cryptographically Secure Pseudo-Random Number Generator based on AES-CTR mode.

src/borg/crypto/low_level.pyx

@@ -2,8 +2,10 @@
 
 API:
 
-    encrypt(data, header=b'', aad_offset=0) -> envelope
-    decrypt(envelope, header_len=0, aad_offset=0) -> data
+    encrypt(data, header=b'', iv=None, aad=b'') -> envelope
+    decrypt(envelope, aad=b'') -> data
+
+header_len and aad_offset are given to the ciphersuite class when creating it, see below.
 
 Envelope layout:
 
@@ -25,12 +27,13 @@ garbage.
 
 Newly designed envelope layouts can just authenticate the whole header.
 
-IV handling:
+IV handling (CS is one of the ciphersuite classes below - the AEAD ones take a single key,
+the legacy AES-CTR ones a mac_key and an enc_key):
 
     iv = ...  # just never repeat!
-    cs = CS(hmac_key, enc_key, iv=iv)
-    envelope = cs.encrypt(data, header, aad_offset)
-    iv = cs.next_iv(len(data))
+    cs = CS(..., iv=iv, header_len=header_len, aad_offset=aad_offset)
+    envelope = cs.encrypt(data, header=header)
+    iv = cs.next_iv()
     (repeat)
 """
 
@@ -41,7 +44,6 @@ from math import ceil
 from cpython cimport PyMem_Malloc, PyMem_Free
 from cpython.buffer cimport PyBUF_SIMPLE, PyObject_GetBuffer, PyBuffer_Release
 from cpython.bytes cimport PyBytes_FromStringAndSize, PyBytes_AsString
-from libc.stdlib cimport malloc, free
 from libc.stdint cimport uint8_t, uint32_t, uint64_t
 from libc.string cimport memset, memcpy
 
@@ -69,8 +71,6 @@ cdef extern from "openssl/evp.h":
 
     EVP_CIPHER_CTX *EVP_CIPHER_CTX_new()
     void EVP_CIPHER_CTX_free(EVP_CIPHER_CTX *a)
-    void EVP_CIPHER_CTX_init(EVP_CIPHER_CTX *a)
-    void EVP_CIPHER_CTX_cleanup(EVP_CIPHER_CTX *a)
 
     int EVP_EncryptInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *cipher, ENGINE *impl,
                            const unsigned char *key, const unsigned char *iv)
@@ -285,6 +285,10 @@ cdef class AES256_CTR_BASE:
         """
         authenticate aad + iv + cdata, decrypt cdata, ignore header bytes up to aad_offset.
         """
+        if len(envelope) < self.header_len + self.mac_len + self.iv_len_short:
+            # truncated data - handle it like any other corruption or tampering, instead of
+            # computing the MAC over negative lengths below.
+            raise IntegrityError('MAC Authentication failed: envelope too short')
         cdef int ilen = len(envelope)
         cdef int hlen = self.header_len
         cdef int aoffset = self.aad_offset
@@ -441,13 +445,13 @@ cdef class _AEAD_BASE:
     @classmethod
     def requirements_check(cls):
         """check whether library requirements for this ciphersuite are satisfied"""
-        raise NotImplemented  # override / implement in child class
+        raise NotImplementedError  # override / implement in child class
 
     def __init__(self, key, iv=None, header_len=0, aad_offset=0):
         """
         init AEAD crypto
 
-        :param key: 256bit encrypt-then-mac key
+        :param key: 256bit AEAD key
         :param iv: 96bit initialisation vector / nonce
         :param header_len: expected length of header
         :param aad_offset: where in the header the authenticated data starts
@@ -481,11 +485,14 @@ cdef class _AEAD_BASE:
         if iv is not None:
             self.set_iv(iv)
         assert self.blocks == 0, 'iv needs to be set before encrypt is called'
-        # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
-        # IV we provide, thus we must not encrypt more than 2^32 cipher blocks with same IV).
+        # CHACHA20 has an internal 32bit block counter (besides the 96bit (12Byte) IV we give it),
+        # thus we must not encrypt more than 2^32 cipher blocks with the same (key, IV) pair.
+        # AES-OCB has no such counter (it derives the per-block offsets from the IV), but we apply
+        # the same limit to both ciphers: the check is cheap and can not trigger for borg messages
+        # anyway, as these are limited to MAX_DATA_SIZE.
         block_count = self.block_count(len(data))
         if block_count > 2**32:
-            raise ValueError('too much data, would overflow internal 32bit counter')
+            raise ValueError('too much data for one message (max 2^32 cipher blocks)')
         cdef int ilen = len(data)
         cdef int hlen = len(header)
         assert hlen == self.header_len_expected
@@ -551,14 +558,18 @@ cdef class _AEAD_BASE:
 
     def decrypt(self, envelope, aad=b''):
         """
-        authenticate aad + header + cdata (from envelope), ignore header bytes up to aad_offset.,
+        authenticate aad + header + cdata (from envelope), ignore header bytes up to aad_offset,
         return decrypted cdata.
         """
-        # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit (12Byte)
-        # IV we provide, thus we must not decrypt more than 2^32 cipher blocks with same IV):
+        # same limit as for encryption, see there: we must not decrypt more than 2^32 cipher
+        # blocks with the same (key, IV) pair.
         approx_block_count = self.block_count(len(envelope))  # sloppy, but good enough for borg
         if approx_block_count > 2**32:
-            raise ValueError('too much data, would overflow internal 32bit counter')
+            raise ValueError('too much data for one message (max 2^32 cipher blocks)')
+        if len(envelope) < self.header_len_expected + self.mac_len:
+            # truncated data - handle it like any other corruption or tampering, instead of
+            # confusing OpenSSL with negative lengths below.
+            raise IntegrityError('Authentication failed: envelope too short')
         cdef int ilen = len(envelope)
         cdef int hlen = self.header_len_expected
         cdef int aoffset = self.aad_offset
@@ -627,8 +638,9 @@ cdef class _AEAD_BASE:
 
     def next_iv(self):
         # call this after encrypt() to get the next iv (int) for the next encrypt() call
-        # AES-OCB, CHACHA20 ciphers all add a internal 32bit counter to the 96bit
-        # (12 byte) IV we provide, thus we only need to increment the IV by 1.
+        # the cipher blocks of a message do not consume IVs here (CHACHA20 counts them in its
+        # internal 32bit block counter, AES-OCB derives the per-block offsets from the 96bit
+        # (12 byte) IV we give it), thus we only need to increment the IV by 1 per message.
         iv = int.from_bytes(self.iv[:self.iv_len], byteorder='big')
         return iv + 1
 

src/borg/legacy/crypto/low_level.pyi

@@ -0,0 +1,12 @@
+# Type stubs for borg.legacy.crypto.low_level
+# This file provides type hints for the Cython extension module
+
+class AES:
+    """A thin wrapper around the OpenSSL EVP cipher API - for legacy key file encryption."""
+
+    def __init__(self, enc_key: bytes, iv: int | bytes | None = None) -> None: ...
+    def encrypt(self, data: bytes, iv: int | bytes | None = None) -> bytes: ...
+    def decrypt(self, data: bytes) -> bytes: ...
+    def block_count(self, length: int) -> int: ...
+    def set_iv(self, iv: int | bytes) -> None: ...
+    def next_iv(self) -> int: ...

src/borg/selftest.py

@@ -26,7 +26,7 @@
 
 SELFTEST_CASES = [CryptoTestCase, ChunkerTestCase, ChunkerFixedTestCase]
 
-SELFTEST_COUNT = 17
+SELFTEST_COUNT = 19
 
 
 class SelfTestResult(TestResult):