feat(application): implement DTLS 1.3 unified-header records (#2467)

docs/source/pcapkit/protocols/application/dtls.rst

@@ -6,8 +6,8 @@ DTLS - Datagram Transport Layer Security
 :mod:`pcapkit.protocols.application.dtls` contains
 :class:`~pcapkit.protocols.application.dtls.DTLS` only, which implements an
 extractor for Datagram Transport Layer Security (DTLS) [*]_ -- DTLS 1.0
-(:rfc:`4347`), DTLS 1.2 (:rfc:`6347`) and DTLS 1.3's legacy-format records
-(:rfc:`9147`) -- at the record layer, plus the epoch/sequence-number fields and
+(:rfc:`4347`), DTLS 1.2 (:rfc:`6347`) and DTLS 1.3's legacy-format and
+unified-header records (:rfc:`9147`) -- at the record layer, plus the epoch/sequence-number fields and
 the handshake-message fragment header that set DTLS apart from
 :class:`~pcapkit.protocols.application.tls.TLS`, whose reliable, ordered
 transport needs neither.
@@ -34,12 +34,18 @@ every version and :meth:`~pcapkit.protocols.application.dtls.DTLS.read` sets
      - ``DTLS 1.3``
    * - any other ``0xFExx``
      - ``DTLS``
+   * - unified header (first octet ``001CSLEE``, :rfc:`9147#section-4.1`)
+     - ``DTLS 1.3``
 
 DTLS 1.3 never puts its own ``0xFEFC`` on the record layer
-(:rfc:`9147#section-4`), so it is told apart only by the
+(:rfc:`9147#section-4`), so a legacy-format record is told apart only by the
 ``supported_versions`` peek. A version the table does not name keeps the
 generic ``DTLS`` alias rather than claiming a specific one.
 
+A unified-header record has no legacy version to read. Its first octet's top
+three bits are ``001``, which no legacy content type (20 through 27) sets, so
+the two formats are told apart by that octet alone.
+
 Known Limitations
 -----------------
 
@@ -49,14 +55,14 @@ Known Limitations
 and its :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.body`
 kept as raw octets.
 
-**DTLS 1.3's unified record header is not decoded** (:issue:`2467`). It has
-no content-type octet (:rfc:`9147#section-4`), so it degrades to
-:class:`~pcapkit.protocols.misc.raw.Raw`. DTLS 1.3's cleartext records use
-the legacy ``DTLSPlaintext`` header and are decoded.
+**A unified-header record with a connection ID is not decoded**
+(:rfc:`9147#section-4.1`, :issue:`2467`). Its length is negotiated, not carried
+in the record, so it degrades to :class:`~pcapkit.protocols.misc.raw.Raw`. The
+same holds for ``tls12_cid`` records (:rfc:`9146`).
 
-**Connection ID records are not decoded** (``tls12_cid``, :rfc:`9146`): the
-connection ID's length is not carried in the record, so they degrade to
-:class:`~pcapkit.protocols.misc.raw.Raw`.
+**A unified-header record carries only the low two bits of the epoch**, and
+its sequence number is header-protected, so the values it reports are the
+ones on the wire.
 
 .. versionadded:: 1.6.0
 
@@ -76,6 +82,7 @@ connection ID's length is not carried in the record, so they degrade to
 
    .. automethod:: read
    .. automethod:: make
+   .. automethod:: unpack
 
    .. automethod:: __length_hint__
 
@@ -90,6 +97,10 @@ Header Schema
    :members:
    :show-inheritance:
 
+.. autoclass:: pcapkit.protocols.schema.application.dtls.DTLSCiphertext
+   :members:
+   :show-inheritance:
+
 Data Models
 -----------
 
@@ -99,6 +110,10 @@ Data Models
    :members:
    :show-inheritance:
 
+.. autoclass:: pcapkit.protocols.data.application.dtls.DTLSCiphertext
+   :members:
+   :show-inheritance:
+
 .. autoclass:: pcapkit.protocols.data.application.dtls.DTLSHandshake
    :members:
    :show-inheritance:

pcapkit/protocols/application/dtls.py

@@ -7,10 +7,11 @@
 :mod:`pcapkit.protocols.application.dtls` contains
 :class:`~pcapkit.protocols.application.dtls.DTLS` only, which implements an
 extractor for the DTLS record-layer family -- DTLS 1.0 (:rfc:`4347`), DTLS
-1.2 (:rfc:`6347`), and DTLS 1.3's legacy-format records (:rfc:`9147`).
-:attr:`UDP.__proto__ <pcapkit.protocols.transport.udp.UDP.__proto__>` binds
-it at port 5684. As with :class:`~pcapkit.protocols.application.tls.TLS`,
-:meth:`~DTLS.read` sets :attr:`~DTLS.alias` from the record's legacy version.
+1.2 (:rfc:`6347`), and DTLS 1.3's legacy-format and unified-header records
+(:rfc:`9147`). :attr:`UDP.__proto__ <pcapkit.protocols.transport.udp.UDP.__proto__>`
+binds it at port 5684. As with :class:`~pcapkit.protocols.application.tls.TLS`,
+:meth:`~DTLS.read` sets :attr:`~DTLS.alias` from the record's legacy version,
+or to ``DTLS 1.3`` for a unified-header record.
 
 Known Limitations
 -----------------
@@ -21,14 +22,14 @@
 and its :attr:`~pcapkit.protocols.data.application.dtls.DTLSHandshake.body`
 kept as raw octets.
 
-**DTLS 1.3's unified record header is not decoded** (:issue:`2467`). It has
-no content-type octet (:rfc:`9147#section-4`), so it degrades to
-:class:`~pcapkit.protocols.misc.raw.Raw`. DTLS 1.3's cleartext records use
-the legacy ``DTLSPlaintext`` header and are decoded.
+**A unified-header record with a connection ID is not decoded**
+(:rfc:`9147#section-4.1`). Its length is negotiated, not carried in the
+record, so it degrades to :class:`~pcapkit.protocols.misc.raw.Raw`. The same
+holds for ``tls12_cid`` records (:rfc:`9146`).
 
-**Connection ID records are not decoded** (``tls12_cid``, :rfc:`9146`): the
-connection ID's length is not carried in the record, so they degrade to
-:class:`~pcapkit.protocols.misc.raw.Raw`.
+**A unified-header record carries only the low two bits of the epoch**, and
+its sequence number is header-protected, so the values it reports are the
+ones on the wire.
 
 .. versionadded:: 1.6.0
 
@@ -38,7 +39,10 @@
 from pcapkit.const.tls.content_type import ContentType as Enum_ContentType
 from pcapkit.protocols.application.ssl import DTLS13_VERSION, SSL
 from pcapkit.protocols.data.application.dtls import DTLS as Data_DTLS
+from pcapkit.protocols.data.application.dtls import DTLSCiphertext as Data_DTLSCiphertext
 from pcapkit.protocols.schema.application.dtls import DTLS as Schema_DTLS
+from pcapkit.protocols.schema.application.dtls import DTLSCiphertext as Schema_DTLSCiphertext
+from pcapkit.utilities.exceptions import ProtocolError
 
 if TYPE_CHECKING:
     from enum import IntEnum as StdlibEnum
@@ -60,14 +64,48 @@
 #: Legacy record versions a DTLS 1.3 record may carry: DTLS 1.2's always, and
 #: DTLS 1.0's on an initial ``ClientHello`` (:rfc:`9147#section-4`).
 _DTLS13_LEGACY_VERSIONS = frozenset({_DTLSV1_0_VERSION, _DTLSV1_2_VERSION})
+#: Top three bits of a unified-header first octet, ``001`` (:rfc:`9147#section-4.1`).
+_UNIFIED_PREFIX = 0b001
+#: Shortest ciphertext a unified-header record may carry: the record-number
+#: sample is taken from its first 16 octets (:rfc:`9147#section-4.2.3`).
+_CIPHERTEXT_MIN_LENGTH = 16
+
+
+def _is_unified_header(octet: int) -> bool:
+    """Whether ``octet`` opens a unified-header record.
+
+    Legacy records open with a content type of 20 through 27, whose top three
+    bits are ``000``, so the ``001`` prefix never matches one of them.
+
+    Args:
+        octet: First octet of the record.
+
+    Returns:
+        Whether the octet's top three bits are the unified-header prefix.
+
+    """
+    return octet >> 5 == _UNIFIED_PREFIX
+
+
+def _ciphertext_header_length(flags: 'dict[str, int]') -> 'int':
+    """Length of a unified header, from its first octet's decoded flags.
+
+    Args:
+        flags: Decoded first octet, as the header schema holds it.
+
+    Returns:
+        First octet, sequence number and, if present, the length field.
+
+    """
+    return 1 + (2 if flags['sequence_long'] else 1) + (2 if flags['length_present'] else 0)
 
 
 class DTLS(SSL[Data_DTLS, Schema_DTLS], schema=Schema_DTLS, data=Data_DTLS):
     """This class implements all protocols in the DTLS record-layer family.
 
     - Datagram Transport Layer Security 1.0 [:rfc:`4347`]
     - Datagram Transport Layer Security 1.2 [:rfc:`6347`]
-    - Datagram Transport Layer Security 1.3, legacy-format records only [:rfc:`9147`]
+    - Datagram Transport Layer Security 1.3, legacy-format and unified-header records [:rfc:`9147`]
 
     Note:
         The subpackage does not track the dispatch tier: DTLS is dispatched
@@ -96,12 +134,15 @@ def alias(self) -> 'Literal["DTLS", "DTLS 1.0", "DTLS 1.2", "DTLS 1.3"]':
         return self._alias
 
     @property
-    def length(self) -> 'Literal[13]':
-        """Header length of current protocol -- the fixed record header,
-        :attr:`~pcapkit.protocols.data.application.dtls.DTLS.length`'s
-        fragment excluded.
+    def length(self) -> 'int':
+        """Header length of current protocol -- the record header, the
+        fragment excluded. Fixed at 13 octets for a legacy record; a unified
+        header's length follows from its first octet.
 
         """
+        schema = getattr(self, '__header__', None)
+        if isinstance(schema, Schema_DTLSCiphertext):
+            return _ciphertext_header_length(schema.flags)
         return 13
 
     ##########################################################################
@@ -148,7 +189,10 @@ def make(self,
              content: 'Any' = b'',
              data: 'Optional[bytes]' = None,
              payload: 'bytes | ProtocolBase | Schema' = b'',
-             **kwargs: 'Any') -> 'Schema_DTLS':
+             ciphertext: 'bool' = False,
+             sequence_long: 'bool' = False,
+             length_present: 'bool' = True,
+             **kwargs: 'Any') -> 'Schema_DTLS | Schema_DTLSCiphertext':
         """Make (construct) packet data, for every tier: ``version`` is written as is.
 
         Args:
@@ -159,25 +203,50 @@ def make(self,
             version: Legacy record version (``major << 8 | minor``), e.g.
                 ``0xFEFD`` for DTLS 1.2 and (the legacy value DTLS 1.3 keeps
                 here) DTLS 1.3.
-            epoch: Epoch.
-            sequence_number: Sequence number within ``epoch`` (48-bit).
+            epoch: Epoch. A unified-header record carries only its low two
+                bits, so a larger value is refused.
+            sequence_number: Sequence number within ``epoch`` (48-bit); for a
+                unified-header record, 8 or 16 bits as ``sequence_long`` says.
             content: Record content: a :obj:`tuple` of
                 :class:`~pcapkit.protocols.data.application.dtls.DTLSHandshake`
                 for ``handshake``, an
                 :class:`~pcapkit.protocols.data.application.tls.Alert`, a
                 :class:`~pcapkit.protocols.data.application.tls.ChangeCipherSpec`,
-                or raw :obj:`bytes`, as :meth:`read` would decode it.
+                or raw :obj:`bytes`, as :meth:`read` would decode it. For a
+                unified-header record, the ciphertext as :obj:`bytes`.
             data: Pre-encoded record fragment. When given, it is used
                 verbatim in place of ``content`` and ``length`` is computed
                 from it.
             payload: Octets after the record, e.g. a further record
                 coalesced in the same buffer.
+            ciphertext: Make a unified-header record (:rfc:`9147#section-4.1`)
+                instead of a legacy one; ``content_type``, ``version`` and
+                ``data`` do not apply.
+            sequence_long: Whether a unified-header record's sequence number is
+                16 bits rather than 8.
+            length_present: Whether a unified-header record carries its length;
+                if not, it runs to the end of the datagram.
             **kwargs: Arbitrary keyword arguments.
 
         Returns:
             Constructed packet data.
 
         """
+        if ciphertext:
+            return Schema_DTLSCiphertext(
+                flags={
+                    'fixed': _UNIFIED_PREFIX,
+                    'connection_id': 0,
+                    'sequence_long': int(sequence_long),
+                    'length_present': int(length_present),
+                    'epoch': epoch,
+                },
+                sequence_number=sequence_number,
+                length=len(content) if length_present else None,
+                fragment=content,
+                payload=payload,
+            )
+
         content_type_ = cast('Enum_ContentType', self._make_index(
             content_type, content_type_default, namespace=content_type_namespace,
             reversed=content_type_reversed, pack=False))
@@ -193,6 +262,29 @@ def make(self,
             payload=payload,
         )
 
+    def unpack(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS | Data_DTLSCiphertext':
+        """Unpack (parse) packet data, with the header schema its first octet selects.
+
+        A unified-header record (:rfc:`9147#section-4.1`) is parsed by
+        :class:`~pcapkit.protocols.schema.application.dtls.DTLSCiphertext`
+        rather than the legacy header; every other record uses the inherited
+        behaviour.
+
+        Args:
+            length: Length of packet data.
+            **kwargs: Arbitrary keyword arguments.
+
+        Returns:
+            Parsed packet data.
+
+        """
+        octets = self._data[:1]
+        if self.__header__ is None and octets and _is_unified_header(octets[0]):
+            packet = kwargs.get('__packet__', {})  # packet data
+            self.__header__ = Schema_DTLSCiphertext.unpack(self._file, length, packet)  # type: ignore[assignment]
+            return self.read(length, **kwargs)
+        return super().unpack(length, **kwargs)
+
     ##########################################################################
     # Data models.
     ##########################################################################
@@ -206,7 +298,7 @@ def __length_hint__(self) -> 'Literal[13]':
     ##########################################################################
 
     @classmethod
-    def _make_data(cls, data: 'Data_DTLS') -> 'dict[str, Any]':  # type: ignore[override]
+    def _make_data(cls, data: 'Data_DTLS | Data_DTLSCiphertext') -> 'dict[str, Any]':  # type: ignore[override]
         """Create key-value pairs from ``data`` for protocol construction.
 
         Args:
@@ -216,6 +308,16 @@ def _make_data(cls, data: 'Data_DTLS') -> 'dict[str, Any]':  # type: ignore[over
             Key-value pairs for protocol construction.
 
         """
+        if isinstance(data, Data_DTLSCiphertext):
+            return {
+                'ciphertext': True,
+                'epoch': data.epoch,
+                'sequence_number': data.sequence_number,
+                'sequence_long': data.sequence_long,
+                'length_present': data.length is not None,
+                'content': data.content,
+                'payload': cls._make_payload(data),
+            }
         return {
             'content_type': data.content_type,
             'version': data.version,
@@ -225,9 +327,11 @@ def _make_data(cls, data: 'Data_DTLS') -> 'dict[str, Any]':  # type: ignore[over
             'payload': cls._make_payload(data),
         }
 
-    def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS':  # pylint: disable=unused-argument
+    def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLS | Data_DTLSCiphertext':  # pylint: disable=unused-argument
         """Read a DTLS record, identifying its concrete tier as it goes.
 
+        A unified-header record is handed to :meth:`_read_dtls_ciphertext`.
+
         Structure of DTLS record layer [:rfc:`9147#section-4`]:
 
         .. code-block:: text
@@ -262,6 +366,9 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') ->
                 malformed for its content type.
 
         """
+        if isinstance(self.__header__, Schema_DTLSCiphertext):
+            return self._read_dtls_ciphertext(length, **kwargs)
+
         schema = self.__header__
         available = length if length is not None else len(self)
         self._validate_tls_record(schema, available=available, dtls=True)
@@ -290,3 +397,46 @@ def _read_dtls_record(self, length: 'Optional[int]' = None, **kwargs: 'Any') ->
 
         remaining = available - (self.length + schema.length)
         return self._decode_next_layer(dtls, -1, remaining)
+
+    def _read_dtls_ciphertext(self, length: 'Optional[int]' = None, **kwargs: 'Any') -> 'Data_DTLSCiphertext':  # pylint: disable=unused-argument
+        """Read a unified-header ciphertext record (:rfc:`9147#section-4.1`).
+
+        Args:
+            length: Length of packet data.
+            **kwargs: Arbitrary keyword arguments.
+
+        Returns:
+            Parsed packet data.
+
+        Raises:
+            ProtocolError: If the header carries a connection ID, whose length
+                is negotiated and not in the record; if the header or its
+                declared length runs past the buffer; or if the ciphertext is
+                shorter than the record-number sample (:rfc:`9147#section-4.2.3`).
+
+        """
+        self._alias = 'DTLS 1.3'
+        schema = self.__header__
+        flags = schema.flags
+        available = length if length is not None else len(self)
+
+        if flags['connection_id']:
+            raise ProtocolError(f'{self.alias}: invalid format: connection ID records are not decoded')
+        needed = self.length + (schema.length or 0)
+        if needed > available:
+            raise ProtocolError(f'{self.alias}: invalid format: record of {needed} octets '
+                                'runs past the end of the buffer')
+        # Receivers MUST reject shorter records as if they failed deprotection.
+        if len(schema.fragment or b'') < _CIPHERTEXT_MIN_LENGTH:
+            raise ProtocolError(f'{self.alias}: invalid format: ciphertext is shorter than the '
+                                'record-number sample (RFC 9147 section 4.2.3)')
+
+        dtls = Data_DTLSCiphertext(
+            epoch=flags['epoch'],
+            sequence_number=schema.sequence_number,
+            sequence_long=bool(flags['sequence_long']),
+            length=schema.length,
+            content=schema.fragment,
+        )
+        remaining = available - (self.length + len(schema.fragment))
+        return self._decode_next_layer(dtls, -1, remaining)