feat(utilities)!: one config module for the tunable limits (#1726)

.github/workflows/unit-tests.yml

@@ -1710,7 +1710,7 @@ jobs:
 
       - name: Run the other tests that read docs/ or Markdown
         timeout-minutes: 5
-        run: python -m unittest -v tests.corekit.test_sentinel_exports_unit
+        run: python -m unittest -v tests.corekit.test_sentinel_exports_unit tests.utilities.test_config_unit
 
   # The verdict a shipping workflow asks for: the full suite, fixtures and
   # all, on one interpreter. It is deliberately not the matrix -- the matrix

docs/source/contributing/pep.rst

@@ -428,9 +428,9 @@ of the frame raw, silently, at a depth that varied with the interpreter and the
 caller's stack (:issue:`1604`, :issue:`1610`).
 
 Recursion stays, with a counted bound:
-:data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` caps a frame at 64 layers,
-and :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT` caps one
-chain of extension headers at 32. Past either, the rest stays
+:data:`~pcapkit.utilities.config.FRAME_LAYER_LIMIT` caps a frame at 64 layers by
+default, and :data:`~pcapkit.utilities.config.EXTENSION_HEADER_LIMIT` caps one
+chain of extension headers at 32 by default. Past either, the rest stays
 :class:`~pcapkit.protocols.misc.raw.Raw`, with one
 :exc:`~pcapkit.utilities.warnings.ProtocolWarning`. Raising
 :func:`sys.setrecursionlimit` instead was rejected: it is process-wide, and the

docs/source/pcapkit/corekit/fields/field.rst

@@ -3,6 +3,20 @@ Base Fields
 
 .. module:: pcapkit.corekit.fields.field
 
+.. versionremoved:: 1.5.0
+
+   Moved to :mod:`pcapkit.utilities.config` (:issue:`1726`):
+
+   * ``pcapkit.corekit.fields.field._MAX_ZERO_PAD_LENGTH``, now
+     :data:`pcapkit.utilities.config.MAX_ZERO_PAD_LENGTH`, set with
+     :envvar:`PCAPKIT_MAX_ZERO_PAD_LENGTH`;
+   * ``pcapkit.corekit.fields.field._MAX_ZERO_PAD_SHORTFALL``, now
+     :data:`pcapkit.utilities.config.MAX_ZERO_PAD_SHORTFALL`, set with
+     :envvar:`PCAPKIT_MAX_ZERO_PAD_SHORTFALL`;
+   * ``pcapkit.corekit.fields.field._ZERO_PAD_BUDGET_RATIO``, now
+     :data:`pcapkit.utilities.config.ZERO_PAD_BUDGET_RATIO`, set with
+     :envvar:`PCAPKIT_ZERO_PAD_BUDGET_RATIO`.
+
 .. autoclass:: pcapkit.corekit.fields.field.Field
    :members:
    :show-inheritance:

docs/source/pcapkit/corekit/io.rst

@@ -10,6 +10,12 @@ I/O Objects
 stream, and :class:`~pcapkit.corekit.io.NamedStream`, which gives a stream a
 ``name``.
 
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.corekit.io._READ_CHUNK``, now
+   :data:`pcapkit.utilities.config.READ_CHUNK`, set with
+   :envvar:`PCAPKIT_READ_CHUNK` (:issue:`1726`).
+
 .. autoclass:: pcapkit.corekit.io.SeekableReader
    :no-members:
    :show-inheritance:

docs/source/pcapkit/dumpkit/common.rst

@@ -7,6 +7,12 @@ Common Utilities
 :mod:`pcapkit.dumpkit`, chiefly customised hooks for :class:`dictdumper.Dumper`
 classes.
 
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.dumpkit.common.FALLBACK_LIMIT``, now
+   :data:`pcapkit.utilities.config.FALLBACK_LIMIT`, set with
+   :envvar:`PCAPKIT_FALLBACK_LIMIT` (:issue:`1726`).
+
 .. autofunction:: pcapkit.dumpkit.common.make_dumper
 
 .. autoclass:: pcapkit.dumpkit.common.Dumper

docs/source/pcapkit/index.rst

@@ -139,14 +139,6 @@ Environment Variables
 
       :data:`pcapkit.utilities.logging.VERBOSE`
 
-.. envvar:: PCAPKIT_HTTP_PROXY
-
-   HTTP proxy address for :mod:`pcapkit.vendor` crawlers.
-
-.. envvar:: PCAPKIT_HTTPS_PROXY
-
-   HTTPS proxy address for :mod:`pcapkit.vendor` crawlers.
-
 .. envvar:: PCAPKIT_SPHINX
 
    If set to ``1``, :mod:`pcapkit` will run with Sphinx additional
@@ -155,3 +147,20 @@ Environment Variables
    .. seealso::
 
       :data:`pcapkit.utilities.logging.SPHINX_TYPE_CHECKING`
+
+The tunable limits, and the :mod:`pcapkit.vendor` crawlers'
+:envvar:`PCAPKIT_VENDOR_RETRY`, :envvar:`PCAPKIT_CI_MODE`,
+:envvar:`PCAPKIT_HTTP_PROXY` and :envvar:`PCAPKIT_HTTPS_PROXY`, are read by
+:mod:`pcapkit.utilities.config`, which lists each with its default, its
+accepted range and its risks.
+
+.. versionchanged:: 1.5.0
+
+   :envvar:`PCAPKIT_HTTP_PROXY` and :envvar:`PCAPKIT_HTTPS_PROXY` are documented
+   with :mod:`pcapkit.utilities.config` rather than here, and are still read on
+   every request. The tunable limits are new as variables (:issue:`1726`). A
+   bad value of :envvar:`PCAPKIT_DEVMODE`, :envvar:`PCAPKIT_VERBOSE`,
+   :envvar:`PCAPKIT_SPHINX`, :envvar:`PCAPKIT_CI_MODE` or
+   :envvar:`PCAPKIT_VENDOR_RETRY` now warns with
+   :exc:`~pcapkit.utilities.warnings.ConfigWarning` instead of being taken
+   silently, or, for :envvar:`PCAPKIT_VENDOR_RETRY`, crashing the import.

docs/source/pcapkit/protocols/internet/internet.rst

@@ -33,7 +33,7 @@ Extension Header Chains
 -----------------------
 
 A chain of IPv6 extension headers is dissected to at most
-:data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT` headers in
+:data:`~pcapkit.utilities.config.EXTENSION_HEADER_LIMIT` headers in
 a row, whether it follows IPv6 or another layer such as IPv4. What follows is
 kept as :class:`~pcapkit.protocols.misc.raw.Raw`, with one
 :exc:`~pcapkit.utilities.warnings.ProtocolWarning` (:issue:`1604`). The
@@ -42,8 +42,12 @@ so it may still find the upper layer past it (:issue:`1609`). So may the
 ``dpkt`` toolkit, which reads the chain as :mod:`dpkt` decodes it. So does the
 ``scapy`` engine behind IPv6, where Scapy dissects each extension header itself;
 behind IPv4, its toolkit steps over them in the octets to find TCP, up to
-:data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT` of them in a
-row. The ``pypcapfile`` toolkit walks no extension header chain, so
-it finds no upper layer behind one.
+:data:`~pcapkit.utilities.config.EXTENSION_HEADER_LIMIT` of them in a row. The
+``pypcapfile`` toolkit walks no extension header chain, so it finds no upper
+layer behind one.
 
-.. autodata:: pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT``, now
+   :data:`pcapkit.utilities.config.EXTENSION_HEADER_LIMIT`, set with
+   :envvar:`PCAPKIT_EXTENSION_HEADER_LIMIT` (:issue:`1726`).

docs/source/pcapkit/protocols/protocol.rst

@@ -105,7 +105,7 @@ Layers per Frame
 ----------------
 
 A frame is dissected to at most
-:data:`~pcapkit.protocols.protocol.FRAME_LAYER_LIMIT` layers of its protocol
+:data:`~pcapkit.utilities.config.FRAME_LAYER_LIMIT` layers of its protocol
 chain, counted from the first layer the frame record carries, through tunnels,
 VLAN tags and IPv6 extension headers alike. What follows is kept as
 :class:`~pcapkit.protocols.misc.raw.Raw`, with one
@@ -115,12 +115,16 @@ default engine's TCP reassembly and flow tracing do not read a TCP segment
 there back out, as they do one the TCP parser rejected (:issue:`1518`). A
 protocol constructed from its own octets is not counted itself, so the count
 starts at the first layer it dissects. One chain of extension headers is also bounded on
-its own, by :data:`~pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT`.
+its own, by :data:`~pcapkit.utilities.config.EXTENSION_HEADER_LIMIT`.
 The ``pyshark``, ``scapy`` and ``pypcapfile`` engines dissect with their own
 parsers, which do not apply the frame's layer bound (:issue:`1609`). The
 ``dpkt`` toolkit reads what :mod:`dpkt` decodes, which applies neither bound.
 
-.. autodata:: pcapkit.protocols.protocol.FRAME_LAYER_LIMIT
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.protocols.protocol.FRAME_LAYER_LIMIT``, now
+   :data:`pcapkit.utilities.config.FRAME_LAYER_LIMIT`, set with
+   :envvar:`PCAPKIT_FRAME_LAYER_LIMIT` (:issue:`1726`).
 
 Data Models
 -----------

docs/source/pcapkit/toolkit/3rdparty.rst

@@ -182,6 +182,12 @@ cannot be used for it.
 
 .. _PyPCAPFile: https://github.com/kisom/pypcapfile
 
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.toolkit.pypcapfile.IPV6_TCP_WARNED_LIMIT``, now
+   :data:`pcapkit.utilities.config.IPV6_TCP_WARNED_LIMIT`, set with
+   :envvar:`PCAPKIT_IPV6_TCP_WARNED_LIMIT` (:issue:`1726`).
+
 .. note::
 
    `PyPCAPFile`_ has no IPv6 decoder, so :func:`~pcapkit.toolkit.pypcapfile.ipv6_reassembly`

docs/source/pcapkit/utilities/chardet.rst

@@ -14,4 +14,10 @@ which is why it lives here rather than beside either of them.
 
 .. autofunction:: pcapkit.utilities.chardet.detect
 
-.. autodata:: pcapkit.utilities.chardet.DETECT_CACHE_SIZE
+The cache holds :data:`~pcapkit.utilities.config.DETECT_CACHE_SIZE` entries.
+
+.. versionremoved:: 1.5.0
+
+   ``pcapkit.utilities.chardet.DETECT_CACHE_SIZE``, now
+   :data:`pcapkit.utilities.config.DETECT_CACHE_SIZE`, set with
+   :envvar:`PCAPKIT_DETECT_CACHE_SIZE` (:issue:`1726`).

docs/source/pcapkit/utilities/config.rst

@@ -0,0 +1,160 @@
+Configuration
+=============
+
+.. module:: pcapkit.utilities.config
+
+:mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit`, and
+the settings of the :mod:`pcapkit.vendor` crawlers (:issue:`1726`). Each limit
+is read from a ``PCAPKIT_*`` environment variable once, when the module is
+imported:
+
+* A value that is not an integer, or is below the variable's minimum, warns with
+  a :exc:`~pcapkit.utilities.warnings.ConfigWarning` and falls back to the
+  default.
+* A limit takes ``-1`` for no limit. That, or a value above the default, is
+  accepted and warns with a :exc:`~pcapkit.utilities.warnings.ConfigWarning`
+  naming the risk below.
+
+Each warning names the variable and the value it was given, e.g.
+``PCAPKIT_FALLBACK_LIMIT='-7' is not an integer of at least 0, or -1 for no
+limit; using the default, 10000``. It obeys the warning filters, so under
+:option:`-W error <-W>` or ``PYTHONWARNINGS=error`` an out-of-range value makes
+``import pcapkit`` raise :exc:`~pcapkit.utilities.warnings.ConfigWarning`.
+
+Readers look each value up as ``config.<NAME>`` at call time, so patching a name
+here reaches every engine and protocol that reads it. The proxy addresses are
+read on every call of :func:`~pcapkit.utilities.config.http_proxy` and
+:func:`~pcapkit.utilities.config.https_proxy` instead.
+
+:envvar:`PCAPKIT_DEVMODE`, :envvar:`PCAPKIT_VERBOSE` and :envvar:`PCAPKIT_SPHINX`
+are read by :mod:`pcapkit.utilities.logging`, which needs them while it loads
+and so cannot import this module. A bad value of one falls back to off, and is
+reported with a :exc:`~pcapkit.utilities.warnings.ConfigWarning` once
+:mod:`pcapkit.utilities.warnings` has loaded.
+
+.. versionadded:: 1.5.0
+
+Limits
+------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Variable
+     - Default
+     - Accepted
+     - Risk past the default, or at ``-1``
+   * - :envvar:`PCAPKIT_FRAME_LAYER_LIMIT`
+     - ``64``
+     - ``>= 0``, or ``-1``
+     - Python's recursion limit: a deep frame may raise :exc:`RecursionError`
+       (:issue:`1610`)
+   * - :envvar:`PCAPKIT_EXTENSION_HEADER_LIMIT`
+     - ``32``
+     - ``>= 0``, or ``-1``
+     - as above, once :envvar:`PCAPKIT_FRAME_LAYER_LIMIT` is raised too
+   * - :envvar:`PCAPKIT_IPV6_TCP_WARNED_LIMIT`
+     - ``32``
+     - ``>= 0``, or ``-1``
+     - each capture remembered pins one packet in memory
+   * - :envvar:`PCAPKIT_FALLBACK_LIMIT`
+     - ``10000``
+     - ``>= 0``, or ``-1``
+     - a large or endless object graph is dumped without bound
+   * - :envvar:`PCAPKIT_DETECT_CACHE_SIZE`
+     - ``1024``
+     - ``>= 0``, or ``-1``
+     - the cache retains every bytestring it was asked about
+   * - :envvar:`PCAPKIT_READ_CHUNK`
+     - ``1048576``
+     - ``>= 1``, or ``-1``
+     - a bogus record length allocates that much in one read
+   * - :envvar:`PCAPKIT_MAX_ZERO_PAD_LENGTH`
+     - ``262144``
+     - ``>= 0``, or ``-1`` to lift the ceiling and the running budget
+     - a hostile capture's declared length is zero-filled in memory
+       (:issue:`554`)
+   * - :envvar:`PCAPKIT_MAX_ZERO_PAD_SHORTFALL`
+     - ``65536``
+     - ``>= 0``, or ``-1`` to disable the running budget
+     - lying lengths amplify into memory without being charged (:issue:`573`)
+   * - :envvar:`PCAPKIT_ZERO_PAD_BUDGET_RATIO`
+     - ``16``
+     - ``>= 0``, or ``-1`` to lift the running budget
+     - as above, per octet actually read (:issue:`573`)
+
+.. envvar:: PCAPKIT_FRAME_LAYER_LIMIT
+
+   Sets :data:`~pcapkit.utilities.config.FRAME_LAYER_LIMIT`.
+
+.. envvar:: PCAPKIT_EXTENSION_HEADER_LIMIT
+
+   Sets :data:`~pcapkit.utilities.config.EXTENSION_HEADER_LIMIT`.
+
+.. envvar:: PCAPKIT_IPV6_TCP_WARNED_LIMIT
+
+   Sets :data:`~pcapkit.utilities.config.IPV6_TCP_WARNED_LIMIT`.
+
+.. envvar:: PCAPKIT_FALLBACK_LIMIT
+
+   Sets :data:`~pcapkit.utilities.config.FALLBACK_LIMIT`.
+
+.. envvar:: PCAPKIT_DETECT_CACHE_SIZE
+
+   Sets :data:`~pcapkit.utilities.config.DETECT_CACHE_SIZE`. Read once, when
+   :mod:`pcapkit.utilities.chardet` is imported.
+
+.. envvar:: PCAPKIT_READ_CHUNK
+
+   Sets :data:`~pcapkit.utilities.config.READ_CHUNK`.
+
+.. envvar:: PCAPKIT_MAX_ZERO_PAD_LENGTH
+
+   Sets :data:`~pcapkit.utilities.config.MAX_ZERO_PAD_LENGTH`.
+
+.. envvar:: PCAPKIT_MAX_ZERO_PAD_SHORTFALL
+
+   Sets :data:`~pcapkit.utilities.config.MAX_ZERO_PAD_SHORTFALL`.
+
+.. envvar:: PCAPKIT_ZERO_PAD_BUDGET_RATIO
+
+   Sets :data:`~pcapkit.utilities.config.ZERO_PAD_BUDGET_RATIO`.
+
+.. autodata:: pcapkit.utilities.config.FRAME_LAYER_LIMIT
+.. autodata:: pcapkit.utilities.config.EXTENSION_HEADER_LIMIT
+.. autodata:: pcapkit.utilities.config.IPV6_TCP_WARNED_LIMIT
+.. autodata:: pcapkit.utilities.config.FALLBACK_LIMIT
+.. autodata:: pcapkit.utilities.config.DETECT_CACHE_SIZE
+.. autodata:: pcapkit.utilities.config.READ_CHUNK
+.. autodata:: pcapkit.utilities.config.MAX_ZERO_PAD_LENGTH
+.. autodata:: pcapkit.utilities.config.MAX_ZERO_PAD_SHORTFALL
+.. autodata:: pcapkit.utilities.config.ZERO_PAD_BUDGET_RATIO
+
+Vendor Crawlers
+---------------
+
+.. envvar:: PCAPKIT_VENDOR_RETRY
+
+   Sets :data:`~pcapkit.utilities.config.VENDOR_RETRY`: an integer ``>= 1``,
+   default ``5``.
+
+.. envvar:: PCAPKIT_CI_MODE
+
+   Sets :data:`~pcapkit.utilities.config.CI_MODE`: ``1``/``yes``/``true``/``on``
+   or ``0``/``no``/``false``/``off``, default off.
+
+.. envvar:: PCAPKIT_HTTP_PROXY
+
+   The HTTP proxy address for the :mod:`pcapkit.vendor` crawlers, read by
+   :func:`~pcapkit.utilities.config.http_proxy` on every call. Unset by default.
+
+.. envvar:: PCAPKIT_HTTPS_PROXY
+
+   The HTTPS proxy address for the :mod:`pcapkit.vendor` crawlers, read by
+   :func:`~pcapkit.utilities.config.https_proxy` on every call. Unset by
+   default.
+
+.. autodata:: pcapkit.utilities.config.VENDOR_RETRY
+.. autodata:: pcapkit.utilities.config.CI_MODE
+.. autofunction:: pcapkit.utilities.config.http_proxy
+.. autofunction:: pcapkit.utilities.config.https_proxy

docs/source/pcapkit/utilities/index.rst

@@ -15,6 +15,7 @@ layer, and wrappers around third-party libraries. It imports nothing from
    exceptions
    warnings
    logging
+   config
    chardet
 
 Version Compatibility

docs/source/pcapkit/utilities/warnings.rst

@@ -166,6 +166,10 @@ configured separately, through the :mod:`logging` module::
    :no-members:
    :show-inheritance:
 
+.. autoexception:: pcapkit.utilities.warnings.ConfigWarning
+   :no-members:
+   :show-inheritance:
+
 :exc:`ResourceWarning` Category
 -------------------------------
 

docs/source/pcapkit/vendor/default.rst

@@ -6,6 +6,17 @@ Base Crawler
 :mod:`pcapkit.vendor.default` contains :class:`~pcapkit.vendor.default.Vendor`
 only, which is the base meta class for all vendor crawlers.
 
+.. versionremoved:: 1.5.0
+
+   Moved to :mod:`pcapkit.utilities.config` (:issue:`1726`):
+
+   * ``pcapkit.vendor.default.MAX_RETRY``, now
+     :data:`pcapkit.utilities.config.VENDOR_RETRY`, set with
+     :envvar:`PCAPKIT_VENDOR_RETRY`;
+   * ``pcapkit.vendor.default.CI_MODE``, now
+     :data:`pcapkit.utilities.config.CI_MODE`, set with
+     :envvar:`PCAPKIT_CI_MODE`.
+
 .. autoclass:: pcapkit.vendor.default.Vendor
    :no-members:
    :show-inheritance: