refactor(vendor): keep the vendor variables in pcapkit.vendor (#1726)

.github/workflows/unit-tests.yml

@@ -1706,7 +1706,7 @@ jobs:
 
       - name: Run the other tests that read docs/ or Markdown
         timeout-minutes: 5
-        run: python -m unittest -v tests.corekit.test_sentinel_exports_unit tests.utilities.test_config_unit
+        run: python -m unittest -v tests.corekit.test_sentinel_exports_unit tests.utilities.test_config_unit tests.vendor.test_vendor_variables_1726_unit
 
   # The verdict a shipping workflow asks for: the full suite, fixtures and
   # all, on one interpreter. It is deliberately not the matrix -- the matrix

docs/source/pcapkit/index.rst

@@ -148,16 +148,16 @@ Environment Variables
 
       :data:`pcapkit.utilities.logging.SPHINX_TYPE_CHECKING`
 
-The tunable limits, and the :mod:`pcapkit.vendor` crawlers'
-:envvar:`PCAPKIT_VENDOR_RETRY`, :envvar:`PCAPKIT_CI_MODE`,
-:envvar:`PCAPKIT_HTTP_PROXY` and :envvar:`PCAPKIT_HTTPS_PROXY`, are read by
-:mod:`pcapkit.utilities.config`, which lists each with its default, its
-accepted range and its risks.
+The tunable limits are read by :mod:`pcapkit.utilities.config`, which lists each
+with its default, its accepted range and its risks. The :mod:`pcapkit.vendor`
+crawlers' :envvar:`PCAPKIT_VENDOR_RETRY`, :envvar:`PCAPKIT_CI_MODE`,
+:envvar:`PCAPKIT_HTTP_PROXY` and :envvar:`PCAPKIT_HTTPS_PROXY` are read by
+:mod:`pcapkit.vendor.default`, and listed under :ref:`vendor-variables`.
 
 .. versionchanged:: 1.5.0
 
    :envvar:`PCAPKIT_HTTP_PROXY` and :envvar:`PCAPKIT_HTTPS_PROXY` are documented
-   with :mod:`pcapkit.utilities.config` rather than here, and are still read on
+   with :mod:`pcapkit.vendor.default` rather than here, and are still read on
    every request. The tunable limits are new as variables (:issue:`1726`). A
    bad value of :envvar:`PCAPKIT_DEVMODE`, :envvar:`PCAPKIT_VERBOSE`,
    :envvar:`PCAPKIT_SPHINX`, :envvar:`PCAPKIT_CI_MODE` or

docs/source/pcapkit/utilities/config.rst

@@ -3,10 +3,9 @@ Configuration
 
 .. module:: pcapkit.utilities.config
 
-:mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit`, and
-the settings of the :mod:`pcapkit.vendor` crawlers (:issue:`1726`). Each limit
-is read from a ``PCAPKIT_*`` environment variable once, when the module is
-imported:
+:mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit`
+(:issue:`1726`). Each limit is read from a ``PCAPKIT_*`` environment variable
+once, when the module is imported:
 
 * A value that is not an integer, or is below the variable's minimum, warns with
   a :exc:`~pcapkit.utilities.warnings.ConfigWarning` and falls back to the
@@ -22,9 +21,13 @@ limit; using the default, 10000``. It obeys the warning filters, so under
 ``import pcapkit`` raise :exc:`~pcapkit.utilities.warnings.ConfigWarning`.
 
 Readers look each value up as ``config.<NAME>`` at call time, so patching a name
-here reaches every engine and protocol that reads it. The proxy addresses are
-read on every call of :func:`~pcapkit.utilities.config.http_proxy` and
-:func:`~pcapkit.utilities.config.https_proxy` instead.
+here reaches every engine and protocol that reads it.
+
+The :mod:`pcapkit.vendor` crawlers' :envvar:`PCAPKIT_VENDOR_RETRY`,
+:envvar:`PCAPKIT_CI_MODE`, :envvar:`PCAPKIT_HTTP_PROXY` and
+:envvar:`PCAPKIT_HTTPS_PROXY` are not held here. They are read by
+:mod:`pcapkit.vendor.default`, the first two with the :ref:`readers
+<config-readers>` below, and documented with it under :ref:`vendor-variables`.
 
 :envvar:`PCAPKIT_DEVMODE`, :envvar:`PCAPKIT_VERBOSE` and :envvar:`PCAPKIT_SPHINX`
 are read by :mod:`pcapkit.utilities.logging`, which needs them while it loads
@@ -130,31 +133,13 @@ Limits
 .. autodata:: pcapkit.utilities.config.MAX_ZERO_PAD_SHORTFALL
 .. autodata:: pcapkit.utilities.config.ZERO_PAD_BUDGET_RATIO
 
-Vendor Crawlers
----------------
-
-.. envvar:: PCAPKIT_VENDOR_RETRY
-
-   Sets :data:`~pcapkit.utilities.config.VENDOR_RETRY`: an integer ``>= 1``,
-   default ``5``.
-
-.. envvar:: PCAPKIT_CI_MODE
-
-   Sets :data:`~pcapkit.utilities.config.CI_MODE`: ``1``/``yes``/``true``/``on``
-   or ``0``/``no``/``false``/``off``, default off.
-
-.. envvar:: PCAPKIT_HTTP_PROXY
-
-   The HTTP proxy address for the :mod:`pcapkit.vendor` crawlers, read by
-   :func:`~pcapkit.utilities.config.http_proxy` on every call. Unset by default.
+.. _config-readers:
 
-.. envvar:: PCAPKIT_HTTPS_PROXY
+Readers
+-------
 
-   The HTTPS proxy address for the :mod:`pcapkit.vendor` crawlers, read by
-   :func:`~pcapkit.utilities.config.https_proxy` on every call. Unset by
-   default.
+The limits above are read with these, and a module that keeps a variable of its
+own reads it with them too, so that a bad value is reported the same way.
 
-.. autodata:: pcapkit.utilities.config.VENDOR_RETRY
-.. autodata:: pcapkit.utilities.config.CI_MODE
-.. autofunction:: pcapkit.utilities.config.http_proxy
-.. autofunction:: pcapkit.utilities.config.https_proxy
+.. autofunction:: pcapkit.utilities.config.read_limit
+.. autofunction:: pcapkit.utilities.config.read_flag

docs/source/pcapkit/vendor/default.rst

@@ -6,17 +6,6 @@ Base Crawler
 :mod:`pcapkit.vendor.default` contains :class:`~pcapkit.vendor.default.Vendor`
 only, which is the base meta class for all vendor crawlers.
 
-.. versionremoved:: 1.5.0
-
-   Moved to :mod:`pcapkit.utilities.config` (:issue:`1726`):
-
-   * ``pcapkit.vendor.default.MAX_RETRY``, now
-     :data:`pcapkit.utilities.config.VENDOR_RETRY`, set with
-     :envvar:`PCAPKIT_VENDOR_RETRY`;
-   * ``pcapkit.vendor.default.CI_MODE``, now
-     :data:`pcapkit.utilities.config.CI_MODE`, set with
-     :envvar:`PCAPKIT_CI_MODE`.
-
 .. autoclass:: pcapkit.vendor.default.Vendor
    :no-members:
    :show-inheritance:
@@ -40,6 +29,44 @@ only, which is the base meta class for all vendor crawlers.
    .. automethod:: _request
    .. automethod:: _dest_path
 
+.. _vendor-variables:
+
+Environment Variables
+---------------------
+
+The crawlers' variables are read here, not by :mod:`pcapkit.utilities.config`
+(:issue:`1726`). :envvar:`PCAPKIT_VENDOR_RETRY` and :envvar:`PCAPKIT_CI_MODE`
+are read once, when this module is imported, with
+:func:`~pcapkit.utilities.config.read_limit` and
+:func:`~pcapkit.utilities.config.read_flag`: a bad value warns with a
+:exc:`~pcapkit.utilities.warnings.ConfigWarning`, naming the variable and the
+value, and falls back to the default. The proxy addresses are read by
+:func:`~pcapkit.vendor.default.get_proxies` on every call, so a proxy set after
+the import is still honoured.
+
+.. envvar:: PCAPKIT_VENDOR_RETRY
+
+   Sets :data:`~pcapkit.vendor.default.MAX_RETRY`: an integer ``>= 1``, default
+   ``5``.
+
+.. envvar:: PCAPKIT_CI_MODE
+
+   Sets :data:`~pcapkit.vendor.default.CI_MODE`: ``1``/``yes``/``true``/``on``
+   or ``0``/``no``/``false``/``off``, default off.
+
+.. envvar:: PCAPKIT_HTTP_PROXY
+
+   The HTTP proxy address for the crawlers, read by
+   :func:`~pcapkit.vendor.default.get_proxies` on every call. Unset by default.
+
+.. envvar:: PCAPKIT_HTTPS_PROXY
+
+   The HTTPS proxy address for the crawlers, read by
+   :func:`~pcapkit.vendor.default.get_proxies` on every call. Unset by default.
+
+.. autodata:: pcapkit.vendor.default.MAX_RETRY
+.. autodata:: pcapkit.vendor.default.CI_MODE
+
 Internal Definitions
 --------------------
 

pcapkit/utilities/config.py

@@ -4,10 +4,9 @@
 
 .. module:: pcapkit.utilities.config
 
-:mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit`,
-and the settings of the :mod:`pcapkit.vendor` crawlers (:issue:`1726`). Each
-limit is read from a ``PCAPKIT_*`` environment variable once, when this module
-is imported, and validated there:
+:mod:`pcapkit.utilities.config` holds every tunable limit of :mod:`pcapkit`
+(:issue:`1726`). Each limit is read from a ``PCAPKIT_*`` environment variable
+once, when this module is imported, and validated there:
 
 * A value that is not an integer, or is below the variable's minimum, is
   reported with a :exc:`~pcapkit.utilities.warnings.ConfigWarning`, and the
@@ -24,8 +23,12 @@
 
 Every reader looks the value up here at call time, as ``config.<NAME>``, so
 patching one name in this module reaches every engine and every protocol that
-reads it. The proxy addresses are the exception: :func:`http_proxy` and
-:func:`https_proxy` read their variables on every call.
+reads it.
+
+:func:`read_limit` and :func:`read_flag` read and validate the variables, and
+are public so that a module keeping variables of its own validates them the same
+way: :mod:`pcapkit.vendor.default` reads the crawlers' variables with them, and
+documents those variables itself.
 
 Besides the standard library, this module imports
 :mod:`pcapkit.utilities.warnings` and nothing else from :mod:`pcapkit`, which is
@@ -50,8 +53,8 @@
     'FRAME_LAYER_LIMIT', 'EXTENSION_HEADER_LIMIT', 'IPV6_TCP_WARNED_LIMIT',
     'FALLBACK_LIMIT', 'DETECT_CACHE_SIZE', 'READ_CHUNK',
     'MAX_ZERO_PAD_LENGTH', 'MAX_ZERO_PAD_SHORTFALL', 'ZERO_PAD_BUDGET_RATIO',
-    # vendor crawlers
-    'VENDOR_RETRY', 'CI_MODE', 'http_proxy', 'https_proxy',
+    # readers
+    'read_limit', 'read_flag',
 ]
 
 #: Strings a boolean variable may take, case-insensitively.
@@ -65,10 +68,14 @@
                    "frame may hit Python's recursion limit and raise RecursionError (#1610)")
 
 
-def _read_limit(name: 'str', default: 'int', *, minimum: 'int' = 0,
+def read_limit(name: 'str', default: 'int', *, minimum: 'int' = 0,
                risk: 'Optional[str]' = None) -> 'int':
     """Read the integer environment variable ``PCAPKIT_<name>``.
 
+    A value that is not an integer, or is below ``minimum``, is reported with a
+    :exc:`~pcapkit.utilities.warnings.ConfigWarning` naming the variable and the
+    value, and ``default`` is used instead.
+
     Args:
         name: Variable name, without the ``PCAPKIT_`` prefix.
         default: Value used when the variable is unset or invalid.
@@ -80,6 +87,8 @@ def _read_limit(name: 'str', default: 'int', *, minimum: 'int' = 0,
     Returns:
         The value of the variable, or ``default``.
 
+    .. versionadded:: 1.5.0
+
     """
     variable = f'PCAPKIT_{name}'
     text = os.environ.get(variable)
@@ -103,16 +112,24 @@ def _read_limit(name: 'str', default: 'int', *, minimum: 'int' = 0,
     return value
 
 
-def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
+def read_flag(name: 'str', default: 'bool' = False) -> 'bool':
     """Read the boolean environment variable ``PCAPKIT_<name>``.
 
+    The value is read with surrounding whitespace stripped and
+    case-insensitively, and is one of ``1``, ``yes``, ``true`` or ``on``, or of
+    ``0``, ``no``, ``false`` or ``off``. Any other value is reported with a
+    :exc:`~pcapkit.utilities.warnings.ConfigWarning` naming the variable and the
+    value, and ``default`` is used instead.
+
     Args:
         name: Variable name, without the ``PCAPKIT_`` prefix.
         default: Value used when the variable is unset or invalid.
 
     Returns:
         The value of the variable, or ``default``.
 
+    .. versionadded:: 1.5.0
+
     """
     variable = f'PCAPKIT_{name}'
     text = os.environ.get(variable)
@@ -153,7 +170,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.protocols.protocol.FRAME_LAYER_LIMIT`` (:issue:`1726`).
-FRAME_LAYER_LIMIT = _read_limit('FRAME_LAYER_LIMIT', 64, risk=_RECURSION_RISK)
+FRAME_LAYER_LIMIT = read_limit('FRAME_LAYER_LIMIT', 64, risk=_RECURSION_RISK)
 
 #: int: Most IPv6 extension headers dissected in one chain, i.e. one after
 #: another with no other layer between them (:issue:`1604`),
@@ -185,7 +202,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.protocols.internet.internet.EXTENSION_HEADER_LIMIT`` (:issue:`1726`).
-EXTENSION_HEADER_LIMIT = _read_limit('EXTENSION_HEADER_LIMIT', 32, risk=_RECURSION_RISK)
+EXTENSION_HEADER_LIMIT = read_limit('EXTENSION_HEADER_LIMIT', 32, risk=_RECURSION_RISK)
 
 #: int: How many captures the ``pypcapfile`` engine remembers having warned
 #: about for TCP over IPv6, :envvar:`PCAPKIT_IPV6_TCP_WARNED_LIMIT`.
@@ -197,7 +214,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.toolkit.pypcapfile.IPV6_TCP_WARNED_LIMIT`` (:issue:`1726`).
-IPV6_TCP_WARNED_LIMIT = _read_limit('IPV6_TCP_WARNED_LIMIT', 32, risk=(
+IPV6_TCP_WARNED_LIMIT = read_limit('IPV6_TCP_WARNED_LIMIT', 32, risk=(
     'each capture remembered pins one of its packets in memory'))
 
 #: int: Most objects :meth:`DictDumper._append_fallback
@@ -212,7 +229,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.dumpkit.common.FALLBACK_LIMIT`` (:issue:`1726`).
-FALLBACK_LIMIT = _read_limit('FALLBACK_LIMIT', 10_000, risk=(
+FALLBACK_LIMIT = read_limit('FALLBACK_LIMIT', 10_000, risk=(
     'an object that expands into a large or endless object graph is written out '
     'in full, which takes memory and time without bound'))
 
@@ -227,7 +244,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.utilities.chardet.DETECT_CACHE_SIZE`` (:issue:`1726`).
-DETECT_CACHE_SIZE = _read_limit('DETECT_CACHE_SIZE', 1024, risk=(
+DETECT_CACHE_SIZE = read_limit('DETECT_CACHE_SIZE', 1024, risk=(
     'the cache keeps every bytestring it was asked about, so a capture full of '
     'never-repeating text is retained in memory'))
 
@@ -241,7 +258,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.corekit.io._READ_CHUNK`` (:issue:`1726`).
-READ_CHUNK = _read_limit('READ_CHUNK', 1 << 20, minimum=1, risk=(
+READ_CHUNK = read_limit('READ_CHUNK', 1 << 20, minimum=1, risk=(
     'a bogus record length allocates up to that many octets in one read'))
 
 #: int: Ceiling on the zero-padding :meth:`FieldBase.unpack
@@ -279,7 +296,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.corekit.fields.field._MAX_ZERO_PAD_LENGTH`` (:issue:`1726`).
-MAX_ZERO_PAD_LENGTH = _read_limit('MAX_ZERO_PAD_LENGTH', 0x40_000, risk=(
+MAX_ZERO_PAD_LENGTH = read_limit('MAX_ZERO_PAD_LENGTH', 0x40_000, risk=(
     'a hostile capture can declare a field that long and have it zero-filled in memory'))
 
 #: int: Shortfall :meth:`FieldBase.unpack
@@ -330,7 +347,7 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.corekit.fields.field._MAX_ZERO_PAD_SHORTFALL`` (:issue:`1726`).
-MAX_ZERO_PAD_SHORTFALL = _read_limit('MAX_ZERO_PAD_SHORTFALL', 0x10_000, risk=(
+MAX_ZERO_PAD_SHORTFALL = read_limit('MAX_ZERO_PAD_SHORTFALL', 0x10_000, risk=(
     'shortfalls up to that size are zero-filled without being charged to the running budget, '
     'so a capture of many lying lengths can amplify into that much memory each'))
 
@@ -360,60 +377,6 @@ def _read_flag(name: 'str', default: 'bool' = False) -> 'bool':
 #: .. versionadded:: 1.5.0
 #:
 #:    Moved from ``pcapkit.corekit.fields.field._ZERO_PAD_BUDGET_RATIO`` (:issue:`1726`).
-ZERO_PAD_BUDGET_RATIO = _read_limit('ZERO_PAD_BUDGET_RATIO', 0x10, risk=(
+ZERO_PAD_BUDGET_RATIO = read_limit('ZERO_PAD_BUDGET_RATIO', 0x10, risk=(
     'a capture whose blocks all lie about their lengths can amplify each octet read '
     'into that many zero octets in memory'))
-
-###############################################################################
-# Vendor Crawlers
-###############################################################################
-
-#: int: How many times a :mod:`pcapkit.vendor` crawler retries a failed request,
-#: :envvar:`PCAPKIT_VENDOR_RETRY`.
-#:
-#: .. versionadded:: 1.5.0
-#:
-#:    Moved from ``pcapkit.vendor.default.MAX_RETRY`` (:issue:`1726`), which
-#:    crashed the import on a value that was not an integer, and read ``0`` as
-#:    ``1``.
-VENDOR_RETRY = _read_limit('VENDOR_RETRY', 5, minimum=1)
-
-#: bool: Whether a :mod:`pcapkit.vendor` crawler runs in CI, where a failed
-#: fetch fails the run instead of prompting, :envvar:`PCAPKIT_CI_MODE`.
-#:
-#: .. versionadded:: 1.5.0
-#:
-#:    Moved from ``pcapkit.vendor.default.CI_MODE`` (:issue:`1726`).
-CI_MODE = _read_flag('CI_MODE')
-
-
-def http_proxy() -> 'Optional[str]':
-    """HTTP proxy address for the :mod:`pcapkit.vendor` crawlers.
-
-    Unlike the limits, :envvar:`PCAPKIT_HTTP_PROXY` is read on every call, as
-    :func:`~pcapkit.vendor.default.get_proxies` always has, so a proxy set after
-    :mod:`pcapkit` is imported is still honoured.
-
-    Returns:
-        The value of :envvar:`PCAPKIT_HTTP_PROXY`, or :data:`None` if unset.
-
-    .. versionadded:: 1.5.0
-
-    """
-    return os.environ.get('PCAPKIT_HTTP_PROXY')
-
-
-def https_proxy() -> 'Optional[str]':
-    """HTTPS proxy address for the :mod:`pcapkit.vendor` crawlers.
-
-    Unlike the limits, :envvar:`PCAPKIT_HTTPS_PROXY` is read on every call, as
-    :func:`~pcapkit.vendor.default.get_proxies` always has, so a proxy set after
-    :mod:`pcapkit` is imported is still honoured.
-
-    Returns:
-        The value of :envvar:`PCAPKIT_HTTPS_PROXY`, or :data:`None` if unset.
-
-    .. versionadded:: 1.5.0
-
-    """
-    return os.environ.get('PCAPKIT_HTTPS_PROXY')

pcapkit/utilities/logging.py

@@ -75,7 +75,7 @@ def _read_flag(variable: 'str') -> 'bool':
     """Read the boolean environment variable ``variable``, off by default.
 
     The value is read with surrounding whitespace stripped and
-    case-insensitively, as :func:`pcapkit.utilities.config._read_flag` reads
+    case-insensitively, as :func:`pcapkit.utilities.config.read_flag` reads
     one. A value that is not then one of :data:`BOOLEAN_STATES` is taken as off
     and recorded in :data:`_BAD_FLAGS`.