diff --git a/CHANGELOG.md b/CHANGELOG.md index 06c1494..453e755 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ diskcache-community forked from DiskCache at upstream commit ebfa37c (3 March 20 ### Docs +- Two new pages: "Fix CVE-2025-69872 in diskcache" gives the steps to read caches with `SafeDisk`, including caches that libraries create, your own classes and entries already in the cache, and lists what any restricted disk has to cover; "Migrating from diskcache 5.6.3" covers the dependency, imports, Django settings, running next to upstream `diskcache`, the behaviour changes since 5.6.3 and going back. Both open the documentation and are linked from the README (PR #27). - A new benchmarks page, measured on 2026-10-12, compares diskcache-community with diskcache 5.6.3, memcached, Redis, lmdb, sqlitedict, shelve and the Django cache backends, with four charts. The scripts that produce it are in `tests/benchmarks` (`driver.py` runs the stores, `report.py` writes the page and charts from the JSON results). The 2019 pages from upstream stay as the old benchmarks, labelled as such and linked from the new page, and the README header, features and documentation list link to the new page (PR #26). - Shared links show a project card: a social preview image in `.github/`, and Open Graph tags on the documentation through `sphinxext-opengraph` (PR #21). - A logo: the README opens with a centered header (logo, name, tagline, links, badges) in the style of the other community forks, and the documentation shows the logo and a favicon (PR #20). diff --git a/README.md b/README.md index 3f81161..a8b75c0 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,8 @@ > next to the upstream `diskcache` package. It carries the fix path for > [CVE-2025-69872](https://github.com/advisories/GHSA-w8v5-vhqr-4h9v) and > reads and writes the same cache directories as 5.6.3. -> See [Why this fork](#why-this-fork) and [Migrating from diskcache](#migrating-from-diskcache). +> See [Why this fork](#why-this-fork), [Migrating from diskcache](#migrating-from-diskcache) +> and [Fix CVE-2025-69872 in diskcache](https://diskcache-community.readthedocs.io/en/latest/fix-cve-2025-69872.html). ## Install @@ -98,6 +99,10 @@ from diskcache_community import Cache, SafeDisk cache = Cache('cache', disk=SafeDisk) ``` +[Fix CVE-2025-69872 in diskcache](https://diskcache-community.readthedocs.io/en/latest/fix-cve-2025-69872.html) covers caches +that libraries create for you, your own classes, and entries already in the +cache. + ## Features - `Cache`, `FanoutCache` and `DjangoCache` for key-value caching with @@ -156,6 +161,9 @@ In Django settings, use `'diskcache_community.DjangoCache'` and `'diskcache_community.SafeDisk'`. Existing cache directories open as they are, and a 5.6.3 process still opens directories written by the fork. +The [migration guide](https://diskcache-community.readthedocs.io/en/latest/migrating.html) lists the few behaviour changes +since 5.6.3 and how to go back. + The upstream `diskcache` package can stay installed, for example when another dependency requires it. Each library uses the package it imports. A library that wants to support both can import whichever is present: @@ -172,6 +180,8 @@ except ImportError: The documentation is at [diskcache-community.readthedocs.io](https://diskcache-community.readthedocs.io/en/latest/): +- [Fix CVE-2025-69872 in diskcache](https://diskcache-community.readthedocs.io/en/latest/fix-cve-2025-69872.html) +- [Migrating from diskcache 5.6.3](https://diskcache-community.readthedocs.io/en/latest/migrating.html) - [Tutorial](https://diskcache-community.readthedocs.io/en/latest/tutorial.html) - [API reference](https://diskcache-community.readthedocs.io/en/latest/api.html) - [Benchmarks](https://diskcache-community.readthedocs.io/en/latest/benchmarks.html) diff --git a/docs/fix-cve-2025-69872.rst b/docs/fix-cve-2025-69872.rst new file mode 100644 index 0000000..6405a17 --- /dev/null +++ b/docs/fix-cve-2025-69872.rst @@ -0,0 +1,223 @@ +Fix CVE-2025-69872 in diskcache +=============================== + +Install diskcache-community and read your caches with :class:`SafeDisk +`: + +.. code-block:: shell + + $ pip install diskcache-community + +.. code-block:: python + + from diskcache_community import Cache, SafeDisk + + cache = Cache('/var/tmp/mycache', disk=SafeDisk) + +The rest of this page explains what the vulnerability is, who is exposed, the +steps for caches you create and for caches libraries create for you, and what +a restricted disk has to cover if you write your own. + +What the vulnerability is +------------------------- + +`CVE-2025-69872`_ (CWE-502) affects every release of the upstream +``diskcache`` package, 5.6.3 included, and there is no fixed upstream release. +The default :class:`Disk ` stores keys and values +that are not plain strings, bytes or numbers as pickles, in ``cache.db`` and in +the value files next to it, and reads them back with :func:`pickle.load`. A +pickle can name any function to call while it is loaded, so anyone who can +write into the cache directory can run code in every process that reads from +it. + +The attacker needs write access to the cache directory. You are exposed when +someone other than the user running the process can write there, for example: + +* a cache directory under ``/tmp`` or another directory shared between users; +* a cache on a shared volume or network drive, or one mounted into a container + that other workloads can write to; +* a cache restored from a CI cache, an artifact or a download that someone + else could have produced. + +A cache that only its own user can write to is not exposed. diskcache-community +creates new cache directories with mode ``0700`` and leaves existing ones as +they are, so check the permissions of directories created by older versions. + +Steps +----- + +1. **Move to diskcache-community.** The fork keeps the API and the files on + disk, and is imported as ``diskcache_community``. :doc:`migrating` covers + the dependency, the imports and the Django settings. + +2. **Select SafeDisk for caches you create.** Pass it as ``disk`` to + :class:`Cache `, :class:`FanoutCache + `, :class:`Deque + ` and :class:`Index + `, and as the ``DISK`` option of + :class:`DjangoCache `: + + .. code-block:: python + + CACHES = { + 'default': { + 'BACKEND': 'diskcache_community.DjangoCache', + 'LOCATION': '/var/tmp/django_cache', + 'DISK': 'diskcache_community.SafeDisk', + } + } + + :class:`SafeDisk ` reads the same files as + :class:`Disk ` but unpickles only standard-library + types such as ``dict``, ``list``, ``datetime`` and ``Decimal``, refuses any + other key or value when it is written, and refuses file paths outside the + cache directory. :ref:`tutorial-safedisk` in the tutorial lists the types. + +3. **Select SafeDisk for caches that libraries create.** A cache created + without a ``disk`` argument uses the process default, which you can set + without changing the library's code: + + .. code-block:: shell + + $ DISKCACHE_DISK=safe python app.py + + or in code, before the caches are created: + + .. code-block:: python + + import diskcache_community + + diskcache_community.configure(diskcache_community.SafeDisk) + + This reaches only caches created through ``diskcache_community``. A library + that imports the upstream ``diskcache`` package keeps reading pickles with + :func:`pickle.load` until it moves to the fork. A library that passes its + own disk keeps it. :ref:`tutorial-default-disk` in the tutorial has the + details. + +4. **Allow your own classes.** If the cache holds dataclasses, named tuples, + enums or pydantic models, allow them with :meth:`SafeDisk.with_types + ` instead of falling back to + :class:`Disk `: + + .. code-block:: python + + from diskcache_community import Cache, SafeDisk + + from myapp.models import Message, Point + + AppDisk = SafeDisk.with_types(Message, Point, modules=('myapp.schemas',)) + cache = Cache('/var/tmp/mycache', disk=AppDisk) + + With ``DISKCACHE_DISK=safe``, the same lists go into + ``DISKCACHE_SAFE_TYPES`` and ``DISKCACHE_SAFE_MODULES``. numpy classes + cannot be allowed, because a crafted numpy pickle can corrupt memory. Store + arrays as bytes instead: + + .. code-block:: python + + import io + + import numpy + + buffer = io.BytesIO() + numpy.save(buffer, array, allow_pickle=False) + cache['embedding'] = buffer.getvalue() + + array = numpy.load(io.BytesIO(cache['embedding']), allow_pickle=False) + +5. **Deal with entries already in the cache.** A cache written with the + default :class:`Disk ` may hold objects that + :class:`SafeDisk ` refuses, and reading them + raises :exc:`pickle.UnpicklingError`. If the entries can be computed again, + clear the cache once with :meth:`clear `, + or let reads drop them with :meth:`with_options + `: + + .. code-block:: python + + AppDisk = SafeDisk.with_options(on_error='miss') + + An unreadable entry is then deleted and treated as missing, and the first + one is logged as a warning on the ``diskcache_community`` logger. See + :ref:`tutorial-unreadable-entries`. + +6. **Check the result.** ``type(cache.disk)`` names the disk a cache uses, and + :func:`default_disk() ` returns the + process default. Run ``pip show diskcache`` to see whether the upstream + package is still installed, and, with pipdeptree installed, + ``pipdeptree -r -p diskcache`` to see which dependency requires it. + +What stays the same +------------------- + +The default :class:`Disk ` still reads pickles with +:func:`pickle.load`, as 5.6.3 does, so every existing cache keeps opening and +caches of arbitrary objects keep working. :class:`SafeDisk +` is a choice you make per cache or per process, +and the choice is never stored in the cache directory: every process that +opens a directory decides for itself. + +Vulnerability scanners match on the distribution name. ``pip-audit``, +Dependabot and similar tools report CVE-2025-69872 for ``diskcache`` and +nothing for ``diskcache-community``, whichever disk you use. A clean scan +means the upstream package is gone, not that your caches use :class:`SafeDisk +`. + +What a restricted disk has to cover +----------------------------------- + +Some projects subclass :class:`Disk ` and replace +:func:`pickle.load` with a restricted unpickler of their own. Restricting how +values are unpickled is not enough. A disk that closes this vulnerability +covers all of the following, and :class:`SafeDisk +` does each one: + +* **Keys as well as values.** Keys that are not plain strings, bytes or + numbers are pickled too, and :meth:`Disk.get ` + unpickles the ``key`` column whenever the cache returns a key, for example + while iterating or in :meth:`peekitem `. + A disk that restricts only :meth:`fetch ` + still runs a crafted key. :class:`SafeDisk ` + reads keys with the same unpickler as values. + +* **The filename column.** Large values live in files whose path is stored in + ``cache.db``. Reads, deletes, expiry and culling all follow that path, and + :meth:`Disk.remove ` also removes the + parent directories it leaves empty. A crafted row can therefore point outside + the cache and have a file read or deleted, including by a delete that runs + after a failed read. :class:`SafeDisk ` refuses + any path that leaves the cache directory, on reads and on removal. + +* **Exact classes, not module prefixes.** With protocol 4 and later, which the + writer of the pickle chooses, a pickle names a global as a dotted path that + the unpickler follows attribute by attribute. Trusting every name under a + package such as ``myapp.types`` therefore also trusts whatever those modules + import: a name like ``os.system`` or ``builtins.eval`` resolves through any + trusted module that imports ``os`` or ``builtins``. + :meth:`SafeDisk.with_types ` + accepts a name only when every step is a class and the final class is + defined in a trusted module, and it refuses dunder attributes. + +* **No numpy.** numpy's array and dtype reconstructors restore internal dtype + state from the pickle without checking it, so a crafted array can make + numpy read arbitrary memory as object pointers. Allowing those names turns + the cache into a crash or memory-corruption vector. + :meth:`SafeDisk.with_types ` + refuses numpy classes even when asked for them. + +* **No imports while reading.** Resolving a name by importing its module runs + that module's top-level code, chosen by the pickle. :class:`SafeDisk + ` looks classes up only in modules that are + already imported. + +* **Refuse on write.** A value the reader would refuse should fail when it is + written, not when a later process reads it. :class:`SafeDisk + ` checks the names in a value's pickle against + the same allowlist before anything is written and raises :exc:`TypeError` + naming the class. + +To report a problem with :class:`SafeDisk `, +follow ``SECURITY.md`` in the repository. + +.. _`CVE-2025-69872`: https://www.cve.org/CVERecord?id=CVE-2025-69872 diff --git a/docs/index.rst b/docs/index.rst index a6c0be2..d0510b9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -21,7 +21,9 @@ Install it with pip and import ``diskcache_community`` where you imported $ pip install diskcache-community -Start with the :doc:`tutorial` and keep the :doc:`api` at hand. The +Coming from diskcache 5.6.3, read :doc:`migrating`; to close the pickle +vulnerability, read :doc:`fix-cve-2025-69872`. Then start with the +:doc:`tutorial` and keep the :doc:`api` at hand. The :doc:`benchmarks` compare DiskCache with memcached, Redis, lmdb and other stores, and the fork with diskcache 5.6.3. @@ -29,6 +31,8 @@ stores, and the fork with diskcache 5.6.3. :maxdepth: 1 :titlesonly: + fix-cve-2025-69872 + migrating tutorial benchmarks cache-benchmarks diff --git a/docs/migrating.rst b/docs/migrating.rst new file mode 100644 index 0000000..e613175 --- /dev/null +++ b/docs/migrating.rst @@ -0,0 +1,142 @@ +Migrating from diskcache 5.6.3 +============================== + +Replace the dependency and the imports. Cache directories need no changes: + +.. code-block:: shell + + $ pip install diskcache-community + +.. code-block:: python + + import diskcache_community as diskcache # was: import diskcache + from diskcache_community import Cache # was: from diskcache import Cache + +diskcache-community keeps the public API of diskcache 5.6.3, the ``cache.db`` +schema, the file layout, the serialization modes, the persisted settings and +the memoize key format. A directory written by 5.6.3 opens with the fork, and +a directory written by the fork opens with 5.6.3. The fork needs Python 3.10 +or later and has no runtime dependencies. + +Steps +----- + +1. **Change the dependency.** Replace ``diskcache`` with + ``diskcache-community`` in ``pyproject.toml``, ``requirements.txt`` or + wherever the project declares it: + + .. code-block:: toml + + dependencies = [ + "diskcache-community>=5.8", + ] + +2. **Change every import.** The package is imported as + ``diskcache_community``. Find the imports with: + + .. code-block:: shell + + $ grep -rn "diskcache" --include="*.py" . + + Change the imports of exceptions and warnings as well as the containers. + When the upstream package stays installed, ``except diskcache.Timeout`` + does not catch the :exc:`Timeout ` that a + ``diskcache_community`` cache raises, because the two packages define + separate classes. The same goes for :class:`Disk + ` subclasses: base them on + ``diskcache_community.Disk`` or ``diskcache_community.JSONDisk``. + +3. **Change the Django settings.** Name the backend + ``'diskcache_community.DjangoCache'``. A ``DISK`` option names + ``'diskcache_community.SafeDisk'`` or the import path of your own disk + class: + + .. code-block:: python + + CACHES = { + 'default': { + 'BACKEND': 'diskcache_community.DjangoCache', + 'LOCATION': '/var/tmp/django_cache', + 'SHARDS': 8, + } + } + +4. **Run the tests.** Existing cache directories open as they are, so a test + suite or a staging environment that uses a cache written by 5.6.3 is a + good check. + +When another dependency still needs diskcache +--------------------------------------------- + +The fork never touches the upstream ``diskcache`` package, so both can be +installed in the same environment. Each library uses the package it imports, +and the objects of one are not instances of the other's classes: pass +``diskcache_community`` caches only to code that imports +``diskcache_community``, and catch the exceptions of the package that created +the cache. + +A library that wants to work with either package can import whichever is +installed: + +.. code-block:: python + + try: + import diskcache_community as diskcache + except ImportError: + import diskcache + +Scanners keep reporting CVE-2025-69872 while the upstream package is +installed. Run ``pip show diskcache`` to see whether it is, and, with +pipdeptree installed, ``pipdeptree -r -p diskcache`` to see which dependency +requires it. + +What behaves differently +------------------------ + +These changes from 5.6.3 can be noticed by an application. The +``CHANGELOG.md`` in the repository lists every change. + +* **New directories are created with mode 0700.** That includes missing + parent directories and the value subdirectories. Existing directories keep + their permissions. If several users share a cache directory, create it with + the permissions you want before the first cache opens it. +* **Read-only caches.** A cache whose database or directory is not writable + opens read-only instead of failing, as does ``Cache(directory, + read_only=True)``. Reads work and every write raises :exc:`ReadOnlyError + `, a subclass of + :exc:`sqlite3.OperationalError`. Opening an existing cache writes nothing + unless a setting differs from the stored one. +* **SQLite builds compiled with SQLITE_DQS=0 work.** On the first open with + write access, the fork rewrites the ``Settings`` triggers of an older cache + to use single-quoted literals. 5.6.3 reads the rewritten triggers as well. +* **Memoize and ignored parameters.** A parameter named in ``ignore`` is now + ignored whether it is passed by keyword or by position. Calls that passed + an ignored parameter by position get a new key, so their first call after + the upgrade is a miss. All other keys are unchanged. +* **Connections and fork.** The SQLite connections of threads that have + exited are closed by :meth:`close ` and + when the cache is garbage collected. A child process created with + ``fork()`` opens new connections instead of using the parent's. +* **FanoutCache.** ``hasattr`` works on a :class:`FanoutCache + `, and ``get`` and ``pop`` return the + documented tuple when a shard times out. +* **Default disk.** The ``disk`` arguments default to ``None``, which means + the type :func:`default_disk ` returns. + That is :class:`Disk `, as before, unless the + process sets ``DISKCACHE_DISK`` or calls :func:`configure + `. + +The fork also adds features that change nothing until you use them: +:class:`SafeDisk ` for caches that untrusted +users can write to, ``store_time=True`` on ``get``, ``iterkeys(tag=...)`` and +:meth:`Disk.with_options ` for entries +that cannot be read. To close CVE-2025-69872 after migrating, see +:doc:`fix-cve-2025-69872`. + +Going back +---------- + +Install ``diskcache==5.6.3`` and change the imports back. Cache directories +written by the fork open with 5.6.3, including entries written through +:class:`SafeDisk `, which are ordinary pickles. +A disk class of your own needs an equivalent subclass of ``diskcache.Disk``.