gh-71549: Add support for JSON serialization of custom types

Add the __json__() and __raw_json__() protocol methods, the
copyreg.json() registration function with copyreg.json_dispatch_table,
the copyreg.RawJSON wrapper for including already encoded JSON in the
output verbatim, and the JSONEncoder.dispatch_table attribute for
overriding the global registry per encoder.

Make deque, mappingproxy, ChainMap, UserDict, UserList and UserString
serializable out of the box.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: serhiy-storchaka

Doc/library/collections.rst

@@ -58,6 +58,10 @@ The class can be used to simulate nested scopes and is useful in templating.
     one of the underlying mappings gets updated, those changes will be reflected
     in :class:`ChainMap`.
 
+    .. versionchanged:: next
+       Added the :meth:`~object.__json__` method;
+       instances are now serializable by the :mod:`json` module.
+
     All of the usual dictionary methods are supported.  In addition, there is a
     *maps* attribute, a method for creating new subcontexts, and a property for
     accessing all but the first mapping:
@@ -486,6 +490,10 @@ or subtracting from an empty counter.
 
     Deques are :ref:`generic <generics>` over the type of their contents.
 
+    .. versionchanged:: next
+       Added the :meth:`~object.__json__` method;
+       instances are now serializable by the :mod:`json` module.
+
 
     Deque objects support the following methods:
 
@@ -1343,6 +1351,10 @@ attribute.
     :class:`!UserDict` instances.  If arguments are provided, they are used to
     initialize :attr:`data`, like a regular dictionary.
 
+    .. versionchanged:: next
+       Added the :meth:`~object.__json__` method;
+       instances are now serializable by the :mod:`json` module.
+
     In addition to supporting the methods and operations of mappings,
     :class:`!UserDict` instances provide the following attribute:
 
@@ -1380,6 +1392,10 @@ to work with because the underlying list is accessible as an attribute.
     defaulting to the empty list ``[]``.  *list* can be any iterable, for
     example a real Python list or a :class:`UserList` object.
 
+    .. versionchanged:: next
+       Added the :meth:`~object.__json__` method;
+       instances are now serializable by the :mod:`json` module.
+
     In addition to supporting the methods and operations of mutable sequences,
     :class:`UserList` instances provide the following attribute:
 
@@ -1429,3 +1445,7 @@ attribute.
     .. versionchanged:: 3.5
        New methods ``__getnewargs__``, ``__rmod__``, ``casefold``,
        ``format_map``, ``isprintable``, and ``maketrans``.
+
+    .. versionchanged:: next
+       Added the :meth:`~object.__json__` method;
+       instances are now serializable by the :mod:`json` module.

Doc/library/copyreg.rst

@@ -1,21 +1,24 @@
-:mod:`!copyreg` --- Register :mod:`!pickle` support functions
-=============================================================
+:mod:`!copyreg` --- Register :mod:`!pickle` and :mod:`!json` support functions
+==============================================================================
 
 .. module:: copyreg
-   :synopsis: Register pickle support functions.
+   :synopsis: Register pickle and JSON support functions.
 
 **Source code:** :source:`Lib/copyreg.py`
 
 .. index::
    pair: module; pickle
    pair: module; copy
+   pair: module; json
 
 --------------
 
-The :mod:`!copyreg` module offers a way to define functions used while pickling
-specific objects.  The :mod:`pickle` and :mod:`copy` modules use those functions
-when pickling/copying those objects.  The module provides configuration
-information about object constructors which are not classes.
+The :mod:`!copyreg` module offers a way to define functions
+used while pickling and JSON-serializing specific objects.
+The :mod:`pickle`, :mod:`copy` and :mod:`json` modules use those functions
+when pickling/copying/serializing those objects.
+The module provides configuration information
+about object constructors which are not classes.
 Such constructors may be factory functions or class instances.
 
 
@@ -39,6 +42,55 @@ Such constructors may be factory functions or class instances.
    object or subclass of :class:`pickle.Pickler` can also be used for
    declaring reduction functions.
 
+
+.. function:: json(type, function)
+
+   Declares that *function* should be used
+   as the JSON serialization function for objects of type *type*.
+   *function* must be callable;
+   it is called with the object as its only argument
+   and must return a substitute object to be serialized,
+   with the same interface as the :meth:`~object.__json__` method.
+   Registration is by exact type:
+   it does not apply to subclasses of *type*.
+   See :ref:`json-protocol` for details.
+
+   .. versionadded:: next
+
+
+.. data:: json_dispatch_table
+
+   The mapping of types to serialization functions
+   filled by :func:`json` and consulted by the :mod:`json` module.
+   It can be overridden for a particular encoder
+   with the :attr:`json.JSONEncoder.dispatch_table` attribute.
+
+   .. versionadded:: next
+
+
+.. class:: RawJSON(encoded_json)
+
+   Wrapper for the already encoded JSON string *encoded_json*.
+   The JSON encoder outputs it verbatim,
+   without validation of its content.
+   ``str()`` of the instance returns *encoded_json*.
+
+   For example, it allows serializing :class:`decimal.Decimal`
+   as a JSON number with full precision:
+
+      >>> import copyreg, decimal, json
+      >>> copyreg.json(decimal.Decimal, lambda d: copyreg.RawJSON(str(d)))
+      >>> json.dumps({'price': decimal.Decimal('1.10')})
+      '{"price": 1.10}'
+
+   A registration can be undone by removing the entry from
+   :data:`json_dispatch_table`:
+
+      >>> del copyreg.json_dispatch_table[decimal.Decimal]
+
+   .. versionadded:: next
+
+
 Example
 -------
 

Doc/library/json.rst

@@ -243,6 +243,11 @@ Basic Usage
    .. versionchanged:: 3.6
       All optional parameters are now :ref:`keyword-only <keyword-only_parameter>`.
 
+   .. versionchanged:: next
+      Serialization can now be customized per type
+      with the :meth:`~object.__json__` method and :func:`copyreg.json`.
+      See :ref:`json-protocol`.
+
 
 .. function:: dumps(obj, *, skipkeys=False, ensure_ascii=True, \
                     check_circular=True, allow_nan=True, cls=None, \
@@ -560,6 +565,14 @@ Encoders and Decoders
    .. versionchanged:: 3.6
       All parameters are now :ref:`keyword-only <keyword-only_parameter>`.
 
+   .. versionchanged:: next
+      The encoder now consults the dispatch table
+      (:attr:`~JSONEncoder.dispatch_table`
+      or :data:`copyreg.json_dispatch_table`)
+      and the :meth:`~object.__json__` method of the object's class
+      before falling back to *default*.
+      See :ref:`json-protocol`.
+
 
    .. method:: default(o)
 
@@ -598,6 +611,106 @@ Encoders and Decoders
             for chunk in json.JSONEncoder().iterencode(bigobject):
                 mysocket.write(chunk)
 
+   .. attribute:: dispatch_table
+
+      A mapping of types to serialization functions,
+      used instead of the global :data:`copyreg.json_dispatch_table`.
+      It can be set as a class attribute of a subclass
+      or as an attribute of an encoder instance.
+      See :ref:`json-protocol`.
+
+      .. versionadded:: next
+
+
+.. _json-protocol:
+
+Serializing custom objects
+--------------------------
+
+Serialization of objects of types that are not supported natively
+(see the :ref:`conversion table <py-to-json-table>`)
+can be customized at three levels:
+
+* The author of a class can define the JSON representation of its instances
+  by giving it a :meth:`~object.__json__` method.
+
+* An application can register a serialization function
+  for instances of a type it does not control
+  with :func:`copyreg.json`.
+
+* A particular call can use its own type-to-function mapping
+  by setting the :attr:`~JSONEncoder.dispatch_table` attribute
+  of a :class:`JSONEncoder` subclass or instance.
+
+The more specific level takes precedence:
+a registered function is used before ``__json__``,
+and an encoder's own dispatch table completely replaces
+the global :data:`copyreg.json_dispatch_table`.
+The *default* function is called last,
+only for objects that none of the above handled.
+
+.. currentmodule:: None
+
+.. method:: object.__json__()
+
+   Return a substitute object to be serialized instead of *self*.
+   Called by the JSON encoder for an object
+   whose type is not supported natively
+   and has no entry in the dispatch table.
+
+   If the result is of a serializable type, it is serialized as usual
+   (but a ``__json__`` method of the result is not consulted).
+   Otherwise the result is interpreted by duck typing:
+
+   * an object with :meth:`~object.__index__` is serialized
+     as a JSON number;
+   * an object with :meth:`~object.__float__` is serialized
+     as a JSON number;
+   * an iterable with a :meth:`!keys` method and
+     :meth:`~object.__getitem__` is serialized as a JSON object;
+   * any other iterable is serialized as a JSON array;
+   * an object with :meth:`~object.__raw_json__` is included
+     in the output verbatim.
+
+   .. versionadded:: next
+
+.. method:: object.__raw_json__()
+
+   Return a string to be included in the JSON output verbatim,
+   without validation of its content.
+   Consulted only for the result of a registered serialization function
+   or a :meth:`~object.__json__` method.
+   To include an already encoded fragment directly in the serialized data,
+   wrap it in :class:`copyreg.RawJSON` instead.
+
+   .. versionadded:: next
+
+.. currentmodule:: json
+
+For example, a class can serialize itself as a JSON object::
+
+   class Point:
+       def __init__(self, x, y):
+           self.x = x
+           self.y = y
+
+       def __json__(self):
+           return {'x': self.x, 'y': self.y}
+
+An application can choose how :class:`decimal.Decimal` is serialized —
+as a JSON string::
+
+   copyreg.json(decimal.Decimal, str)
+
+or as a JSON number with full precision, using :class:`copyreg.RawJSON`::
+
+   copyreg.json(decimal.Decimal, lambda d: copyreg.RawJSON(str(d)))
+
+A number of standard library container types define
+:meth:`~object.__json__` and are therefore serializable out of the box.
+
+.. versionadded:: next
+
 
 Exceptions
 ----------

Doc/library/types.rst

@@ -397,6 +397,10 @@ Standard names are defined for the following types:
       Updated to support the new union (``|``) operator from :pep:`584`, which
       simply delegates to the underlying mapping.
 
+   .. versionchanged:: next
+      Added the :meth:`~object.__json__` method;
+      instances are now serializable by the :mod:`json` module.
+
    .. describe:: key in proxy
 
       Return ``True`` if the underlying mapping has a key *key*, else

Doc/whatsnew/3.16.rst

@@ -96,6 +96,16 @@ codecs
   even when a built-in codec of the same name exists.
   (Contributed by Serhiy Storchaka in :gh:`152997`.)
 
+copyreg
+-------
+
+* Add :func:`copyreg.json`, :data:`copyreg.json_dispatch_table`
+  and :class:`copyreg.RawJSON`
+  for registering JSON serialization functions
+  for types that cannot be modified;
+  see the :mod:`json` entry below.
+  (Contributed by Serhiy Storchaka in :gh:`71549`.)
+
 curses
 ------
 
@@ -293,6 +303,30 @@ Add :meth:`~ipaddress.IPv4Network.next_network` and
 network with a specific prefix size.
 
 
+json
+----
+
+* JSON serialization of custom types can now be customized
+  without subclassing :class:`~json.JSONEncoder` or passing *default*:
+  a class can define its JSON representation
+  with the new :meth:`~object.__json__` method,
+  an application can register a serialization function
+  for a type it does not control with the new :func:`copyreg.json`,
+  and a particular encoder can use its own type-to-function mapping
+  via the new :attr:`~json.JSONEncoder.dispatch_table` attribute.
+  The new :class:`copyreg.RawJSON` wrapper includes
+  an already encoded JSON string in the output verbatim,
+  which allows, for example, serializing :class:`decimal.Decimal`
+  as a JSON number with full precision.
+  Standard library containers with an unambiguous JSON form
+  (:class:`collections.deque`, :class:`types.MappingProxyType`,
+  :class:`collections.ChainMap`, :class:`collections.UserDict`,
+  :class:`collections.UserList` and :class:`collections.UserString`)
+  are now serializable out of the box.
+  See :ref:`json-protocol`.
+  (Contributed by Serhiy Storchaka in :gh:`71549`.)
+
+
 logging
 -------
 

Include/internal/pycore_global_objects_fini_generated.h

@@ -169,6 +169,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(__iter__)
         STRUCT_FOR_ID(__itruediv__)
         STRUCT_FOR_ID(__ixor__)
+        STRUCT_FOR_ID(__json__)
         STRUCT_FOR_ID(__lazy_import__)
         STRUCT_FOR_ID(__lazy_modules__)
         STRUCT_FOR_ID(__le__)
@@ -206,6 +207,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(__qualname__)
         STRUCT_FOR_ID(__radd__)
         STRUCT_FOR_ID(__rand__)
+        STRUCT_FOR_ID(__raw_json__)
         STRUCT_FOR_ID(__rdivmod__)
         STRUCT_FOR_ID(__reduce__)
         STRUCT_FOR_ID(__reduce_ex__)