O01, the object header read byte by byte

GLOSSARY.md

@@ -8,7 +8,7 @@ This file is generated from `pyxray/src/pyxray/glossary.py`. Edit that and run `
 
 ## Index
 
-[ASDL](#asdl) | [Argument Clinic](#argument-clinic) | [EXTENDED_ARG](#extended_arg) | [JIT](#jit) | [PEG parser](#peg-parser) | [Pyodide](#pyodide) | [WebAssembly](#webassembly) | [abstract syntax tree](#abstract-syntax-tree) | [adaptive instruction](#adaptive-instruction) | [arena](#arena) | [assembler](#assembler) | [backtrace](#backtrace) | [basic block](#basic-block) | [binding](#binding) | [block](#block) | [blurb](#blurb) | [borrowed reference](#borrowed-reference) | [bytecode](#bytecode) | [cell](#cell) | [class cell](#class-cell) | [closure](#closure) | [code generation](#code-generation) | [code object](#code-object) | [coding cookie](#coding-cookie) | [cold block](#cold-block) | [computed goto](#computed-goto) | [configure](#configure) | [constant folding](#constant-folding) | [control flow graph](#control-flow-graph) | [cycle collector](#cycle-collector) | [deallocation](#deallocation) | [debug build](#debug-build) | [deoptimization](#deoptimization) | [devguide](#devguide) | [dispatch](#dispatch) | [environment changed](#environment-changed) | [eval loop](#eval-loop) | [evaluation order](#evaluation-order) | [exception table](#exception-table) | [f string](#f-string) | [finalizer](#finalizer) | [frame](#frame) | [free threaded build](#free-threaded-build) | [free variable](#free-variable) | [gdb](#gdb) | [generated file](#generated-file) | [generation](#generation) | [grammar](#grammar) | [header file](#header-file) | [immortal object](#immortal-object) | [indent and dedent](#indent-and-dedent) | [inline cache](#inline-cache) | [instance dictionary](#instance-dictionary) | [instruction](#instruction) | [interning](#interning) | [left recursion](#left-recursion) | [line table](#line-table) | [magic number](#magic-number) | [marshal](#marshal) | [monitoring events](#monitoring-events) | [new reference](#new-reference) | [object](#object) | [object header](#object-header) | [obmalloc](#obmalloc) | [oparg](#oparg) | [opcode](#opcode) | [parser generator](#parser-generator) | [pdb](#pdb) | [pointer](#pointer) | [pool](#pool) | [product type](#product-type) | [profile guided optimization](#profile-guided-optimization) | [pseudo instruction](#pseudo-instruction) | [pyc file](#pyc-file) | [pyconfig](#pyconfig) | [reference count](#reference-count) | [reference cycle](#reference-cycle) | [reference leak](#reference-leak) | [regen](#regen) | [regrtest](#regrtest) | [replacement field](#replacement-field) | [resource](#resource) | [scope](#scope) | [segmentation fault](#segmentation-fault) | [short circuiting](#short-circuiting) | [small integer cache](#small-integer-cache) | [soft keyword](#soft-keyword) | [specialization](#specialization) | [stack depth](#stack-depth) | [stack effect](#stack-effect) | [stolen reference](#stolen-reference) | [struct](#struct) | [sum type](#sum-type) | [symbol table](#symbol-table) | [symbol table pass](#symbol-table-pass) | [t string](#t-string) | [test case](#test-case) | [tier one](#tier-one) | [tier two](#tier-two) | [token](#token) | [tokenizer](#tokenizer) | [trace function](#trace-function) | [type object](#type-object) | [underflow](#underflow) | [value stack](#value-stack) | [weak reference](#weak-reference)
+[ASDL](#asdl) | [Argument Clinic](#argument-clinic) | [EXTENDED_ARG](#extended_arg) | [GC pre header](#gc-pre-header) | [JIT](#jit) | [PEG parser](#peg-parser) | [PyVarObject](#pyvarobject) | [Pyodide](#pyodide) | [WebAssembly](#webassembly) | [abstract syntax tree](#abstract-syntax-tree) | [adaptive instruction](#adaptive-instruction) | [arena](#arena) | [assembler](#assembler) | [backtrace](#backtrace) | [basic block](#basic-block) | [binding](#binding) | [block](#block) | [blurb](#blurb) | [borrowed reference](#borrowed-reference) | [bytecode](#bytecode) | [cell](#cell) | [class cell](#class-cell) | [closure](#closure) | [code generation](#code-generation) | [code object](#code-object) | [coding cookie](#coding-cookie) | [cold block](#cold-block) | [computed goto](#computed-goto) | [configure](#configure) | [constant folding](#constant-folding) | [control flow graph](#control-flow-graph) | [cycle collector](#cycle-collector) | [deallocation](#deallocation) | [debug build](#debug-build) | [deoptimization](#deoptimization) | [devguide](#devguide) | [dispatch](#dispatch) | [environment changed](#environment-changed) | [eval loop](#eval-loop) | [evaluation order](#evaluation-order) | [exception table](#exception-table) | [f string](#f-string) | [finalizer](#finalizer) | [frame](#frame) | [free threaded build](#free-threaded-build) | [free variable](#free-variable) | [gdb](#gdb) | [generated file](#generated-file) | [generation](#generation) | [grammar](#grammar) | [header file](#header-file) | [immortal object](#immortal-object) | [indent and dedent](#indent-and-dedent) | [inline cache](#inline-cache) | [instance dictionary](#instance-dictionary) | [instruction](#instruction) | [interning](#interning) | [left recursion](#left-recursion) | [line table](#line-table) | [magic number](#magic-number) | [marshal](#marshal) | [monitoring events](#monitoring-events) | [new reference](#new-reference) | [object](#object) | [object header](#object-header) | [obmalloc](#obmalloc) | [oparg](#oparg) | [opcode](#opcode) | [parser generator](#parser-generator) | [pdb](#pdb) | [pointer](#pointer) | [pool](#pool) | [product type](#product-type) | [profile guided optimization](#profile-guided-optimization) | [pseudo instruction](#pseudo-instruction) | [pyc file](#pyc-file) | [pyconfig](#pyconfig) | [reference count](#reference-count) | [reference cycle](#reference-cycle) | [reference leak](#reference-leak) | [regen](#regen) | [regrtest](#regrtest) | [replacement field](#replacement-field) | [resource](#resource) | [scope](#scope) | [segmentation fault](#segmentation-fault) | [short circuiting](#short-circuiting) | [small integer cache](#small-integer-cache) | [soft keyword](#soft-keyword) | [specialization](#specialization) | [stack depth](#stack-depth) | [stack effect](#stack-effect) | [stolen reference](#stolen-reference) | [struct](#struct) | [sum type](#sum-type) | [symbol table](#symbol-table) | [symbol table pass](#symbol-table-pass) | [t string](#t-string) | [test case](#test-case) | [tier one](#tier-one) | [tier two](#tier-two) | [token](#token) | [tokenizer](#tokenizer) | [trace function](#trace-function) | [type object](#type-object) | [underflow](#underflow) | [value stack](#value-stack) | [weak reference](#weak-reference)
 
 ## Reading the source
 
@@ -634,6 +634,14 @@ It is a real dict and you can look at it, which is why `self.name = name` works
 
 Also written `__dict__`. First met in T08. See also [object](#object), [type object](#type-object).
 
+### PyVarObject
+
+**An object header with a length field welded on the end of it.**
+
+Tuples, lists and bytes objects all hold a count of how many items they have, and rather than each of them inventing a field for it the header itself grows by one machine word called `ob_size`. It is the same trick as the header: put the thing everybody needs in a fixed place so that generic code can read it without knowing the type. Strings keep their length in the same place without being one of these, and integers used to and no longer do.
+
+Also written variable sized object, `ob_size`. First met in O01. See also [object header](#object-header), [object](#object). In the source: [`Include/object.h:174-178@v3.15.0rc1#PyVarObject`](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L174-L178).
+
 ## Memory
 
 Where objects come from and what happens to them afterwards. T09 is the lesson.
@@ -718,6 +726,14 @@ The collector runs finalizers on the objects in a cycle before it frees any of t
 
 Also written `__del__`. First met in T09. See also [deallocation](#deallocation), [cycle collector](#cycle-collector). In the source: [`Python/gc.c:1041-1074@v3.15.0rc1#finalize_garbage`](https://github.com/python/cpython/blob/v3.15.0rc1/Python/gc.c#L1041-L1074).
 
+### GC pre header
+
+**Two words allocated in front of an object, holding its place in the collector's list.**
+
+Only the types the cycle collector tracks get one, and the object's own address points past it, so nothing that reads the header ever sees it. You can still measure it: `sys.getsizeof` adds it and the object's own `__sizeof__` does not, so the gap between those two is exactly this.
+
+Also written `PyGC_Head`. First met in O01. See also [cycle collector](#cycle-collector), [object header](#object-header). In the source: [`Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1#PyGC_Head`](https://github.com/python/cpython/blob/v3.15.0rc1/Include/internal/pycore_interp_structs.h#L158-L169).
+
 ## Building the interpreter
 
 The words that turn out to be about the binary rather than about the language. B01 through B04 are the lessons, and several numbers in the earlier lessons move when the build does.

README.md

@@ -116,6 +116,7 @@ A lesson can also end with a boss fight, which is a problem the text does not so
 | F10 | [Inside a code object](lessons/f10-inside-a-code-object/f10.ipynb) | The record the compiler hands back, walked from a module down to a method, the seven slots behind three tuples that add up to eight, why a closure's first instruction runs before line one, every bit in co_flags, what equality compares and what it deliberately ignores, and what happens when you try to change one | M3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f10-inside-a-code-object/f10.ipynb) |
 | F11 | [Two tables on the side](lessons/f11-two-tables-on-the-side/f11.ipynb) | The two blobs beside the bytecode, why wrapping a loop in a try changes not one instruction, the four numbers in an exception table entry and the marker bit that makes a variable length table binary searchable, the six forms a location entry can take, and two decoders written here that agree with the interpreter's | M3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f11-two-tables-on-the-side/f11.ipynb) |
 | F12 | [What ends up on disk](lessons/f12-what-ends-up-on-disk/f12.ipynb) | The other direction: a code object flattened into bytes and read straight back, the sixteen byte header decoded field by field, the type byte and the reference table that make marshal small, a fifty line reader checked against the real one, and a handmade .pyc that imports with no .py anywhere | M3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f12-what-ends-up-on-disk/f12.ipynb) |
+| O01 | [The header, byte by byte](lessons/o01-the-header-byte-by-byte/o01.ipynb) | The two fields everybody quotes, read out of your own interpreter with ctypes, and the first of them turning out to be three: a thirty two bit count, sixteen bits named and unused, and the flags that say whether an object can ever be freed, plus why an immortal starts in the middle of the range and where a length lives when a type has one | M4 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o01-the-header-byte-by-byte/o01.ipynb) |
 
 More are landing in order. [lessons/README.md](lessons/README.md) explains how one is put together and how to run them locally.
 

citations.lock.json

@@ -155,6 +155,16 @@
       "first_line": "struct _PyCfgBuilder* _PyCfg_FromInstructionSequence(_PyInstructionSequence *seq);",
       "lines": 5
     },
+    "Include/internal/pycore_global_strings.h:30-38@v3.15.0rc1": {
+      "digest": "6e8ba9cbf4bb5ee4",
+      "first_line": "/* The following is auto-generated by Tools/build/generate_global_objects.py. */",
+      "lines": 9
+    },
+    "Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1": {
+      "digest": "dfd8f4ee4f87e6a6",
+      "first_line": "",
+      "lines": 12
+    },
     "Include/internal/pycore_interp_structs.h:271-286@v3.15.0rc1": {
       "digest": "84ddd939ee6f307f",
       "first_line": "#ifndef Py_GIL_DISABLED",
@@ -180,6 +190,11 @@
       "first_line": "#define PYC_MAGIC_NUMBER 3666",
       "lines": 6
     },
+    "Include/internal/pycore_object.h:83-89@v3.15.0rc1": {
+      "digest": "70fca12f791324cc",
+      "first_line": "#if SIZEOF_VOID_P > 4",
+      "lines": 7
+    },
     "Include/internal/pycore_obmalloc.h:128-146@v3.15.0rc1": {
       "digest": "2b3a0f329f4a22fe",
       "first_line": "/*",
@@ -265,16 +280,41 @@
       "first_line": "#define LOCAL 1",
       "lines": 6
     },
+    "Include/object.h:127-149@v3.15.0rc1": {
+      "digest": "c7e17988b3729573",
+      "first_line": "struct _object {",
+      "lines": 23
+    },
     "Include/object.h:127-150@v3.15.0rc1": {
       "digest": "355a45f89904b83a",
       "first_line": "struct _object {",
       "lines": 24
     },
+    "Include/object.h:156-167@v3.15.0rc1": {
+      "digest": "5dfbd32621883aa8",
+      "first_line": "struct _object {",
+      "lines": 12
+    },
     "Include/object.h:156-170@v3.15.0rc1": {
       "digest": "3721c9256c01c28f",
       "first_line": "struct _object {",
       "lines": 15
     },
+    "Include/object.h:174-178@v3.15.0rc1": {
+      "digest": "f70e693e6b6d5dc5",
+      "first_line": "struct PyVarObject {",
+      "lines": 5
+    },
+    "Include/object.h:237-244@v3.15.0rc1": {
+      "digest": "dc45c7bd6a91c2b3",
+      "first_line": "// bpo-39573: The Py_SET_SIZE() function must be used to set an object size.",
+      "lines": 8
+    },
+    "Include/object.h:580-583@v3.15.0rc1": {
+      "digest": "323df69ccd90ebf9",
+      "first_line": "// Flag values for ob_flags (16 bits available, if SIZEOF_VOID_P > 4).",
+      "lines": 4
+    },
     "Include/object.h:623-629@v3.15.0rc1": {
       "digest": "cd6bd3f1f3e26b0c",
       "first_line": "/* Macro for returning Py_None from a function.",
@@ -295,6 +335,16 @@
       "first_line": "#ifndef _Py_OPAQUE_PYOBJECT",
       "lines": 12
     },
+    "Include/refcount.h:126-136@v3.15.0rc1": {
+      "digest": "dd1efebab1b34716",
+      "first_line": "static inline Py_ALWAYS_INLINE int _Py_IsImmortal(PyObject *op)",
+      "lines": 11
+    },
+    "Include/refcount.h:23-50@v3.15.0rc1": {
+      "digest": "6f9e131a08baae2f",
+      "first_line": "#if SIZEOF_VOID_P > 4",
+      "lines": 28
+    },
     "Include/refcount.h:417-429@v3.15.0rc1": {
       "digest": "a2af4e3c17465fe0",
       "first_line": "static inline Py_ALWAYS_INLINE void Py_DECREF(PyObject *op)",
@@ -585,6 +635,11 @@
       "first_line": "static PyObject *",
       "lines": 6
     },
+    "Objects/object.c:2773-2791@v3.15.0rc1": {
+      "digest": "84c3b29a29ccb125",
+      "first_line": "void",
+      "lines": 19
+    },
     "Objects/object.c:3282-3300@v3.15.0rc1": {
       "digest": "6b3a6c6e934a76fc",
       "first_line": "/*",

lessons/CLAIMS.md

@@ -12,7 +12,7 @@ header, what the allocator does with a freed block, the shape of the eval loop.
 marked with the reason, and a lesson is allowed at most 3 of them. The cap is the point.
 Without it the exception becomes the rule and this goes back to being a book.
 
-243 claims across 28 lessons, 19 of them not observable from Python.
+250 claims across 29 lessons, 20 of them not observable from Python.
 
 ## B01. Building CPython, and whether you need to
 
@@ -183,6 +183,18 @@ Without it the exception becomes the rule and this goes back to being a book.
 | a .pyc assembled by hand imports and runs with no source file on disk | [`f12-26`](f12-what-ends-up-on-disk/f12.ipynb) |
 | a wrong magic number raises and a wrong timestamp is fixed silently | [`f12-29`](f12-what-ends-up-on-disk/f12.ipynb) |
 
+## O01. The header, byte by byte
+
+| Claim | Proved by |
+| --- | --- |
+| id() is the address an object lives at, and the two machine words there are its reference count and a pointer to its type | [`o01-07`](o01-the-header-byte-by-byte/o01.ipynb) |
+| the reference count is thirty two bits wide, and the other half of that word holds two more fields | [`o01-10`](o01-the-header-byte-by-byte/o01.ipynb) |
+| an immortal object's count starts halfway between the immortality line and the top of the field, leaving about a billion of slack in each direction | [`o01-13`](o01-the-header-byte-by-byte/o01.ipynb) |
+| a string literal that happens to be one of CPython's own identifiers is immortal, and the same characters CPython does not use are not | [`o01-16`](o01-the-header-byte-by-byte/o01.ipynb) |
+| for a tuple, a list, a bytes and a str, the machine word after the header is the length | [`o01-19`](o01-the-header-byte-by-byte/o01.ipynb) |
+| sys.getsizeof reports two machine words more than an object's own __sizeof__ for the types the cycle collector tracks, and nothing extra for the types it does not | [`o01-21`](o01-the-header-byte-by-byte/o01.ipynb) |
+| the free threaded build gives every object a thirty two byte header, split so the owning thread can increment without an atomic instruction | not observable from Python: the fields only exist in a build configured with --disable-gil, and this notebook is not running one |
+
 ## T01. One line, seven stages
 
 | Claim | Proved by |

lessons/README.md

@@ -32,6 +32,7 @@ Each lesson is a notebook you can run. There is nothing to install and nothing t
 | [F10. Inside a code object](f10-inside-a-code-object/f10.ipynb) | What compiling a file actually hands you, the code objects that live in other code objects' constants, the one array behind co_varnames and co_cellvars and co_freevars, the instructions that run before your first line, what co_flags remembers, and why two code objects can be equal without being the same one | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f10-inside-a-code-object/f10.ipynb) |
 | [F11. Two tables on the side](f11-two-tables-on-the-side/f11.ipynb) | Why a try you never trip costs nothing, the exception table decoded byte by byte and checked against dis, how a raise binary searches a table whose entries are not the same length, the six shapes a source location can take, and a hand written line table decoder run against every code object in dis.py | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f11-two-tables-on-the-side/f11.ipynb) |
 | [F12. What ends up on disk](f12-what-ends-up-on-disk/f12.ipynb) | What a .pyc actually holds, the four header fields and which two decide staleness, why half the magic number is a carriage return and a newline, the one byte that names every marshalled object, how a repeated value costs five bytes, a marshal reader written here, and a .pyc assembled by hand and imported with no source file on disk | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f12-what-ends-up-on-disk/f12.ipynb) |
+| [O01. The header, byte by byte](o01-the-header-byte-by-byte/o01.ipynb) | Reading the object header out of memory with ctypes and checking every field against something Python will tell you another way, the thirty two bit count and the two fields sharing its word, why an immortal object parks at 3 << 30 rather than at the top, the two flag bits that say how it got there, why a string CPython uses itself is immortal and yours is not, and the two words in front of the header that only the collector sees | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o01-the-header-byte-by-byte/o01.ipynb) |
 
 ## The three programs