From 66e80a45d380912877790c3b2aff063e80003a52 Mon Sep 17 00:00:00 2001 From: tamnd <1218621+tamnd@users.noreply.github.com> Date: Mon, 31 Aug 2026 19:31:04 +0700 Subject: [PATCH] O01, the object header read byte by byte The first lesson of the object model part. T08 said every object starts with a reference count and a pointer to its type. That is what almost every article about CPython says and it is not quite what is in memory. The lesson reads the bytes with ctypes out of the reader's own interpreter and checks each field against something Python will tell you another way. On an ordinary sixty four bit build the first word is a thirty two bit ob_refcnt, sixteen bits called ob_overflow that are declared in the struct and referenced nowhere else in the source tree, and sixteen bits of ob_flags. It then works through why an immortal object parks at 3 << 30 rather than at the top of the field, which is so that an extension compiled against 3.11 can increment and decrement without checking and be a billion out either way; the two flag bits that record whether an object was compiled into the binary or promoted while running, and why shutting down has to tell them apart; the identifier strings CPython ships, which is why a literal "self" in your code is immortal and a string CPython has never seen is not; where a length lives for the types that have one, including the assert that keeps Py_SIZE away from integers; and the two words in front of the header that only the cycle collector sees, measured as the gap between __sizeof__ and getsizeof. Every offset is computed from the pointer size rather than hardcoded, so the same cells run in the browser probe, where Python is a thirty two bit build and every number halves. 3.14 and 3.15 print identical output for the whole lesson. Six diagrams as excalidraw and svg. Two glossary terms, PyVarObject and GC pre header, which takes GLOSSARY.md to 108. Both README tables updated. --- GLOSSARY.md | 18 +- README.md | 1 + citations.lock.json | 55 + lessons/CLAIMS.md | 14 +- lessons/README.md | 1 + lessons/o01-the-header-byte-by-byte/build.py | 339 ++++ .../o01-the-header-byte-by-byte/diagrams.py | 144 ++ .../how-far-off-you-can-be.excalidraw | 1035 ++++++++++++ .../diagrams/how-far-off-you-can-be.svg | 1 + .../the-first-sixteen-bytes.excalidraw | 473 ++++++ .../diagrams/the-first-sixteen-bytes.svg | 1 + .../diagrams/the-same-header-wider.excalidraw | 823 ++++++++++ .../diagrams/the-same-header-wider.svg | 1 + .../three-fields-in-one-word.excalidraw | 602 +++++++ .../diagrams/three-fields-in-one-word.svg | 1 + .../two-ways-to-become-immortal.excalidraw | 753 +++++++++ .../diagrams/two-ways-to-become-immortal.svg | 1 + .../diagrams/where-a-length-lives.excalidraw | 1435 +++++++++++++++++ .../diagrams/where-a-length-lives.svg | 1 + lessons/o01-the-header-byte-by-byte/o01.ipynb | 441 +++++ probes/pyodide/lessons.json | 68 +- probes/pyodide/lessons.md | 3 +- pyxray/src/pyxray/glossary.py | 18 + 23 files changed, 6214 insertions(+), 15 deletions(-) create mode 100644 lessons/o01-the-header-byte-by-byte/build.py create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams.py create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.excalidraw create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.svg create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.excalidraw create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.svg create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.excalidraw create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.svg create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.excalidraw create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.svg create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.excalidraw create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.svg create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.excalidraw create mode 100644 lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.svg create mode 100644 lessons/o01-the-header-byte-by-byte/o01.ipynb diff --git a/GLOSSARY.md b/GLOSSARY.md index b9db73f..2c62b1c 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -8,7 +8,7 @@ This file is generated from `pyxray/src/pyxray/glossary.py`. Edit that and run ` ## Index -[ASDL](#asdl) | [Argument Clinic](#argument-clinic) | [EXTENDED_ARG](#extended_arg) | [JIT](#jit) | [PEG parser](#peg-parser) | [Pyodide](#pyodide) | [WebAssembly](#webassembly) | [abstract syntax tree](#abstract-syntax-tree) | [adaptive instruction](#adaptive-instruction) | [arena](#arena) | [assembler](#assembler) | [backtrace](#backtrace) | [basic block](#basic-block) | [binding](#binding) | [block](#block) | [blurb](#blurb) | [borrowed reference](#borrowed-reference) | [bytecode](#bytecode) | [cell](#cell) | [class cell](#class-cell) | [closure](#closure) | [code generation](#code-generation) | [code object](#code-object) | [coding cookie](#coding-cookie) | [cold block](#cold-block) | [computed goto](#computed-goto) | [configure](#configure) | [constant folding](#constant-folding) | [control flow graph](#control-flow-graph) | [cycle collector](#cycle-collector) | [deallocation](#deallocation) | [debug build](#debug-build) | [deoptimization](#deoptimization) | [devguide](#devguide) | [dispatch](#dispatch) | [environment changed](#environment-changed) | [eval loop](#eval-loop) | [evaluation order](#evaluation-order) | [exception table](#exception-table) | [f string](#f-string) | [finalizer](#finalizer) | [frame](#frame) | [free threaded build](#free-threaded-build) | [free variable](#free-variable) | [gdb](#gdb) | [generated file](#generated-file) | [generation](#generation) | [grammar](#grammar) | [header file](#header-file) | [immortal object](#immortal-object) | [indent and dedent](#indent-and-dedent) | [inline cache](#inline-cache) | [instance dictionary](#instance-dictionary) | [instruction](#instruction) | [interning](#interning) | [left recursion](#left-recursion) | [line table](#line-table) | [magic number](#magic-number) | [marshal](#marshal) | [monitoring events](#monitoring-events) | [new reference](#new-reference) | [object](#object) | [object header](#object-header) | [obmalloc](#obmalloc) | [oparg](#oparg) | [opcode](#opcode) | [parser generator](#parser-generator) | [pdb](#pdb) | [pointer](#pointer) | [pool](#pool) | [product type](#product-type) | [profile guided optimization](#profile-guided-optimization) | [pseudo instruction](#pseudo-instruction) | [pyc file](#pyc-file) | [pyconfig](#pyconfig) | [reference count](#reference-count) | [reference cycle](#reference-cycle) | [reference leak](#reference-leak) | [regen](#regen) | [regrtest](#regrtest) | [replacement field](#replacement-field) | [resource](#resource) | [scope](#scope) | [segmentation fault](#segmentation-fault) | [short circuiting](#short-circuiting) | [small integer cache](#small-integer-cache) | [soft keyword](#soft-keyword) | [specialization](#specialization) | [stack depth](#stack-depth) | [stack effect](#stack-effect) | [stolen reference](#stolen-reference) | [struct](#struct) | [sum type](#sum-type) | [symbol table](#symbol-table) | [symbol table pass](#symbol-table-pass) | [t string](#t-string) | [test case](#test-case) | [tier one](#tier-one) | [tier two](#tier-two) | [token](#token) | [tokenizer](#tokenizer) | [trace function](#trace-function) | [type object](#type-object) | [underflow](#underflow) | [value stack](#value-stack) | [weak reference](#weak-reference) +[ASDL](#asdl) | [Argument Clinic](#argument-clinic) | [EXTENDED_ARG](#extended_arg) | [GC pre header](#gc-pre-header) | [JIT](#jit) | [PEG parser](#peg-parser) | [PyVarObject](#pyvarobject) | [Pyodide](#pyodide) | [WebAssembly](#webassembly) | [abstract syntax tree](#abstract-syntax-tree) | [adaptive instruction](#adaptive-instruction) | [arena](#arena) | [assembler](#assembler) | [backtrace](#backtrace) | [basic block](#basic-block) | [binding](#binding) | [block](#block) | [blurb](#blurb) | [borrowed reference](#borrowed-reference) | [bytecode](#bytecode) | [cell](#cell) | [class cell](#class-cell) | [closure](#closure) | [code generation](#code-generation) | [code object](#code-object) | [coding cookie](#coding-cookie) | [cold block](#cold-block) | [computed goto](#computed-goto) | [configure](#configure) | [constant folding](#constant-folding) | [control flow graph](#control-flow-graph) | [cycle collector](#cycle-collector) | [deallocation](#deallocation) | [debug build](#debug-build) | [deoptimization](#deoptimization) | [devguide](#devguide) | [dispatch](#dispatch) | [environment changed](#environment-changed) | [eval loop](#eval-loop) | [evaluation order](#evaluation-order) | [exception table](#exception-table) | [f string](#f-string) | [finalizer](#finalizer) | [frame](#frame) | [free threaded build](#free-threaded-build) | [free variable](#free-variable) | [gdb](#gdb) | [generated file](#generated-file) | [generation](#generation) | [grammar](#grammar) | [header file](#header-file) | [immortal object](#immortal-object) | [indent and dedent](#indent-and-dedent) | [inline cache](#inline-cache) | [instance dictionary](#instance-dictionary) | [instruction](#instruction) | [interning](#interning) | [left recursion](#left-recursion) | [line table](#line-table) | [magic number](#magic-number) | [marshal](#marshal) | [monitoring events](#monitoring-events) | [new reference](#new-reference) | [object](#object) | [object header](#object-header) | [obmalloc](#obmalloc) | [oparg](#oparg) | [opcode](#opcode) | [parser generator](#parser-generator) | [pdb](#pdb) | [pointer](#pointer) | [pool](#pool) | [product type](#product-type) | [profile guided optimization](#profile-guided-optimization) | [pseudo instruction](#pseudo-instruction) | [pyc file](#pyc-file) | [pyconfig](#pyconfig) | [reference count](#reference-count) | [reference cycle](#reference-cycle) | [reference leak](#reference-leak) | [regen](#regen) | [regrtest](#regrtest) | [replacement field](#replacement-field) | [resource](#resource) | [scope](#scope) | [segmentation fault](#segmentation-fault) | [short circuiting](#short-circuiting) | [small integer cache](#small-integer-cache) | [soft keyword](#soft-keyword) | [specialization](#specialization) | [stack depth](#stack-depth) | [stack effect](#stack-effect) | [stolen reference](#stolen-reference) | [struct](#struct) | [sum type](#sum-type) | [symbol table](#symbol-table) | [symbol table pass](#symbol-table-pass) | [t string](#t-string) | [test case](#test-case) | [tier one](#tier-one) | [tier two](#tier-two) | [token](#token) | [tokenizer](#tokenizer) | [trace function](#trace-function) | [type object](#type-object) | [underflow](#underflow) | [value stack](#value-stack) | [weak reference](#weak-reference) ## Reading the source @@ -634,6 +634,14 @@ It is a real dict and you can look at it, which is why `self.name = name` works Also written `__dict__`. First met in T08. See also [object](#object), [type object](#type-object). +### PyVarObject + +**An object header with a length field welded on the end of it.** + +Tuples, lists and bytes objects all hold a count of how many items they have, and rather than each of them inventing a field for it the header itself grows by one machine word called `ob_size`. It is the same trick as the header: put the thing everybody needs in a fixed place so that generic code can read it without knowing the type. Strings keep their length in the same place without being one of these, and integers used to and no longer do. + +Also written variable sized object, `ob_size`. First met in O01. See also [object header](#object-header), [object](#object). In the source: [`Include/object.h:174-178@v3.15.0rc1#PyVarObject`](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L174-L178). + ## Memory Where objects come from and what happens to them afterwards. T09 is the lesson. @@ -718,6 +726,14 @@ The collector runs finalizers on the objects in a cycle before it frees any of t Also written `__del__`. First met in T09. See also [deallocation](#deallocation), [cycle collector](#cycle-collector). In the source: [`Python/gc.c:1041-1074@v3.15.0rc1#finalize_garbage`](https://github.com/python/cpython/blob/v3.15.0rc1/Python/gc.c#L1041-L1074). +### GC pre header + +**Two words allocated in front of an object, holding its place in the collector's list.** + +Only the types the cycle collector tracks get one, and the object's own address points past it, so nothing that reads the header ever sees it. You can still measure it: `sys.getsizeof` adds it and the object's own `__sizeof__` does not, so the gap between those two is exactly this. + +Also written `PyGC_Head`. First met in O01. See also [cycle collector](#cycle-collector), [object header](#object-header). In the source: [`Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1#PyGC_Head`](https://github.com/python/cpython/blob/v3.15.0rc1/Include/internal/pycore_interp_structs.h#L158-L169). + ## Building the interpreter The words that turn out to be about the binary rather than about the language. B01 through B04 are the lessons, and several numbers in the earlier lessons move when the build does. diff --git a/README.md b/README.md index 2440ff9..ab44be4 100644 --- a/README.md +++ b/README.md @@ -116,6 +116,7 @@ A lesson can also end with a boss fight, which is a problem the text does not so | F10 | [Inside a code object](lessons/f10-inside-a-code-object/f10.ipynb) | The record the compiler hands back, walked from a module down to a method, the seven slots behind three tuples that add up to eight, why a closure's first instruction runs before line one, every bit in co_flags, what equality compares and what it deliberately ignores, and what happens when you try to change one | M3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f10-inside-a-code-object/f10.ipynb) | | F11 | [Two tables on the side](lessons/f11-two-tables-on-the-side/f11.ipynb) | The two blobs beside the bytecode, why wrapping a loop in a try changes not one instruction, the four numbers in an exception table entry and the marker bit that makes a variable length table binary searchable, the six forms a location entry can take, and two decoders written here that agree with the interpreter's | M3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f11-two-tables-on-the-side/f11.ipynb) | | F12 | [What ends up on disk](lessons/f12-what-ends-up-on-disk/f12.ipynb) | The other direction: a code object flattened into bytes and read straight back, the sixteen byte header decoded field by field, the type byte and the reference table that make marshal small, a fifty line reader checked against the real one, and a handmade .pyc that imports with no .py anywhere | M3 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f12-what-ends-up-on-disk/f12.ipynb) | +| O01 | [The header, byte by byte](lessons/o01-the-header-byte-by-byte/o01.ipynb) | The two fields everybody quotes, read out of your own interpreter with ctypes, and the first of them turning out to be three: a thirty two bit count, sixteen bits named and unused, and the flags that say whether an object can ever be freed, plus why an immortal starts in the middle of the range and where a length lives when a type has one | M4 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o01-the-header-byte-by-byte/o01.ipynb) | More are landing in order. [lessons/README.md](lessons/README.md) explains how one is put together and how to run them locally. diff --git a/citations.lock.json b/citations.lock.json index 9589552..75b02b9 100644 --- a/citations.lock.json +++ b/citations.lock.json @@ -155,6 +155,16 @@ "first_line": "struct _PyCfgBuilder* _PyCfg_FromInstructionSequence(_PyInstructionSequence *seq);", "lines": 5 }, + "Include/internal/pycore_global_strings.h:30-38@v3.15.0rc1": { + "digest": "6e8ba9cbf4bb5ee4", + "first_line": "/* The following is auto-generated by Tools/build/generate_global_objects.py. */", + "lines": 9 + }, + "Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1": { + "digest": "dfd8f4ee4f87e6a6", + "first_line": "", + "lines": 12 + }, "Include/internal/pycore_interp_structs.h:271-286@v3.15.0rc1": { "digest": "84ddd939ee6f307f", "first_line": "#ifndef Py_GIL_DISABLED", @@ -180,6 +190,11 @@ "first_line": "#define PYC_MAGIC_NUMBER 3666", "lines": 6 }, + "Include/internal/pycore_object.h:83-89@v3.15.0rc1": { + "digest": "70fca12f791324cc", + "first_line": "#if SIZEOF_VOID_P > 4", + "lines": 7 + }, "Include/internal/pycore_obmalloc.h:128-146@v3.15.0rc1": { "digest": "2b3a0f329f4a22fe", "first_line": "/*", @@ -265,16 +280,41 @@ "first_line": "#define LOCAL 1", "lines": 6 }, + "Include/object.h:127-149@v3.15.0rc1": { + "digest": "c7e17988b3729573", + "first_line": "struct _object {", + "lines": 23 + }, "Include/object.h:127-150@v3.15.0rc1": { "digest": "355a45f89904b83a", "first_line": "struct _object {", "lines": 24 }, + "Include/object.h:156-167@v3.15.0rc1": { + "digest": "5dfbd32621883aa8", + "first_line": "struct _object {", + "lines": 12 + }, "Include/object.h:156-170@v3.15.0rc1": { "digest": "3721c9256c01c28f", "first_line": "struct _object {", "lines": 15 }, + "Include/object.h:174-178@v3.15.0rc1": { + "digest": "f70e693e6b6d5dc5", + "first_line": "struct PyVarObject {", + "lines": 5 + }, + "Include/object.h:237-244@v3.15.0rc1": { + "digest": "dc45c7bd6a91c2b3", + "first_line": "// bpo-39573: The Py_SET_SIZE() function must be used to set an object size.", + "lines": 8 + }, + "Include/object.h:580-583@v3.15.0rc1": { + "digest": "323df69ccd90ebf9", + "first_line": "// Flag values for ob_flags (16 bits available, if SIZEOF_VOID_P > 4).", + "lines": 4 + }, "Include/object.h:623-629@v3.15.0rc1": { "digest": "cd6bd3f1f3e26b0c", "first_line": "/* Macro for returning Py_None from a function.", @@ -295,6 +335,16 @@ "first_line": "#ifndef _Py_OPAQUE_PYOBJECT", "lines": 12 }, + "Include/refcount.h:126-136@v3.15.0rc1": { + "digest": "dd1efebab1b34716", + "first_line": "static inline Py_ALWAYS_INLINE int _Py_IsImmortal(PyObject *op)", + "lines": 11 + }, + "Include/refcount.h:23-50@v3.15.0rc1": { + "digest": "6f9e131a08baae2f", + "first_line": "#if SIZEOF_VOID_P > 4", + "lines": 28 + }, "Include/refcount.h:417-429@v3.15.0rc1": { "digest": "a2af4e3c17465fe0", "first_line": "static inline Py_ALWAYS_INLINE void Py_DECREF(PyObject *op)", @@ -585,6 +635,11 @@ "first_line": "static PyObject *", "lines": 6 }, + "Objects/object.c:2773-2791@v3.15.0rc1": { + "digest": "84c3b29a29ccb125", + "first_line": "void", + "lines": 19 + }, "Objects/object.c:3282-3300@v3.15.0rc1": { "digest": "6b3a6c6e934a76fc", "first_line": "/*", diff --git a/lessons/CLAIMS.md b/lessons/CLAIMS.md index 99c9591..378786c 100644 --- a/lessons/CLAIMS.md +++ b/lessons/CLAIMS.md @@ -12,7 +12,7 @@ header, what the allocator does with a freed block, the shape of the eval loop. marked with the reason, and a lesson is allowed at most 3 of them. The cap is the point. Without it the exception becomes the rule and this goes back to being a book. -243 claims across 28 lessons, 19 of them not observable from Python. +250 claims across 29 lessons, 20 of them not observable from Python. ## B01. Building CPython, and whether you need to @@ -183,6 +183,18 @@ Without it the exception becomes the rule and this goes back to being a book. | a .pyc assembled by hand imports and runs with no source file on disk | [`f12-26`](f12-what-ends-up-on-disk/f12.ipynb) | | a wrong magic number raises and a wrong timestamp is fixed silently | [`f12-29`](f12-what-ends-up-on-disk/f12.ipynb) | +## O01. The header, byte by byte + +| Claim | Proved by | +| --- | --- | +| id() is the address an object lives at, and the two machine words there are its reference count and a pointer to its type | [`o01-07`](o01-the-header-byte-by-byte/o01.ipynb) | +| the reference count is thirty two bits wide, and the other half of that word holds two more fields | [`o01-10`](o01-the-header-byte-by-byte/o01.ipynb) | +| an immortal object's count starts halfway between the immortality line and the top of the field, leaving about a billion of slack in each direction | [`o01-13`](o01-the-header-byte-by-byte/o01.ipynb) | +| a string literal that happens to be one of CPython's own identifiers is immortal, and the same characters CPython does not use are not | [`o01-16`](o01-the-header-byte-by-byte/o01.ipynb) | +| for a tuple, a list, a bytes and a str, the machine word after the header is the length | [`o01-19`](o01-the-header-byte-by-byte/o01.ipynb) | +| sys.getsizeof reports two machine words more than an object's own __sizeof__ for the types the cycle collector tracks, and nothing extra for the types it does not | [`o01-21`](o01-the-header-byte-by-byte/o01.ipynb) | +| the free threaded build gives every object a thirty two byte header, split so the owning thread can increment without an atomic instruction | not observable from Python: the fields only exist in a build configured with --disable-gil, and this notebook is not running one | + ## T01. One line, seven stages | Claim | Proved by | diff --git a/lessons/README.md b/lessons/README.md index 51b90b7..d384103 100644 --- a/lessons/README.md +++ b/lessons/README.md @@ -32,6 +32,7 @@ Each lesson is a notebook you can run. There is nothing to install and nothing t | [F10. Inside a code object](f10-inside-a-code-object/f10.ipynb) | What compiling a file actually hands you, the code objects that live in other code objects' constants, the one array behind co_varnames and co_cellvars and co_freevars, the instructions that run before your first line, what co_flags remembers, and why two code objects can be equal without being the same one | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f10-inside-a-code-object/f10.ipynb) | | [F11. Two tables on the side](f11-two-tables-on-the-side/f11.ipynb) | Why a try you never trip costs nothing, the exception table decoded byte by byte and checked against dis, how a raise binary searches a table whose entries are not the same length, the six shapes a source location can take, and a hand written line table decoder run against every code object in dis.py | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f11-two-tables-on-the-side/f11.ipynb) | | [F12. What ends up on disk](f12-what-ends-up-on-disk/f12.ipynb) | What a .pyc actually holds, the four header fields and which two decide staleness, why half the magic number is a carriage return and a newline, the one byte that names every marshalled object, how a repeated value costs five bytes, a marshal reader written here, and a .pyc assembled by hand and imported with no source file on disk | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/f12-what-ends-up-on-disk/f12.ipynb) | +| [O01. The header, byte by byte](o01-the-header-byte-by-byte/o01.ipynb) | Reading the object header out of memory with ctypes and checking every field against something Python will tell you another way, the thirty two bit count and the two fields sharing its word, why an immortal object parks at 3 << 30 rather than at the top, the two flag bits that say how it got there, why a string CPython uses itself is immortal and yours is not, and the two words in front of the header that only the collector sees | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o01-the-header-byte-by-byte/o01.ipynb) | ## The three programs diff --git a/lessons/o01-the-header-byte-by-byte/build.py b/lessons/o01-the-header-byte-by-byte/build.py new file mode 100644 index 0000000..9da038f --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/build.py @@ -0,0 +1,339 @@ +#!/usr/bin/env python +"""O01. The header, byte by byte. + +The first lesson of the object model part, and the twenty seventh overall. T08 said that every +object starts with a reference count and a pointer to its type, which is true and is what +almost every article about CPython says. This lesson reads the actual bytes, and finds that +the first of those two fields is really three. + +The count is thirty two bits, not sixty four. Above it sit sixteen bits with a name and no +users, and above those sixteen bits of flags, two of which decide whether an object is +immortal and whether shutting down is allowed to free it. + +Everything here is read with ctypes out of the reader's own interpreter, so none of it is +taken on trust. The offsets are computed from the pointer size rather than hardcoded, which is +what lets the same cells run in a browser, where Python is a thirty two bit build and the +header is half the size. + +Run this file to regenerate the notebook, or `just build-lessons` to regenerate all of them. +`just lessons` checks that the committed notebook still matches this file. +""" + +from nbbuild import BANNER, Lesson +from nbdiagram import Diagrams + +lesson = Lesson("o01-the-header-byte-by-byte", "o01") +badge = lesson.badge +cite = lesson.cite +term = lesson.term +figure = Diagrams("o01-the-header-byte-by-byte").figure + + +lesson.md(f""" +# O01. The header, byte by byte + +{badge} + +Every article about CPython tells you the same thing about the {term("object header")}: two fields, a {term("reference count")} and a pointer to the type. It is a good summary and it is what T08 said. + +It is also missing something. On an ordinary sixty four bit build the count is thirty two bits wide, not sixty four, and the other half of that word is two more fields nobody mentions. One of them decides whether the object can ever be freed. + +This lesson reads the bytes out of your own interpreter with `ctypes` and checks each field against something Python will tell you another way. Nothing below is taken on trust. + +{figure("the-first-sixteen-bytes", "the header as five stacked fields, the first word split into three")} +""") + + +lesson.md(""" +## About the source references + +Now and then this lesson points at CPython's own source, like this: `Include/object.h:127-149@v3.15.0rc1#_object`. + +Read it as three parts: the file, the lines, and the release those line numbers belong to. Sometimes there is a fourth part after a `#`, which is the name of the thing those lines are inside. + +Every reference is a link, and every one is checked against the pinned source on each change, so a stale reference fails the build instead of sending you somewhere wrong. You never have to read any of it. The references are there so you can go deeper when you want to, and so you can check that this lesson is not making things up. + +## Setup + +Colab does not come with the small package these lessons use, so the next cell installs it. If you are running this from a checkout of the repository it is already installed and the cell does nothing. +""") + + +lesson.code(""" +import sys + +if sys.version_info < (3, 14): + print("This lesson needs CPython 3.14 or newer.") + print(f"This runtime is {sys.version.split()[0]}, and the cells below will not run on it.") +else: + try: + import pyxray + except ImportError: + %pip install -q "pyxray @ git+https://github.com/tamnd/cpython-internals@main#subdirectory=pyxray" + import pyxray +""") + + +lesson.md(""" +## Which Python is this + +Everything below was checked against the version this cell prints and against 3.14. The two agree on every number in this lesson, which is unusual and worth saying. +""") + + +lesson.code( + """ +import pyxray + +pyxray.show() +""", + differs=BANNER, + quiet=True, +) + + +lesson.md(f""" +## Where an object actually is + +`id()` is documented as a number that is unique for the lifetime of an object. In CPython it is the address, and `ctypes` will read memory at an address, so the two together are enough to look at any object's header from Python. + +Everything below computes its offsets from the size of a {term("pointer")} rather than writing 8 and 16 into the code. That is not fussiness. In a browser this notebook is running a thirty two bit build and every offset halves. + +{lesson.claim("id() is the address an object lives at, and the two machine words there are its reference count and a pointer to its type")} +""") + + +lesson.code( + """ +import ctypes + +WORD = ctypes.sizeof(ctypes.c_void_p) + + +class Thing: + pass + + +it = Thing() + +print(f" a pointer here is {WORD} bytes, so the header is {2 * WORD}") +print(f" it lives at {id(it)}") +print() +raw = ctypes.c_uint32.from_address(id(it)).value +counted = sys.getrefcount(it) - 1 + +print(f" first word, read as a count {raw}") +print(f" sys.getrefcount, less its own argument {counted}") +print(f" the same number: {raw == counted}") +print() +second = ctypes.c_size_t.from_address(id(it) + WORD).value +print(f" second word {second}") +print(f" id(Thing) is {id(Thing)}") +print(f" so the second word is the type pointer: {second == id(Thing)}") +print() + +also = it +print(f" after a second name, the count reads {ctypes.c_uint32.from_address(id(it)).value}") +del also +print(f" after deleting that name, it reads {ctypes.c_uint32.from_address(id(it)).value}") +""", + varies="The address is wherever your interpreter happened to put the object, and a pointer is four bytes in a browser rather than eight.", +) + + +lesson.md(f""" +## The count is not a whole word + +That first read used `c_uint32`, four bytes, and got the right answer. Try `c_uint64` on an ordinary object and you get the same answer, because the top half is zero. Try it on `None` and you do not. + +The C is {cite("Include/object.h:127-149@v3.15.0rc1#_object")}, and on a machine where a pointer is more than four bytes the first word is a union of three fields: `ob_refcnt` at thirty two bits, `ob_overflow` at sixteen, and `ob_flags` at sixteen. + +{figure("three-fields-in-one-word", "the first word of None split into flags, overflow and count")} + +`ob_overflow` is worth a moment. It is declared in that struct and it appears nowhere else in the entire source tree. Sixteen bits with a name, reserved and not yet used. + +{lesson.claim("the reference count is thirty two bits wide, and the other half of that word holds two more fields")} +""") + + +lesson.code( + """ +def header(value): + \"\"\"The first machine word of an object, split the way the C struct splits it.\"\"\" + word = ctypes.c_size_t.from_address(id(value)).value + return word & 0xFFFFFFFF, (word >> 32) & 0xFFFF, (word >> 48) & 0xFFFF + + +for label, value in [("a Thing", it), ("the float 1.5", 1.5), ("None", None), ("str", str)]: + count, overflow, flags = header(value) + print(f" {label:14} count {count:>12} overflow {overflow} flags {flags}") +""", + varies="The two ordinary counts depend on what else in your session is holding those objects. And in a browser Python is a thirty two bit build, where the whole word is the count and the other two fields are not in the struct at all, so those columns read zero.", +) + + +lesson.md(f""" +## A billion either way + +`None` came back with a count of 3221225472, which is `3 << 30`. Nothing is holding `None` three billion times. It is an {term("immortal object")}: the count is parked at a value the interpreter never decrements, so `None` is never freed and, more to the point, nothing ever writes to that cache line. + +The test is a comparison, {cite("Include/refcount.h:126-136@v3.15.0rc1#_Py_IsImmortal")}, and the interesting part is which two numbers it uses. The line is at `2 ** 31`, and the starting value is `3 << 30`, which is not the top of the field and not the line either. + +{figure("how-far-off-you-can-be", "the four numbers that matter in a thirty two bit reference count")} + +The comment at {cite("Include/refcount.h:23-50@v3.15.0rc1#_Py_IMMORTAL_INITIAL_REFCNT")} says why. An extension compiled against Python 3.11 has an old `Py_INCREF` in it that knows nothing about immortality, so it increments and decrements `None` like anything else. Parking the count in the middle of the immortal range means that extension can be off by about a billion in either direction and `None` stays immortal. + +{lesson.claim("an immortal object's count starts halfway between the immortality line and the top of the field, leaving about a billion of slack in each direction")} +""") + + +lesson.code( + """ +INITIAL = 3 << 30 +MINIMUM = 1 << 31 +TOP = 2**32 + +print(f" the count parked in None {sys.getrefcount(None):>12}") +print(f" 3 << 30 {INITIAL:>12}") +print(f" the immortality line, 1 << 31 {MINIMUM:>12}") +print(f" the top of the field, 2 ** 32 {TOP:>12}") +print() +print(f" slack below the start {INITIAL - MINIMUM:>13,}") +print(f" slack above the start {TOP - INITIAL:>13,}") +""", + varies="A thirty two bit build draws the line at 1 << 30 and parks a static immortal at 7 << 28, so in a browser the first number and the two amounts of slack are different. The shape of the argument is the same.", +) + + +lesson.md(f""" +## Two ways to never be freed + +`None` also came back with `ob_flags` of 5. Three of those sixteen bits have meanings, {cite("Include/object.h:580-583@v3.15.0rc1#_Py_STATICALLY_ALLOCATED_FLAG")}, and 5 is bit 0 and bit 2: immortal, and statically allocated. + +Statically allocated means the object is a `PyObject` written out in the C source and compiled into the binary. {cite("Include/internal/pycore_object.h:83-89@v3.15.0rc1#_PyObject_HEAD_INIT")} is the initialiser that sets both bits at once. + +There is another way to become immortal. {cite("Objects/object.c:2773-2791@v3.15.0rc1#_Py_SetImmortalUntracked")} parks the count on an object that was allocated normally, and it sets bit 0 only. That distinction is not cosmetic: when the interpreter shuts down it has to free the promoted ones and must leave the static ones alone, because the static ones are not on the heap. + +{figure("two-ways-to-become-immortal", "static immortals with both bits against promoted immortals with one")} + +The surprise is in the strings. CPython ships a table of the identifier strings it uses itself, {cite("Include/internal/pycore_global_strings.h:30-38@v3.15.0rc1#_Py_global_strings")}, generated by a script and compiled into the binary. Write `"self"` in your own code and the compiler {term("interning", "interns")} it to that table entry, so your string is a static immortal. Write something CPython has never heard of and you get an ordinary heap string with an ordinary count. + +{lesson.claim("a string literal that happens to be one of CPython's own identifiers is immortal, and the same characters CPython does not use are not")} +""") + + +lesson.code( + """ +for text in ["self", "append", "flags", "zzz nobody uses this one"]: + count, _, flags = header(text) + print(f" {text!r:28} count {count:>12} flags {flags}") + +print() +print(f" sys.intern on a fresh string leaves flags at {header(sys.intern('q w e r t y'))[2]}") +print(" so interning and immortalising are two different things") +""", + varies="The counts depend on what else in your session is holding these strings. Which of them are immortal does not.", +) + + +lesson.md(f""" +## The third word, when there is one + +After the two header words, some types put a length. Those are the variable sized ones, {cite("Include/object.h:174-178@v3.15.0rc1#PyVarObject")}, and the field is `ob_size`. A tuple of three, a list of seven and a `bytes` of four all put their length right there. + +{figure("where-a-length-lives", "which types keep a length in the third word and which do not")} + +Two things on that picture are worth pointing at. A `str` is not a {term("PyVarObject")} at all, and it still keeps its length in the same place, because `PyASCIIObject` puts a plain `length` field immediately after the header. And an `int` does not, which is why the accessor asserts about it: {cite("Include/object.h:237-244@v3.15.0rc1#_Py_SIZE_impl")} refuses to run on a `PyLong` or a `PyBool`, because since 3.12 an integer packs its sign and its digit count into one field with a different meaning. + +{lesson.claim("for a tuple, a list, a bytes and a str, the machine word after the header is the length")} +""") + + +lesson.code(""" +def third_word(value): + \"\"\"Whatever is in the word after the header, read as a signed count.\"\"\" + return ctypes.c_ssize_t.from_address(id(value) + 2 * WORD).value + + +for value in [(1, 2, 3), [0] * 7, b"abcd", bytearray(3), "hello"]: + kind = type(value).__name__ + print(f" {kind:10} {value!r:26} len {len(value)} third word {third_word(value)}") +""") + + +lesson.md(f""" +## What sits in front of the header + +There is one more thing in the sixteen bytes story, and it is not in the sixteen bytes. It is in front of them. + +Types the {term("cycle collector")} tracks get a {term("GC pre header")} allocated immediately before the object, {cite("Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1#PyGC_Head")}, two words holding the links that thread every tracked object into a list. The object's own address points past it, so nothing that reads the header ever sees it. + +You can see it as a gap. `value.__sizeof__()` is what the type says about itself. `sys.getsizeof(value)` calls that and then adds the pre header if the type has one. + +{lesson.claim("sys.getsizeof reports two machine words more than an object's own __sizeof__ for the types the cycle collector tracks, and nothing extra for the types it does not")} +""") + + +lesson.code( + """ +everything = [ + ("object()", object()), + ("1.5", 1.5), + ("'hello'", "hello"), + ("()", ()), + ("(1, 2, 3)", (1, 2, 3)), + ("[1, 2, 3]", [1, 2, 3]), + ("{'a': 1}", {"a": 1}), +] + +for label, value in everything: + own = value.__sizeof__() + both = sys.getsizeof(value) + print(f" {label:10} __sizeof__ {own:>3} getsizeof {both:>3} gap {both - own}") +""", + varies="Every number here is counted in pointers, so they roughly halve in a browser. The gap is two pointers wherever it is not zero.", +) + + +lesson.md(f""" +## The same header, wider + +Everything above is the ordinary build. The {term("free threaded build")} has a different header, {cite("Include/object.h:156-167@v3.15.0rc1#_object")}, and it is twice the size. + +{figure("the-same-header-wider", "the sixteen byte header against the thirty two byte free threaded one")} + +The reason is the count. Incrementing a shared counter means writing to a cache line, and if two threads hold the same object then every increment on one core invalidates the other's copy. So the free threaded header splits the count in two: `ob_ref_local`, which only the owning thread touches and therefore needs no atomic instruction, and `ob_ref_shared`, which everybody else uses. `ob_tid` says which thread owns it, and there is a one byte mutex in there as well for the per object lock. + +{lesson.claim("the free threaded build gives every object a thirty two byte header, split so the owning thread can increment without an atomic instruction", unobservable="the fields only exist in a build configured with --disable-gil, and this notebook is not running one")} + +That is the whole design of free threaded CPython in one struct, and the concurrency lessons come back to it. + +## Try it yourself + +Three things to poke at. + +Walk `gc.get_objects()` and read the flags word on every one of them. Nothing tracked by the cycle collector is immortal, because `_Py_SetImmortal` untracks first. Seeing that come out as a clean result rather than a claim is worth the six lines. + +Find the highest reference count in your session. Sort by the count field, look at the top ten, and see whether you can explain each one. The empty tuple and the interpreter's own small integers will be near the top and they will be parked rather than counted, so filter those out first. + +Take a class with `__slots__` and one without, and compare `sys.getsizeof` on an instance of each. The gap tells you what an {term("instance dictionary")} costs, and the header is the same either way. + +## What just happened + +Two fields is a good summary of the header and it is not what is in memory. On an ordinary sixty four bit build the first word is a thirty two bit count, sixteen bits reserved under the name `ob_overflow` and used nowhere, and sixteen bits of flags. The second word is the type pointer. + +Immortality is a comparison against `2 ** 31`, and the value it parks at is `3 << 30`, deliberately in the middle so that an old extension incrementing and decrementing without checking can be a billion out either way and the object stays immortal. + +Two of the flag bits say how the object became immortal, because shutting down has to free the promoted ones and must not touch the ones compiled into the binary. Every identifier string CPython uses itself is in that second category, which is why `"self"` in your code is immortal and a string CPython has never seen is not. + +After the header, variable sized types put their length. Strings are not variable sized types and put it in the same place anyway. Integers used to and no longer do, and the accessor asserts about it. + +And in front of the header, for the types the cycle collector tracks, there are two more words that the object's own address points past. + +## What is next + +The type pointer is the field everything else in this part hangs off. O02 follows it, into the type object: what a `PyTypeObject` holds, which of its hundred or so fields are slots and which are bookkeeping, and how a type written in Python and a type written in C end up being the same kind of thing. +""") + + +raise SystemExit(lesson.save()) diff --git a/lessons/o01-the-header-byte-by-byte/diagrams.py b/lessons/o01-the-header-byte-by-byte/diagrams.py new file mode 100644 index 0000000..79a8f7c --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python +"""The diagrams for O01, the object header read one field at a time. + +Each scene is written out twice, as an editable `.excalidraw` and as the `.svg` the lesson +embeds. Run this file to regenerate them, or `just build-diagrams` for every lesson. + +The one carrying the lesson is `the-first-sixteen-bytes`. Everybody learns the header as two +fields, a count and a type pointer, and that is close enough until you look at the bytes. The +count is thirty two bits, not sixty four, and the rest of that word is doing other jobs. +""" + +from nbdiagram import Gallery, figures + +gallery = Gallery("o01-the-header-byte-by-byte") + +gallery.add( + figures.stack( + "the-first-sixteen-bytes", + [ + "bytes 0 to 3, ob_refcnt, the count, thirty two bits of it", + "bytes 4 to 5, ob_overflow, a name with no users yet", + "bytes 6 to 7, ob_flags, sixteen bits, three of them defined", + "bytes 8 to 15, ob_type, one pointer to the type object", + "byte 16 onwards, whatever this particular type needs", + ], + title="What every object in a running Python starts with", + note="Two fields is the usual summary, and the first of the two is really three.", + ) +) + + +gallery.add( + figures.spans( + "three-fields-in-one-word", + "0005 0000 c0000000", + [ + (0, 4, "ob_flags is 5"), + (5, 9, "ob_overflow is 0"), + (10, 18, "ob_refcnt is 3 << 30"), + ], + title="The first word of None, written the way you would read a number", + caption="Bits 48 and up are flags, the middle sixteen bits are named and unused, and the bottom half is the count.", + ) +) + + +gallery.add( + figures.table( + "how-far-off-you-can-be", + ["value", "the name in the source", "what it means"], + [ + ["0", "nothing holds this", "the object is about to be freed"], + ["2 ** 31", "_Py_IMMORTAL_MINIMUM_REFCNT", "at or above this, treated as immortal"], + ["3 << 30", "_Py_IMMORTAL_INITIAL_REFCNT", "where an immortal object starts"], + [ + "2 ** 32", + "the top of a thirty two bit field", + "increments stop here rather than wrap", + ], + ], + title="Why an immortal object starts in the middle and not at the top", + caption="An old extension can be a billion out either way and the object is still immortal.", + tones=["quiet", "focus", "focus", "quiet"], + ) +) + + +gallery.add( + figures.compare( + "two-ways-to-become-immortal", + ( + "built into the binary", + [ + "a static PyObject in the C source", + "ob_flags is 5, so both bits are on", + "None, True, small ints, every type", + "and the identifiers CPython ships", + ], + ), + ( + "promoted while running", + [ + "_Py_SetImmortal parks the count", + "ob_flags is 1, immortal bit only", + "sys.intern does not do this", + "nor does anything you can call", + ], + ), + title="Immortal by construction, and immortal by promotion", + verdict="Not decoration. Shutting down frees the promoted ones and leaves the static ones alone.", + verdict_tone="focus", + ) +) + + +gallery.add( + figures.table( + "where-a-length-lives", + ["type", "the third word", "how len gets it"], + [ + ["tuple", "ob_size", "a PyVarObject, read straight out"], + ["list", "ob_size", "the length, and not the capacity"], + ["bytes", "ob_size", "a PyVarObject too"], + ["str", "length", "not a PyVarObject, same offset anyway"], + ["int", "lv_tag", "sign and digit count packed together"], + ["object, float", "there is none", "sixteen and twenty four bytes, and they stop"], + ], + title="The third word, for the types that have one", + caption="Py_SIZE asserts the type is not an int and not a bool, because those two moved their length elsewhere.", + tones=["quiet", "focus", "quiet", "focus", "focus", "warning"], + ) +) + + +gallery.add( + figures.compare( + "the-same-header-wider", + ( + "ordinary build, sixteen bytes", + [ + "ob_refcnt, thirty two bits", + "ob_overflow, sixteen bits", + "ob_flags, sixteen bits", + "ob_type, one pointer", + ], + ), + ( + "free threaded build, thirty two bytes", + [ + "ob_tid, which thread owns it", + "ob_flags, and a one byte mutex", + "ob_gc_bits", + "ob_ref_local and ob_ref_shared", + "ob_type, one pointer", + ], + ), + title="One object, two headers, and the choice is made at build time", + verdict="Every object costs sixteen more bytes so that two threads can hold the same one without fighting over a cache line.", + verdict_tone="focus", + ) +) + + +raise SystemExit(gallery.save()) diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.excalidraw b/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.excalidraw new file mode 100644 index 0000000..9859790 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.excalidraw @@ -0,0 +1,1035 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://github.com/tamnd/cpython-internals", + "elements": [ + { + "id": "feb37d3a790b6f0c1c91", + "type": "text", + "x": 0.0, + "y": 0.0, + "width": 835.56, + "height": 30.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "Why an immortal object starts in the middle and not at the top", + "originalText": "Why an immortal object starts in the middle and not at the top", + "fontSize": 24, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "159610957f8012cacb6d", + "type": "text", + "x": 14.0, + "y": 50.0, + "width": 44.4, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "value", + "originalText": "value", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "32a6e7d65c97cafbb08c", + "type": "text", + "x": 109.2, + "y": 50.0, + "width": 202.56, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "the name in the source", + "originalText": "the name in the source", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "73f5ef0f69bd9f59c46a", + "type": "text", + "x": 454.0, + "y": 50.0, + "width": 125.52, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "what it means", + "originalText": "what it means", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "aa000ed1c9e9a5c83760", + "type": "line", + "x": 0.0, + "y": 76.0, + "width": 823.2, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "54f78cdbba737ede1e4c", + "type": "line", + "x": 0.0, + "y": 76.0, + "width": 823.2, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ], + [ + 823.2, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "e254125f7fabc3a532bc", + "type": "text", + "x": 14.0, + "y": 86.0, + "width": 9.6, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "0", + "originalText": "0", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "2a3b0a928b7be3f546d5", + "type": "text", + "x": 109.2, + "y": 86.0, + "width": 172.79999999999998, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "nothing holds this", + "originalText": "nothing holds this", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "ff039534de90200bb68f", + "type": "text", + "x": 454.0, + "y": 86.0, + "width": 297.59999999999997, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "the object is about to be freed", + "originalText": "the object is about to be freed", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "eb7f4bd251220caba245", + "type": "line", + "x": 0.0, + "y": 116.0, + "width": 823.2, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "f09168d4dd4a856a3b26", + "type": "line", + "x": 0.0, + "y": 116.0, + "width": 823.2, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ], + [ + 823.2, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "23b3e36507af0b86b3f2", + "type": "text", + "x": 14.0, + "y": 126.0, + "width": 67.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "2 ** 31", + "originalText": "2 ** 31", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "bfd1ee111d930c9e9bfd", + "type": "text", + "x": 109.2, + "y": 126.0, + "width": 259.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "_Py_IMMORTAL_MINIMUM_REFCNT", + "originalText": "_Py_IMMORTAL_MINIMUM_REFCNT", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "3e36f63628b19ced8c7c", + "type": "text", + "x": 454.0, + "y": 126.0, + "width": 355.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "at or above this, treated as immortal", + "originalText": "at or above this, treated as immortal", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "99f3b17c57b86995928a", + "type": "line", + "x": 0.0, + "y": 156.0, + "width": 823.2, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "c0a255c49a0250d01039", + "type": "line", + "x": 0.0, + "y": 156.0, + "width": 823.2, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ], + [ + 823.2, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "54a59a6b66ce7ffbba0a", + "type": "text", + "x": 14.0, + "y": 166.0, + "width": 67.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "3 << 30", + "originalText": "3 << 30", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "4b92773b550a651149ba", + "type": "text", + "x": 109.2, + "y": 166.0, + "width": 259.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "_Py_IMMORTAL_INITIAL_REFCNT", + "originalText": "_Py_IMMORTAL_INITIAL_REFCNT", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "18cee259573435b514b7", + "type": "text", + "x": 454.0, + "y": 166.0, + "width": 297.59999999999997, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "where an immortal object starts", + "originalText": "where an immortal object starts", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "d64400cdf1c11c16da71", + "type": "line", + "x": 0.0, + "y": 196.0, + "width": 823.2, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "bfde93b6501557877175", + "type": "line", + "x": 0.0, + "y": 196.0, + "width": 823.2, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ], + [ + 823.2, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "79a4aa194a048689d766", + "type": "text", + "x": 14.0, + "y": 206.0, + "width": 67.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "2 ** 32", + "originalText": "2 ** 32", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "befcc7878c8327ab651d", + "type": "text", + "x": 109.2, + "y": 206.0, + "width": 316.8, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "the top of a thirty two bit field", + "originalText": "the top of a thirty two bit field", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "0d2ef15a14a02190b23c", + "type": "text", + "x": 454.0, + "y": 206.0, + "width": 355.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "increments stop here rather than wrap", + "originalText": "increments stop here rather than wrap", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "ad625f6fc54128213af5", + "type": "line", + "x": 0.0, + "y": 236.0, + "width": 823.2, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 823.2, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "2917ee634a067f8f47cd", + "type": "text", + "x": 0.0, + "y": 256.0, + "width": 718.08, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "An old extension can be a billion out either way and the object is still immortal.", + "originalText": "An old extension can be a billion out either way and the object is still immortal.", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + } + ], + "appState": { + "gridSize": 20, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.svg b/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.svg new file mode 100644 index 0000000..63f5503 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.svg @@ -0,0 +1 @@ +Why an immortal object starts in the middle and not at the topvaluethe name in the sourcewhat it means0nothing holds thisthe object is about to be freed2 ** 31_Py_IMMORTAL_MINIMUM_REFCNTat or above this, treated as immortal3 << 30_Py_IMMORTAL_INITIAL_REFCNTwhere an immortal object starts2 ** 32the top of a thirty two bit fieldincrements stop here rather than wrapAn old extension can be a billion out either way and the object is still immortal. diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.excalidraw b/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.excalidraw new file mode 100644 index 0000000..f69ec4f --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.excalidraw @@ -0,0 +1,473 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://github.com/tamnd/cpython-internals", + "elements": [ + { + "id": "1e9dc094dfd7b705a270", + "type": "text", + "x": 0.0, + "y": 0.0, + "width": 652.6800000000001, + "height": 30.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "What every object in a running Python starts with", + "originalText": "What every object in a running Python starts with", + "fontSize": 24, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "75c05adfb3752245b036", + "type": "rectangle", + "x": 0.0, + "y": 50.0, + "width": 740.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#e8590c", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "94d086f465eaba1ce687", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "94d086f465eaba1ce687", + "type": "text", + "x": 16.0, + "y": 66.0, + "width": 708.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "byte 16 onwards, whatever this particular type needs", + "originalText": "byte 16 onwards, whatever this particular type needs", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "75c05adfb3752245b036", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "6ef753cd3b09602ad3d3", + "type": "text", + "x": 756.0, + "y": 66.0, + "width": 25.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "top", + "originalText": "top", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "248a85e369761ef12975", + "type": "rectangle", + "x": 0.0, + "y": 102.0, + "width": 740.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#6741d9", + "backgroundColor": "#d0bfff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "f542ce20b19fff1ee116", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "f542ce20b19fff1ee116", + "type": "text", + "x": 16.0, + "y": 118.0, + "width": 708.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bytes 8 to 15, ob_type, one pointer to the type object", + "originalText": "bytes 8 to 15, ob_type, one pointer to the type object", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "248a85e369761ef12975", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "e2ba363847713f6800c4", + "type": "rectangle", + "x": 0.0, + "y": 154.0, + "width": 740.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#6741d9", + "backgroundColor": "#d0bfff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "f7654e6525f8d534ca24", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "f7654e6525f8d534ca24", + "type": "text", + "x": 16.0, + "y": 170.0, + "width": 708.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bytes 6 to 7, ob_flags, sixteen bits, three of them defined", + "originalText": "bytes 6 to 7, ob_flags, sixteen bits, three of them defined", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "e2ba363847713f6800c4", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "4d85f0fffa389a8c8f7f", + "type": "rectangle", + "x": 0.0, + "y": 206.0, + "width": 740.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#6741d9", + "backgroundColor": "#d0bfff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "942c345417078de8a751", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "942c345417078de8a751", + "type": "text", + "x": 16.0, + "y": 222.0, + "width": 708.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bytes 4 to 5, ob_overflow, a name with no users yet", + "originalText": "bytes 4 to 5, ob_overflow, a name with no users yet", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "4d85f0fffa389a8c8f7f", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "ad4449f0b958b2dd8965", + "type": "rectangle", + "x": 0.0, + "y": 258.0, + "width": 740.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#6741d9", + "backgroundColor": "#d0bfff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "aafcfd05512b0508bbcf", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "aafcfd05512b0508bbcf", + "type": "text", + "x": 16.0, + "y": 274.0, + "width": 708.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bytes 0 to 3, ob_refcnt, the count, thirty two bits of it", + "originalText": "bytes 0 to 3, ob_refcnt, the count, thirty two bits of it", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "ad4449f0b958b2dd8965", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "15713b20329e034f61c7", + "type": "text", + "x": 0.0, + "y": 330.0, + "width": 650.6400000000001, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "Two fields is the usual summary, and the first of the two is really three.", + "originalText": "Two fields is the usual summary, and the first of the two is really three.", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + } + ], + "appState": { + "gridSize": 20, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.svg b/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.svg new file mode 100644 index 0000000..6cd9357 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.svg @@ -0,0 +1 @@ +What every object in a running Python starts withbyte 16 onwards, whatever this particular type needstopbytes 8 to 15, ob_type, one pointer to the type objectbytes 6 to 7, ob_flags, sixteen bits, three of them definedbytes 4 to 5, ob_overflow, a name with no users yetbytes 0 to 3, ob_refcnt, the count, thirty two bits of itTwo fields is the usual summary, and the first of the two is really three. diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.excalidraw b/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.excalidraw new file mode 100644 index 0000000..ad3d037 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.excalidraw @@ -0,0 +1,823 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://github.com/tamnd/cpython-internals", + "elements": [ + { + "id": "3b568523b98fc76f334a", + "type": "text", + "x": 0.0, + "y": 0.0, + "width": 828.72, + "height": 30.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "One object, two headers, and the choice is made at build time", + "originalText": "One object, two headers, and the choice is made at build time", + "fontSize": 24, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "d012d05045356f587ea1", + "type": "text", + "x": 84.25, + "y": 50.0, + "width": 307.5, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "ordinary build, sixteen bytes", + "originalText": "ordinary build, sixteen bytes", + "fontSize": 20, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "7c9c4225da7d30ac5fd4", + "type": "rectangle", + "x": 0.0, + "y": 87.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#1971c2", + "backgroundColor": "#a5d8ff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "2b8cc692b45f31aeca24", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "2b8cc692b45f31aeca24", + "type": "text", + "x": 16.0, + "y": 103.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_refcnt, thirty two bits", + "originalText": "ob_refcnt, thirty two bits", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "7c9c4225da7d30ac5fd4", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "0eb2e0f13dbe687503e3", + "type": "rectangle", + "x": 0.0, + "y": 147.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "192cfe47f4d32229806a", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "192cfe47f4d32229806a", + "type": "text", + "x": 16.0, + "y": 163.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_overflow, sixteen bits", + "originalText": "ob_overflow, sixteen bits", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "0eb2e0f13dbe687503e3", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "239b1beebe1c41928bb0", + "type": "rectangle", + "x": 0.0, + "y": 207.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "6cc95b6a909ae03a2500", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "6cc95b6a909ae03a2500", + "type": "text", + "x": 16.0, + "y": 223.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_flags, sixteen bits", + "originalText": "ob_flags, sixteen bits", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "239b1beebe1c41928bb0", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "17eceabcad2a197c56aa", + "type": "rectangle", + "x": 0.0, + "y": 267.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "3d50df39dce92007ac62", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "3d50df39dce92007ac62", + "type": "text", + "x": 16.0, + "y": 283.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_type, one pointer", + "originalText": "ob_type, one pointer", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "17eceabcad2a197c56aa", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "eb822d301bd5e9de2663", + "type": "text", + "x": 577.6500000000001, + "y": 50.0, + "width": 392.69999999999993, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "free threaded build, thirty two bytes", + "originalText": "free threaded build, thirty two bytes", + "fontSize": 20, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "46ae4cd42a7ad0444917", + "type": "rectangle", + "x": 536.0, + "y": 87.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#099268", + "backgroundColor": "#96f2d7", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "f10b58e30b92ee0b54da", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "f10b58e30b92ee0b54da", + "type": "text", + "x": 552.0, + "y": 103.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_tid, which thread owns it", + "originalText": "ob_tid, which thread owns it", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "46ae4cd42a7ad0444917", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "f13fab5287327b80449c", + "type": "rectangle", + "x": 536.0, + "y": 147.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "f0b5a9be67c5068e341d", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "f0b5a9be67c5068e341d", + "type": "text", + "x": 552.0, + "y": 163.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_flags, and a one byte mutex", + "originalText": "ob_flags, and a one byte mutex", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "f13fab5287327b80449c", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "14eff2f290f44d1806c8", + "type": "rectangle", + "x": 536.0, + "y": 207.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "9b9aa96cd0384d4aba27", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "9b9aa96cd0384d4aba27", + "type": "text", + "x": 552.0, + "y": 223.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_gc_bits", + "originalText": "ob_gc_bits", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "14eff2f290f44d1806c8", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "835e9e887a4883cf5f49", + "type": "rectangle", + "x": 536.0, + "y": 267.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "7c9b5b3c78b08449589e", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "7c9b5b3c78b08449589e", + "type": "text", + "x": 552.0, + "y": 283.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_ref_local and ob_ref_shared", + "originalText": "ob_ref_local and ob_ref_shared", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "835e9e887a4883cf5f49", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "02fa2c8ac1e078358760", + "type": "rectangle", + "x": 536.0, + "y": 327.0, + "width": 476.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "79fe9285673ffec3d643", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "79fe9285673ffec3d643", + "type": "text", + "x": 552.0, + "y": 343.0, + "width": 444.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_type, one pointer", + "originalText": "ob_type, one pointer", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "02fa2c8ac1e078358760", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "47d5f87c19ea0763a12e", + "type": "rectangle", + "x": 0.0, + "y": 404.0, + "width": 1012.0, + "height": 82.0, + "angle": 0, + "strokeColor": "#e8590c", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "3fc8bd1651c8ede2ec4b", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "3fc8bd1651c8ede2ec4b", + "type": "text", + "x": 16.0, + "y": 420.0, + "width": 980.0, + "height": 50.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Every object costs sixteen more bytes so that two threads can hold the same one\nwithout fighting over a cache line.", + "originalText": "Every object costs sixteen more bytes so that two threads can hold the same one\nwithout fighting over a cache line.", + "fontSize": 20, + "fontFamily": 2, + "textAlign": "center", + "verticalAlign": "middle", + "containerId": "47d5f87c19ea0763a12e", + "lineHeight": 1.25, + "autoResize": false + } + ], + "appState": { + "gridSize": 20, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.svg b/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.svg new file mode 100644 index 0000000..2f5384a --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.svg @@ -0,0 +1 @@ +One object, two headers, and the choice is made at build timeordinary build, sixteen bytesob_refcnt, thirty two bitsob_overflow, sixteen bitsob_flags, sixteen bitsob_type, one pointerfree threaded build, thirty two bytesob_tid, which thread owns itob_flags, and a one byte mutexob_gc_bitsob_ref_local and ob_ref_sharedob_type, one pointerEvery object costs sixteen more bytes so that two threads can hold the same onewithout fighting over a cache line. diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.excalidraw b/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.excalidraw new file mode 100644 index 0000000..24535e1 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.excalidraw @@ -0,0 +1,602 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://github.com/tamnd/cpython-internals", + "elements": [ + { + "id": "88ca3e15c667a3ff7306", + "type": "text", + "x": 0.0, + "y": 0.0, + "width": 869.4000000000001, + "height": 30.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "The first word of None, written the way you would read a number", + "originalText": "The first word of None, written the way you would read a number", + "fontSize": 24, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "0206470a437b541a6122", + "type": "text", + "x": 0.0, + "y": 50.0, + "width": 216.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": { + "textLength": 216.0 + }, + "text": "0005 0000 c0000000", + "originalText": "0005 0000 c0000000", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "29b02b1fff83fe7f3d50", + "type": "line", + "x": 0.0, + "y": 81.0, + "width": 146.0, + "height": 72.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#a5d8ff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 40, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0.0, + 0.0 + ], + [ + 48.0, + 0.0 + ], + [ + 146.0, + 72.0 + ], + [ + 0.0, + 72.0 + ], + [ + 0.0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "e9b18a7a7c89ba4f1ee8", + "type": "line", + "x": 60.0, + "y": 81.0, + "width": 270.96, + "height": 72.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#d0bfff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 40, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0.0, + 0.0 + ], + [ + 48.0, + 0.0 + ], + [ + 270.96, + 72.0 + ], + [ + 94.0, + 72.0 + ], + [ + 0.0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "3c9abf50a17960835323", + "type": "line", + "x": 120.0, + "y": 81.0, + "width": 428.55999999999995, + "height": 72.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#96f2d7", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 40, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0.0, + 0.0 + ], + [ + 96.0, + 0.0 + ], + [ + 428.55999999999995, + 72.0 + ], + [ + 218.95999999999998, + 72.0 + ], + [ + 0.0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "b6cfc67f90a6e036a8f2", + "type": "line", + "x": 0.0, + "y": 81.0, + "width": 48.0, + "height": 0.0, + "angle": 0, + "strokeColor": "#1971c2", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0.0, + 0.0 + ], + [ + 48.0, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "d19ade0b8701325c22f8", + "type": "line", + "x": 60.0, + "y": 81.0, + "width": 48.0, + "height": 0.0, + "angle": 0, + "strokeColor": "#6741d9", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0.0, + 0.0 + ], + [ + 48.0, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "d206abdf4763b42359bb", + "type": "line", + "x": 120.0, + "y": 81.0, + "width": 96.0, + "height": 0.0, + "angle": 0, + "strokeColor": "#099268", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0.0, + 0.0 + ], + [ + 96.0, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "19084591a37bb54474e8", + "type": "rectangle", + "x": 0.0, + "y": 153.0, + "width": 146.0, + "height": 52.0, + "angle": 0, + "strokeColor": "#1971c2", + "backgroundColor": "#a5d8ff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "a87dcb6e5921969bde98", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "a87dcb6e5921969bde98", + "type": "text", + "x": 16.0, + "y": 169.0, + "width": 114.0, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_flags is 5", + "originalText": "ob_flags is 5", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "center", + "verticalAlign": "middle", + "containerId": "19084591a37bb54474e8", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "a7da81ebae89622c6e61", + "type": "rectangle", + "x": 154.0, + "y": 153.0, + "width": 176.95999999999998, + "height": 52.0, + "angle": 0, + "strokeColor": "#6741d9", + "backgroundColor": "#d0bfff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "e2c66b3830eb81519062", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "e2c66b3830eb81519062", + "type": "text", + "x": 170.0, + "y": 169.0, + "width": 144.95999999999998, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_overflow is 0", + "originalText": "ob_overflow is 0", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "center", + "verticalAlign": "middle", + "containerId": "a7da81ebae89622c6e61", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "a33545755f5b50e991fd", + "type": "rectangle", + "x": 338.96, + "y": 153.0, + "width": 209.6, + "height": 52.0, + "angle": 0, + "strokeColor": "#099268", + "backgroundColor": "#96f2d7", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "b762a7cd6eb439c784e5", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "b762a7cd6eb439c784e5", + "type": "text", + "x": 354.96, + "y": 169.0, + "width": 177.6, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_refcnt is 3 << 30", + "originalText": "ob_refcnt is 3 << 30", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "center", + "verticalAlign": "middle", + "containerId": "a33545755f5b50e991fd", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "e2990a463f8b6a4ae51e", + "type": "text", + "x": 0.0, + "y": 225.0, + "width": 938.88, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "Bits 48 and up are flags, the middle sixteen bits are named and unused, and the bottom half is the count.", + "originalText": "Bits 48 and up are flags, the middle sixteen bits are named and unused, and the bottom half is the count.", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + } + ], + "appState": { + "gridSize": 20, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.svg b/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.svg new file mode 100644 index 0000000..fcd6365 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.svg @@ -0,0 +1 @@ +The first word of None, written the way you would read a number0005 0000 c0000000ob_flags is 5ob_overflow is 0ob_refcnt is 3 << 30Bits 48 and up are flags, the middle sixteen bits are named and unused, and the bottom half is the count. diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.excalidraw b/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.excalidraw new file mode 100644 index 0000000..3dedab7 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.excalidraw @@ -0,0 +1,753 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://github.com/tamnd/cpython-internals", + "elements": [ + { + "id": "f47137652ddd5ad0da6d", + "type": "text", + "x": 0.0, + "y": 0.0, + "width": 691.2, + "height": 30.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "Immortal by construction, and immortal by promotion", + "originalText": "Immortal by construction, and immortal by promotion", + "fontSize": 24, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "ec05b37ef0cd0168aa4d", + "type": "text", + "x": 112.0, + "y": 50.0, + "width": 216.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "built into the binary", + "originalText": "built into the binary", + "fontSize": 20, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "208027f03953f1f71b07", + "type": "rectangle", + "x": 0.0, + "y": 87.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#1971c2", + "backgroundColor": "#a5d8ff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "29947175d54aaba89707", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "29947175d54aaba89707", + "type": "text", + "x": 16.0, + "y": 103.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "a static PyObject in the C source", + "originalText": "a static PyObject in the C source", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "208027f03953f1f71b07", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "6132a1a08ca0c45c5682", + "type": "rectangle", + "x": 0.0, + "y": 147.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "5bd6e0d2ec8ee41cc07d", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "5bd6e0d2ec8ee41cc07d", + "type": "text", + "x": 16.0, + "y": 163.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_flags is 5, so both bits are on", + "originalText": "ob_flags is 5, so both bits are on", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "6132a1a08ca0c45c5682", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "c7011f96e9c3d776de88", + "type": "rectangle", + "x": 0.0, + "y": 207.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "97d96cd503976edb3ac8", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "97d96cd503976edb3ac8", + "type": "text", + "x": 16.0, + "y": 223.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "None, True, small ints, every type", + "originalText": "None, True, small ints, every type", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "c7011f96e9c3d776de88", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "09caa5ce9b218035ce9d", + "type": "rectangle", + "x": 0.0, + "y": 267.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "13dae1eb59c4b69838d1", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "13dae1eb59c4b69838d1", + "type": "text", + "x": 16.0, + "y": 283.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "and the identifiers CPython ships", + "originalText": "and the identifiers CPython ships", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "09caa5ce9b218035ce9d", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "b534a5571ca9410ce4c4", + "type": "text", + "x": 594.3, + "y": 50.0, + "width": 251.4, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "promoted while running", + "originalText": "promoted while running", + "fontSize": 20, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "6d19f98ff85a8a72a2ac", + "type": "rectangle", + "x": 500.0, + "y": 87.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#099268", + "backgroundColor": "#96f2d7", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "b4b72a7a463d929e9185", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "b4b72a7a463d929e9185", + "type": "text", + "x": 516.0, + "y": 103.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "_Py_SetImmortal parks the count", + "originalText": "_Py_SetImmortal parks the count", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "6d19f98ff85a8a72a2ac", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "bb198829e97bfb7a1d77", + "type": "rectangle", + "x": 500.0, + "y": 147.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "312c0579e3ed6cdb3631", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "312c0579e3ed6cdb3631", + "type": "text", + "x": 516.0, + "y": 163.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ob_flags is 1, immortal bit only", + "originalText": "ob_flags is 1, immortal bit only", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "bb198829e97bfb7a1d77", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "00a86271b58d9bea5119", + "type": "rectangle", + "x": 500.0, + "y": 207.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "5d3b1e0722a9538bd202", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "5d3b1e0722a9538bd202", + "type": "text", + "x": 516.0, + "y": 223.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sys.intern does not do this", + "originalText": "sys.intern does not do this", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "00a86271b58d9bea5119", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "76f09e07948610583ca9", + "type": "rectangle", + "x": 500.0, + "y": 267.0, + "width": 440.0, + "height": 57.0, + "angle": 0, + "strokeColor": "#495057", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "5daa04c0c15cde5720c1", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "5daa04c0c15cde5720c1", + "type": "text", + "x": 516.0, + "y": 283.0, + "width": 408.0, + "height": 25.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "nor does anything you can call", + "originalText": "nor does anything you can call", + "fontSize": 20, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "middle", + "containerId": "76f09e07948610583ca9", + "lineHeight": 1.25, + "autoResize": false + }, + { + "id": "ae81700d72e6915b3a56", + "type": "rectangle", + "x": 0.0, + "y": 344.0, + "width": 940.0, + "height": 82.0, + "angle": 0, + "strokeColor": "#e8590c", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [ + { + "id": "1ef76387b21b35c55a34", + "type": "text" + } + ], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "1ef76387b21b35c55a34", + "type": "text", + "x": 16.0, + "y": 360.0, + "width": 908.0, + "height": 50.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Not decoration. Shutting down frees the promoted ones and leaves the static ones\nalone.", + "originalText": "Not decoration. Shutting down frees the promoted ones and leaves the static ones\nalone.", + "fontSize": 20, + "fontFamily": 2, + "textAlign": "center", + "verticalAlign": "middle", + "containerId": "ae81700d72e6915b3a56", + "lineHeight": 1.25, + "autoResize": false + } + ], + "appState": { + "gridSize": 20, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.svg b/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.svg new file mode 100644 index 0000000..da419f0 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.svg @@ -0,0 +1 @@ +Immortal by construction, and immortal by promotionbuilt into the binarya static PyObject in the C sourceob_flags is 5, so both bits are onNone, True, small ints, every typeand the identifiers CPython shipspromoted while running_Py_SetImmortal parks the countob_flags is 1, immortal bit onlysys.intern does not do thisnor does anything you can callNot decoration. Shutting down frees the promoted ones and leaves the static onesalone. diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.excalidraw b/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.excalidraw new file mode 100644 index 0000000..c34095b --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.excalidraw @@ -0,0 +1,1435 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://github.com/tamnd/cpython-internals", + "elements": [ + { + "id": "051f07e9d65fcdc57d7f", + "type": "text", + "x": 0.0, + "y": 0.0, + "width": 568.4399999999999, + "height": 30.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "The third word, for the types that have one", + "originalText": "The third word, for the types that have one", + "fontSize": 24, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "d2929e2f9bf61bf3032d", + "type": "text", + "x": 14.0, + "y": 50.0, + "width": 34.8, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "type", + "originalText": "type", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "3a3c64dbdb70f7499b1a", + "type": "text", + "x": 166.8, + "y": 50.0, + "width": 122.16, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "the third word", + "originalText": "the third word", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "82351274de6b41868254", + "type": "text", + "x": 329.20000000000005, + "y": 50.0, + "width": 135.36, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "how len gets it", + "originalText": "how len gets it", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "287ec08923e005124cbd", + "type": "line", + "x": 0.0, + "y": 76.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "66ab6c19b9c5398182bf", + "type": "line", + "x": 0.0, + "y": 76.0, + "width": 765.6, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ], + [ + 765.6, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "3bd9a687937c3a9baa1e", + "type": "text", + "x": 14.0, + "y": 86.0, + "width": 48.0, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "tuple", + "originalText": "tuple", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "e32d5ed338805ed62d64", + "type": "text", + "x": 166.8, + "y": 86.0, + "width": 67.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "ob_size", + "originalText": "ob_size", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "b4df2311043f960363f7", + "type": "text", + "x": 329.20000000000005, + "y": 86.0, + "width": 307.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "a PyVarObject, read straight out", + "originalText": "a PyVarObject, read straight out", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "6dec2e6b3aaca73b60ad", + "type": "line", + "x": 0.0, + "y": 116.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "e2335302cfedf1371712", + "type": "line", + "x": 0.0, + "y": 116.0, + "width": 765.6, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ], + [ + 765.6, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "2b1ff4c0d319f9707774", + "type": "text", + "x": 14.0, + "y": 126.0, + "width": 38.4, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "list", + "originalText": "list", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "f57d0734466447ffa4a2", + "type": "text", + "x": 166.8, + "y": 126.0, + "width": 67.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "ob_size", + "originalText": "ob_size", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "0d8c9157700b36cadb72", + "type": "text", + "x": 329.20000000000005, + "y": 126.0, + "width": 307.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "the length, and not the capacity", + "originalText": "the length, and not the capacity", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "3e380a15e648b87f159d", + "type": "line", + "x": 0.0, + "y": 156.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "2d1417181f63d375ef19", + "type": "line", + "x": 0.0, + "y": 156.0, + "width": 765.6, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#e9ecef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ], + [ + 765.6, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "08d0e7adb87fd64812e9", + "type": "text", + "x": 14.0, + "y": 166.0, + "width": 48.0, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "bytes", + "originalText": "bytes", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "29e794630904379e2e69", + "type": "text", + "x": 166.8, + "y": 166.0, + "width": 67.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "ob_size", + "originalText": "ob_size", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "e02b5e6874acef8c06bb", + "type": "text", + "x": 329.20000000000005, + "y": 166.0, + "width": 163.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "a PyVarObject too", + "originalText": "a PyVarObject too", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "5a5387b2c70c4359736c", + "type": "line", + "x": 0.0, + "y": 196.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "b4a90f4ef0b942e8e635", + "type": "line", + "x": 0.0, + "y": 196.0, + "width": 765.6, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ], + [ + 765.6, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "c205e4495b07c69dec07", + "type": "text", + "x": 14.0, + "y": 206.0, + "width": 28.799999999999997, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "str", + "originalText": "str", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "0bf51e2133f82ae1633b", + "type": "text", + "x": 166.8, + "y": 206.0, + "width": 57.599999999999994, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "length", + "originalText": "length", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "67ad83fcc36b1fd60a3e", + "type": "text", + "x": 329.20000000000005, + "y": 206.0, + "width": 355.2, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "not a PyVarObject, same offset anyway", + "originalText": "not a PyVarObject, same offset anyway", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "93d071eb31ba8a793e3b", + "type": "line", + "x": 0.0, + "y": 236.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "7bb4d62d02aa4dea76bc", + "type": "line", + "x": 0.0, + "y": 236.0, + "width": 765.6, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#ffd8a8", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ], + [ + 765.6, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "8673f0f29a44249c5b03", + "type": "text", + "x": 14.0, + "y": 246.0, + "width": 28.799999999999997, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "int", + "originalText": "int", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "90b5b2e7122af8912d25", + "type": "text", + "x": 166.8, + "y": 246.0, + "width": 57.599999999999994, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "lv_tag", + "originalText": "lv_tag", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "260ae9740504ad49155a", + "type": "text", + "x": 329.20000000000005, + "y": 246.0, + "width": 345.59999999999997, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "sign and digit count packed together", + "originalText": "sign and digit count packed together", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "f7eb6034dedc5cb1ce7e", + "type": "line", + "x": 0.0, + "y": 276.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "47e83f2fb2bf16c6e48f", + "type": "line", + "x": 0.0, + "y": 276.0, + "width": 765.6, + "height": 40.0, + "angle": 0, + "strokeColor": "transparent", + "backgroundColor": "#ffc9c9", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ], + [ + 765.6, + 40.0 + ], + [ + 0, + 40.0 + ], + [ + 0, + 0.0 + ] + ], + "lastCommittedPoint": null, + "polygon": true + }, + { + "id": "77cce97c5fa001709eea", + "type": "text", + "x": 14.0, + "y": 286.0, + "width": 124.8, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "object, float", + "originalText": "object, float", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "42af1a9f6b454dc73c23", + "type": "text", + "x": 166.8, + "y": 286.0, + "width": 124.8, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "there is none", + "originalText": "there is none", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "1082b1c70332f46e8780", + "type": "text", + "x": 329.20000000000005, + "y": 286.0, + "width": 422.4, + "height": 20.0, + "angle": 0, + "strokeColor": "#1e1e1e", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "sixteen and twenty four bytes, and they stop", + "originalText": "sixteen and twenty four bytes, and they stop", + "fontSize": 16, + "fontFamily": 3, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + }, + { + "id": "d8375bf0c7178d46a723", + "type": "line", + "x": 0.0, + "y": 316.0, + "width": 765.6, + "height": 0.0, + "angle": 0, + "strokeColor": "#ced4da", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0.0 + ], + [ + 765.6, + 0.0 + ] + ], + "lastCommittedPoint": null + }, + { + "id": "2f01ed3d73613b0080ee", + "type": "text", + "x": 0.0, + "y": 336.0, + "width": 917.28, + "height": 20.0, + "angle": 0, + "strokeColor": "#5c5f66", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 0, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1, + "version": 1, + "versionNonce": 1, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "customData": null, + "text": "Py_SIZE asserts the type is not an int and not a bool, because those two moved their length elsewhere.", + "originalText": "Py_SIZE asserts the type is not an int and not a bool, because those two moved their length elsewhere.", + "fontSize": 16, + "fontFamily": 2, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "lineHeight": 1.25, + "autoResize": true + } + ], + "appState": { + "gridSize": 20, + "viewBackgroundColor": "#ffffff" + }, + "files": {} +} diff --git a/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.svg b/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.svg new file mode 100644 index 0000000..525dc7c --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.svg @@ -0,0 +1 @@ +The third word, for the types that have onetypethe third wordhow len gets ittupleob_sizea PyVarObject, read straight outlistob_sizethe length, and not the capacitybytesob_sizea PyVarObject toostrlengthnot a PyVarObject, same offset anywayintlv_tagsign and digit count packed togetherobject, floatthere is nonesixteen and twenty four bytes, and they stopPy_SIZE asserts the type is not an int and not a bool, because those two moved their length elsewhere. diff --git a/lessons/o01-the-header-byte-by-byte/o01.ipynb b/lessons/o01-the-header-byte-by-byte/o01.ipynb new file mode 100644 index 0000000..1bd78a1 --- /dev/null +++ b/lessons/o01-the-header-byte-by-byte/o01.ipynb @@ -0,0 +1,441 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "o01-01", + "metadata": {}, + "source": [ + "# O01. The header, byte by byte\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o01-the-header-byte-by-byte/o01.ipynb)\n", + "\n", + "Every article about CPython tells you the same thing about the [object header](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#object-header): two fields, a [reference count](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#reference-count) and a pointer to the type. It is a good summary and it is what T08 said.\n", + "\n", + "It is also missing something. On an ordinary sixty four bit build the count is thirty two bits wide, not sixty four, and the other half of that word is two more fields nobody mentions. One of them decides whether the object can ever be freed.\n", + "\n", + "This lesson reads the bytes out of your own interpreter with `ctypes` and checks each field against something Python will tell you another way. Nothing below is taken on trust.\n", + "\n", + "![the header as five stacked fields, the first word split into three](https://raw.githubusercontent.com/tamnd/cpython-internals/main/lessons/o01-the-header-byte-by-byte/diagrams/the-first-sixteen-bytes.svg)" + ] + }, + { + "cell_type": "markdown", + "id": "o01-02", + "metadata": {}, + "source": [ + "## About the source references\n", + "\n", + "Now and then this lesson points at CPython's own source, like this: `Include/object.h:127-149@v3.15.0rc1#_object`.\n", + "\n", + "Read it as three parts: the file, the lines, and the release those line numbers belong to. Sometimes there is a fourth part after a `#`, which is the name of the thing those lines are inside.\n", + "\n", + "Every reference is a link, and every one is checked against the pinned source on each change, so a stale reference fails the build instead of sending you somewhere wrong. You never have to read any of it. The references are there so you can go deeper when you want to, and so you can check that this lesson is not making things up.\n", + "\n", + "## Setup\n", + "\n", + "Colab does not come with the small package these lessons use, so the next cell installs it. If you are running this from a checkout of the repository it is already installed and the cell does nothing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-03", + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "if sys.version_info < (3, 14):\n", + " print(\"This lesson needs CPython 3.14 or newer.\")\n", + " print(f\"This runtime is {sys.version.split()[0]}, and the cells below will not run on it.\")\n", + "else:\n", + " try:\n", + " import pyxray\n", + " except ImportError:\n", + " %pip install -q \"pyxray @ git+https://github.com/tamnd/cpython-internals@main#subdirectory=pyxray\"\n", + " import pyxray" + ] + }, + { + "cell_type": "markdown", + "id": "o01-04", + "metadata": {}, + "source": [ + "## Which Python is this\n", + "\n", + "Everything below was checked against the version this cell prints and against 3.14. The two agree on every number in this lesson, which is unusual and worth saying." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-05", + "metadata": { + "cpython_internals": { + "differs": "This prints the interpreter you are on, so it is different for everybody." + } + }, + "outputs": [], + "source": [ + "import pyxray\n", + "\n", + "pyxray.show()" + ] + }, + { + "cell_type": "markdown", + "id": "o01-06", + "metadata": {}, + "source": [ + "## Where an object actually is\n", + "\n", + "`id()` is documented as a number that is unique for the lifetime of an object. In CPython it is the address, and `ctypes` will read memory at an address, so the two together are enough to look at any object's header from Python.\n", + "\n", + "Everything below computes its offsets from the size of a [pointer](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#pointer) rather than writing 8 and 16 into the code. That is not fussiness. In a browser this notebook is running a thirty two bit build and every offset halves.\n", + "\n", + "id() is the address an object lives at, and the two machine words there are its reference count and a pointer to its type" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-07", + "metadata": { + "cpython_internals": { + "varies": "The address is wherever your interpreter happened to put the object, and a pointer is four bytes in a browser rather than eight." + } + }, + "outputs": [], + "source": [ + "import ctypes\n", + "\n", + "WORD = ctypes.sizeof(ctypes.c_void_p)\n", + "\n", + "\n", + "class Thing:\n", + " pass\n", + "\n", + "\n", + "it = Thing()\n", + "\n", + "print(f\" a pointer here is {WORD} bytes, so the header is {2 * WORD}\")\n", + "print(f\" it lives at {id(it)}\")\n", + "print()\n", + "raw = ctypes.c_uint32.from_address(id(it)).value\n", + "counted = sys.getrefcount(it) - 1\n", + "\n", + "print(f\" first word, read as a count {raw}\")\n", + "print(f\" sys.getrefcount, less its own argument {counted}\")\n", + "print(f\" the same number: {raw == counted}\")\n", + "print()\n", + "second = ctypes.c_size_t.from_address(id(it) + WORD).value\n", + "print(f\" second word {second}\")\n", + "print(f\" id(Thing) is {id(Thing)}\")\n", + "print(f\" so the second word is the type pointer: {second == id(Thing)}\")\n", + "print()\n", + "\n", + "also = it\n", + "print(f\" after a second name, the count reads {ctypes.c_uint32.from_address(id(it)).value}\")\n", + "del also\n", + "print(f\" after deleting that name, it reads {ctypes.c_uint32.from_address(id(it)).value}\")" + ] + }, + { + "cell_type": "markdown", + "id": "o01-08", + "metadata": {}, + "source": [ + "> **Version note.** The address is wherever your interpreter happened to put the object, and a pointer is four bytes in a browser rather than eight." + ] + }, + { + "cell_type": "markdown", + "id": "o01-09", + "metadata": {}, + "source": [ + "## The count is not a whole word\n", + "\n", + "That first read used `c_uint32`, four bytes, and got the right answer. Try `c_uint64` on an ordinary object and you get the same answer, because the top half is zero. Try it on `None` and you do not.\n", + "\n", + "The C is [Include/object.h:127-149@v3.15.0rc1#_object](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L127-L149), and on a machine where a pointer is more than four bytes the first word is a union of three fields: `ob_refcnt` at thirty two bits, `ob_overflow` at sixteen, and `ob_flags` at sixteen.\n", + "\n", + "![the first word of None split into flags, overflow and count](https://raw.githubusercontent.com/tamnd/cpython-internals/main/lessons/o01-the-header-byte-by-byte/diagrams/three-fields-in-one-word.svg)\n", + "\n", + "`ob_overflow` is worth a moment. It is declared in that struct and it appears nowhere else in the entire source tree. Sixteen bits with a name, reserved and not yet used.\n", + "\n", + "the reference count is thirty two bits wide, and the other half of that word holds two more fields" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-10", + "metadata": { + "cpython_internals": { + "varies": "The two ordinary counts depend on what else in your session is holding those objects. And in a browser Python is a thirty two bit build, where the whole word is the count and the other two fields are not in the struct at all, so those columns read zero." + } + }, + "outputs": [], + "source": [ + "def header(value):\n", + " \"\"\"The first machine word of an object, split the way the C struct splits it.\"\"\"\n", + " word = ctypes.c_size_t.from_address(id(value)).value\n", + " return word & 0xFFFFFFFF, (word >> 32) & 0xFFFF, (word >> 48) & 0xFFFF\n", + "\n", + "\n", + "for label, value in [(\"a Thing\", it), (\"the float 1.5\", 1.5), (\"None\", None), (\"str\", str)]:\n", + " count, overflow, flags = header(value)\n", + " print(f\" {label:14} count {count:>12} overflow {overflow} flags {flags}\")" + ] + }, + { + "cell_type": "markdown", + "id": "o01-11", + "metadata": {}, + "source": [ + "> **Version note.** The two ordinary counts depend on what else in your session is holding those objects. And in a browser Python is a thirty two bit build, where the whole word is the count and the other two fields are not in the struct at all, so those columns read zero." + ] + }, + { + "cell_type": "markdown", + "id": "o01-12", + "metadata": {}, + "source": [ + "## A billion either way\n", + "\n", + "`None` came back with a count of 3221225472, which is `3 << 30`. Nothing is holding `None` three billion times. It is an [immortal object](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#immortal-object): the count is parked at a value the interpreter never decrements, so `None` is never freed and, more to the point, nothing ever writes to that cache line.\n", + "\n", + "The test is a comparison, [Include/refcount.h:126-136@v3.15.0rc1#_Py_IsImmortal](https://github.com/python/cpython/blob/v3.15.0rc1/Include/refcount.h#L126-L136), and the interesting part is which two numbers it uses. The line is at `2 ** 31`, and the starting value is `3 << 30`, which is not the top of the field and not the line either.\n", + "\n", + "![the four numbers that matter in a thirty two bit reference count](https://raw.githubusercontent.com/tamnd/cpython-internals/main/lessons/o01-the-header-byte-by-byte/diagrams/how-far-off-you-can-be.svg)\n", + "\n", + "The comment at [Include/refcount.h:23-50@v3.15.0rc1#_Py_IMMORTAL_INITIAL_REFCNT](https://github.com/python/cpython/blob/v3.15.0rc1/Include/refcount.h#L23-L50) says why. An extension compiled against Python 3.11 has an old `Py_INCREF` in it that knows nothing about immortality, so it increments and decrements `None` like anything else. Parking the count in the middle of the immortal range means that extension can be off by about a billion in either direction and `None` stays immortal.\n", + "\n", + "an immortal object's count starts halfway between the immortality line and the top of the field, leaving about a billion of slack in each direction" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-13", + "metadata": { + "cpython_internals": { + "varies": "A thirty two bit build draws the line at 1 << 30 and parks a static immortal at 7 << 28, so in a browser the first number and the two amounts of slack are different. The shape of the argument is the same." + } + }, + "outputs": [], + "source": [ + "INITIAL = 3 << 30\n", + "MINIMUM = 1 << 31\n", + "TOP = 2**32\n", + "\n", + "print(f\" the count parked in None {sys.getrefcount(None):>12}\")\n", + "print(f\" 3 << 30 {INITIAL:>12}\")\n", + "print(f\" the immortality line, 1 << 31 {MINIMUM:>12}\")\n", + "print(f\" the top of the field, 2 ** 32 {TOP:>12}\")\n", + "print()\n", + "print(f\" slack below the start {INITIAL - MINIMUM:>13,}\")\n", + "print(f\" slack above the start {TOP - INITIAL:>13,}\")" + ] + }, + { + "cell_type": "markdown", + "id": "o01-14", + "metadata": {}, + "source": [ + "> **Version note.** A thirty two bit build draws the line at 1 << 30 and parks a static immortal at 7 << 28, so in a browser the first number and the two amounts of slack are different. The shape of the argument is the same." + ] + }, + { + "cell_type": "markdown", + "id": "o01-15", + "metadata": {}, + "source": [ + "## Two ways to never be freed\n", + "\n", + "`None` also came back with `ob_flags` of 5. Three of those sixteen bits have meanings, [Include/object.h:580-583@v3.15.0rc1#_Py_STATICALLY_ALLOCATED_FLAG](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L580-L583), and 5 is bit 0 and bit 2: immortal, and statically allocated.\n", + "\n", + "Statically allocated means the object is a `PyObject` written out in the C source and compiled into the binary. [Include/internal/pycore_object.h:83-89@v3.15.0rc1#_PyObject_HEAD_INIT](https://github.com/python/cpython/blob/v3.15.0rc1/Include/internal/pycore_object.h#L83-L89) is the initialiser that sets both bits at once.\n", + "\n", + "There is another way to become immortal. [Objects/object.c:2773-2791@v3.15.0rc1#_Py_SetImmortalUntracked](https://github.com/python/cpython/blob/v3.15.0rc1/Objects/object.c#L2773-L2791) parks the count on an object that was allocated normally, and it sets bit 0 only. That distinction is not cosmetic: when the interpreter shuts down it has to free the promoted ones and must leave the static ones alone, because the static ones are not on the heap.\n", + "\n", + "![static immortals with both bits against promoted immortals with one](https://raw.githubusercontent.com/tamnd/cpython-internals/main/lessons/o01-the-header-byte-by-byte/diagrams/two-ways-to-become-immortal.svg)\n", + "\n", + "The surprise is in the strings. CPython ships a table of the identifier strings it uses itself, [Include/internal/pycore_global_strings.h:30-38@v3.15.0rc1#_Py_global_strings](https://github.com/python/cpython/blob/v3.15.0rc1/Include/internal/pycore_global_strings.h#L30-L38), generated by a script and compiled into the binary. Write `\"self\"` in your own code and the compiler [interns](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#interning) it to that table entry, so your string is a static immortal. Write something CPython has never heard of and you get an ordinary heap string with an ordinary count.\n", + "\n", + "a string literal that happens to be one of CPython's own identifiers is immortal, and the same characters CPython does not use are not" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-16", + "metadata": { + "cpython_internals": { + "varies": "The counts depend on what else in your session is holding these strings. Which of them are immortal does not." + } + }, + "outputs": [], + "source": [ + "for text in [\"self\", \"append\", \"flags\", \"zzz nobody uses this one\"]:\n", + " count, _, flags = header(text)\n", + " print(f\" {text!r:28} count {count:>12} flags {flags}\")\n", + "\n", + "print()\n", + "print(f\" sys.intern on a fresh string leaves flags at {header(sys.intern('q w e r t y'))[2]}\")\n", + "print(\" so interning and immortalising are two different things\")" + ] + }, + { + "cell_type": "markdown", + "id": "o01-17", + "metadata": {}, + "source": [ + "> **Version note.** The counts depend on what else in your session is holding these strings. Which of them are immortal does not." + ] + }, + { + "cell_type": "markdown", + "id": "o01-18", + "metadata": {}, + "source": [ + "## The third word, when there is one\n", + "\n", + "After the two header words, some types put a length. Those are the variable sized ones, [Include/object.h:174-178@v3.15.0rc1#PyVarObject](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L174-L178), and the field is `ob_size`. A tuple of three, a list of seven and a `bytes` of four all put their length right there.\n", + "\n", + "![which types keep a length in the third word and which do not](https://raw.githubusercontent.com/tamnd/cpython-internals/main/lessons/o01-the-header-byte-by-byte/diagrams/where-a-length-lives.svg)\n", + "\n", + "Two things on that picture are worth pointing at. A `str` is not a [PyVarObject](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#pyvarobject) at all, and it still keeps its length in the same place, because `PyASCIIObject` puts a plain `length` field immediately after the header. And an `int` does not, which is why the accessor asserts about it: [Include/object.h:237-244@v3.15.0rc1#_Py_SIZE_impl](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L237-L244) refuses to run on a `PyLong` or a `PyBool`, because since 3.12 an integer packs its sign and its digit count into one field with a different meaning.\n", + "\n", + "for a tuple, a list, a bytes and a str, the machine word after the header is the length" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-19", + "metadata": {}, + "outputs": [], + "source": [ + "def third_word(value):\n", + " \"\"\"Whatever is in the word after the header, read as a signed count.\"\"\"\n", + " return ctypes.c_ssize_t.from_address(id(value) + 2 * WORD).value\n", + "\n", + "\n", + "for value in [(1, 2, 3), [0] * 7, b\"abcd\", bytearray(3), \"hello\"]:\n", + " kind = type(value).__name__\n", + " print(f\" {kind:10} {value!r:26} len {len(value)} third word {third_word(value)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "o01-20", + "metadata": {}, + "source": [ + "## What sits in front of the header\n", + "\n", + "There is one more thing in the sixteen bytes story, and it is not in the sixteen bytes. It is in front of them.\n", + "\n", + "Types the [cycle collector](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#cycle-collector) tracks get a [GC pre header](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#gc-pre-header) allocated immediately before the object, [Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1#PyGC_Head](https://github.com/python/cpython/blob/v3.15.0rc1/Include/internal/pycore_interp_structs.h#L158-L169), two words holding the links that thread every tracked object into a list. The object's own address points past it, so nothing that reads the header ever sees it.\n", + "\n", + "You can see it as a gap. `value.__sizeof__()` is what the type says about itself. `sys.getsizeof(value)` calls that and then adds the pre header if the type has one.\n", + "\n", + "sys.getsizeof reports two machine words more than an object's own __sizeof__ for the types the cycle collector tracks, and nothing extra for the types it does not" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "o01-21", + "metadata": { + "cpython_internals": { + "varies": "Every number here is counted in pointers, so they roughly halve in a browser. The gap is two pointers wherever it is not zero." + } + }, + "outputs": [], + "source": [ + "everything = [\n", + " (\"object()\", object()),\n", + " (\"1.5\", 1.5),\n", + " (\"'hello'\", \"hello\"),\n", + " (\"()\", ()),\n", + " (\"(1, 2, 3)\", (1, 2, 3)),\n", + " (\"[1, 2, 3]\", [1, 2, 3]),\n", + " (\"{'a': 1}\", {\"a\": 1}),\n", + "]\n", + "\n", + "for label, value in everything:\n", + " own = value.__sizeof__()\n", + " both = sys.getsizeof(value)\n", + " print(f\" {label:10} __sizeof__ {own:>3} getsizeof {both:>3} gap {both - own}\")" + ] + }, + { + "cell_type": "markdown", + "id": "o01-22", + "metadata": {}, + "source": [ + "> **Version note.** Every number here is counted in pointers, so they roughly halve in a browser. The gap is two pointers wherever it is not zero." + ] + }, + { + "cell_type": "markdown", + "id": "o01-23", + "metadata": {}, + "source": [ + "## The same header, wider\n", + "\n", + "Everything above is the ordinary build. The [free threaded build](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#free-threaded-build) has a different header, [Include/object.h:156-167@v3.15.0rc1#_object](https://github.com/python/cpython/blob/v3.15.0rc1/Include/object.h#L156-L167), and it is twice the size.\n", + "\n", + "![the sixteen byte header against the thirty two byte free threaded one](https://raw.githubusercontent.com/tamnd/cpython-internals/main/lessons/o01-the-header-byte-by-byte/diagrams/the-same-header-wider.svg)\n", + "\n", + "The reason is the count. Incrementing a shared counter means writing to a cache line, and if two threads hold the same object then every increment on one core invalidates the other's copy. So the free threaded header splits the count in two: `ob_ref_local`, which only the owning thread touches and therefore needs no atomic instruction, and `ob_ref_shared`, which everybody else uses. `ob_tid` says which thread owns it, and there is a one byte mutex in there as well for the per object lock.\n", + "\n", + "the free threaded build gives every object a thirty two byte header, split so the owning thread can increment without an atomic instruction\n", + "\n", + "That is the whole design of free threaded CPython in one struct, and the concurrency lessons come back to it.\n", + "\n", + "## Try it yourself\n", + "\n", + "Three things to poke at.\n", + "\n", + "Walk `gc.get_objects()` and read the flags word on every one of them. Nothing tracked by the cycle collector is immortal, because `_Py_SetImmortal` untracks first. Seeing that come out as a clean result rather than a claim is worth the six lines.\n", + "\n", + "Find the highest reference count in your session. Sort by the count field, look at the top ten, and see whether you can explain each one. The empty tuple and the interpreter's own small integers will be near the top and they will be parked rather than counted, so filter those out first.\n", + "\n", + "Take a class with `__slots__` and one without, and compare `sys.getsizeof` on an instance of each. The gap tells you what an [instance dictionary](https://github.com/tamnd/cpython-internals/blob/main/GLOSSARY.md#instance-dictionary) costs, and the header is the same either way.\n", + "\n", + "## What just happened\n", + "\n", + "Two fields is a good summary of the header and it is not what is in memory. On an ordinary sixty four bit build the first word is a thirty two bit count, sixteen bits reserved under the name `ob_overflow` and used nowhere, and sixteen bits of flags. The second word is the type pointer.\n", + "\n", + "Immortality is a comparison against `2 ** 31`, and the value it parks at is `3 << 30`, deliberately in the middle so that an old extension incrementing and decrementing without checking can be a billion out either way and the object stays immortal.\n", + "\n", + "Two of the flag bits say how the object became immortal, because shutting down has to free the promoted ones and must not touch the ones compiled into the binary. Every identifier string CPython uses itself is in that second category, which is why `\"self\"` in your code is immortal and a string CPython has never seen is not.\n", + "\n", + "After the header, variable sized types put their length. Strings are not variable sized types and put it in the same place anyway. Integers used to and no longer do, and the accessor asserts about it.\n", + "\n", + "And in front of the header, for the types the cycle collector tracks, there are two more words that the object's own address points past.\n", + "\n", + "## What is next\n", + "\n", + "The type pointer is the field everything else in this part hangs off. O02 follows it, into the type object: what a `PyTypeObject` holds, which of its hundred or so fields are slots and which are bookkeeping, and how a type written in Python and a type written in C end up being the same kind of thing." + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/probes/pyodide/lessons.json b/probes/pyodide/lessons.json index e1fe5df..db6f8a8 100644 --- a/probes/pyodide/lessons.json +++ b/probes/pyodide/lessons.json @@ -70,7 +70,7 @@ { "name": "b03-07", "status": "ok", - "printed": "test_adding (__main__.Arithmetic.test_adding) ... ok\ntest_dividing (__main__.Arithmetic.test_dividing) ... ok\ntest_powers (__main__.Arithmetic.test_powers) ... FAIL\n\n======================================================================\nFAIL: test_powers (__main__.Arithmetic.test_powers)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"b03-07\", line 13, in test_powers\nAssertionError: 1024 != 1000\n\n----------------------------------------------------------------------\nRan 3 tests in 0.001s\n\nFAILED (failures=1)\n\n3 tests run, 1 failed, 0 errored\n" + "printed": "test_adding (__main__.Arithmetic.test_adding) ... ok\ntest_dividing (__main__.Arithmetic.test_dividing) ... ok\ntest_powers (__main__.Arithmetic.test_powers) ... FAIL\n\n======================================================================\nFAIL: test_powers (__main__.Arithmetic.test_powers)\n----------------------------------------------------------------------\nTraceback (most recent call last):\n File \"b03-07\", line 13, in test_powers\nAssertionError: 1024 != 1000\n\n----------------------------------------------------------------------\nRan 3 tests in 0.002s\n\nFAILED (failures=1)\n\n3 tests run, 1 failed, 0 errored\n" }, { "name": "b03-10", @@ -539,7 +539,7 @@ { "name": "f10-11", "status": "ok", - "printed": " -- MAKE_CELL 0 (a)\n MAKE_CELL 6 (n)\n\n 1 RESUME 0\n\n 2 LOAD_SMALL_INT 0\n STORE_DEREF 6 (n)\n\n 4 LOAD_FAST_BORROW 0 (a)\n LOAD_FAST_BORROW 6 (n)\n BUILD_TUPLE 2\n LOAD_CONST 1 ()\n MAKE_FUNCTION\n SET_FUNCTION_ATTRIBUTE 8 (closure)\n STORE_FAST 5 (inner)\n\n 7 LOAD_FAST_BORROW 5 (inner)\n RETURN_VALUE\n\nDisassembly of :\n -- COPY_FREE_VARS 2\n\n 4 RESUME 0\n\n 5 LOAD_DEREF 0 (a)\n LOAD_DEREF 1 (n)\n BINARY_OP 0 (+)\n RETURN_VALUE\n" + "printed": " -- MAKE_CELL 0 (a)\n MAKE_CELL 6 (n)\n\n 1 RESUME 0\n\n 2 LOAD_SMALL_INT 0\n STORE_DEREF 6 (n)\n\n 4 LOAD_FAST_BORROW 0 (a)\n LOAD_FAST_BORROW 6 (n)\n BUILD_TUPLE 2\n LOAD_CONST 1 ()\n MAKE_FUNCTION\n SET_FUNCTION_ATTRIBUTE 8 (closure)\n STORE_FAST 5 (inner)\n\n 7 LOAD_FAST_BORROW 5 (inner)\n RETURN_VALUE\n\nDisassembly of :\n -- COPY_FREE_VARS 2\n\n 4 RESUME 0\n\n 5 LOAD_DEREF 0 (a)\n LOAD_DEREF 1 (n)\n BINARY_OP 0 (+)\n RETURN_VALUE\n" }, { "name": "f10-13", @@ -631,12 +631,12 @@ { "name": "f12-07", "status": "ok", - "printed": " greet.py is 54 bytes of text\n greet.cpython-314.pyc is 251 bytes\n\n header 2b 0e 0d 0a 00 00 00 00 5a 6e 95 6a 36 00 00 00\n the rest 235 bytes, and marshal.loads gives back a\n code object\n" + "printed": " greet.py is 54 bytes of text\n greet.cpython-314.pyc is 251 bytes\n\n header 2b 0e 0d 0a 00 00 00 00 44 73 95 6a 36 00 00 00\n the rest 235 bytes, and marshal.loads gives back a\n code object\n" }, { "name": "f12-10", "status": "ok", - "printed": " magic 168627755\n flags 0\n mtime 1788178010 source 1788178010\n size 54 source 54\n" + "printed": " magic 168627755\n flags 0\n mtime 1788179268 source 1788179268\n size 54 source 54\n" }, { "name": "f12-13", @@ -670,7 +670,51 @@ { "name": "f12-29", "status": "ok", - "printed": " one bit flipped in the magic: bad magic number in 'broken': b'*\\x0e\\r\\n'\n\n put a wrong mtime in greet.cpython-314.pyc: 1788178085\n imported it with no complaint: 'hi world'\n and the header now says 1788178085\n the file was quietly rewritten: False\n" + "printed": " one bit flipped in the magic: bad magic number in 'broken': b'*\\x0e\\r\\n'\n\n put a wrong mtime in greet.cpython-314.pyc: 1788179387\n imported it with no complaint: 'hi world'\n and the header now says 1788179387\n the file was quietly rewritten: False\n" + } + ] + }, + { + "slug": "o01-the-header-byte-by-byte", + "cells": [ + { + "name": "o01-03", + "status": "ok" + }, + { + "name": "o01-05", + "status": "ok", + "printed": "cpython 3.14.2 on Emscripten wasm32, WebAssembly\n note: this is 3.14, and everything here is written against 3.15, so bytecode and some object layouts will differ from the prose\n" + }, + { + "name": "o01-07", + "status": "ok", + "printed": " a pointer here is 4 bytes, so the header is 8\n it lives at 24114960\n\n first word, read as a count 1\n sys.getrefcount, less its own argument 1\n the same number: True\n\n second word 23644248\n id(Thing) is 23644248\n so the second word is the type pointer: True\n\n after a second name, the count reads 2\n after deleting that name, it reads 1\n" + }, + { + "name": "o01-10", + "status": "ok", + "printed": " a Thing count 4 overflow 0 flags 0\n the float 1.5 count 3 overflow 0 flags 0\n None count 1879048192 overflow 0 flags 0\n str count 1879048192 overflow 0 flags 0\n" + }, + { + "name": "o01-13", + "status": "ok", + "printed": " the count parked in None 1879048192\n 3 << 30 3221225472\n the immortality line, 1 << 31 2147483648\n the top of the field, 2 ** 32 4294967296\n\n slack below the start 1,073,741,824\n slack above the start 1,073,741,824\n" + }, + { + "name": "o01-16", + "status": "ok", + "printed": " 'self' count 1879048192 flags 0\n 'append' count 1879048192 flags 0\n 'flags' count 1879048192 flags 0\n 'zzz nobody uses this one' count 3 flags 0\n\n sys.intern on a fresh string leaves flags at 0\n so interning and immortalising are two different things\n" + }, + { + "name": "o01-19", + "status": "ok", + "printed": " tuple (1, 2, 3) len 3 third word 3\n list [0, 0, 0, 0, 0, 0, 0] len 7 third word 7\n bytes b'abcd' len 4 third word 4\n bytearray bytearray(b'\\x00\\x00\\x00') len 3 third word 3\n str 'hello' len 5 third word 5\n" + }, + { + "name": "o01-21", + "status": "ok", + "printed": " object() __sizeof__ 8 getsizeof 8 gap 0\n 1.5 __sizeof__ 16 getsizeof 16 gap 0\n 'hello' __sizeof__ 26 getsizeof 26 gap 0\n () __sizeof__ 16 getsizeof 24 gap 8\n (1, 2, 3) __sizeof__ 28 getsizeof 36 gap 8\n [1, 2, 3] __sizeof__ 36 getsizeof 44 gap 8\n {'a': 1} __sizeof__ 100 getsizeof 108 gap 8\n" } ] }, @@ -1056,7 +1100,7 @@ { "name": "t04-30", "status": "ok", - "printed": "free variables: ('total',)\nthe cells: (,)\nfirst call: 1\nsecond call: 2\nthe cell now: 2\n" + "printed": "free variables: ('total',)\nthe cells: (,)\nfirst call: 1\nsecond call: 2\nthe cell now: 2\n" }, { "name": "t04-33", @@ -1333,7 +1377,7 @@ { "name": "t07-38", "status": "ok", - "printed": "asked twice, got the same object: True\nand it is still here after the call returned: \nits name: make_one\n" + "printed": "asked twice, got the same object: True\nand it is still here after the call returned: \nits name: make_one\n" }, { "name": "t07-40", @@ -1367,12 +1411,12 @@ { "name": "t08-11", "status": "ok", - "printed": "NoneType at 0x30d7d8, refcount is parked, this object is never freed, 8 bytes, not tracked\nint at 0x31a7cc, refcount is parked, this object is never freed, 16 bytes, not tracked\nstr at 0x16ead00, 3 reference(s), 26 bytes, not tracked\nlist at 0x1786278, 2 reference(s), 44 bytes, tracked by the cycle collector\ndict at 0x170f4f8, 2 reference(s), 108 bytes, tracked by the cycle collector\nfunction at 0x1658b10, 4 reference(s), 84 bytes, tracked by the cycle collector\n" + "printed": "NoneType at 0x30d7d8, refcount is parked, this object is never freed, 8 bytes, not tracked\nint at 0x31a7cc, refcount is parked, this object is never freed, 16 bytes, not tracked\nstr at 0x150cd38, 3 reference(s), 26 bytes, not tracked\nlist at 0x1817c80, 2 reference(s), 44 bytes, tracked by the cycle collector\ndict at 0x1860a90, 2 reference(s), 108 bytes, tracked by the cycle collector\nfunction at 0x1021488, 4 reference(s), 84 bytes, tracked by the cycle collector\n" }, { "name": "t08-14", "status": "ok", - "printed": "a == b True same contents\na is b False different objects\na is c True same object\n\nid(a) 0x14ebb00\nid(b) 0x1812d48\nid(c) 0x14ebb00\n" + "printed": "a == b True same contents\na is b False different objects\na is c True same object\n\nid(a) 0x12ee6b0\nid(b) 0x17611a0\nid(c) 0x12ee6b0\n" }, { "name": "t08-17", @@ -1412,7 +1456,7 @@ { "name": "t08-35", "status": "ok", - "printed": "dict at 0x134a068\ndict at 0xfe4bd0\nlist at 0x18bf170\n" + "printed": "dict at 0x185ea18\ndict at 0xfc1440\nlist at 0x18948a0\n" }, { "name": "t08-37", @@ -1471,7 +1515,7 @@ { "name": "t09-21", "status": "ok", - "printed": "Node at 0x1045240 -> Node at 0x1648a48 -> Node at 0x18bf560 -> Node at 0x1045240\n" + "printed": "Node at 0x17cfcc0 -> Node at 0x13a2df0 -> Node at 0x128a4b0 -> Node at 0x17cfcc0\n" }, { "name": "t09-23", @@ -1501,7 +1545,7 @@ { "name": "t09-37", "status": "ok", - "printed": "first object was at 0x14f6390\nsecond object is at 0x14f6390\nsame address reused -> True\n" + "printed": "first object was at 0x1536090\nsecond object is at 0x1536090\nsame address reused -> True\n" } ] }, diff --git a/probes/pyodide/lessons.md b/probes/pyodide/lessons.md index 8d650ac..d8884a8 100644 --- a/probes/pyodide/lessons.md +++ b/probes/pyodide/lessons.md @@ -2,7 +2,7 @@ Generated by `just build-probe`. Do not edit by hand, the change will be overwritten. -28 lesson(s) on Pyodide 3.14.2: 28 ran end to end, 302 cell(s) in total. +29 lesson(s) on Pyodide 3.14.2: 29 ran end to end, 310 cell(s) in total. The checks in `report.md` next to this ask whether a surface exists. This runs the lessons themselves: every code cell of every notebook, in order, in one Pyodide runtime, with `pyxray` mounted off the disk rather than installed. The install cell is the one thing changed, and only its `%pip` line, which a reader in a browser does not need either. @@ -24,6 +24,7 @@ The checks in `report.md` next to this ask whether a surface exists. This runs t | f10-inside-a-code-object | 8 | runs end to end | | f11-two-tables-on-the-side | 11 | runs end to end | | f12-what-ends-up-on-disk | 11 | runs end to end | +| o01-the-header-byte-by-byte | 8 | runs end to end | | t01-one-line-seven-stages | 18 | runs end to end | | t02-text-becomes-tokens | 32 | runs end to end | | t03-tokens-become-a-tree | 13 | runs end to end | diff --git a/pyxray/src/pyxray/glossary.py b/pyxray/src/pyxray/glossary.py index 05f18fa..07a67c7 100644 --- a/pyxray/src/pyxray/glossary.py +++ b/pyxray/src/pyxray/glossary.py @@ -734,6 +734,15 @@ def anchor(name: str) -> str: see=("object", "type object"), met="T08", ), + Term( + name="PyVarObject", + short="An object header with a length field welded on the end of it.", + long="Tuples, lists and bytes objects all hold a count of how many items they have, and rather than each of them inventing a field for it the header itself grows by one machine word called `ob_size`. It is the same trick as the header: put the thing everybody needs in a fixed place so that generic code can read it without knowing the type. Strings keep their length in the same place without being one of these, and integers used to and no longer do.", + cite="Include/object.h:174-178@v3.15.0rc1#PyVarObject", + also=("variable sized object", "`ob_size`"), + see=("object header", "object"), + met="O01", + ), ), ) @@ -823,6 +832,15 @@ def anchor(name: str) -> str: see=("deallocation", "cycle collector"), met="T09", ), + Term( + name="GC pre header", + short="Two words allocated in front of an object, holding its place in the collector's list.", + long="Only the types the cycle collector tracks get one, and the object's own address points past it, so nothing that reads the header ever sees it. You can still measure it: `sys.getsizeof` adds it and the object's own `__sizeof__` does not, so the gap between those two is exactly this.", + cite="Include/internal/pycore_interp_structs.h:158-169@v3.15.0rc1#PyGC_Head", + also=("`PyGC_Head`",), + see=("cycle collector", "object header"), + met="O01", + ), ), )