M4 The object model

[tamnd]

Roughly ten weeks and fourteen lessons, which makes it the largest content milestone in the project. It is also where the specification stops being a description and starts being something you could implement from, because the object model is where a reimplementation either matches CPython or quietly does not.

Exit criterion. BP-DICT specifies the probe sequence, the growth policy and compaction precisely enough that an independent implementation reproduces CPython's exact dk_indices array for a given sequence of insertions and deletions, verified by test. The coverage ledger shows every PyTypeObject slot owned by a blueprint.

Content

• O01 through O14: the object header, type objects, slots, the MRO, attribute lookup, descriptors, the dict, the split table layout, integers, strings, lists and tuples, the free lists, weak references and finalization

Blueprints

• BP-OBJECT through BP-FINALIZE, twelve of them

Tooling

• DictLab and SpecializationHeat widgets
• The hash and repr golden corpora
• Struct layout generation in bpc working across all three build configurations, which is Q5: can bpc extract struct layouts across all three build configurations #5

Gates

• Q5: can bpc extract struct layouts across all three build configurations #5 whether bpc can extract struct layouts reliably, and the DWARF fallback if header parsing is not good enough

[tamnd]

O01 is merged in #131, and M4 is open.

T08 back in part one said an object is a count and a type pointer. That is the right summary and it is not what is in memory, so this lesson reads the sixteen bytes one field at a time with ctypes and checks every number against something you can get another way.

What is in the lesson:

• the first machine word is not one field, it is a thirty two bit ob_refcnt, sixteen bits called ob_overflow that nothing in the source tree reads, and sixteen bits of ob_flags
• reading the low half of that word as a c_uint32 gives you the same number sys.getrefcount does, less its own argument, and the lesson watches it move when you bind and delete a name
• an immortal object parks its count at 3 << 30 while the immortality test is a comparison against 2 ** 31, which leaves exactly 2 ** 30 of slack in each direction so a stable ABI extension can be a billion increments out either way and the object stays immortal
• there are two kinds of immortal and the flags word tells them apart: ob_flags of 5 means built into the binary, ob_flags of 1 means promoted by _Py_SetImmortal while running
• "self", "append" and "flags" come back immortal because CPython ships a generated table of its own identifier strings and the compiler interns literals to it, while sys.intern on a fresh string does not immortalise anything
• the third word is ob_size for a PyVarObject, is a length at the same offset for str without str being one, and is lv_tag for int, which is why Py_SIZE asserts the type is not an int
• the gap between __sizeof__ and sys.getsizeof is the GC pre header, two words in front of the address you get from id
• and the free threaded build replaces the whole sixteen bytes with a thirty two byte header that splits the count into a local half and a shared half

Everything is cited against v3.15.0rc1: _object in both its forms, PyVarObject, _Py_SIZE_impl, _Py_STATICALLY_ALLOCATED_FLAG, _Py_IMMORTAL_INITIAL_REFCNT, _Py_IsImmortal, _PyObject_HEAD_INIT, _Py_global_strings, PyGC_Head and _Py_SetImmortalUntracked. Six diagrams, and the notebook runs in Colab and in the browser probe.

Every cell computes its offsets from ctypes.sizeof(ctypes.c_void_p) rather than hardcoding eight, because Pyodide is a thirty two bit build where the header is eight bytes and the whole first word is the count. The five cells that read differently there carry a note saying so.

Two glossary terms went in with it: PyVarObject and GC pre header. That takes GLOSSARY.md to 108.

Next is O02, the type object, following the pointer in the second word to see what is on the other end.

[tamnd]

O02 is merged in #132.

O01 stopped at the second machine word and said it points at the type. This lesson follows that pointer and looks at what is on the other end, which turns out to be the biggest struct in the interpreter and mostly a set of pre computed answers to questions the interpreter asks about instances.

What is in the lesson:

• the second word of any object is the address of its type, and reading it with ctypes for 3, "hi", int and type gives int, str, type and type, so following the pointer stops finding anything new after two steps
• a type is an object, which is why it has a header of its own and why type(type) is type closes the loop rather than being a special case
• tp_basicsize and tp_itemsize are the two numbers a type keeps about its instances, and basicsize + itemsize * count reproduces __sizeof__ exactly for tuple and bytes, misses for list because the items live in a separate array, and explains the 944 and 40 on type itself
• static types are one PyTypeObject in the binary and heap types are a malloc'd PyHeapTypeObject, and Py_TPFLAGS_HEAPTYPE at bit 9 is the one flag almost everything else follows from
• tp_dictoffset has four meanings: zero for no dict, a positive byte offset, a negative offset counted back from the end of a var sized object, and exactly -1 as a sentinel meaning the VM manages the dict itself
• a class statement gives you that -1, so the dict pointer sits at a fixed spot in front of the object at minus 24, with the weakref list at minus 32 and the two words of GC pre header at minus 16, and sys.getsizeof minus __sizeof__ comes out at 32 for a plain class and 16 for a __slots__ class, matching _PyType_PreHeaderSize line for line
• there is no class opcode, only a call: LOAD_BUILD_CLASS, MAKE_FUNCTION around the class body, CALL, and the metaclass builds the type

Twelve citations against v3.15.0rc1: _typeobject, PyHeapTypeObject, tp_weaklistoffset, tp_dictoffset, PyVarObject, Py_TPFLAGS_HEAPTYPE, PyType_Type, _PyType_AllocNoTrack, _PyObject_ComputedDictPointer, MANAGED_DICT_OFFSET, _PyType_PreHeaderSize and builtin___build_class__. Six diagrams, and the notebook runs in Colab and in the browser probe.

One number moves between versions: type.__basicsize__ is 936 on 3.14 and 944 on 3.15, because the type struct gained one more machine word. That cell carries a note. The disassembly of a class statement carries one too, because 3.15 emits LOAD_COMMON_CONSTANT where 3.14 emits LOAD_CONST.

The one thing the lesson claims and cannot show you is that _PyType_AllocNoTrack also reserves room after the object for the inline values array, which neither __sizeof__ nor getsizeof counts. That is O08.

Three glossary terms went in with it: static type, heap type and metaclass, and the existing type object entry picked up a citation. That takes GLOSSARY.md to 111.

[tamnd]

O03 is merged in #133.

O02 walked past seventy function pointer fields in the type struct without saying where they come from. This lesson is about the one table that connects them to dunder names, and about the fact that the table gets walked in both directions.

What is in the lesson:

• slotdefs in typeobject.c is ninety four rows, and each row holds a name, the offset of a slot inside PyHeapTypeObject, the dispatcher to install when Python defines the name, the wrapper to expose when C defines the slot, and a docstring
• when you write a class, fixup_slot_dispatchers reads your __repr__ out of the class dict and puts a generic slot_tp_repr into tp_repr, which then looks your function back up by name
• when a type is written in C, add_operators goes the other way and puts a wrapper_descriptor into the class dict for every slot that already has a function, which is where int.__add__ comes from
• so setting __repr__ on an instance does nothing at all: the slot dispatcher looks the name up on the type and never touches the instance dict, and the lesson shows q.__repr__() returning your string while repr(q) ignores it
• the mapping is not one to one in either direction, and that is where every surprising rule comes from: __len__ fills both mp_length and sq_length, __getitem__ fills both mp_subscript and sq_item which is why a class with only __getitem__ is iterable and supports in, and all six comparison dunders share tp_richcompare with an op argument
• __add__ and __radd__ are two names on one slot, and the reflected one goes first only when the right operand's type is a proper subclass of the left one and defines __radd__ itself
• and defining __eq__ takes __hash__ away in three steps you can point at: overrides_hash sees __eq__ in the class dict, so inherit_slots does not copy tp_hash down, so type_ready_set_hash finds the slot empty and writes a literal None into your class dict

Eleven citations against v3.15.0rc1: slotdefs, TPSLOT, MPSLOT, SLOT0, SLOT1BINFULL, add_operators, fixup_slot_dispatchers, update_slot, overrides_hash, type_ready_set_hash and the __len__ rows themselves. Six diagrams, and the notebook runs in Colab and in the browser probe.

Every cell in this one prints the same thing on 3.14, on 3.15.0rc1 and in the browser, so it needed no version notes beyond the standard banner.

Two glossary terms went in with it: slot and slot wrapper. That takes GLOSSARY.md to 113.

Next is O04, the MRO, which is the other half of how a name gets found: O03 covers which dict the interpreter reads, O04 covers the order it reads them in.

[tamnd]

O04 is merged in #134.

O03 said a slot dispatcher looks a dunder name up on the type, and that a class inherits slots from its base. That is fine with one base. With two there is no obvious next link, and this lesson is about what CPython does instead.

The answer is that it does not search. Every class carries a flat list of classes, computed once when the class is made and stored on the type, and every lookup reads that list front to back.

What is in the lesson:

• the diamond, where D().where() gives C rather than A, straight out of D.__mro__ rather than out of any search
• swapping the declared bases gives a different list, so the bases tuple is an input to the computation and not just a record of what you typed
• the three constraints the list satisfies, checked against classes from the notebook and from the standard library: a class before its bases, declared order preserved, and both of those for every entry rather than only the first
• the C3 merge written out in twenty lines of Python, with an assertion that it reproduces __mro__ exactly for eleven classes including bool, io.StringIO, collections.OrderedDict and type
• the single base fast path in mro_implementation_unlocked, which skips the merge and just prepends, so ordinary single inheritance never runs C3 at all
• the two ways a class statement fails before the class exists: contradictory orders, where the message names the classes the merge got stuck between rather than the bases you wrote, and a repeated base, which check_duplicates catches earlier with a different message
• super() reading the MRO of the instance rather than of the class the call appears in, so the same unchanged method in Left returns Base for one instance and dispatches to Right for another, which is the whole of cooperative multiple inheritance and is one list lookup
• and the two ways the list changes afterwards: assigning __bases__, which rebuilds it for the class and every subclass while existing instances pick up the new method at once, and a metaclass with its own mro(), which can hand back an order C3 would never produce at the cost of the method cache

Thirteen citations against v3.15.0rc1: pmerge, tail_contains, check_duplicates, set_mro_error, mro_implementation_unlocked and its fast path, mro_invoke, mro_hierarchy_for_complete_type, type_mro_modified, _PySuper_LookupDescr and find_name_in_mro. Six diagrams, and the notebook runs in Colab and in the browser probe.

Every cell prints the same thing on 3.14, on 3.15.0rc1 and in the browser, so this one needed no version notes beyond the standard banner.

Two glossary terms went in with it: MRO and C3 linearization. That takes GLOSSARY.md to 115.

Next is O05, attribute lookup, which is the loop that reads the list this lesson built.

[tamnd]

O05 has landed in #135.

This is the lesson the last three were building towards. PyObject_GenericGetAttr reads four places in a fixed order, and once you know the order every rule people memorise about attributes is predictable rather than memorised.

What it covers:

• the four steps in the order the C runs them, and the one if at Objects/object.c:1925 that is the entire precedence rule
• a table of who wins for every combination of what is on the type and what is in the instance dict, with only one row that surprises anybody
• __getattr__ and __getattribute__, which differ by five characters and by everything else, plus _Py_slot_tp_getattr_hook swapping itself out of tp_getattro when a class has no __getattr__
• _Py_type_getattro_stackref, the separate path a class lookup takes, which checks the metatype MRO first and passes None where the instance would go
• the method cache: a version tag per type, a hash of the tag xored with the name pointer, and type_modified_unlocked zeroing the tag on the class and every subclass rather than sweeping anything
• LOAD_ATTR specialising into four different instructions, and the two extra observations it takes to re specialise after a property is assigned onto the class
• where the "did you mean" suggestion is actually computed, which is not in str(error)

Fourteen citations, six diagrams, one new glossary term, data descriptor.

The M4 checklist is now four of fourteen done. O06 is next and is already in progress: descriptors properly, which turns out to be one protocol with property, classmethod, staticmethod and bound methods all sitting on top of it.

[tamnd]

O06 has landed in #136.

O05 said a data descriptor beats the instance dict and never said what one was. This is that lesson.

The angle that made it worth writing: property, classmethod, staticmethod and the self in every method you have ever written are one protocol, and three of the four are under ten lines of C. sm_descr_get is six lines that hand the callable back untouched. cm_descr_get is eight that call the same PyMethod_New a plain function does, with the class in place of the instance. func_descr_get is seven. Only property is long, and most of its length is error messages.

What it covers:

• a descriptor written by hand so the __get__, __set__ and __set_name__ calls are visible as they happen
• PyDescr_IsData, one line asking whether tp_descr_set is non null, and the whole precedence rule resting on it
• raw.__get__(g, Greeter) doing by hand exactly what the dot does, and a fresh bound method on every read
• classmethod binding to the class the lookup started from, which is what makes Sub.cm() get Sub
• a property with a live value sitting in the instance dict being ignored, and the messages that name which of the three functions is missing
• the five descriptor types C code builds because it cannot write a class body, which two are data descriptors, and the type check descr_setcheck does before writing at a fixed offset
• a descriptor in an instance dict doing nothing at all, and the branch that explains why
• type_new_set_names walking a copy of the class dict once, and the descriptor assigned after the class statement that never learns its name

Sixteen citations, six diagrams, two new glossary terms, descriptor and bound method.

Every cell runs identically on 3.14, 3.15.0rc1 and Pyodide 3.14.2, so the only version note in the lesson is the banner.

The M4 checklist is now five of fourteen. O07 is open in #137: the dict itself, which every lesson so far has been treating as one simple step.