Skip to content

O07: two arrays and a hash #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tamnd merged 1 commit into from
Aug 31, 2026
Merged

O07: two arrays and a hash #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion GLOSSARY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This file is generated from `pyxray/src/pyxray/glossary.py`. Edit that and run `

## Index

[ASDL](#asdl) | [Argument Clinic](#argument-clinic) | [C3 linearization](#c3-linearization) | [EXTENDED_ARG](#extended_arg) | [GC pre header](#gc-pre-header) | [JIT](#jit) | [MRO](#mro) | [PEG parser](#peg-parser) | [PyVarObject](#pyvarobject) | [Pyodide](#pyodide) | [WebAssembly](#webassembly) | [abstract syntax tree](#abstract-syntax-tree) | [adaptive instruction](#adaptive-instruction) | [arena](#arena) | [assembler](#assembler) | [backtrace](#backtrace) | [basic block](#basic-block) | [binding](#binding) | [block](#block) | [blurb](#blurb) | [borrowed reference](#borrowed-reference) | [bound method](#bound-method) | [bytecode](#bytecode) | [cell](#cell) | [class cell](#class-cell) | [closure](#closure) | [code generation](#code-generation) | [code object](#code-object) | [coding cookie](#coding-cookie) | [cold block](#cold-block) | [computed goto](#computed-goto) | [configure](#configure) | [constant folding](#constant-folding) | [control flow graph](#control-flow-graph) | [cycle collector](#cycle-collector) | [data descriptor](#data-descriptor) | [deallocation](#deallocation) | [debug build](#debug-build) | [deoptimization](#deoptimization) | [descriptor](#descriptor) | [devguide](#devguide) | [dispatch](#dispatch) | [environment changed](#environment-changed) | [eval loop](#eval-loop) | [evaluation order](#evaluation-order) | [exception table](#exception-table) | [f string](#f-string) | [finalizer](#finalizer) | [frame](#frame) | [free threaded build](#free-threaded-build) | [free variable](#free-variable) | [gdb](#gdb) | [generated file](#generated-file) | [generation](#generation) | [grammar](#grammar) | [header file](#header-file) | [heap type](#heap-type) | [immortal object](#immortal-object) | [indent and dedent](#indent-and-dedent) | [inline cache](#inline-cache) | [instance dictionary](#instance-dictionary) | [instruction](#instruction) | [interning](#interning) | [left recursion](#left-recursion) | [line table](#line-table) | [magic number](#magic-number) | [marshal](#marshal) | [metaclass](#metaclass) | [monitoring events](#monitoring-events) | [new reference](#new-reference) | [object](#object) | [object header](#object-header) | [obmalloc](#obmalloc) | [oparg](#oparg) | [opcode](#opcode) | [parser generator](#parser-generator) | [pdb](#pdb) | [pointer](#pointer) | [pool](#pool) | [product type](#product-type) | [profile guided optimization](#profile-guided-optimization) | [pseudo instruction](#pseudo-instruction) | [pyc file](#pyc-file) | [pyconfig](#pyconfig) | [reference count](#reference-count) | [reference cycle](#reference-cycle) | [reference leak](#reference-leak) | [regen](#regen) | [regrtest](#regrtest) | [replacement field](#replacement-field) | [resource](#resource) | [scope](#scope) | [segmentation fault](#segmentation-fault) | [short circuiting](#short-circuiting) | [slot](#slot) | [slot wrapper](#slot-wrapper) | [small integer cache](#small-integer-cache) | [soft keyword](#soft-keyword) | [specialization](#specialization) | [stack depth](#stack-depth) | [stack effect](#stack-effect) | [static type](#static-type) | [stolen reference](#stolen-reference) | [struct](#struct) | [sum type](#sum-type) | [symbol table](#symbol-table) | [symbol table pass](#symbol-table-pass) | [t string](#t-string) | [test case](#test-case) | [tier one](#tier-one) | [tier two](#tier-two) | [token](#token) | [tokenizer](#tokenizer) | [trace function](#trace-function) | [type object](#type-object) | [underflow](#underflow) | [value stack](#value-stack) | [weak reference](#weak-reference)
[ASDL](#asdl) | [Argument Clinic](#argument-clinic) | [C3 linearization](#c3-linearization) | [EXTENDED_ARG](#extended_arg) | [GC pre header](#gc-pre-header) | [JIT](#jit) | [MRO](#mro) | [PEG parser](#peg-parser) | [PyVarObject](#pyvarobject) | [Pyodide](#pyodide) | [WebAssembly](#webassembly) | [abstract syntax tree](#abstract-syntax-tree) | [adaptive instruction](#adaptive-instruction) | [arena](#arena) | [assembler](#assembler) | [backtrace](#backtrace) | [basic block](#basic-block) | [binding](#binding) | [block](#block) | [blurb](#blurb) | [borrowed reference](#borrowed-reference) | [bound method](#bound-method) | [bytecode](#bytecode) | [cell](#cell) | [class cell](#class-cell) | [closure](#closure) | [code generation](#code-generation) | [code object](#code-object) | [coding cookie](#coding-cookie) | [cold block](#cold-block) | [compact dict](#compact-dict) | [computed goto](#computed-goto) | [configure](#configure) | [constant folding](#constant-folding) | [control flow graph](#control-flow-graph) | [cycle collector](#cycle-collector) | [data descriptor](#data-descriptor) | [deallocation](#deallocation) | [debug build](#debug-build) | [deoptimization](#deoptimization) | [descriptor](#descriptor) | [devguide](#devguide) | [dispatch](#dispatch) | [environment changed](#environment-changed) | [eval loop](#eval-loop) | [evaluation order](#evaluation-order) | [exception table](#exception-table) | [f string](#f-string) | [finalizer](#finalizer) | [frame](#frame) | [free threaded build](#free-threaded-build) | [free variable](#free-variable) | [gdb](#gdb) | [generated file](#generated-file) | [generation](#generation) | [grammar](#grammar) | [header file](#header-file) | [heap type](#heap-type) | [immortal object](#immortal-object) | [indent and dedent](#indent-and-dedent) | [inline cache](#inline-cache) | [instance dictionary](#instance-dictionary) | [instruction](#instruction) | [interning](#interning) | [left recursion](#left-recursion) | [line table](#line-table) | [magic number](#magic-number) | [marshal](#marshal) | [metaclass](#metaclass) | [monitoring events](#monitoring-events) | [new reference](#new-reference) | [object](#object) | [object header](#object-header) | [obmalloc](#obmalloc) | [oparg](#oparg) | [opcode](#opcode) | [parser generator](#parser-generator) | [pdb](#pdb) | [pointer](#pointer) | [pool](#pool) | [probe sequence](#probe-sequence) | [product type](#product-type) | [profile guided optimization](#profile-guided-optimization) | [pseudo instruction](#pseudo-instruction) | [pyc file](#pyc-file) | [pyconfig](#pyconfig) | [reference count](#reference-count) | [reference cycle](#reference-cycle) | [reference leak](#reference-leak) | [regen](#regen) | [regrtest](#regrtest) | [replacement field](#replacement-field) | [resource](#resource) | [scope](#scope) | [segmentation fault](#segmentation-fault) | [short circuiting](#short-circuiting) | [slot](#slot) | [slot wrapper](#slot-wrapper) | [small integer cache](#small-integer-cache) | [soft keyword](#soft-keyword) | [specialization](#specialization) | [stack depth](#stack-depth) | [stack effect](#stack-effect) | [static type](#static-type) | [stolen reference](#stolen-reference) | [struct](#struct) | [sum type](#sum-type) | [symbol table](#symbol-table) | [symbol table pass](#symbol-table-pass) | [t string](#t-string) | [test case](#test-case) | [tier one](#tier-one) | [tier two](#tier-two) | [token](#token) | [tokenizer](#tokenizer) | [trace function](#trace-function) | [type object](#type-object) | [underflow](#underflow) | [value stack](#value-stack) | [weak reference](#weak-reference)

## Reading the source

Expand Down Expand Up @@ -722,6 +722,22 @@ A class statement compiles to a call to `__build_class__`, which runs the class

First met in O02. See also [type object](#type-object), [heap type](#heap-type). In the source: [`Python/bltinmodule.c:102-108@v3.15.0rc1#builtin___build_class__`](https://github.com/python/cpython/blob/v3.15.0rc1/Python/bltinmodule.c#L102-L108).

### compact dict

**A dict laid out as a small array of slot numbers in front of an entry array.**

The slot array, `dk_indices`, holds a row number into the entry array, or -1 for a position never used, or -2 for one that used to hold something. The entry array is appended to and never reordered, so iterating it top to bottom gives insertion order with no bookkeeping at all. Only the small array has holes, which is why the layout is called compact: a mostly empty hash table costs one byte per slot rather than a whole row.

Also written compact ordered dict. First met in O07. See also [probe sequence](#probe-sequence), [instance dictionary](#instance-dictionary). In the source: [`Include/internal/pycore_dict.h:196-235@v3.15.0rc1#_dictkeysobject`](https://github.com/python/cpython/blob/v3.15.0rc1/Include/internal/pycore_dict.h#L196-L235).

### probe sequence

**The order of slots a dict lookup visits when the first one is a collision.**

The first slot is the hash masked down to the table size. After that the step is `i = mask & (i * 5 + perturb + 1)`, where `perturb` starts as the whole hash and is shifted right by 5 each round. The `i * 5 + 1` part visits every slot exactly once and in an order unrelated to how consecutive keys arrive, and `perturb` brings back the high bits of the hash the mask discarded. For a size 8 table starting at slot 0 the order is 0, 1, 6, 7, 4, 5, 2, 3.

First met in O07. See also [compact dict](#compact-dict). In the source: [`Objects/dictobject.c:1078-1101@v3.15.0rc1#do_lookup`](https://github.com/python/cpython/blob/v3.15.0rc1/Objects/dictobject.c#L1078-L1101).

## Memory

Where objects come from and what happens to them afterwards. T09 is the lesson.
Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ A lesson can also end with a boss fight, which is a problem the text does not so
| O04 | [The order things are found in](lessons/o04-the-order-things-are-found-in/o04.ipynb) | Why a class with two bases carries a flat list instead of searching, the twenty line merge that builds it and matches CPython exactly on classes out of the standard library, the two ways a class statement can fail before the class exists, why super goes somewhere the class it is written in has never heard of, and the two ways the list changes after the fact | M4 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o04-the-order-things-are-found-in/o04.ipynb) |
| O05 | [What a dot does](lessons/o05-what-a-dot-does/o05.ipynb) | The four places x.name looks and the fixed order it looks in them, why a property cannot be shadowed by an instance attribute and a method can, why __getattr__ and __getattribute__ are not two versions of the same hook, why a class lookup starts at the metaclass and passes None where the instance would go, and the version tag that lets four thousand cache entries go stale without anybody clearing them | M4 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o05-what-a-dot-does/o05.ipynb) |
| O06 | [What property actually is](lessons/o06-what-property-actually-is/o06.ipynb) | The three method protocol that property, classmethod, staticmethod and the self in every method you have written all turn out to be, the one line of C that decides whether a descriptor beats the instance dict or loses to it, the seven lines that build a bound method, the five descriptor types C code makes because it cannot write a class body, why a descriptor in an instance dict does nothing at all, and the pass over the class dict that tells each descriptor the name it was assigned to | M4 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o06-what-property-actually-is/o06.ipynb) |
| O07 | [Two arrays and a hash](lessons/o07-two-arrays-and-a-hash/o07.ipynb) | The small array of slot numbers in front of the append only array of entries, why insertion order comes free out of that split, the three lines that decide where a lookup looks next and the slot order they produce, the sixty lines of Python that grow at exactly the same key counts as the real dict, why deleting frees nothing and can even make a dict bigger, and why a dict of strings does not store its hashes | M4 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tamnd/cpython-internals/blob/main/lessons/o07-two-arrays-and-a-hash/o07.ipynb) |

More are landing in order. [lessons/README.md](lessons/README.md) explains how one is put together and how to run them locally.

Expand Down
80 changes: 80 additions & 0 deletions citations.lock.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,21 @@
"first_line": "static inline int",
"lines": 6
},
"Include/internal/pycore_dict.h:184-185@v3.15.0rc1": {
"digest": "1c99b541e59b4656",
"first_line": "#define DKIX_EMPTY (-1)",
"lines": 2
},
"Include/internal/pycore_dict.h:196-235@v3.15.0rc1": {
"digest": "f371282052e3ed7f",
"first_line": "struct _dictkeysobject {",
"lines": 40
},
"Include/internal/pycore_dict.h:79-90@v3.15.0rc1": {
"digest": "c3ccf876d4deb3d8",
"first_line": "",
"lines": 12
},
"Include/internal/pycore_flowgraph.h:27-31@v3.15.0rc1": {
"digest": "5f318ca1d3438caa",
"first_line": "struct _PyCfgBuilder* _PyCfg_FromInstructionSequence(_PyInstructionSequence *seq);",
Expand Down Expand Up @@ -695,6 +710,71 @@
"first_line": "static PyObject *",
"lines": 37
},
"Objects/dictobject.c:1078-1101@v3.15.0rc1": {
"digest": "4385385afec4f891",
"first_line": "do_lookup(PyDictObject *mp, PyDictKeysObject *dk, PyObject *key, Py_hash_t hash,",
"lines": 24
},
"Objects/dictobject.c:2141-2175@v3.15.0rc1": {
"digest": "344810b229e48b6b",
"first_line": "dictresize(PyDictObject *mp,",
"lines": 35
},
"Objects/dictobject.c:2904-2941@v3.15.0rc1": {
"digest": "7a0ffae503ffbf9f",
"first_line": "delitem_common(PyDictObject *mp, Py_hash_t hash, Py_ssize_t ix,",
"lines": 38
},
"Objects/dictobject.c:329-329@v3.15.0rc1": {
"digest": "49bfe45f364ef798",
"first_line": "#define PERTURB_SHIFT 5",
"lines": 1
},
"Objects/dictobject.c:360-398@v3.15.0rc1": {
"digest": "725a960e3d558884",
"first_line": "The first half of collision resolution is to visit table indices via this",
"lines": 39
},
"Objects/dictobject.c:37-49@v3.15.0rc1": {
"digest": "3c2ccab161989419",
"first_line": "Size of indices is dk_size. Type of each index in indices varies with dk_size:",
"lines": 13
},
"Objects/dictobject.c:581-581@v3.15.0rc1": {
"digest": "142bef72e1124e8d",
"first_line": "#define USABLE_FRACTION(n) (((n) << 1)/3)",
"lines": 1
},
"Objects/dictobject.c:584-604@v3.15.0rc1": {
"digest": "ad8228aa7b7cea82",
"first_line": "static inline uint8_t",
"lines": 21
},
"Objects/dictobject.c:606-615@v3.15.0rc1": {
"digest": "e05e8d8144b5bcd0",
"first_line": "/* estimate_keysize is reverse function of USABLE_FRACTION.",
"lines": 10
},
"Objects/dictobject.c:618-628@v3.15.0rc1": {
"digest": "96e2b280745dc3c5",
"first_line": "/* GROWTH_RATE. Growth rate upon hitting maximum load.",
"lines": 11
},
"Objects/dictobject.c:630-640@v3.15.0rc1": {
"digest": "3275c5fcfd6e274a",
"first_line": "/* This immutable, empty PyDictKeysObject is used for PyDict_Clear()",
"lines": 11
},
"Objects/dictobject.c:66-89@v3.15.0rc1": {
"digest": "dc6f596fb1dd0c0a",
"first_line": "There are four kinds of slots in the table (slot is index, and",
"lines": 24
},
"Objects/dictobject.c:91-106@v3.15.0rc1": {
"digest": "1049da1243d995b5",
"first_line": "/*",
"lines": 16
},
"Objects/funcobject.c:1264-1270@v3.15.0rc1": {
"digest": "991dc84999f49f93",
"first_line": "func_descr_get(PyObject *func, PyObject *obj, PyObject *type)",
Expand Down
11 changes: 10 additions & 1 deletion lessons/CLAIMS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ header, what the allocator does with a freed block, the shape of the eval loop.
marked with the reason, and a lesson is allowed at most 3 of them. The cap is the point.
Without it the exception becomes the rule and this goes back to being a book.

287 claims across 34 lessons, 22 of them not observable from Python.
291 claims across 35 lessons, 22 of them not observable from Python.

## B01. Building CPython, and whether you need to

Expand Down Expand Up @@ -257,6 +257,15 @@ Without it the exception becomes the rule and this goes back to being a book.
| a descriptor placed in an instance dict is returned as itself, because the lookup that would call __get__ only runs for objects found on the type | [`o06-24`](o06-what-property-actually-is/o06.ipynb) |
| __set_name__ runs once for every entry in the class dict during class creation, and never runs again for later assignments to the class | [`o06-26`](o06-what-property-actually-is/o06.ipynb) |

## O07. Two arrays and a hash

| Claim | Proved by |
| --- | --- |
| the probe order for a size 8 table starting at slot 0 is 0, 1, 6, 7, 4, 5, 2, 3, which visits every slot exactly once before repeating | [`o07-08`](o07-two-arrays-and-a-hash/o07.ipynb) |
| a dict grows when its entry array is full rather than when its length crosses a threshold, so the resize points fall at 6, 11, 22, 43 and 86 keys | [`o07-10`](o07-two-arrays-and-a-hash/o07.ipynb) |
| a dict that is deleted from and inserted into in equal measure still resizes, because the entry array only ever gets compacted by a resize | [`o07-16`](o07-two-arrays-and-a-hash/o07.ipynb) |
| a dict with only exact str keys uses a smaller entry layout, and putting one non string key in converts the whole table to the larger one | [`o07-21`](o07-two-arrays-and-a-hash/o07.ipynb) |

## T01. One line, seven stages

| Claim | Proved by |
Expand Down
Loading
Loading