F12. What ends up on disk

[tamnd]

The last lesson of the front end part, and the twenty sixth overall. F01 to F11 all went one direction: text in, a code object out, eleven stages in between. This one goes the other way, and it is the reason importing a module a second time is fast.

What the lesson does

It compiles a two line file with py_compile, then reads the .pyc back as bytes and takes it apart. Sixteen bytes of header, four fields, four bytes each, and everything after that is one marshalled code object and nothing else.

The header cell prints all four fields next to what they are supposed to match, so the staleness rule is visible rather than described: the mtime and the length in the header are the source file's, and if either drifts the file is thrown away.

Then the magic number, which is the part I did not expect. Only two of its four bytes are the number. The other two are a carriage return and a newline, glued on by PYC_MAGIC_NUMBER_TOKEN. That is a trap on purpose: if something copies a .pyc in text mode on a system that translates line endings, those are the bytes that change, so the file fails the magic check loudly instead of loading and doing something strange three hundred bytes later.

The middle of the lesson is marshal itself. One byte per object, seven low bits holding an ascii letter naming the type, top bit meaning "put this in a numbered table, something later may point at it". A cell prints the first byte of nine different values so the letters are visible. Then the sharing: a 66 character string said twice costs 63 bytes less than saying it twice, because the second one is a five byte back reference, and a nested code object costs the same five.

Type c is sixteen fields in a fixed order, and the order is the entire specification. The lesson writes a fifty line reader for the subset a .pyc uses and points it at the file from the top, then checks the filename, the bytecode and the line table against marshal.loads. Worth noticing where the last two fields are: the line table and the exception table F11 decoded, sitting at the end of every function in the file.

It finishes by writing a .pyc by hand and importing it with SourcelessFileLoader, with no .py anywhere on disk, and then breaking one in each of the two ways. A wrong magic number raises. A wrong timestamp does not raise at all, it just gets compiled again and the file quietly rewritten, and the cell proves that by reading the bytes back afterwards.

Checks

Three cells carry differs or varies notes, because the byte counts move between 3.14 and 3.15 and the timestamps are whenever you ran it. Everything that has to be exact, the four matches lines and the two Trues at the end, says the same on both.

One thing that took a fix: the ('abcd', 'abcd') cell was printing a different first byte on each version, because w_ref refuses to share an object with exactly one reference to it and a bare literal passed straight to dumps has a different refcount in the two runtimes. Binding it to a name first makes it stable, and it also demonstrates the rule the prose describes.

Three new glossary terms: marshal, pyc file, magic number. Six diagrams, .excalidraw and .svg, all under 1100 px. Runs end to end in Pyodide and in Colab. Both READMEs have their rows. just check and just versions are green locally.

This closes the twelve front end lessons on M3, issue #22.