_testinternalcapi exports three compiler hooks. pyxray.compiler uses two of them. The third, assemble_code_object, is the one that would let a lesson build a code object by hand from an instruction sequence it had just watched the optimizer rewrite, which is the natural ending for the front end lessons. It is not wired up, and this is why.
What happens
Passing the output of compiler_codegen straight to assemble_code_object does not raise. It aborts.
Assertion failed: (PyDict_Check(umd.u_consts)), function _testinternalcapi_assemble_code_object_impl, file _testinternalcapi.c, line 818.
An abort is a process kill. In a notebook that takes the kernel with it and loses whatever the reader had done in that session, which is the worst failure mode this material can produce, and it happens to the reader who is experimenting rather than the one following along exactly. So pyxray.compiler.assemble raises NotImplementedError with the reason written out, and stages() gets its finished code object from an ordinary compile(), which runs the same three stages inside the interpreter and so is the real output of the pipeline rather than a reconstruction.
The two traps
compiler_codegen returns consts as a list. assemble_code_object requires a dict mapping each constant to its index. Handing one to the other trips the assertion above.
compiler_codegen does not return names, varnames, cellvars or freevars at all. For any code that touches a name there is therefore no way to build correct metadata out of the previous stage's output alone, which is the thing that would make the demonstration honest. Reconstructing them by walking the instruction sequence is possible but it is a reimplementation of part of the symbol table, and if it is wrong the failure is an abort rather than a wrong answer.
What needs doing
Validate every key and its type before the call, so that a bad input produces a Python exception the reader can read instead of a signal. That means knowing the full expected shape of the metadata dictionary, which is fourteen keys and undocumented, and deciding what to do about the four missing ones.
If that turns out to be more than a small piece of work, the alternative is to leave the refusal in place and have the lesson explain what the third stage does using the code object from compile(), which is what it does today. That is a weaker ending but it is not a wrong one.
Not blocking any milestone. Worth doing before M3, where the compile explorer lives.
_testinternalcapiexports three compiler hooks.pyxray.compileruses two of them. The third,assemble_code_object, is the one that would let a lesson build a code object by hand from an instruction sequence it had just watched the optimizer rewrite, which is the natural ending for the front end lessons. It is not wired up, and this is why.What happens
Passing the output of
compiler_codegenstraight toassemble_code_objectdoes not raise. It aborts.An abort is a process kill. In a notebook that takes the kernel with it and loses whatever the reader had done in that session, which is the worst failure mode this material can produce, and it happens to the reader who is experimenting rather than the one following along exactly. So
pyxray.compiler.assembleraisesNotImplementedErrorwith the reason written out, andstages()gets its finished code object from an ordinarycompile(), which runs the same three stages inside the interpreter and so is the real output of the pipeline rather than a reconstruction.The two traps
compiler_codegenreturnsconstsas a list.assemble_code_objectrequires a dict mapping each constant to its index. Handing one to the other trips the assertion above.compiler_codegendoes not returnnames,varnames,cellvarsorfreevarsat all. For any code that touches a name there is therefore no way to build correct metadata out of the previous stage's output alone, which is the thing that would make the demonstration honest. Reconstructing them by walking the instruction sequence is possible but it is a reimplementation of part of the symbol table, and if it is wrong the failure is an abort rather than a wrong answer.What needs doing
Validate every key and its type before the call, so that a bad input produces a Python exception the reader can read instead of a signal. That means knowing the full expected shape of the metadata dictionary, which is fourteen keys and undocumented, and deciding what to do about the four missing ones.
If that turns out to be more than a small piece of work, the alternative is to leave the refusal in place and have the lesson explain what the third stage does using the code object from
compile(), which is what it does today. That is a weaker ending but it is not a wrong one.Not blocking any milestone. Worth doing before M3, where the compile explorer lives.