Raise on stale source in DSL jit preprocessing - #3522
Open
VaggelisGian wants to merge 1 commit into
Open
Conversation
The jit preprocessor re-reads function sources from disk at first
compile and slices at the code object's recorded line number. When a
module changed after import, for example an in-place package upgrade
while a long-lived process kept the old modules loaded, the slice no
longer starts with the decorated definition. The preprocessor then
silently skipped staging and the kernel failed later far from the
cause, with a NameError from unstaged execution or a ValueError about
free variables during code replacement.
Turn that case into an immediate error naming the file whose source
went stale. Functions decorated programmatically without a decorator
in their source take the same path; they were never staged before
either, so nothing that worked regresses.
Add test/python/CuTeDSL/test_stale_source.py covering slices landing
on an undecorated def, on an import line, and recovery once the file
matches again.
Test Plan:
docker python:3.12-slim, editable install of python/CuTeDSL:
module modified after import, unfixed code:
ValueError: kernel_fn() requires a code object with 0 free vars, not 2
fixed code: immediate error naming mod_under_test.py and suggesting
a process restart
python test/python/CuTeDSL/test_stale_source.py -v:
Ran 3 tests ... OK (all three fail on unfixed code)
unmodified module still compiles
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3395
The jit preprocessor re-reads function sources from disk at first compile and slices at the code object's recorded line number. When the module source changed after import, two silent failure modes existed:
transform_functionreturned[], and the function was silently skipped from staging;ValueError: kernel_fn() requires a code object with 0 free vars.ast_preprocessor.transform_functionnow raisesDSLUserCodeErrornaming the stale file when the re-read slice does not start with the decorator, covering both cases with one loud error at the right place.Programmatic decoration without a matching source decorator also raises loudly now (it was never staged before either); aliased decorators (
jj = cute.jit) take the same path.Test plan
New
test/python/CuTeDSL/test_stale_source.py(3 tests, CPU-only): a module whose source changed after import raises the new loud error naming the file; unmodified modules compile normally; a failed-then-restored source compiles again (no preprocessor state poisoning).Run in a
python:3.12-slimcontainer against this branch (wheel natives bridged, branch tree first onsys.path):With only
ast_preprocessor.pyreverted to the parent commit, same environment:End-to-end repro of issue case 2 before the fix:
after the fix the failure names the stale file and suggests re-importing it.