fix(dynamo): keep the complex decomposition inside the TRT op set - #4580
Open
shoumikhin wants to merge 2 commits into
Open
fix(dynamo): keep the complex decomposition inside the TRT op set#4580shoumikhin wants to merge 2 commits into
shoumikhin wants to merge 2 commits into
Conversation
The complex lowering pass hands the interior expansion to PyTorch's decompose_complex_in_graph, which re-traces the module with make_fx. The module that comes back is therefore built from whatever the complex tensor subclass dispatched to, not from the ops that survived the earlier stages of the same pass list, and three kinds of leftovers reach the partitioner: * aten.stack and aten.alias, which the decomposition table run at export time expands and drops, are back; * constants that constant folding had already reduced to one frozen tensor are live chains of ops again, and the converters for those ops want a TRT tensor rather than a weight; * aten.view_as_real and aten.view_as_complex are left at the seams. The last one only shows up when the complex region is interior, that is when the graph takes real tensors and returns real tensors, as a rotary embedding does. The boundary normalizer only knew about aten.complex, aten.real and aten.imag, which is what a graph with complex inputs or outputs meets upstream at. None of these have a TensorRT converter, so the partitioner cut the graph around them: a rotary attention module that used to build one engine came back as three, with PyTorch blocks in between. Pass the decomposition table to the retrace, fold constants again after it, and teach the normalizer the two view_as_real seams. The rotary attention module builds a single engine again and matches eager output.
The repository lint job runs `black --check .` across the whole tree, so any file that does not match the formatter fails CI for every open pull request, not only the one that touched it. `tests/py/dynamo/conversion/test_cumsum_aten.py` is currently not black-conformant on main, which turns the Python Linting check red here. Reformat that one file with black. This is a formatting-only change: two statements that fit on a single line are un-wrapped. No test logic changes. Verified by running `black --check .` on the full tree: all files pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is broken
tests/py/dynamo/lowering/test_aten_lowering_passes.py::TestComplexSubgraph::test_complex_subgraphis red onmain. The test compiles a rotary attention module and expects one TensorRT engine; it gets three, with PyTorch blocks in between.Why
The complex lowering pass hands the interior expansion to PyTorch's
decompose_complex_in_graph, which re-traces the module withmake_fx. The module that comes back is therefore built from whatever the complex tensor subclass dispatched to, not from the ops that survived the earlier stages of the same pass list. Three kinds of leftovers reach the partitioner:aten.stackandaten.alias, which the decomposition table run at export time expands and drops, are back;aten.view_as_realandaten.view_as_complexare left at the seams.The last one only shows up when the complex region is interior, meaning the graph takes real tensors and returns real tensors, which is what a rotary embedding does. The boundary normalizer only knew about
aten.complex,aten.realandaten.imag, which is what a graph with complex inputs or outputs meets at its edges.None of these have a TensorRT converter, so the partitioner cuts the graph around them.
Fix
Three parts, all in the lowering pass:
view_as_realseams:view_as_real(view_as_complex(x))folds tox, andview_as_real(complex(re, im))folds to the interleaved real layout the rest of the flow expects.Tests
The failing test passes again: the rotary attention module builds a single engine and matches eager output.
Full runs:
tests/py/dynamo/lowering/326 passed, and the complex, rope and graph break suites 189 passed.Three new unit tests cover the added folds and the post-retrace constant folding. All three fail without this change and pass with it.