fix(runtime): back a python engine runtime cache with the python handle - #4585
Open
shoumikhin wants to merge 2 commits into
Open
fix(runtime): back a python engine runtime cache with the python handle#4585shoumikhin wants to merge 2 commits into
shoumikhin wants to merge 2 commits into
Conversation
…ndle RuntimeCache picked its backing from whether the C++ library is loaded, not from which runtime consumes the cache. A python TRTEngine built directly from a packed engine tuple therefore got the torchbind handle, whose cache is materialized on the C++ side, so ensure_cache returned None and set_runtime_cache(None) raised a TypeError. Let the caller state the consuming runtime, and hold the wrapper on the config so the IRuntimeCache outlives the IRuntimeConfig that points at it.
shoumikhin
force-pushed
the
fix/python-runtime-cache-backing
branch
from
August 26, 2026 03:16
1b8c287 to
5805c84
Compare
The repository lint job runs `black --check .` across the whole tree, so any file that does not match the formatter fails CI for every open pull request, not only the one that touched it. `tests/py/dynamo/conversion/test_cumsum_aten.py` is currently not black-conformant on main, which turns the Python Linting check red here. Reformat that one file with black. This is a formatting-only change: two statements that fit on a single line are un-wrapped. No test logic changes. Verified by running `black --check .` on the full tree: all files pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is broken
Two tests are red on
main, RTX only:runtime/test_aliased_io.py::TestPythonRuntimeAliasedIO::test_in_place_write_throughruntime/test_aliased_io.py::TestPythonRuntimeAliasedIO::test_streaming_state_accumulatesWhy
RuntimeCachechooses its backing from whether the C++ library is loaded:A loaded library does not mean a C++ engine is the consumer. These tests build
a python
TRTEnginestraight from a packed engine tuple, which is a documentedway to use it. Its default settings carry a cache path string, so
TRTRuntimeConfig._apply_settingswraps that path, gets the torchbind handle,and asks it for a live cache. The torchbind handle materializes its cache on
the C++ side and cannot pass an
IRuntimeCacheback to python, soensure_cachereturnsNone, andNoneis not a cache.Fix
Let the caller say which runtime will consume the cache.
_apply_settingsonly ever runs for a python engine, so it asks for the python backing. Nothing
else changes: every existing caller keeps the old default.
One related detail:
IRuntimeConfigdoes not own the cache attached to it, sothe wrapper now lives on the config instead of being dropped at the end of the
call. Before this change the wrapper was released immediately, which was
harmless only because the attach never happened.
Tested
The RTX path cannot run on this machine. What was checked here, with the C++
library loaded:
ensure_cachereturnsNone, which is the exact input theerror message reports
ensure_cachereturns the live cachetest_000_runtime_cache.py,test_004_runtime_settings.pyandtest_aliased_io.pygive the same result before and after the change onnon-RTX TensorRT, 28 passed and 27 skipped.