Skip to content

fix(runtime): back a python engine runtime cache with the python handle - #4585

Open
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:fix/python-runtime-cache-backing
Open

fix(runtime): back a python engine runtime cache with the python handle#4585
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:fix/python-runtime-cache-backing

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

What is broken

Two tests are red on main, RTX only:

  • runtime/test_aliased_io.py::TestPythonRuntimeAliasedIO::test_in_place_write_through
  • runtime/test_aliased_io.py::TestPythonRuntimeAliasedIO::test_streaming_state_accumulates
TypeError: set_runtime_cache(): incompatible function arguments.
  1. (self: IRuntimeConfig, cache: nvinfer1::IRuntimeCache = None) -> bool
Invoked with: <IRuntimeConfig object at 0x...>, None

Why

RuntimeCache chooses its backing from whether the C++ library is loaded:

if ENABLED_FEATURES.torch_tensorrt_runtime:
    self._handle = torch.classes.tensorrt.RuntimeCacheHandle(path)
else:
    self._handle = _RuntimeCacheHandle(path=path)

A loaded library does not mean a C++ engine is the consumer. These tests build
a python TRTEngine straight from a packed engine tuple, which is a documented
way to use it. Its default settings carry a cache path string, so
TRTRuntimeConfig._apply_settings wraps that path, gets the torchbind handle,
and asks it for a live cache. The torchbind handle materializes its cache on
the C++ side and cannot pass an IRuntimeCache back to python, so
ensure_cache returns None, and None is not a cache.

Fix

Let the caller say which runtime will consume the cache. _apply_settings
only ever runs for a python engine, so it asks for the python backing. Nothing
else changes: every existing caller keeps the old default.

One related detail: IRuntimeConfig does not own the cache attached to it, so
the wrapper now lives on the config instead of being dropped at the end of the
call. Before this change the wrapper was released immediately, which was
harmless only because the attach never happened.

Tested

The RTX path cannot run on this machine. What was checked here, with the C++
library loaded:

  • default backing: ensure_cache returns None, which is the exact input the
    error message reports
  • python backing: ensure_cache returns the live cache

test_000_runtime_cache.py, test_004_runtime_settings.py and
test_aliased_io.py give the same result before and after the change on
non-RTX TensorRT, 28 passed and 27 skipped.

@meta-cla meta-cla Bot added the cla signed label Aug 26, 2026
@github-actions github-actions Bot added the component: api [Python] Issues re: Python API label Aug 26, 2026
@github-actions
github-actions Bot requested a review from narendasan August 26, 2026 02:39
…ndle

RuntimeCache picked its backing from whether the C++ library is loaded, not
from which runtime consumes the cache. A python TRTEngine built directly from
a packed engine tuple therefore got the torchbind handle, whose cache is
materialized on the C++ side, so ensure_cache returned None and
set_runtime_cache(None) raised a TypeError.

Let the caller state the consuming runtime, and hold the wrapper on the config
so the IRuntimeCache outlives the IRuntimeConfig that points at it.
@shoumikhin
shoumikhin force-pushed the fix/python-runtime-cache-backing branch from 1b8c287 to 5805c84 Compare August 26, 2026 03:16
The repository lint job runs `black --check .` across the whole tree, so any
file that does not match the formatter fails CI for every open pull request,
not only the one that touched it. `tests/py/dynamo/conversion/test_cumsum_aten.py`
is currently not black-conformant on main, which turns the Python Linting check
red here.

Reformat that one file with black. This is a formatting-only change: two
statements that fit on a single line are un-wrapped. No test logic changes.

Verified by running `black --check .` on the full tree: all files pass.
@github-actions github-actions Bot added the component: tests Issues re: Tests label Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant