Ship only the TensorRT delegate in the ExecuTorch runtime wheel - #4567
Open
shoumikhin wants to merge 16 commits into
Open
Ship only the TensorRT delegate in the ExecuTorch runtime wheel#4567shoumikhin wants to merge 16 commits into
shoumikhin wants to merge 16 commits into
Conversation
github-actions
Bot
added
component: tests
7 tasks
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
3 times, most recently
from
August 23, 2026 19:00
44796ff to
3c104cb
Compare
shoumikhin
marked this pull request as ready for review
August 23, 2026 19:08
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
from
August 23, 2026 19:27
3c104cb to
4adc20b
Compare
github-actions
Bot
added
the
documentation
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
from
August 23, 2026 19:30
4adc20b to
7cac1af
Compare
shoumikhin
marked this pull request as draft
August 23, 2026 19:31
shoumikhin
marked this pull request as ready for review
August 23, 2026 19:57
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
7 times, most recently
from
August 24, 2026 17:00
ff3379e to
a008221
Compare
lanluo-nvidia
added
the
ci: nightly
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
4 times, most recently
from
August 25, 2026 06:55
49b052e to
370c368
Compare
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
2 times, most recently
from
August 25, 2026 18:48
50b85c1 to
29f8db4
Compare
…nce runner gate
The ExecuTorch gate exports one program, x + 1, and TensorRT takes it whole. So
nothing in CI ever runs a program where TensorRT and ExecuTorch's own CUDA
backend each own part of the same graph. That coalesced case is the whole point
of combining the two backends, and it is not covered end to end today. There is
a composition test that checks both delegates land in the file, but it never
loads or runs the program.
This adds the missing run.
A new example, examples/torchtrt_executorch_example/export_coalesced.py, exports
cos(erfinv(tanh(x))). TensorRT has no converter for erfinv, so a CudaPartitioner
catch-all gives that operator to the CUDA backend while TensorRT keeps the rest.
The script fails if the saved .pte does not carry both a TensorRTBackend and a
CudaBackend delegate, so a partitioning change cannot quietly turn this into a
TensorRT-only run that still passes.
The script also writes <model>.expected next to the .pte, holding the output
shape and the eager reference value for an all-ones input. Both reference
runners fill inputs with 1.0 and this model is elementwise, so one number
describes the whole expected output. Reading it from a file, instead of
hard-coding a number in the shell script, keeps the expectation tied to the
model.
verify-executorch-reference-runner.sh now takes an optional third argument, the
coalesced .pte. When given, it runs both the CMake-built runner and the packaged
runner on it and compares every printed value against that reference. TensorRT,
AOTInductor and eager PyTorch use different kernels for the same math, so the
comparison uses a tolerance of 0.001 rather than matching printed digits.
The existing x + 1 assertions keep the same strength. They now go through the
same helper with a zero tolerance, because x + 1 on ones is exact in float32.
Usage:
python examples/torchtrt_executorch_example/export_coalesced.py \
--model_path=coalesced.pte
.github/scripts/verify-executorch-reference-runner.sh \
model.pte kv_cache_decode.pte coalesced.pte
Test plan
On a Linux x86_64 host with an NVIDIA A100 GPU:
- Ran export_coalesced.py. It reported delegates
['TensorRTBackend', 'CudaBackend', 'TensorRTBackend'] and wrote "[64,64]" and
"0.6722" into the .expected file.
- Ran the resulting .pte through the reference runner. It printed
"output[0] shape=[64,64]" and first 8 values of 0.6722, an exact match to the
eager result.
- Deleted the aoti_cuda_blob.ptd that the CUDA backend writes and ran again.
Same output, so this model needs no external weight file.
- Exercised the new shell assertion helper against captured runner output:
correct output passes; one wrong value fails; a wrong shape fails; a missing
values line fails; a value inside the tolerance passes and one outside it
fails.
- shellcheck, bash -n, black and isort are clean on the changed files.
Not yet observed in CI: the ExecuTorch runtime build job currently fails on main
when the packaged reference runner aborts on the existing x + 1 model, and the
test job is skipped while that is true. Both happen before this new code runs.
ExecuTorch 1.4.1 ships no linkable C++ runtime: its wheel contains zero shared libraries, its CMake package exports only a static _portable_lib, and no CUDA wheel exists for it on any channel. That is why the runtime wheel rebuilds ExecuTorch from source today, and it is the blocker for shipping only the TensorRT delegate. The prebuilt runtime landed on ExecuTorch main on 2026-08-20, six days after 1.4.1 was tagged, so no release carries it yet. Move the pin to the nightly line that does, keeping the release-line range on installable metadata so the same range prefers 1.5.0 over any dev build the day it ships, with no edit needed. The two pins now have to name one ExecuTorch rather than two that look close, because the delegate compiles headers from the source tree and links the runtime out of the wheel. Every wheel records its source commit, so add a test asserting the pinned commit is the pinned wheel's own git_version. Nothing else was enforcing that, and a mismatch is silent: both pins look plausible and the build succeeds. Deriving the range with a three-field split raised on the nightly form, so derive it from the release line the first two fields name. ExecuTorch's CUDA wheels are published only on the PyTorch nightly index, so every install site gains that channel. No site gains --pre: the pin names an exact dev build, which pip installs from an explicit version without it, and the existing --pre uses here are for torch. CI derives the channel from the row's own CU_VERSION, which keeps the runtime the delegate links to the same CUDA build as the rest of the job.
The CI installer globs torch_tensorrt*.whl, which also matches the ExecuTorch runtime wheel, whose install_requires names a dev build published only on the nightly channel. With no index on that line the whole pip invocation failed, and because line 1's set -e is commented out the failure was swallowed and the job died later with a confusing ImportError. The two range sites installed a range against the nightly channel, which gains a member every day, so they resolved to whatever was newest while the delegate is compiled from the commit the pin names. Both now request the pin exactly, which is the pairing the drift test exists to check; it was written to skip in exactly the state the ranges produced, so nothing reported it. setup.py keeps its range, because a published requirement has to stay resolvable for users off the same line. The two shapes now differ deliberately and test_derived_requirements_match_the_pin checks each for its own. Six printed install instructions gave a bare pip install of the executorch extra, which cannot resolve a dev pin from PyPI. They name the channel now. The discovery regex saw only == and >=, so a site added with any other PEP 440 operator was invisible to the drift check. It now recognises all of them.
The executorch suite is nightly-only, and the nightly matrix runs cu132 rows as
well as cu130 ones, so the fixed cu130 channel in tests/ci/runner.py would
install a CUDA 13.0 ExecuTorch into a CUDA 13.2 job. PR builds are pinned to
cu130 by filter-matrix.py, which is why watching PR CI could never show this.
Derived from CU_VERSION now, with cu130 as the local default, matching what the
two workflow files already did.
Three example docstrings still printed a bare `pip install -e ".[executorch]"`.
That resolved off PyPI before this pin moved to a dev build; it cannot now, so
they name the nightly index too. The runtime's ImportError advice and the
reference runner README already did.
The installer's hardcoded nightly channel gets the reason written down: a .dev
wheel exists on no other channel, so deriving it from ${CHANNEL} like the lines
above would break the install on exactly the test and release runs the index was
added for.
…alls from Raising the executorch floor to a dev build made `uv lock` fail outright. The lock's win32 required-environment in pyproject.toml resolved the extra from PyPI, whose executorch stops at 1.4.1 -- uv.lock records five win_amd64 wheels for it -- so nothing satisfied the new range and uv errors rather than falling back. Reproduced against a probe project: without a marker uv reports the win32 split unsatisfiable, with one it resolves. The requirement now carries `platform_system == 'Linux'`, the shape EXECUTORCH_RUNTIME_REQUIREMENT already uses, which also stops pip reporting no matching distribution for Windows users of the extra. The delegate is a Linux object and ExecuTorch publishes CUDA wheels for no other platform, so the marker states what was already true. docgen installed the extra with --pre against the nightly channel, so it resolved through the range and took whichever dev build was newest that morning while the delegate compiled from the pinned commit. It names the pin now, read out of dev_dep_versions.yml. The pip line that installs both wheels gets `|| exit 1`. linux-test.yml concatenates this installer ahead of the user script and line 1's `set -e` is commented out, so a failure there was discarded and the job died later with an unrelated-looking ImportError; measured with `false` in place of the pip call, exit was 0 and the user script still ran. Two tests were checking source text rather than behaviour. The CUDA-row test now calls _setup_commands with CU_VERSION set and unset and reads the URL, which catches keeping the os.environ.get line while hardcoding the channel -- the mutation the string match passed. The drift check now asserts the set of files that pin ExecuTorch, because a site changing to bare `executorch` stops matching the search entirely and left the old `assert found` satisfied. Also corrects two claims: 1.4.1 does ship _portable_lib.so, so the comment says its executorch/lib carries no standalone linkable runtime, and no install site gains --pre, since an exact .dev pin needs none.
…tup fails The reference-runner README and the runtime's ImportError advice both printed `pip install "torch-tensorrt[executorch]"` with no index, and the commit that introduced the pin claimed otherwise. That claim was checked against the wrong branch: the fix existed only on the stacked runtime-wheel change, so this branch kept shipping the bare command. It matters more here than a docs nit, because this branch is what raises the floor above PyPI's newest executorch, so the bare command now cannot resolve at all. All seven printed install instructions carry the channel. The drift checks were counting the wrong thing. The requirement test asserted a set of paths, but two files carry two sites each, so either could drop one and stay in the set: turning `executorch-build-linux.yml:88` or `:128` into bare `executorch` both survived. The commit test only asserted nonzero, so any single MODULE.bazel could switch to `branch = "nightly"` unnoticed. Both now assert a per-file site count through one helper, as a minimum rather than an exact number so it holds on the stacked branch too, which removes one README site. All five mutations are caught and each names the file. Counting also surfaced a fifth commit site the nonzero check could not see: the reference-runner README's EXECUTORCH_REF shell default, correctly pinned but unaccounted for. docgen's pin was invisible to both: it is built by a shell substitution, so `$(` is not a digit and the literal search never saw it, and deleting the line survived. The derived-requirement test now runs the command docgen embeds and compares what it prints. A failed setup step printed `::warning::` and fell through to pytest. Most of the executorch suite gates on pytest.importorskip, so a failed ExecuTorch install skipped those files, left the rest passing, and reported success with a populated junit xml -- green exactly when the suite could not test what it exists to test. Driving the real run_suite with a failing setup step reproduced it, and returning the code makes it red without invoking pytest. Pre-existing, but this branch makes it likely to fire, since a nightly pin is eventually pruned from the channel. The pin tests themselves ran on nightly only, so none of this drift machinery ran on a PR or a push to main -- when a pin actually goes stale. They need no GPU, no ExecuTorch and not even torch, so they move to their own l0 suite in every lane, and the nightly suite excludes them by keyword so nothing runs twice. Also: the reference-runner README no longer says the extra installs the runtime wheel, since that requirement is commented out in setup.py.
…U lane The drift checks read derived strings and never the values CI consumes, so three ways of silently shipping no ExecuTorch all stayed green. Dropping the requirement from the runner's setup command left the step succeeding with nothing installed, after which the suite skips on importorskip; emptying EXTRAS_REQUIRE["executorch"] broke every documented `pip install "torch-tensorrt[executorch]"`; and the runtime README was recorded as carrying one pin site when it carries two, so either could go bare while the other satisfied the count -- the exact hole the per-file counts were added to close. The checks now assert the argument list the runner builds, the extras entries by AST, and the true per-file counts. All five mutations fail now. run_suite had no test at all, so replacing its `return rc` with `continue` restored the silent-green behaviour the fail-closed change exists to prevent. It is driven directly now, asserting both the propagated exit code and that pytest never runs once setup has failed. The pin suite was landing on a GPU runner: Suite.runner defaults to the matrix validation runner, so a five-second text check became one CUDA-container job per python and CUDA row, behind a wheel build. It runs in the Python lint job instead, which is already ubuntu-latest and needs none of that. The claim that it needs "not even torch" was also wrong -- tests/py/dynamo/conftest.py imports torch at module scope, which is why the lint invocation passes --noconftest. The shell tier that runs the whole executorch directory now excludes the pin file too, so the dedup claim is true of both paths rather than just the manifest one. uv.lock still records the pre-bump range with no platform marker. uv-update.yml regenerates it on pushes to main touching setup.py, and only that workflow runs `uv sync --locked`, so this breaks nothing -- but the drift was invisible, since the lock writes a bare specifier the pin search cannot match. A strict=False xfail records it and turns into a real failure via XPASS once the lock is refreshed. Editing the lock by hand was the wrong fix: its resolved entry and hashes come from a resolver run against the nightly index. Also removes internal shorthand from the PR description, and corrects a line citation for the one deliberate range in executorch-build-linux.yml.
The lint step added for these checks could not execute. It invokes pytest, and the job installs .github/scripts/requirements.txt (PyGithub) plus the lint dependency group (black, clang-format); neither carries pytest, so the step exited 1 on "No module named pytest" before running a single assertion. pyyaml is needed too, because reading the pin file shells out to a yaml import. Both are installed now, and a test asserts the step exists and installs them, since deleting it is otherwise invisible: every assertion here still passes locally while nothing runs it on a pull request. Reproduced the failure in a stdlib-only venv and confirmed the fixed command passes with only those two. The step also gets if: always(), so an unrelated formatting failure earlier in the job no longer hides the pin check. Three properties the checks are supposed to protect had no coverage: Deleting both published extras from EXTRAS_REQUIRE left everything green. The loop iterated whatever keys existed, so removing them iterated nothing and was indistinguishable from them being correct. It now requires the two published keys to be present, and only those, which also stops an unrelated future extra from turning this red for naming no ExecuTorch. The workflow opt-out marker was ordinary prose, "verify the end user's workflow". Pasting that sentence above a requirement and widening it to a range passed. It is an explicit token now, and the upward scan walks through comment lines to find it, so a cosmetic line between the opt-out and the requirement neither reclassifies the site nor fails the build. Nothing asserted that printed install instructions name the nightly channel, which is why that regressed and was re-fixed three times in this change without anything noticing. One test covers all of them by reading whole blocks rather than single lines, since every instruction wraps and the index lands on a continuation. It catches the CI install of the locally built wheel too, which carries no extra and is the site that broke most often. Generated docs under docs/ are excluded: corrections belong in docsrc/, and the committed Sphinx output is stale there independently. Also: the executorch requirement now strips its local version label like the other four, so the wheel does not bind itself to one CUDA train; the lockfile xfail is strict, since a non-strict xfail reports XPASS and ignores it and so could never fail; the fail-closed comment says it covers every setup step rather than implying only executorch; an empty frozenset and the dead branch reading it are gone; and the sys.path mutations use monkeypatch so they do not leak between tests.
The check that the two pins name one ExecuTorch could not run anywhere. It skips unless the installed wheel is exactly the pinned version, so it means something only on the nightly GPU lane, and that lane deselected it. The deselection is written as "not test_executorch_pin" to skip the source-consistency checks in the same file, but -k matches the module name in the test id, so it dropped every test in the module including this one. Both deselection sites now keep it by name. Proved it on a host with the pinned wheel installed, whose recorded git_version is the pinned commit: the check passes at the correct pins, fails when the commit pin names a different tree, and fails when the commit pin is deleted outright. Before this it was deselected in all three states. Bumping the version alone still skips, correctly, because the installed wheel is then not the one the pin names and its provenance says nothing about whether the two pins agree. A test asserts both sites keep it, since re-tightening either one to a bare module name is a small and plausible edit that would silently restore the gap.
Every guard added in this change asserted that a string appeared somewhere in a file, so each certified the state it was written to prevent. The keyword guard grepped for the kept test's name. Changing "or" to "and" in both -k expressions left it green, and that expression collects nothing at all, which is worse than the bug the guard exists to catch. Reverting the expressions and leaving the name behind in a comment also left it green, and a comment explaining the keyword sits directly above it, which is where an editor would naturally write that name. It now runs pytest's own collection under each expression and requires exactly the pairing test to come back. The CI guard searched the workflow as one blob, so it could not tell which job it was reading. The same commit that fixed the lint failure also added pytest and pyyaml to cpp-linting, which has no pin check, so deleting them from the job that does run it stayed green and would have restored the original failure invisibly. Neutralising the command while leaving its filename in a shell comment, and setting a falsy step condition, were also green. It now parses the workflow, finds the job that actually invokes pytest on this file, and requires the installs in an earlier step of that same job. The unused installs are gone from cpp-linting. The requirement pattern captured an equality prefix and stopped, so "executorch==PIN,!=PIN", a specifier that excludes the version it appears to pin, compared equal to the pin. The same truncation rejected the legal PEP 508 spelling with spaces around the operator. Requirements are parsed now and compared as specifier sets, with a check that the pinned version actually satisfies them. The site scanner counted raw search hits, so gutting a pin to a bare "executorch" while putting the exact pin in a comment in the same file kept the per-file minimum satisfied. Comments no longer count, except in the bazel repositories, where the annotation beside the pinned commit is the only record of which wheel that commit belongs to. Also corrected two claims this change made: the executorch tier is reachable from a pull request through executorch-test-linux.yml as well as the nightly manifest, so it is not the only route, and the shell helper now says why one test is kept out of the deselection.
The uv.lock check was a strict xfail. uv.lock records ">=1.4.1,<1.5" while the pin derives ">=1.5.0.dev20260822,<1.6", so the assertion fails and the xfail is satisfied. Refresh the lock and the assertion passes, and a strict xfail reports that pass as a failure. The lint step runs this file with if: always() on every pull request, so one lock refresh would have made the lint job red on every subsequent pull request, for a file none of them touched, until someone edited this test. Measured: baseline 1 xfailed, and 1 failed once the specifier is bumped. My own docstring claimed the lock is machine-generated and not edited by hand. Two hand refreshes landed on 2026-08-23, inside ordinary version-bump changes, so that was wrong as well. It now accepts both resting states and only fails where something is actually wrong: a recorded range whose lower bound is above the pin, which means the lock names an ExecuTorch this repository does not pin. Behind the pin passes, the derived range passes, and ">=1.6,<1.7", an open-ended ">=1.7" and "==1.9.0" all fail. Comparing lower bounds rather than probing the specifier with sample versions: an upper-bound test missed the open-ended case, and a low sentinel version called the ordinary behind-the-pin state a failure.
test_derived_requirements_match_the_pin extracted the python3 -c one-liner from docgen.yml and ran it. Whatever that line said got executed on every pull request: rewriting it to write a file left the test green and the file written. Same class as the bash -c problem fixed in test_api.py last round, still live here. It now compares the command as text against the exact form that reads __executorch_version__ out of dev_dep_versions.yml. Four mutations caught, including a payload that writes a file and still prints the right version, with nothing executed. The CI reachability guard tested the raw string for "--collect-only", so it accepted "--co", pytest's own documented short form, which collects and asserts nothing. It also could not see an exit status being discarded. Now tokenised: --collect-only, --co, -h, --help, a "||" short-circuit and continue-on-error are all rejected, and all five are caught where four previously survived. The comment exemption for .md/.rst/.txt defeated exactly the threat its docstring names. Install commands live in prose files, so exempting them made a comment count as a pin there: the runtime README's install line gutted to a bare "executorch" passed as long as a decoy "# executorch==<pin>" sat beside it, and failed only with no comment present. The exemption is gone, and trailing comments no longer count either, since a decoy after a live requirement on the same line kept the per-file count satisfied. Five mutations caught, baseline green.
…it resolves The nightly-index guard matched only the named-distribution spelling, so the four sites that write "pip install .[executorch]" were unguarded: docgen.yml and the three export examples. The nightly index could be deleted from all four with the test green. Each of the four is now caught individually. Its second half was a bare substring test for the host, which proves a string sits nearby rather than that the instruction resolves. Rewriting every channel in the tree, 18 files, to a nonexistent cu999 left it green. The CUDA suffix is now checked against the set the project publishes for. Deliberately not compared against __cuda_version__: five sites legitimately say cu130 while the pin says 13.2, and I confirmed against the live index that cu130 and cu132 both carry 38 ExecuTorch wheels while cu999 carries none.
The printed install commands resolved no ExecuTorch. "torch-tensorrt[executorch]" with no version pin resolves the stable PyPI wheel, which carries no executorch extra, so the command exited 0 and installed nothing the feature needs. Add --pre to the six commands that name the extra and assert its presence in the guard that already reads them. Close four ways to neutralise the pin check while its guard stayed green: a ";" or "&" terminator after pytest, continue-on-error or a falsy if: on the owning job, and reducing the workflow trigger so it never runs on pull requests. The trigger check also handles PyYAML reading the unquoted "on" key as the boolean True. Close both ways to strip the pairing check while its guard stayed green: assert the workflow actually calls trt_tier_executorch, and validate suite lane names against the known set so a typo raises at import instead of silently dropping the suite from every matrix. Also: anchor the docgen pin check to a live line so a commented-out install no longer satisfies it; fix the lockfile range check crashing on a legal "==1.4.*" clause; correct the range comment to describe what the range admits; and note in the install advice that the feature is published for Linux only.
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
4 times, most recently
from
August 25, 2026 23:13
7f7a295 to
cbde0f4
Compare
The delegate is built against one ExecuTorch: __executorch_version__ selects the wheel it links against and __executorch_commit__ selects the tree it compiles from. Those two values repeat across the build workflows, the bazel modules, the docker and toolchain copies, and the docs, so they can drift apart or fall behind upstream with nothing to notice. Add a script and a daily workflow that move both pins to the newest ExecuTorch wheel on the nightly index. The source commit is read from the chosen wheel's own version.py, so the two pins always name one ExecuTorch rather than two that happen to be close. The update lands as a pull request, so the pin consistency checks and the delegate build and test lane decide whether the new wheel is usable before it reaches main. A day with no new nightly rewrites nothing and opens nothing. On a release branch the schedule is a no-op and the pin moves only by a manual run pointed at the stable line, so a cut release does not drift. Back the mechanism with consistency checks that run under the linter. Every requirement and comment that names ExecuTorch is asserted to match the pinned version, including the variable-index install once the variable's assignment is resolved and extensionless install files like justfile. The source commit is checked against the wheel's own provenance wherever that wheel is installed, and commits left in comments are not mistaken for pins. The wheel-content and CI-invocation checks measure effect, running the workflow's own step against a passing and a failing stub and requiring the exit status to follow, rather than enumerating bypass spellings. Install the built wheel in the runtime README rather than an unpublished package.
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
from
August 26, 2026 03:26
cbde0f4 to
6eba5e9
Compare
The torch-tensorrt-executorch-runtime wheel shipped a full ExecuTorch Python runtime alongside the TensorRT delegate. This ships only the delegate: a single shared library that registers TensorRTBackend with the ExecuTorch runtime that the executorch distribution already provides, rather than bundling a second copy of that runtime. The native build produces just the delegate library, its RUNPATH points at the executorch package the delegate links against, and setup.py packages the one shared object. The runtime dependency stays commented out in the top-level setup.py because the delegate wheel is not published to any index yet, so the docs and the load-time and save-time errors direct users to build it from py/torch-tensorrt-executorch-runtime/README.md. The delegate links libstdc++ statically. The build toolchain is newer than the libstdc++ present on a user's machine, and an optimized build emits out-of-line calls into the newer runtime (for example std::string::_M_replace_cold, an unversioned reference). Linking libstdc++ dynamically left those resolving against the older system libstdc++, which does not define them, so the library failed to load with an undefined std::string symbol on the very host it targets. Static linking pulls that code in and leaves no libstdc++ dependency to be too old, the way ExecuTorch's own runner links, and the build guard requires the result so a dynamic link cannot slip back in. The wheel is tagged py3-none rather than per-interpreter, because the delegate is a plain shared object with no Python ABI and one build serves every CPython. test_api.py checks the shipped layout: the delegate resolves through the loader in the layout that ships, the wheel's RUNPATH is compared whole against the one the build asks for, the symbol versions and the C++ runtime dependency are compared against the runtime the delegate links, and the wheel's own metadata is checked. The reachability scans that assert the import and static-C++ checks run in CI parse each language's grammar rather than matching text, and none of them execute the workflow they inspect.
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
from
August 26, 2026 05:48
6eba5e9 to
339f92d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
This change makes the
torch-tensorrt-executorch-runtimewheel much smaller and safer.Today the wheel rebuilds all of ExecuTorch from source and ships its own renamed copy of the ExecuTorch runtime. After this change, the wheel ships only one thing: the TensorRT delegate, a single shared library. It reuses the ExecuTorch runtime from the
executorchwheel the user already has.The problem
The delegate needs the ExecuTorch runtime to run. The runtime is the C++ code that actually runs a model. The old wheel got that runtime by building its own copy and shipping it.
That causes a real bug. Two copies of the same runtime can end up in one process at the same time, and they can disagree. The old wheel worked around this by loading first and replacing the runtime that was already there. So the order of loading mattered. A wrong order broke things.
Building a whole second copy of ExecuTorch is also slow and large, and it only existed to solve a problem that should not exist in the first place.
The solution
The delegate now works the same way ExecuTorch's own delegates work. It links
libexecutorch.sofrom the installedexecutorchwheel, and it registers itself into that runtime when it loads. Nothing is replaced. There is only ever one runtime in the process. The whole class of load-order bugs goes away.Python loads the delegate with
ctypes, which is a plain shared-library load, not a Python import. Then it checks that the backend showed up in the runtime.What the wheel ships now
Before, the wheel shipped four native libraries. All four were rebuilt copies of ExecuTorch pieces:
_portable_lib.sodata_loader.solibextension_cuda.solibaoti_cuda_shims.soAfter, it ships one:
libexecutorch_backend_tensorrt.soThe file has the right name now
Before,
setup.pydeclared the delegate as a Python extension module. So it shipped with a Python-style name:_executorch_backend_tensorrt.cpython-312-x86_64-linux-gnu.so. That name says "Python module", but the file has no Python symbols. It is just a shared library thatctypesloads.It now ships under its real name, which matches ExecuTorch's own backends like
libexecutorch_backend_cuda.so:_executorch_backend_tensorrt.cpython-312-x86_64-linux-gnu.solibexecutorch_backend_tensorrt.soThe old declaration did one useful thing: it marked the wheel as platform-specific, not pure Python. That is now set directly, so the wheel is still tagged for the platform (
cp312-cp312-linux_x86_64).What we could delete
native/CMakeLists.txtdrops from 372 lines to 207. Most of the removed code was only there to rebuild ExecuTorch, for example:Two safety checks kept, done properly
One removed piece was a real safety check. It is now done in a cleaner way.
The old build linked the C++ standard library statically, because the build container and the run container might have different versions. The new delegate is a thin shim. It needs a lower set of C++ library symbols than
libexecutorch.soitself needs. So instead of static linking, a check after the build compares the symbol versions the delegate needs against the pinnedlibexecutorch.so. If the delegate ever needs a higher one, the build fails.The build also removed an absolute path that leaked into the wheel. When you link ExecuTorch, it adds the build machine's own path to the library search list. That path sorted ahead of the relative ones. So on the build machine the delegate could find
libexecutorch.sothrough a path that would not exist for a real user.patchelfnow rewrites the search list so it only has relative entries, and a check confirms no absolute entry is left.One new build check
On Linux, a shared library links even when some symbols are missing. So an under-linked delegate would build fine and only fail later, at load time, with a confusing "backend unavailable".
check_imports_executorch_runtime.shcatches this at build time. It confirms the delegate callsregister_backendfrom outside (not from a private copy it defined itself) and that it linkslibexecutorch.so.Type of change
This is flagged as possibly breaking for two reasons:
torch_tensorrt_executorch_runtime._portable_libor.data_loaderdirectly will no longer find them, because the wheel no longer ships them.activate()also no longer returns a module. The documented API (get_runtime(),BACKEND_NAME,torch_tensorrt.load(...)) does not change.executorchwheel at runtime, not just at build time, because it links a library that only CUDA builds ship. If a CPU-onlyexecutorchis installed, the user now gets a clear message fromactivate()instead of a confusing load error.Checklist
Testing
Verified on a Linux GPU host against the real pinned ExecuTorch CUDA wheel. The main test builds a small third-party delegate against only the wheel, installs it into a simulated
site-packageslayout, and loads it with nothing onLD_LIBRARY_PATH:libexecutorch.so, and it rejects a library that really needs a higher C++ symbol version. Symbol versions are compared as numbers, so3.4.21correctly ranks above3.4.9.readelfacross 31 artifact shapes, including one that checks the numeric version sort on its own, and gives it the same arguments the build uses.setup.pycontains exactlytorch_tensorrt_executorch_runtime/libexecutorch_backend_tensorrt.soand is tagged for the platform. The CI wheel-contents check runs against that real wheel and compares the exact filename, so a wrong name would fail it.setup.pyand__init__.pyeach hold the delegate filename, and a test asserts they match, because renaming only one would build a wheel CI accepts but the runtime cannot load.test_python_runtime.py: 25 cases across 15 functions, all pass with torch installed. black, isort, and strict mypy are clean.Notes for reviewers
ci: nightlylabel, notci: full. The ExecuTorch build and test jobs already run on the ordinary pull request lane. The nightly lane is needed only for theexecutorchpytest suite, which is where the delegate is actually run end to end.Known gaps
pip installline that cannot work.test_api.pymeta-tests, which assert the CI guard steps still exist, run on the nightly lane. The real properties are enforced on every pull request: the runtime build and test jobs, the wheel-content checker, the ELF guard, and the runtime check all run whenever the lane is notskip. So a deleted guard is caught by those jobs going red. The nightly lane only delays the meta-test that flags a guard being removed. These tests need no GPU, no ExecuTorch, and no built wheel.