Myelin NVRTC compilation failure on Jetson Thor (SM110) building Fast-FoundationStereo - reproduces on TRT 10.14.1 and 11.2.1.2

[juliusekela]

Title: Myelin NVRTC compilation failure on Jetson Thor (SM110) building Fast-FoundationStereo — reproduces on TRT 10.14.1 and 11.2.1.2

Environment

• Jetson Thor (SM110), JetPack 7 / L4T, unified memory 122 GB
• TensorRT 10.14.1 (JetPack tarball trtexec) and 11.2.1.2 (pip tensorrt cu13 aarch64, python API)
• Model: Fast-FoundationStereo, exported from the NGC checkpoint (nvidia/tao/fast-foundationstereo v1.2, model_best_bp2_serialize.pth) with the official scripts/make_single_onnx.py (NVlabs/Fast-FoundationStereo, opset 17), --height 288 --width 480 --valid_iters 4, exporter run in nvcr.io/nvidia/pytorch:25.06-py3-igpu

Failure (identical on both TRT versions, fp32 — no precision flags involved)

TRT 10.14.1:

Error[9]: Skipping tactic 0x0 due to exception [myelin_graph.h:1182] MyelinCheckException:
nvrtc_compile.cpp:1118: CHECK(success) failed. NVRTC Compilation failure
Error[10]: IBuilder::buildSerializedNetworkToStream: Error Code 10: Internal Error
(Could not find any implementation for node {ForeignNode[/Concat.../Unsqueeze_166]}.)

TRT 11.2.1.2:

MyelinCheckException: nvrtc_compile.cpp:1122: CHECK(success) failed. NVRTC Compilation failure
Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/Concat.../Cast_118]}.)

Additional data points

• An older single-ONNX export of the same model (576x960, iters_8, exported ~May 2026 with an older torch) builds and runs correctly in fp32 on the same machine/TRT 10.14.1 (1199 ms) — the failure is specific to graph patterns emitted by the newer exporter/torch.
• fp16 builds of the older export segfault the TRT compiler on SM110 (10.13.3 and 10.14.1; July 2026 finding).
• Full FoundationStereo-S (NGC dynamic ONNX) at 288x480 builds fp32 on 10.14.1 (1052 ms) — so small dims per se are fine.
• Looks related to TRT 10.15.1 Myelin failure on RTX PRO 6000 (sm_120) with Fast-FoundationStereo post_runner #4715 (sm_120: Myelin "no valid tactics" on Fast-FS post_runner at all precisions, fused 3D-ConvTranspose/Cast pattern).

Questions

1. Is there a fix or workaround for the SM110 Myelin/NVRTC codegen failure (builder flags, tactic source restrictions, graph rewrites)?
2. Is the repo's plugin path (make_plugin_onnx.py + cpp plugin) the intended deployment route on Thor today?
3. Is fp16 expected to work for this model on SM110, given the exporter's own printed recipe suggests trtexec --fp16?

Full build logs and both ONNX files available on request.

[juliusekela]

Additional data point: graph-side workarounds do not help. We tried (a) onnx-simplifier on the export, (b) marking intermediate tensors inside the failing region as network outputs to split the Myelin fusion — one barrier at a time at /Concat, /Cast_11, /Unsqueeze_16, /ReduceSum boundaries, and finally all 210 candidate tensors at once. Every variant fails with the same MyelinCheckException (NVRTC compilation failure), with the ForeignNode span shifting but never dissolving. This suggests the failing NVRTC kernel is generated for an individual op's lowering on SM110 rather than a cross-op fusion. Happy to bisect further with a debug build or share the ONNX.