Qualcomm: HTP hangs in fp16 on a dilated conv when the dilated extent exceeds the input and cin is large

[msluszniak]

Repro

torch.nn.Conv2d(960, 256, kernel_size=3, padding=36, dilation=36, bias=False)  # input (1, 960, 65, 65)

Lowered with generate_htp_compiler_spec(use_fp16=True) for SM8850. Lowering succeeds; execution never returns and a watchdog kills it after a constant ~11.47 s:

[ERROR] [Qnn ExecuTorch]: QnnDsp <E> Graph  failed in execution with err 1007
[ERROR] [Qnn ExecuTorch]: qnn_graph_execute failed. Error 1007        (1011 in some builds)

ExecuTorch 1.6.0, QAIRT 2.47.0.260601, Galaxy S26 Ultra (SM8850, Hexagon v81), qnn_executor_runner from the same tree.

Bracketing

Both conditions are required: the dilated extent (1 + 36*2 = 73) must exceed the 65px input, and the input-channel count must be large.

fp16 conv at 65x65	result
960->256, d36, one op	hang
960->256, d36, as 2x(480->256) summed	6.83 ms
960->256, d36, as 4x(240->256) summed	13.38 ms
512->256, d36	3.89 ms
960->256, d24	15.06 ms
960->256, 1x1	4.81 ms
same model in a16w8	3.25 ms

The 2-way split is the same math (conv is linear in cin), which points at input-channel volume at that dilation rather than at the op itself.

Does not help

use_multi_contexts, use_slc_allocator, use_dlbc, kHtpSustainedHighPerformance. Input size is irrelevant: 520x520 and 256x256 hang identically.

Impact

DeepLabV3's ASPP uses dilations 12/24/36, so every DeepLabV3 is unusable in fp16 on HTP; the backbone alone runs at 3.77 ms and the full head hangs. a16w8 compiles the same convolution and runs fine, so it looks fp16-specific.

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin @abhinaykukkadapu @psiddh

[mergennachin]

cc @psiddh

[qti-horodnic]

@msluszniak

Thank you for the detailed report, this is a valid issue which I was also able to reproduce locally with qairt 2.49.
The root cause seems to be on the HTP side: graph prepare cannot tile this convolution into VTCM. Compiling the identical shape in a16w8 makes it explicit, getting error: q::DilatedConvLayer_s1.opt not sufficiently tiled to fit in TCM. Requires 8638464 bytes against 8388608 available on SM8850. The quantized path rejects this at finalize; fp16 skips the check and emits a context binary that cannot execute, hence a dead DSP instead of a compile error. I also confirmed it's not an ET problem: running the extracted context binary through qnn-net-run reproduces it without ET.

I've cut a ticket to the backend team to fix the root cause. I'll add a diagnostic on the ET side: the partitioner will log a warning naming the convolution when its weights plus the receptive field of one output pixel exceed the SoC's VTCM, so the failure is attributable from the export log instead of being silent. It does not prevent the crash and the estimate is approximate, so it can also fire on convolutions that do run. The workaround is splitting along cin and summing; I will detail that in the README. Btw, 2 refinements to your bracketing: the channel threshold is 512 < cin ≤ 768 at d36, and d32 already fails (dilated extent 65, i.e. equal to the input, not exceeding it).