feat: enable QASYMM8_SIGNED to F32 assembly dequantization by morgolock · Pull Request #1302 · ARM-software/ComputeLibrary

morgolock · 2026-07-01T11:38:39Z

Enable QASYMM8_SIGNED input and weights to use the F32 DequantizeFloat assembly output stage, including the direct convolution selection path.

Propagate input and weight zero-points plus the unrounded mathematical K depth so asymmetric offset correction uses the real GEMM/convolution depth rather than arm_gemm's padded internal K.

Add NEON validation coverage for the direct I8S8F32 convolution path.

Performance was checked on a A76, pinned to CPU 4 with one thread. The change is neutral on large workloads and improves the small NHWC signed int8 to F32 convolution path: QASYMM8_SIGNED/RunSmallDequantizeF32 NHWC/no-activation cases show a 1.36x geomean speedup over github/main, with the tiniest single-batch cases improving by roughly 1.45x to 3.13x.

Change-Id: Ie723d3da629d48de6de737c425bf7ad48e0f7feb

Enable QASYMM8_SIGNED input and weights to use the F32 DequantizeFloat assembly output stage, including the direct convolution selection path. Propagate input and weight zero-points plus the unrounded mathematical K depth so asymmetric offset correction uses the real GEMM/convolution depth rather than arm_gemm's padded internal K. Fix the interleaved no-merge DequantizeFloat scheduler stride so kernels that do not pack row-sum slots advance between A panels correctly. Guard the symmetric no-merge dequant support helper with the SME/SME2 feature macros. The helper is only referenced when those no-merge dequantized kernels are compiled, so non-SME builds must not define it unconditionally under Werror. Add NEON validation coverage for the direct I8S8F32 convolution path. Performance was checked on a Cortex-A76, pinned to CPU 4 with one thread. The change is neutral on large workloads and improves the small NHWC signed int8 to F32 convolution path: QASYMM8_SIGNED/RunSmallDequantizeF32 NHWC/no-activation cases show a 1.36x geomean speedup over github/main, with the tiniest single-batch cases improving by roughly 1.45x to 3.13x. Validate GEMM3D convolution probes with the real weights and output tensor metadata instead of reusing the input data type for every dummy tensor. This prevents QASYMM8 input with QSYMM8_PER_CHANNEL weights from selecting a 3D path that the real lowp operation cannot safely configure. When QASYMM8 output is handled through the signed lowp assembly path, route fused assembly through the signed intermediate output, keep the adjusted output stage in sync for configure/validate/runtime quantization updates, and convert back to QASYMM8 after the fused assembly path as well as the fallback path. This fixes quantized per-channel GEMM convolution and deconvolution validation crashes introduced by admitting the int8 per-channel RHS assembly path. Signed-off-by: Pablo Marquez Tello <pablo.tello@arm.com> Change-Id: I945c24e5cd3d21d857b68de90d27aa18a31f7547

Passavee-Losripat · 2026-07-14T16:26:47Z

Hi, I've been testing this PR for OpenVINO's ARM CPU plugin int8 enablement and found a possible validation gap in the new QASYMM8_SIGNED->F32 dequantization path.

QSYMM8_PER_CHANNEL weights are correctly rejected by validate() on this path. However, QASYMM8_SIGNED weights whose QuantizationInfo carries a per-channel scale vector are accepted and the kernel then dequantizes every output channel with scale[0] (the scalar uniform().scale product in create_arm_gemm_dequant).

I create minimal repo to test this: all-ones i8 input, all-ones 3×3×3 i8 weights, bias 0, distinct per-channel scales s[c] -> each interior output should be 27·s[c]:

channel    got      expected(27*s[c])    ratio
c00      0.0270        0.0270            1.0000
c01      0.0270        0.0540            0.5000
c02      0.0270        0.1080            0.2500
...
c11      0.0270       55.2960            0.0005

validate() returns OK, but every channel is computed with s[0]. Since quantization frameworks (NNCF, PyTorch, TF) produce per-channel weight scales by default, this input shape is what integrations like OpenVINO naturally pass, so it fails silently on real models rather than erroring out.

I am thinking of rejecting scale().size() > 1 on this path the same way QSYMM8_PER_CHANNEL is rejected, unless per-channel support is planned. I can share the full standalone test program if useful.

gunes-arm · 2026-07-21T13:52:34Z

                    "We could not find an optimized kernel for U8/QASYMM8 input and U32 output");
            }
-            else if (b->data_type() == DataType::QASYMM8_SIGNED)
+            else if (b->data_type() == DataType::S8 || b->data_type() == DataType::QASYMM8_SIGNED)


Why do we add S8 data type to here?

gunes-arm · 2026-07-21T13:59:32Z

     * |:--------------|:--------------|:--------------|:--------------|
     * |QASYMM8        |QASYMM8        |S32            |QASYMM8        |
     * |QASYMM8_SIGNED |QASYMM8_SIGNED |S32            |QASYMM8_SIGNED |
+     * |QASYMM8_SIGNED |QASYMM8_SIGNED |F32            |F32            |


How about Fp16? And, Do we support mixed sign inputs? Such as UInt8 inputs and Int8 weights. That's what I felt from the changes in the glue layer. I also don't see the tests for mixed sign inputs.

gunes-arm · 2026-07-21T14:04:15Z

+                        {TensorInfo(TensorShape(8U, 6U, 6U), 1, DataType::F32, DataLayout::NHWC),
+                         TensorInfo(TensorShape(16U, 4U, 4U), 1, DataType::F32, DataLayout::NHWC),
+                         TensorInfo(TensorShape(8U, 6U, 6U), 1, DataType::F32, DataLayout::NHWC),
+                         TensorInfo(TensorShape(8U, 6U, 6U), 1, DataType::F32, DataLayout::NHWC)}),


Shouldn't we make some F16 & adjust the expected in the body with cpu_supports_dtypes?

gunes-arm · 2026-07-21T14:04:38Z

+    ARM_COMPUTE_EXPECT(bool(s) == expected, framework::LogLevel::ERRORS);
+}
+
+/** Verify GEMM_CONV2D is returned automatically for NHWC QASYMM8_SIGNED to F32 dequantized convolution. */


gunes-arm · 2026-07-21T14:07:29Z

+    const QuantizationInfo qi(0.25f, 0);
+    TensorInfo             src_info(TensorShape(16U, 8U, 8U), 1, DataType::QASYMM8_SIGNED, DataLayout::NHWC);
+    TensorInfo             wgt_info(TensorShape(16U, 3U, 3U, 8U), 1, DataType::QASYMM8_SIGNED, DataLayout::NHWC);
+    TensorInfo             dst_info(TensorShape(8U, 6U, 6U), 1, DataType::F32, DataLayout::NHWC);


I'd say, loop over F32 and F16 and back with cpu_supports_dtypes function.

gunes-arm · 2026-07-21T14:12:18Z

+                       NEDequantizeFloatConvolutionFixture,
+                       framework::DatasetMode::ALL,
+                       combine(zip(make("InputShape", {TensorShape(8U, 8U, 16U), TensorShape(16U, 16U, 32U)}),
+                                   make("WeightsShape", {TensorShape(3U, 3U, 16U, 8U), TensorShape(3U, 3U, 32U, 16U)}),


Could we add another conv size other than 3x3? As far as I can see we only test 3x3 and 1x1.

Or better, why don't we directly use something like SmallConvolutionLayerDataset?

By using a pre-defined dataset like SmallConvolutionLayerDataset, I'm talking about removing all the different combs below, like padded, strided etc. Not just for this particular suite of tests. I think it'll reduce the test code we maintain. Please let me know if we're missing any coverage by not using those pre-defined datasets.

Also, I'd like to repeat my comments about Fp16 output here.

morgolock force-pushed the pr/asm_dequant_f32 branch 5 times, most recently from 5aa921d to 002c426 Compare July 2, 2026 09:50

gunes-arm requested changes Jul 2, 2026

View reviewed changes

This was referenced Jul 8, 2026

feat: optimized QASYMM8_SIGNED->F32 direct convolution path #1298

Closed

feat: enable QASYMM8_SIGNED→F32 in CpuGemmAssemblyDispatch #1297

Closed

morgolock force-pushed the pr/asm_dequant_f32 branch from 002c426 to 8381408 Compare July 8, 2026 13:28

morgolock requested a review from gunes-arm July 8, 2026 13:33

gunes-arm requested changes Jul 21, 2026

View reviewed changes

alvoron mentioned this pull request Jul 21, 2026

[CPU][ARM] Int8 conv swish fq fusion openvinotoolkit/openvino#34931

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: enable QASYMM8_SIGNED to F32 assembly dequantization#1302

feat: enable QASYMM8_SIGNED to F32 assembly dequantization#1302
morgolock wants to merge 1 commit into
mainfrom
pr/asm_dequant_f32

morgolock commented Jul 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Passavee-Losripat commented Jul 14, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

gunes-arm Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

morgolock commented Jul 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Passavee-Losripat commented Jul 14, 2026

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants