Skip to content

[OpenVINO] Add Support for GroupedMM in AWQ and Scale Estimation for OpenVINO Backend - #4176

Open
anzr299 wants to merge 15 commits into
openvinotoolkit:developfrom
anzr299:an/ov/grouped_mm_support
Open

[OpenVINO] Add Support for GroupedMM in AWQ and Scale Estimation for OpenVINO Backend#4176
anzr299 wants to merge 15 commits into
openvinotoolkit:developfrom
anzr299:an/ov/grouped_mm_support

Conversation

@anzr299

@anzr299 anzr299 commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Changes

GroupedMatMul applies one weight matrix per group to a single shared 2D activation. Data-aware compression assumed a 3D weight always comes with a matching 3D activation, so AWQ, scale estimation and GPTQ failed.

AWQ

For weights, The experts axis is folded into the output channels, the search then minimizes the error over all experts at once, and the scale is broadcast back over the experts axis.

For activation, the same scale is applied over the hidden dim of the activation in non mergible case, For mergible scale, it is reshaped but still only applied over the hidden dim of the previous matmul weights.

Scale estimation

Broadcasts the pooled statistics over the group axis, so the scale stays per group.

Reason for changes

to support grouped_mm based MoE models in OpenVINO backend.

Related tickets

191895

Tests

test_scale_estimation and test_awq_scale_reference extended for this case.
test_node_utils extended for the weight channel axes and activation channel axis for GroupedMatMul.

WWB Eval

Model: Qwen3.6 35B A3B
mode: INT4_ASYM; GS: 64;

AWQ=True

MoE op WWB Accuracy
BatchedMM 0.956345
GroupedMM 0.952167

Scale Estimation=True

MoE op WWB Accuracy
BatchedMM 0.9534
GroupedMM 0.9587

Test examples - success

Weight compression - success

@anzr299
anzr299 requested a review from a team as a code owner August 21, 2026 14:03
@github-actions github-actions Bot added documentation Improvements or additions to documentation NNCF PT Pull requests that updates NNCF PyTorch NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF ONNX Pull requests that updates NNCF ONNX labels Aug 21, 2026

@staticmethod
def get_moe_scale_estimation_ref(check_sampling_activation_stats_flow):
def get_moe_scale_estimation_ref(check_sampling_activation_stats_flow, grouped_mm=False):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to move reference to json file ? We have examples in tests.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good idea. I will move it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have combined all the refs for SE in 1 function (earlier MoE refs had another getter). Then moved it to a reference JSON file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Code Freeze documentation Improvements or additions to documentation NNCF ONNX Pull requests that updates NNCF ONNX NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PT Pull requests that updates NNCF PyTorch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants