Skip to content

Fix CPU Huffman truncation of wide quantization outliers - #260

Merged
JieyangChen7 merged 1 commit into
CODARcode:masterfrom
VarShankar:fix-cpu-huffman-int64-outliers
Sep 30, 2026
Merged

JieyangChen7 merged 1 commit into
CODARcode:masterfrom
VarShankar:fix-cpu-huffman-int64-outliers

Conversation

@VarShankar

Copy link
Copy Markdown
Contributor

Summary

The MGARD-CPU Huffman path accepts quantized coefficients as long int, but its outlier (miss) stream narrows every value to int. On LP64 systems, valid quantization indices outside the signed 32-bit range are therefore silently truncated, and decompression can violate the requested error tolerance by orders of magnitude.

This change:

  • keeps legacy int-width outlier streams when all misses fit in int;
  • stores the complete miss stream as long int when any miss requires the wider type;
  • decodes both legacy and wide streams using the recorded miss-byte count; and
  • rejects inconsistent miss-stream sizes.

Ordinary existing streams retain their previous representation. A new regression test covers both the legacy-width path and values outside the int range.

Validation

  • The focused reproducer fails on unpatched master and passes with this change.
  • The new Catch2 regression test passes.
  • All 310 configured MGARD-CPU and serial MGARD-X tests pass.

@JieyangChen7

Copy link
Copy Markdown
Collaborator

@VarShankar Thanks a lot for your contribution to fixing this issue for MGARD-CPU. We are going to merge your PR.

@JieyangChen7
JieyangChen7 merged commit b9b3f8d into CODARcode:master Sep 30, 2026
6 checks passed
@VarShankar

Copy link
Copy Markdown
Contributor Author

Of course! I ran into this in a problem I was working on, so I figured I should do a PR. Thanks for maintaining MGARD!

@VarShankar
VarShankar deleted the fix-cpu-huffman-int64-outliers branch October 6, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants