Repository navigation
Mdr lossless fixes - #261
Merged
Merged
Conversation
…lengths CompressPrimary's target_cr check summed freq_subarray[i] * CL_subarray[i] after GetCodebook, but GetCodebook leaves freq_subarray sorted ascending (SortByKey writes in place) and GenerateCW reverses CL_subarray, so the sum paired unrelated symbols. For skewed inputs the most frequent symbol got the longest code length: on MDR-X bitplane groups with ~87% zero bytes the estimate was 0.73-0.89x while byte Huffman actually reaches 3.7-4.5x, so HybridLevelCompressor stored those groups raw. Estimate from the unsorted histogram (_d_freq_copy_subarray) and the symbol-indexed codebook, whose top byte is the codeword length. Only callers passing target_cr > 1 (MDR-X level compressors) are affected; the mgard-x pipeline calls Huffman with target_cr = 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With Config::lossless == Huffman_Zstd, HybridLevelCompressor compresses groups above size_threshold with ZSTD (the existing, previously unused zstd member) instead of RLE/byte Huffman, and keeps the result whenever it is smaller than the raw group. ZSTD groups carry a 7-byte "MGXZSTD" signature and are detected on decompression. The default (Huffman) path is unchanged. mdr-x gains the refactor option -l/--lossless <huffman|huffman-zstd>. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GenerateRequest rounds each level's bitplane count up to a whole group of num_merged_bitplanes (4) with n = ((n - 1) / m + 1) * m, which maps n = 0 to 4: a level the size interpreter decided to skip still fetched a full group, capping the retrieval ratio at loose tolerances (CESM PSL, L-inf at 0.5 * range: CR 12.6 instead of 249). Round with (n + m - 1) / m * m, which keeps 0 at 0. With zero-bitplane levels allowed, CurrFinalLevel() can be below the finest level, and LoadMetadata only refreshed level_num_bitplanes up to it while ProgressiveReconstruct decodes all levels, so a reconstructor reused for another dataset/subdomain decoded stale increments (bound exceeded 30x in a reuse test). LoadMetadata now refreshes every level. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.