Skip to content

fix(encoding): read wrapped miniblock level counts - #8410

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-6936-1
Open

fix(encoding): read wrapped miniblock level counts#8410
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-6936-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • let block decompressors report a payload-derived value count when the encoding can determine it
  • recover v6.0.1 RLE structural levels only when the decoded/header difference is an exact u16 wrap, while preserving strict corruption checks
  • add a version-pinned v6.0.1 generator, historical stable-format fixture, and unit/end-to-end regressions

Root cause

Pylance 6.0.1 cast a miniblock's structural level count to u16. For list pages with a dense prefix and many trailing empty lists, the header wrapped while the RLE payload retained every level. Readers trusted the truncated count and either short-read the list column or rejected the complete payload as an RLE overflow. Current writers already prevent these oversized miniblock chunks; this change restores stable-format read compatibility for files already written.

Validation

  • cargo test -p lance-encoding miniblock_levels --lib
  • cargo test -p lance-encoding block_rle_infers_value_count_from_run_lengths --lib
  • cargo test -p lance test_v6_0_1_miniblock_level_count_overflow --lib
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings
  • make build from python/
  • uv run make lint from python/
  • local Python scan and merge-insert against the historical fixture

Fixes #6936

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG:Incorrect array length for StructArray field "captions", expected 8192 got 5364

0 participants