Skip to content

perf(parquet): use typed memo insertion for fixed-width dictionaries - #1312

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-typed-memo-flba-int96
Sep 15, 2026
Merged

zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-typed-memo-flba-int96

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • Use BinaryMemoTable.InsertOrGet for FLBA dictionary Put and PutDictionary.
  • Use the same typed insertion path for DictInt96Encoder.Put.
  • Keep empty Put calls as no-ops.
  • Add Int96 dictionary round-trip coverage and FLBA/Int96 benchmarks.

Why

These encoders already use the binary memo table, but still called GetOrInsert through the generic interface for every value. The typed entry point avoids the extra interface conversion. Dictionary order and encoded output stay unchanged.

Benchmark

Apple M1 Pro, macOS arm64, Go 1.26.3. 65,535 values and 100 unique values, median of five runs.

Case Before After Change
FLBA, width 16 4.59 ms 3.09 ms 33% faster
Int96 5.41 ms 3.87 ms 29% faster
FLBA allocations 131,117 65,581 50% fewer
Int96 allocations 196,655 131,118 33% fewer

Allocated bytes also fell by about 42% for FLBA and 33% for Int96.

Tests

  • go test ./... with Arrow and Parquet test data
  • go test ./parquet/internal/encoding/...
  • go test -race ./parquet/internal/encoding/...
  • go vet ./parquet/internal/encoding/...
  • go test -tags noasm ./parquet/internal/encoding/...

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typed memo insertion preserves fixed-width value ownership, sliced offsets, and Int96 dictionary encoding. Focused dictionary encoding tests passed (go test -p 2 -count=1 ./parquet/internal/encoding -run Dict).

@zeroshade
zeroshade merged commit eda27cd into apache:main Sep 15, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants