Skip to content

perf(parquet): decode nullable plain values into valid runs - #1314

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-direct-nullable-plain
Open

fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-direct-nullable-plain

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this change?

  • PlainDecoder.DecodeSpaced used to decode all non-null values into a dense prefix first, then move them into their bitmap positions.
  • For simple validity patterns, decode each valid run directly into its final output range.
  • Keep the existing dense plus spaced expansion path for highly fragmented validity.
  • Use the reverse bitmap reader so trailing-null pages stop after finding the dense prefix.
  • Add edge-case tests and a focused benchmark.

Benchmark

Apple M1 Pro, Go 1.26.3, 65,536 int32 values, 8 runs, 300ms per sample:

Validity pattern Before After
leading null 11.6 us/op 7.1 us/op
clustered nulls 8.6 us/op 7.2 us/op
trailing null 5.6 us/op 5.6 us/op

Random 10% nulls and alternating validity keep the existing fallback path. All cases stayed at 0 B/op and 0 allocs/op.

Tests

  • PARQUET_TEST_DATA=parquet-testing/data ARROW_TEST_DATA=arrow-testing/data go test -p 2 ./... -count=1
  • go test -race -p 2 ./parquet/internal/encoding ./parquet/file
  • go vet -p 2 ./parquet/internal/encoding ./parquet/file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant