Skip to content

perf(arrow/array): append dictionary indices directly - #1313

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-append-indices
Open

perf(arrow/array): append dictionary indices directly#1313
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-append-indices

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this change?

  • DictionaryBuilder.AppendIndices used to allocate a new typed slice on every call.
  • Convert the input values directly into the final typed index buffer.
  • Keep the existing bulk validity bitmap path.
  • Add coverage for all eight index types, nullable appends, split appends, and invalid validity lengths.
  • No public API changes.

Benchmark

Command:

go test ./arrow/array -run '^$' -bench '^BenchmarkDictionaryBuilderAppendIndices$' -benchmem -benchtime=300ms -count=8

Apple M1 Pro, Go 1.26.3. The benchmark uses 65,536 indices and a 256-entry dictionary.

Case Before ns/op After ns/op Before B/op After B/op Allocs
int32, non-null 110,370 75,398 815,908 553,762 18 -> 17
int32, nullable 116,578 75,895 815,907 553,762 18 -> 17

This is about 32% less time and allocation bytes for the non-null case, and about 35% less time for the nullable case.

The Parquet decoder's separate []uint64 -> []int scratch conversion is intentionally outside this PR.

Tests

  • go test ./...
  • go test -race ./arrow/array
  • go vet ./arrow/array

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant