Skip to content

perf(compute): use typed memo insertion for binary hashes - #1315

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/compute-dict-binary-memo
Open

fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/compute-dict-binary-memo

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

What does this change?

  • doAppendBinary and doAppendFixedSize still called MemoTable.GetOrInsert for every value.
  • The memo is already a TypedMemoTable[[]byte].
  • Use InsertOrGet directly, like the numeric hash paths.
  • Null handling is unchanged.

Why

This removes the []byte to interface{} call-site conversion from the compute dictionary and unique kernels. It also makes the typed binary memo path reachable from these hot loops.

Benchmark

Command:

go test -p 1 -vet=off ./arrow/compute -run '^$' -bench '^BenchmarkDictionaryEncodeBinary$' -benchmem -benchtime=500ms -count=7

Apple M1 Pro, macOS arm64, Go 1.26.3. The numbers below are medians of 7 runs. Baseline uses GetOrInsert at both call sites. After uses TypedMemoTable[[]byte].InsertOrGet.

Case Before After Change
string, 65,536 values 4,837,326 ns/op 3,151,276 ns/op 34.9% faster
fixed-size binary 16, 65,536 values 4,445,131 ns/op 3,134,294 ns/op 29.5% faster
string B/op 3,444,557 1,871,270 45.7% fewer
fixed-size binary B/op 3,444,128 1,870,749 45.7% fewer
string allocs/op 131,150 65,613 50.0% fewer
fixed-size binary allocs/op 131,146 65,609 50.0% fewer

Coverage

  • Existing string and binary dictionary tests cover the variable-width path.
  • Added a fixed-size-binary dictionary test.
  • Added string and 16-byte fixed-size-binary benchmarks.
  • go test -p 1 ./arrow/compute -count=1

No public API changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant