Skip to content

Export VarBin to Arrow without canonicalizing the encoding first - #9370

Open
robert3005 wants to merge 1 commit into
rk/string-arrow-benchesfrom
rk/arrowvarbinnoexecute
Open

Export VarBin to Arrow without canonicalizing the encoding first#9370
robert3005 wants to merge 1 commit into
rk/string-arrow-benchesfrom
rk/arrowvarbinnoexecute

Conversation

@robert3005

@robert3005 robert3005 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

to_arrow_byte_array used a private ArrowByteExportable matcher listing VarBin, Chunked and Constant. That list is closed, so an encoding such as FSST, OnPair or Zstd was executed all the way to a canonical VarBinView, and the builder then had to re-lay out every value the encoding could have written directly.

It now stops at dyn VarBinExportable, so operators like Filter and Slice still run (and push down) while the encoding underneath them keeps its specialized append_to_builder, and out-of-crate encodings participate by claiming the capability.

Alongside the dynamic export target we fix the validation of null values (required by arrow) and redundant casting of offsets

@robert3005
robert3005 force-pushed the rk/arrowvarbinnoexecute branch from 875083a to 82f37ac Compare August 11, 2026 23:23
@robert3005 robert3005 changed the title Teach ArrayVTable about capabilities and let arrays provide extensible capabilities Export VarBin to Arrow without canonicalizing the encoding first Aug 11, 2026
@robert3005
robert3005 changed the base branch from develop to rk/string-arrow-benches August 11, 2026 23:24
@codspeed-hq

codspeed-hq Bot commented Aug 11, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 15.45%

⚡ 2 improved benchmarks
✅ 1960 untouched benchmarks
🆕 4 new benchmarks
⏩ 89 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation cold_misaligned[(64, 256)] 5.2 ms 4.4 ms +19.31%
Simulation slice_primitive_tight_loop[10000] 473.8 µs 424.1 µs +11.71%
🆕 Simulation file_to_offset_array[binary_array] N/A 317.6 ms N/A
🆕 Simulation file_to_offset_array[string_array] N/A 317.6 ms N/A
🆕 Simulation file_to_offset_batches[binary_array] N/A 195.6 ms N/A
🆕 Simulation file_to_offset_batches[string_array] N/A 196.1 ms N/A

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/arrowvarbinnoexecute (a39053c) with develop (ca7f626)2

Open in CodSpeed

Footnotes

  1. 89 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on rk/string-arrow-benches (4fce0bc) during the generation of this report, so develop (ca7f626) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

`to_arrow_byte_array` used a private `ArrowByteExportable` matcher listing
`VarBin`, `Chunked` and `Constant`. That list is closed, so an encoding such as
FSST or Zstd was executed all the way to a canonical `VarBinView` and the
builder then had to re-lay out every value the encoding could have written
directly.

Stop at `dyn VarBinExportable` instead, so operators like Filter and Slice still
run and push down while the encoding underneath keeps its specialized
`append_to_builder`.

Also reinterpret unsigned `VarBin` offsets instead of casting them, checking
only the last offset, since offsets are non-decreasing. UTF-8 validation now
covers null slots too: Arrow's `value(i)` reinterprets a null slot's bytes just
like any other slot's, so bytes spanned by the offsets have to be valid even
where the slot is null.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the rk/arrowvarbinnoexecute branch from 82f37ac to a39053c Compare August 11, 2026 23:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant