[Experimental][WIP] Pfor delta encoding - #51150

Draft

prtkgaur wants to merge 93 commits into

apache:mainfrom

prtkgaur:pfor-delta-encoding

Commits on Oct 2, 2026

Add PFOR core algorithm and tests
sfc-gh-pgaur
committed
7cc6bf5
Integrate PFOR encoding into parquet encoder/decoder
sfc-gh-pgaur
committed
378b3ee
Add PFOR encoding benchmark
sfc-gh-pgaur
committed
6a1d6fb
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur
committed
003818c
Take buffer parameters as spans in the PFOR wire routines
sfc-gh-pgaur
committed
f2459cd
Return Result<T>/Status on decode paths instead of ARROW_DCHECK
sfc-gh-pgaur
committed
df91383
Assert little-endian and replace reinterpret_cast with SafeCopy
sfc-gh-pgaur
committed
5950438
Unpack a PFOR vector in one call instead of a batch loop
sfc-gh-pgaur
committed
7dfaa76
Add pragma GCC unroll/ivdep to decode loops for better vectorization
sfc-gh-pgaur
committed
ad3dc1e
Add PforEncodedVectorView for zero-copy decode path
sfc-gh-pgaur
committed
4699aab
Let the encode path choose its vector size
sfc-gh-pgaur
committed
8976c6e
Fix pfor_test.cc: unwrap Result<> from PforVectorInfo::Load()
sfc-gh-pgaur
committed
ab7852e
Convert PforWrapper::LoadHeader to return Result<PforHeader>
sfc-gh-pgaur
committed
d779ef4
Use SafeLoadAs/SafeStore for header and offset array in PforWrapper
sfc-gh-pgaur
committed
64fe429
Validate header fields in PforWrapper::LoadHeader
sfc-gh-pgaur
committed
5bc0368
Use int64_t and a qualified name at the PFOR call sites
sfc-gh-pgaur
committed
470be2c
Use uint8_t* consistently in PforWrapper API instead of char*
sfc-gh-pgaur
committed
045e4c5
Make PforVectorInfo a class with validated loads
sfc-gh-pgaur
committed
d5dbef5
Convert PforEncodedVector and PforEncodedVectorView to classes
sfc-gh-pgaur
committed
c42de3c
Validate num_exceptions in Load and construct PforVectorInfo directly
sfc-gh-pgaur
committed
116d020
Build the PFOR comparison benchmark
sfc-gh-pgaur
committed
ed2c22b
Use bit_util helpers in PFOR and note the incremental paths
sfc-gh-pgaur
committed
218c8f1
Drop Snowflake attribution from PFOR header comments
sfc-gh-pgaur
committed
f5fdcdf
Fix IsPowerOf2(int32_t) ambiguity in pfor_wrapper.cc
sfc-gh-pgaur
committed
0864578
Add FastLanes auto-vectorized bit-packing library
sfc-gh-pgaur
committed
269188c
Add FastLanes-FOR to parquet-pfor-comparison-benchmark
sfc-gh-pgaur
committed
f08c620
Add a flat-output decode for FastLanes-FOR and benchmark it
sfc-gh-pgaur
committed
74050c4
FastLanes: add toTransposed32; correct fromTransposed32 docstring
sfc-gh-pgaur
committed
1990356
Give PFOR a per-vector packing mode
sfc-gh-pgaur
committed
e611c9c
Add a transposed output order to PFOR decode
sfc-gh-pgaur
committed
6efe862
Add ARROW_RELEASE_O3 to keep CMake's default Release -O3
sfc-gh-pgaur
committed
581380a
Cover bit_width 0 and 32 in the transposed-output tests
sfc-gh-pgaur
committed
4d45609
Adapt PFOR to updated Arrow util APIs
sfc-gh-pgaur
committed
27df502
Migrate PFOR to std::span
sfc-gh-pgaur
committed
53be58b
Add a packing mode that interleaves without FL_ORDER
sfc-gh-pgaur
committed
82faad3
Decode BitPack into stack scratch instead of a heap buffer
sfc-gh-pgaur
committed
9d34600
Vectorize the frame-of-reference add in BitPack decode
sfc-gh-pgaur
committed
542133d
Speed up PFOR encode with a wider histogram and stack deltas
sfc-gh-pgaur
committed
08b33a2
Skip FOR-add pass in PFOR decode when frame-of-reference is 0
sfc-gh-pgaur
committed
7814e2d
Add TPC-H, more TPC-DS, and NYC-taxi numeric columns to PFOR benchmark
sfc-gh-pgaur
committed
aea8f29
Remove FastLanes packing modes from PFOR (BitPack only)
sfc-gh-pgaur
committed
2bfb25a
Add int64 (BIGINT) columns to the PFOR comparison benchmark
sfc-gh-pgaur
committed
35f20fe
Note the frame-of-reference bias fold as a TODO on the decode path
sfc-gh-pgaur
committed
2a5f4cd
Add a bias parameter to arrow::internal::unpack
sfc-gh-pgaur
committed
1e24ce6
Keep unpack_full at memcpy speed when it carries a bias
sfc-gh-pgaur
committed
8627855
Reflow an unpack call site to clang-format 18
sfc-gh-pgaur
committed
f1a89f0
Store the PFOR bit width in 7 bits, not 6
sfc-gh-pgaur
committed
f0c88cd
Fold the frame-of-reference add into the PFOR unpacker
sfc-gh-pgaur
committed
2d67bf8
Spell the restrict qualifier portably in unpack_full
sfc-gh-pgaur
committed
6b169b6
Fix two errors PFOR raises at the checkin warning level
sfc-gh-pgaur
committed
f04b208
List PFOR in SupportedEncodings for INT32 and INT64
sfc-gh-pgaur
committed
394e47d
Use std::bit_width instead of __builtin_clz in PFOR
sfc-gh-pgaur
committed
b7924c3
Apply clang-format 18 to the PFOR sources
sfc-gh-pgaur
committed
1b096df
Reject a PFOR page header that disagrees with its buffer
sfc-gh-pgaur
committed
617ab78
Count PFOR exceptions in an unsigned field
sfc-gh-pgaur
committed
b63da79
Put the output buffer last in PforWrapper::Decode
sfc-gh-pgaur
committed
95e9243
Return Status from PforWrapper::Encode
sfc-gh-pgaur
committed
44920ed
Decode PFOR pages with null slots through the Arrow path
sfc-gh-pgaur
committed
6662f82
Note that incremental PFOR encode and decode come later
sfc-gh-pgaur
committed
ea78f8d
Correct the bit_width mask comments in PforVectorInfo
sfc-gh-pgaur
committed
e017193
Validate PFOR exception positions and counts before patching
sfc-gh-pgaur
committed
bba5367
Drop the cached PFOR page when the decoder is given a new one
sfc-gh-pgaur
committed
ec77675
Validate PFOR wire metadata and enforce the output buffer size
sfc-gh-pgaur
committed
689b56e
Build PFOR into libarrow instead of recompiling it per target
sfc-gh-pgaur
committed
6d39bca
Stop PforDecoder from shadowing its base class page state
sfc-gh-pgaur
committed
bdcd30f
Derive PFOR layout sizes from the fields they describe
sfc-gh-pgaur
committed
7ad1b9a
Drop the unused PFOR vector view; harden SerializeVector
sfc-gh-pgaur
committed
49b45b0
Note the missing PLAIN fallback in PforEncoder
sfc-gh-pgaur
committed
413380a
Type-parameterize the PFOR tests over int32 and int64
sfc-gh-pgaur
committed
69b2e66
Take the PFOR value count from the page's own header
sfc-gh-pgaur
committed
cf67bdf
Validate the PFOR element count and offset chain before decoding
sfc-gh-pgaur
committed
995e30d
Encode an all-null PFOR page as a bare header
sfc-gh-pgaur
committed
1bcfc0b
Serialize PFOR little-endian instead of refusing to build
sfc-gh-pgaur
committed
eb45f7a
Build the PFOR sources under meson too
sfc-gh-pgaur
committed
6ce2e19
Record what a batched PFOR read costs today
sfc-gh-pgaur
committed
9d7e487
Add end-to-end PFOR tests through the Arrow reader and writer
sfc-gh-pgaur
committed
9239b9c
Name the PFOR headers _internal.h
sfc-gh-pgaur
committed
19106ff
Bound the sequential unpacker at the page, not at the vector
sfc-gh-pgaur
committed
3801f14
Use Arrow's section-rule style for the PFOR comments
sfc-gh-pgaur
committed
1c9ef2b
Decode PFOR pages one vector at a time
sfc-gh-pgaur
committed
4758312
Give PFOR a delta mode and a frame it can search
sfc-gh-pgaur
committed
cfea389
Cut the frame search's cost, and lower its frame onto a real value
sfc-gh-pgaur
committed
3048774
Add the delta-mode column shapes to the codec comparison benchmark
sfc-gh-pgaur
committed
d2a3717
Decline the PFOR delta mode from a sampled estimate
sfc-gh-pgaur
committed
81a096e
Gate PFOR and its delta mode behind writer properties
sfc-gh-pgaur
committed
db23c81
Harden the PFOR delta flag against a corrupt page
sfc-gh-pgaur
committed
ae68c9d
Note that the delta prefix sum could fold into the unpack kernel
sfc-gh-pgaur
committed
a20b4f7
Measure the unpacker's page bound from the header this vector wrote
sfc-gh-pgaur
committed
3018114
Extend the comment sweep to the delta-mode files
sfc-gh-pgaur
committed
5162b28
Name the frame-at-minimum alternative in the low-outlier test
sfc-gh-pgaur
committed
de64182
Let a caller require the delta representation
sfc-gh-pgaur
committed
e5025b8
[C++][Parquet] Coalesce equal-width DELTA_BINARY_PACKED miniblocks
sfc-gh-pgaur
committed
9b57f1d
[C++][Parquet] Scan DELTA_BINARY_PACKED deltas a vector at a time
sfc-gh-pgaur
committed
e7f5049