Skip to content

[Issue #1279] Replace C++ metadata serialization with FlatBuffers - #1390

Open
whzruc wants to merge 1 commit into
pixelsdb:masterfrom
whzruc:rebuild/cpp-flatbuffers-serialization
Open

[Issue #1279] Replace C++ metadata serialization with FlatBuffers#1390
whzruc wants to merge 1 commit into
pixelsdb:masterfrom
whzruc:rebuild/cpp-flatbuffers-serialization

Conversation

@whzruc

@whzruc whzruc commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Closes #1279.

This PR supersedes the closed and unmerged #1282. It rebuilds the FlatBuffers migration on the latest master and includes the correctness and integration fixes found during validation.

Summary

  • add the shared Pixels metadata schema at flatbuffers/pixels.fbs
  • vendor the FlatBuffers toolchain and generate C++ sources during the build
  • replace Protobuf metadata handling in the C++ reader, writer, footer cache, statistics, CLI, and DuckDB integration paths
  • fix integer decoding, 64-bit RLE handling, asynchronous buffer reads, and CLI error handling uncovered by round-trip testing
  • extend writer and CLI test coverage

The existing proto/*.proto definitions remain unchanged for Java and gRPC users. This change only replaces the C++ Pixels file-metadata serialization path.

Validation

  • make -j
  • IntegerWriterTest: 4/4 passed
  • PixelsWriterTest --gtest_also_run_disabled_tests: 7/7 passed
  • pixels-cli LOAD generated a FlatBuffers-based .pxl file
  • DuckDB pixels_scan successfully read the generated file and returned the expected 3-row aggregate and values

Compatibility note

Files produced by this C++ implementation use FlatBuffers metadata and therefore require the corresponding FlatBuffers-aware C++ reader.

@whzruc
whzruc force-pushed the rebuild/cpp-flatbuffers-serialization branch 2 times, most recently from adaa077 to ce3304a Compare August 27, 2026 09:47
@whzruc

whzruc commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

Null handling was tightened during round-trip validation:

  • ColumnWriter now preserves each pixel\u0027s numberOfValues and hasNull fields in the FlatBuffers statistics snapshot.
  • IntColumnReader and LongColumnReader expand the compact null bitmap into the output vector and only consume a value for a null row when nullsPadding is enabled. This prevents values after an unpadded null from shifting.

Validated with the 4 integer writer tests, all 7 PixelsWriter tests, and a CLI-to-DuckDB round trip producing 1001, NULL, 1003 for a LONG column.

…fers

Replace the C++ Protobuf metadata path with FlatBuffers while keeping the Java and gRPC Protobuf definitions unchanged.

- add the shared Pixels metadata schema under flatbuffers/ and vendor the FlatBuffers toolchain
- generate and link FlatBuffers sources from the C++ build
- migrate reader, writer, footer, statistics, CLI, and DuckDB integration paths
- fix integer decoding, 64-bit RLE handling, async reads, and CLI error handling
- extend writer and CLI round-trip coverage

Tests:
- make -j
- IntegerWriterTest (4 tests)
- PixelsWriterTest --gtest_also_run_disabled_tests (7 tests)
- pixels-cli LOAD followed by DuckDB pixels_scan round trip
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[pixels-cpp] Use flatbuffers instead of protobuf for serialization

1 participant