Repository navigation
feat: use released tpcgen-cli 4.0.0 from PyPI - #114
Open
Kevin Liu (kevinjqliu) wants to merge 1 commit into
Open
Kevin Liu (kevinjqliu) wants to merge 1 commit into
Kevin Liu (kevinjqliu) wants to merge 1 commit into
Conversation
Kevin Liu (kevinjqliu)
force-pushed
the
kevinjqliu-use-tpcgen-cli-4-0
branch
2 times, most recently
from
October 20, 2026 03:07
1c48bb8 to
83ab200
Compare
Replace the vendored pre-release tpcgen-cli binaries with the tpcgen-cli 4.0.0 package from PyPI (tpch_datagen / tpcds_datagen extras), and reduce the integration to a thin wrapper around the CLI. - One _tpcgen module builds the tpcgen-cli command and runs it. Options (compression, row-group size, threads) are passed straight through. - Part counts come from one measured SF1000 size table per benchmark, targeting ~256 MiB files. The compression-factor, native-ratio, and TPC-DS row-count scaling tables are removed. - Output keeps tpcgen-cli's layout (<table>/<table>.<part>.<ext>); the renaming to LakeBench and dbgen/dsdgen file names is removed. - Remove native/tpcgen, the hatch build hook, wheel build scripts, and the native-wheels workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Kevin Liu (kevinjqliu)
force-pushed
the
kevinjqliu-use-tpcgen-cli-4-0
branch
from
October 20, 2026 03:14
83ab200 to
7875d4c
Compare
Kevin Liu (kevinjqliu)
left a comment
Kevin Liu (kevinjqliu)
left a comment
Author
There was a problem hiding this comment.
removed a lot of wrapper code 😄
There was a problem hiding this comment.
note, we just released https://pypi.org/project/tpcgen-cli/ today so msft's internal pypi mirror hasn't picked it up yet. I'll regenerated the file again once its picked up
Contributor
|
Tom Zeng (@tomz) - FYI this was anticipated work once tpcgen-cli was released. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this change is needed
tpcgen-cli4.0.0 is released on PyPI. LakeBench currently vendors a pre-release binary built from a fork, plus a lot of logic to size, rename, and package its output.How
Use the PyPI package and make the integration a thin wrapper around the CLI.
tpch_datagen/tpcds_datageninstalltpcgen-cli==4.0.0. Removes the vendored binaries, hatch build hook, wheel scripts, andnative-wheelsworkflow.datagen/_tpcgen.pybuilds the command and runs it.compression,target_row_group_size_mb, andnum_threadsare passed straight through.<table>/<table>.<part>.<ext>.Behavior changes:
lineitem-00001.zstd.parquet→lineitem.1.parquet. Native files are no longer renamed todbgen/dsdgennames. Loading is unaffected.target_row_group_size_mbis now the uncompressed size (tpcgen-cli's--row-group-bytes).compression_factoris removed.num_threads.Test
pytest tests/ --ignore=tests/integration), also on Python 3.8.DeltaScan serialization not implemented) fail the same way with the legacy DuckDB generator.Closes issue
N/A