fix(spark): report the written file size in the commit message - #9325
Merged
robert3005 merged 1 commit intoAug 11, 2026
Merged
Conversation
bytesWritten accumulated FieldVector.getBufferSize() per batch, which is the uncompressed in-memory Arrow size, and handed that to VortexWriterCommitMessage as "the number of bytes written". VortexWriter.finish() already returns a summary carrying the file's physical size, and close() was only calling finish() and discarding it. Take the size from there and drop the per-batch accumulation, which also removes work from the write path. Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005
approved these changes
Aug 11, 2026
robert3005
enabled auto-merge (squash)
August 11, 2026 12:48
Merging this PR will improve performance by 13.04%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
bytesWrittenaccumulatedFieldVector.getBufferSize()per batch — the uncompressed in-memory Arrow size — and handed that toVortexWriterCommitMessage, whose javadoc documents it as "the number of bytes written". For a compressed file the two differ by the compression ratio.The real value was already available:
VortexWriter.finish()returns a summary carrying the file's physical size, andclose()was only callingfinish()and discarding the result.What changes are included in this PR?
Take the size from
finish().fileSize()and drop the per-batch accumulation, which also removes work from the write path../gradlew :vortex-spark_2.13:test --tests '…VortexDataSourceWriteTest' --tests '…VortexDataSourceBasicTest'— 15 pass:vortex-spark_2.12:test— 15 passspotlessCheck(both variants) andjavadoc— cleanNo new test: the value only becomes observable through a commit message a driver collects, which the suite has no fixture for.
What APIs are changed? Are there any user-facing changes?
None.
VortexWriterCommitMessage.bytesWritten()now reports the file size rather than the in-memory size, which is what its javadoc already promised.AI assistance
Prepared with agentic AI assistance. I read the writer's finalization path and
VortexWriter.finish()to confirm the summary is produced there and thatclose()discarded it.