Skip to content

fix(spark): report the written file size in the commit message - #9325

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-commit-message-file-size
Aug 11, 2026
Merged

fix(spark): report the written file size in the commit message#9325
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-commit-message-file-size

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

bytesWritten accumulated FieldVector.getBufferSize() per batch — the uncompressed in-memory Arrow size — and handed that to VortexWriterCommitMessage, whose javadoc documents it as "the number of bytes written". For a compressed file the two differ by the compression ratio.

The real value was already available: VortexWriter.finish() returns a summary carrying the file's physical size, and close() was only calling finish() and discarding the result.

What changes are included in this PR?

Take the size from finish().fileSize() and drop the per-batch accumulation, which also removes work from the write path.

  • ./gradlew :vortex-spark_2.13:test --tests '…VortexDataSourceWriteTest' --tests '…VortexDataSourceBasicTest' — 15 pass
  • same on :vortex-spark_2.12:test — 15 pass
  • spotlessCheck (both variants) and javadoc — clean

No new test: the value only becomes observable through a commit message a driver collects, which the suite has no fixture for.

What APIs are changed? Are there any user-facing changes?

None. VortexWriterCommitMessage.bytesWritten() now reports the file size rather than the in-memory size, which is what its javadoc already promised.

AI assistance

Prepared with agentic AI assistance. I read the writer's finalization path and VortexWriter.finish() to confirm the summary is produced there and that close() discarded it.

bytesWritten accumulated FieldVector.getBufferSize() per batch, which is
the uncompressed in-memory Arrow size, and handed that to
VortexWriterCommitMessage as "the number of bytes written".

VortexWriter.finish() already returns a summary carrying the file's
physical size, and close() was only calling finish() and discarding it.
Take the size from there and drop the per-batch accumulation, which also
removes work from the write path.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/fix A bug fix label Aug 11, 2026
@robert3005
robert3005 enabled auto-merge (squash) August 11, 2026 12:48
@codspeed-hq

codspeed-hq Bot commented Aug 11, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 13.04%

⚡ 1 improved benchmark
✅ 1943 untouched benchmarks
⏩ 85 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (1000, 16)] 73.3 µs 64.9 µs +13.04%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-commit-message-file-size (7ce0bfe) with develop (4b37237)

Open in CodSpeed

Footnotes

  1. 85 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@robert3005
robert3005 merged commit d900b57 into vortex-data:develop Aug 11, 2026
95 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants