Skip to content

fix(spark): name the Vortex files when a read fails - #9326

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-reader-ioexception-path
Aug 11, 2026
Merged

fix(spark): name the Vortex files when a read fails#9326
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-reader-ioexception-path

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

The three IOException wrappers in VortexPartitionReader threw a bare RuntimeException(e). When a scan fails mid-flight the executor log then shows the cause's stack trace but not the one fact it cannot carry: which Vortex files were being read. On a scan spanning many partitions that is the difference between a one-line diagnosis and grepping task logs.

The reader already holds the paths — it opens the DataSource with them.

What changes are included in this PR?

Each wrapper now names the partition's paths and what the reader was doing (load a batch from / read the loaded batch of / close the reader over), through one small helper.

  • ./gradlew :vortex-spark_2.13:test — 167 tests, only the pre-existing VortexDataSourceS3MockTest fails here for lack of a Docker environment
  • same on :vortex-spark_2.12:test
  • spotlessCheck (both variants) and javadoc — clean

No new test: these branches need a mid-scan native I/O failure, which the suite has no fixture for.

What APIs are changed? Are there any user-facing changes?

None. The exception type is unchanged; only the message gains the paths.

AI assistance

Prepared with agentic AI assistance. I read the reader's three failure paths and confirmed spark.paths() is already available at each of them.

The three IOException wrappers in the reader threw a bare
RuntimeException, so a failure mid-scan surfaced without the one fact a
stack trace does not carry: which files were being read.

The reader already holds them, so include the partition's paths and what
it was doing when the failure occurred.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/chore A trivial change label Aug 11, 2026
@robert3005
robert3005 enabled auto-merge (squash) August 11, 2026 12:52
@codspeed-hq

codspeed-hq Bot commented Aug 11, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 13.04%

⚡ 1 improved benchmark
✅ 1943 untouched benchmarks
⏩ 85 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (1000, 16)] 73.3 µs 64.9 µs +13.04%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-reader-ioexception-path (37a52ff) with develop (4b37237)

Open in CodSpeed

Footnotes

  1. 85 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@robert3005
robert3005 merged commit 402a5ac into vortex-data:develop Aug 11, 2026
104 of 105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants