fix(spark): name the Vortex files when a read fails - #9326
Merged
robert3005 merged 1 commit intoAug 11, 2026
Conversation
The three IOException wrappers in the reader threw a bare RuntimeException, so a failure mid-scan surfaced without the one fact a stack trace does not carry: which files were being read. The reader already holds them, so include the partition's paths and what it was doing when the failure occurred. Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005
approved these changes
Aug 11, 2026
robert3005
enabled auto-merge (squash)
August 11, 2026 12:52
Merging this PR will improve performance by 13.04%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The three
IOExceptionwrappers inVortexPartitionReaderthrew a bareRuntimeException(e). When a scan fails mid-flight the executor log then shows the cause's stack trace but not the one fact it cannot carry: which Vortex files were being read. On a scan spanning many partitions that is the difference between a one-line diagnosis and grepping task logs.The reader already holds the paths — it opens the
DataSourcewith them.What changes are included in this PR?
Each wrapper now names the partition's paths and what the reader was doing (
load a batch from/read the loaded batch of/close the reader over), through one small helper../gradlew :vortex-spark_2.13:test— 167 tests, only the pre-existingVortexDataSourceS3MockTestfails here for lack of a Docker environment:vortex-spark_2.12:testspotlessCheck(both variants) andjavadoc— cleanNo new test: these branches need a mid-scan native I/O failure, which the suite has no fixture for.
What APIs are changed? Are there any user-facing changes?
None. The exception type is unchanged; only the message gains the paths.
AI assistance
Prepared with agentic AI assistance. I read the reader's three failure paths and confirmed
spark.paths()is already available at each of them.