Skip to content

test(spark): characterize partition transform resolution - #9323

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-transform-tests
Aug 11, 2026
Merged

test(spark): characterize partition transform resolution#9323
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-transform-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

resolveTransforms turns Spark partition transforms into the directory keys and column indices a writer uses, and had no tests. It is a pure static function, so the behaviour is cheap to pin.

What changes are included in this PR?

Test only. Covers the directory keys derived for identity, years/months/days/hours and bucket; the type guards (temporal transforms reject a non-temporal column, hours also rejects a date); the bucket count and multi-column index resolution; and the rejections for an unknown transform name, a missing numBuckets argument, a zero-reference transform and a column outside the schema.

Also covers the Java serialization round trip of ResolvedTransform — the reason resolution happens eagerly on the driver, since Spark's Transform objects cannot cross to executors.

Verified non-vacuous: disabling the temporal type guard fails the guard cases.

  • ./gradlew :vortex-spark_2.13:test --tests '…PartitionedVortexDataWriterTest' — 11 pass
  • same on :vortex-spark_2.12:test — 11 pass
  • spotlessCheck (both variants) and javadoc — clean

What APIs are changed? Are there any user-facing changes?

None.

AI assistance

Prepared with agentic AI assistance. I read the resolution code and its callers, confirmed each assertion against the current implementation, and verified the guard tests fail when the guard is removed.

resolveTransforms had no tests. Pin the directory keys it derives, the
type guards on the temporal transforms, the bucket count and multi-column
resolution, and the rejections for an unknown transform name, a missing
numBuckets argument and a column outside the schema.

Also cover the Java serialization round trip of ResolvedTransform, which
is the reason the resolution happens eagerly on the driver: Spark's
Transform objects are Scala case classes that cannot cross to executors.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@codspeed-hq

codspeed-hq Bot commented Aug 10, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 10.85%

❌ 2 regressed benchmarks
✅ 1942 untouched benchmarks
⏩ 85 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress[u64, (1000, 16)] 64 µs 72.1 µs -11.19%
Simulation take[small_m/shuffled/primitive/nonnull/chunks=16384/indices=16] 1.2 ms 1.3 ms -10.51%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:spark-partition-transform-tests (a70e7a0) with develop (17b190b)

Open in CodSpeed

Footnotes

  1. 85 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@robert3005 robert3005 added the changelog/chore A trivial change label Aug 11, 2026
@robert3005
robert3005 enabled auto-merge (squash) August 11, 2026 12:47
@robert3005
robert3005 merged commit 8979e32 into vortex-data:develop Aug 11, 2026
93 of 95 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants