[SPARK-59250][SQL] Translate an empty runtime IN filter to AlwaysFalse in DSv2 runtime filtering - #58524
[SPARK-59250][SQL] Translate an empty runtime IN filter to AlwaysFalse in DSv2 runtime filtering#58524pan3793 wants to merge 3 commits into
Conversation
…e in DSv2 runtime filtering Assisted-by: Claude Fable 5
…tering Assisted-by: Claude Fable 5
d7b2aaa to
5ccb9f1
Compare
|
cc @szehon-ho |
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
Optimization makes sense to me
| spark.conf.set("spark.sql.catalog.testcat", classOf[InMemoryTableWithV2FilterCatalog].getName) | ||
| } | ||
|
|
||
| import testImplicits._ |
There was a problem hiding this comment.
nit: can we put it on top or more natural place ?
There was a problem hiding this comment.
Moved the import and helper into the shared DynamicPartitionPruningV2Suite.
| scans.head | ||
| } | ||
|
|
||
| test("SPARK-59250: DPP prunes all DSv2 partitions when the runtime IN filter is empty", |
There was a problem hiding this comment.
Could we also cover the SupportsRuntimeFiltering (V1 Filter) path? This PR updates InMemoryBaseTable.InMemoryBatchScan to handle AlwaysFalse, but these tests run only with InMemoryTableWithV2FilterCatalog, so they exercise InMemoryV2FilterBatchScan and leave the V1 compatibility path untested. It may be possible to share the tests with DynamicPartitionPruningV2Suite so both implementations run them.
There was a problem hiding this comment.
Moved the tests into DynamicPartitionPruningV2Suite, so they now run through V1 Filter, V2 Predicate, and Catalyst runtime filtering.
Assisted-by: Claude Fable 5
2 participants
What changes were proposed in this pull request?
In
DataSourceV2Strategy.translateRuntimeFilterV2, match anInSubqueryExecwhose subquery result is empty before the pushable-column check, and translate it toAlwaysFalse.expr IN ()is false for every row regardless ofexpr, so the translation is valid even whenexpritself is untranslatable, e.g. a join key wrapped in a cast inserted by type coercion (cast(col as bigint) IN dynamicpruning#x).The test connector scans
InMemoryTableWithV2FilterandInMemoryBaseTable.InMemoryBatchScanare updated to honorAlwaysFalseby pruning all partitions.Why are the changes needed?
When the build side of a DPP join produces no rows, the runtime filter degenerates to
key IN (). Today this is either emitted as a zero-valueINpredicate that connectors cannot interpret meaningfully (Spark's ownInMemoryTableWithV2Filterignores it: itsfilter()requireschildren().length > 1), or, when the key is wrapped in a cast, dropped entirely with "Can't translate ... unsupported expression". Either way the scan reads every partition to produce zero output rows.Translating to
AlwaysFalseprunes everything and makes the connector contract explicit: connectors receive a well-defined predicate instead of a degenerate IN.Does this PR introduce any user-facing change?
No correctness change. DSv2 scans receiving an empty runtime IN filter can now prune all partitions instead of reading everything.
How was this patch tested?
New tests in
DynamicPartitionPruningV2Suitecovering a bare pruning key and a cast-wrapped pruning key; both assertfilteredPartitionsis empty. The tests run through V1Filter, V2Predicate, and Catalyst runtime filtering. Previously all 25 partitions were read.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5