fix: describe CometInMemoryTableScan by the scan it replaces in EXPLAIN - #6453
Conversation
CometInMemoryTableScanExec printed every constructor field in its plan string, among them the CachedRDDBuilder with the whole cached plan, physical and logical, inline and with its raw newlines. That broke the tree of every plan that read the cache, in EXPLAIN and EXPLAIN FORMATTED alike. Print the Spark InMemoryTableScanExec it replaces instead, which shows the table's name when it has one, the attributes read and any pruning predicates.
sunchao
left a comment
There was a problem hiding this comment.
Summary
- Prior state and problem: Cache scans printed
CachedRDDBuilder, serializer identity and multiline cached plans insideEXPLAINoutput, disrupting its structure. - Design approach: Override
stringArgswithIterator(originalPlan)to reuse Spark’s scan description. - Correctness / compatibility analysis: Checked Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources.
TreeNode.argStringrenders the embedded scan throughsimpleString(maxFields), preserving table names, attributes, predicates and truncation. Execution and canonicalization remain unchanged. No introduced P1/P2 issues found within this review. - Key design decisions: The single override follows existing Comet rendering conventions without adding an abstraction or version branch. Keeping
innerChildrenunchanged avoids including cached-plan operators in fallback reporting. - Implementation sketch: One rendering override and a regression test covering the scan description, pruning predicates, tree output and
EXPLAIN FORMATTED. - Behavioral changes worth calling out: Internal cache objects disappear from printed arguments. The change removes their formatting work without adding work to batch execution.
- Suggested improvements: None meeting the P1/P2 reporting threshold.
Reviewed the full two-file diff from 8e4bded5578c4fb2f9d199f0ce892556911a183a to 2a400a0ad0201ea10c735adc0fe12487ac5552c3. The PR is not a draft. Read repository instructions and the supplied discussion snapshot. The existing approval raises no unresolved concerns.
Routed skills: review-comet-pr. No sibling skill applies to this formatting-only change.
Exact-head CI: 25 successful checks, 15 skipped, no failures or unfinished checks. The Linux Spark 4.1 execution log confirms the new regression test and cache suites ran successfully. That job reports 1,151 passing tests and zero failures.
Validation limits: No local JVM/native build or tests were run because this checkout has no build artifacts. Runtime evidence comes from exact-head CI. Other supported Spark versions were source-checked. Upstream Spark SQL suites, macOS and Iceberg suites were skipped.
Which issue does this PR close?
Closes #6452.
Rationale for this change
The cache scan's plan string dumped its
CachedRDDBuilder, and with it the whole cached plan, into the middle of every plan that read the cache, breakingEXPLAINandEXPLAIN FORMATTED. The issue has an example.What changes are included in this PR?
CometInMemoryTableScanExecoverridesstringArgsto print the SparkInMemoryTableScanExecit replaces, the way other Comet operators leave theiroriginalPlanout. That shows the table's name when it has one, the attributes read and any pruning predicates, for exampleCometInMemoryTableScan Scan In-memory table named_t [k#1L].Spark's own scan also lists the
InMemoryRelationas an inner child, so the cached plan is drawn as an indented subtree below it. This PR does not do that, becauseExtendedExplainInfowalksinnerChildren, so the cached plan's operators and fallback reasons would then count toward every query that reads the cache. That can be decided separately.How are these changes tested?
A new test in
CometInMemoryCacheSuitechecks the scan's line, including its pruning predicates, and checks that neither the tree string norEXPLAIN FORMATTEDprints theCachedRDDBuilderor the serializer. It fails without the override.CometInMemoryCacheSuite,CometInMemoryCacheKryoSuiteandCometInMemoryCachePruningSuitepass on Spark 3.4 and 4.1.