Skip to content

fix: describe CometInMemoryTableScan by the scan it replaces in EXPLAIN - #6453

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:fix/cache-scan-explain
Sep 30, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:fix/cache-scan-explain

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #6452.

Rationale for this change

The cache scan's plan string dumped its CachedRDDBuilder, and with it the whole cached plan, into the middle of every plan that read the cache, breaking EXPLAIN and EXPLAIN FORMATTED. The issue has an example.

What changes are included in this PR?

CometInMemoryTableScanExec overrides stringArgs to print the Spark InMemoryTableScanExec it replaces, the way other Comet operators leave their originalPlan out. That shows the table's name when it has one, the attributes read and any pruning predicates, for example CometInMemoryTableScan Scan In-memory table named_t [k#1L].

Spark's own scan also lists the InMemoryRelation as an inner child, so the cached plan is drawn as an indented subtree below it. This PR does not do that, because ExtendedExplainInfo walks innerChildren, so the cached plan's operators and fallback reasons would then count toward every query that reads the cache. That can be decided separately.

How are these changes tested?

A new test in CometInMemoryCacheSuite checks the scan's line, including its pruning predicates, and checks that neither the tree string nor EXPLAIN FORMATTED prints the CachedRDDBuilder or the serializer. It fails without the override. CometInMemoryCacheSuite, CometInMemoryCacheKryoSuite and CometInMemoryCachePruningSuite pass on Spark 3.4 and 4.1.

CometInMemoryTableScanExec printed every constructor field in its plan
string, among them the CachedRDDBuilder with the whole cached plan, physical
and logical, inline and with its raw newlines. That broke the tree of every
plan that read the cache, in EXPLAIN and EXPLAIN FORMATTED alike.

Print the Spark InMemoryTableScanExec it replaces instead, which shows the
table's name when it has one, the attributes read and any pruning predicates.
@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading labels Sep 30, 2026

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

  • Prior state and problem: Cache scans printed CachedRDDBuilder, serializer identity and multiline cached plans inside EXPLAIN output, disrupting its structure.
  • Design approach: Override stringArgs with Iterator(originalPlan) to reuse Spark’s scan description.
  • Correctness / compatibility analysis: Checked Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources. TreeNode.argString renders the embedded scan through simpleString(maxFields), preserving table names, attributes, predicates and truncation. Execution and canonicalization remain unchanged. No introduced P1/P2 issues found within this review.
  • Key design decisions: The single override follows existing Comet rendering conventions without adding an abstraction or version branch. Keeping innerChildren unchanged avoids including cached-plan operators in fallback reporting.
  • Implementation sketch: One rendering override and a regression test covering the scan description, pruning predicates, tree output and EXPLAIN FORMATTED.
  • Behavioral changes worth calling out: Internal cache objects disappear from printed arguments. The change removes their formatting work without adding work to batch execution.
  • Suggested improvements: None meeting the P1/P2 reporting threshold.

Reviewed the full two-file diff from 8e4bded5578c4fb2f9d199f0ce892556911a183a to 2a400a0ad0201ea10c735adc0fe12487ac5552c3. The PR is not a draft. Read repository instructions and the supplied discussion snapshot. The existing approval raises no unresolved concerns.

Routed skills: review-comet-pr. No sibling skill applies to this formatting-only change.

Exact-head CI: 25 successful checks, 15 skipped, no failures or unfinished checks. The Linux Spark 4.1 execution log confirms the new regression test and cache suites ran successfully. That job reports 1,151 passing tests and zero failures.

Validation limits: No local JVM/native build or tests were run because this checkout has no build artifacts. Runtime evidence comes from exact-head CI. Other supported Spark versions were source-checked. Upstream Spark SQL suites, macOS and Iceberg suites were skipped.

@andygrove
andygrove added this pull request to the merge queue Sep 30, 2026
@andygrove

Copy link
Copy Markdown
Member Author

Thanks @sunchao and @rich7420

Merged via the queue into apache:main with commit 353a384 Sep 30, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scan Parquet scan / data reading bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CometInMemoryTableScan prints its whole cached plan inline in EXPLAIN

3 participants