[spark] Add tests for catalog-managed Format Table partitions - #9119
Open
sundapeng wants to merge 5 commits into
Open
[spark] Add tests for catalog-managed Format Table partitions#9119sundapeng wants to merge 5 commits into
sundapeng wants to merge 5 commits into
Conversation
TRUNCATE TABLE, TRUNCATE TABLE PARTITION and ALTER TABLE ... RENAME TO PARTITION all refuse a format table with catalog-managed partitions, and these tests pin that they refuse without moving anything: a registration pointing at a directory that is gone, or data under a spec nobody registered, would both be worse than the refusal. The refusal TRUNCATE PARTITION produced described the table wrongly. "Only FileStoreTable supports partitions" is reached by a table that has partitions and lists them through this very trait, which sends the reader looking for the wrong problem; it now says which operations do manage those partitions. A null partition value registers under the default partition name, keeps a directory of that name, reads back as null, is discovered by MSCK REPAIR and is dropped by that name. SHOW PARTITIONS prints it as dt=null rather than the default name, which is what Spark's own ShowPartitionsExec does with a null value; a native Paimon table prints the same thing, so the test records where that behaviour comes from instead of working around it here. The remaining probes cover ADD/DROP PARTITION idempotence, prefix DROP, partial specs in SHOW PARTITIONS, case-insensitive partition column names and empty string partition values, all of which already behaved this way.
Contributor
|
CI failed |
Member
Author
|
Local Spark 3.5 / Scala 2.12 evidence:
Fix: 22e5269. |
sundapeng
changed the title
sundapeng
changed the title
leaves12138
requested changes
Aug 11, 2026
leaves12138
left a comment
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
Thanks for adding this coverage. I found two issues that should be addressed before merging.
leaves12138
approved these changes
Aug 11, 2026
leaves12138
left a comment
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Thanks for addressing the comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
A catalog-managed Format Table keeps the visible partition set in the catalog while the rows remain
in partition directories. A partition command must therefore update both sides consistently, or
leave both unchanged.
ADD PARTITION,DROP PARTITION,SHOW PARTITIONS, writes, andMSCK REPAIR TABLEalready supportthis table type. This PR adds end-to-end coverage for their Spark SQL behavior, including duplicate
ADD handling,
DROP IF EXISTS, partial partition specs, case-insensitive column resolution,writer-side registration, a registered partition whose directory is missing, and default partition
values.
TRUNCATE TABLE,TRUNCATE TABLE PARTITION, andRENAME PARTITIONare currently unsupported for aFormat Table. This PR does not add that support. A rejected command must leave catalog registrations
and rows unchanged. If support is added later, TRUNCATE must keep registrations consistent with the
remaining rows, and RENAME must move the registration and directory together without losing rows.
The only production behavior change is the error for an unsupported partition operation. The old
message,
Only FileStoreTable supports partitions, is misleading because the Format Table ispartitioned. The new message says that the operation is limited to a native Paimon table and points
Format Table users to
ADD PARTITION,DROP PARTITION, andMSCK REPAIR TABLE.SHOW PARTITIONSrenders a null value asdt=nullon Spark's V2 path; a native Paimon table has thesame behavior, so this PR records it instead of adding Format Table-specific rendering. Empty string
partition values follow Paimon's existing default-partition encoding and read back as null.
There is no public API, REST contract, or storage format change.
ANALYZE TABLEis out of scope.Tests
case-insensitive partition columns, sorted and filtered SHOW output, writer registration, and a
registered partition whose directory is missing.
consistently or leaving state unchanged, null and empty values, default-partition MSCK discovery,
and default-partition DROP.
Spotless, and Enforcer passed.