[core] Derive partition layout from active files

[dwangatt]

Purpose

Extracted from #9370 as a prerequisite for supporting different bucket counts across partitions.

A partition's current bucket layout must be derived from the files that are active in the current snapshot. It must not be inferred from data-file creation timestamps, because deleted files from a later rescale can otherwise override the layout restored by a rollback.

Changes

• Update AbstractFileStoreScan#readPartitionEntries to merge manifest entries first and discard deleted files before aggregating partition entries.
• Preserve the existing generic PartitionEntry aggregation semantics; layout resolution is handled at the snapshot scan boundary where file liveness is known.
• Add a regression test covering:
  1. a partition written with 2 buckets;
  2. overwrite with 4 buckets;
  3. rollback to the first snapshot;
  4. verification that the visible partition layout returns to 2 buckets.

Testing

• FileStoreScanPartitionBucketEntryTest#testReadPartitionEntriesRestoresBucketLayoutAfterRollback

Related

• Extracted from [WIP] Split partition-level bucket-count support into focused PRs #9370
• Prerequisite for the follow-up layout-aware BatchWriteBuilder overwrite routing work.

[JingsongLi]

Deriving partition metadata from live files looks like the right semantic fix. Aggregating ADD and DELETE entries can cancel the counts, but it cannot reliably undo fields such as totalBuckets and lastFileCreationTime.

Could we preserve that behavior while reducing the cost of readAndMergeFileEntries(..., false)? A manifest containing both additions and deletions is read twice, and completed reads can retain full ManifestEntry lists until they are consumed. This may increase latency and peak heap for large tables. Aggregating within each manifest task, or consuming results in bounded batches, would help.

I’d also strengthen the regression test. After rollbackTo(1), the old implementation reads snapshot 1 and may return 2 as well. A deterministic test with deleted and live files in the same scanned snapshot would establish that this change fixes the original behavior.

[dwangatt]

Thanks @JingsongLi Updated PR with:

• using readManifest() instead of readAndMergeFileEntries(). Filter out DELETE manifest entries and only loads ADD entries
• Added a test that work with new logic, but fail with original logic.

[JingsongLi]

Requirement fit: SUPPORTED. This is already an observable correctness fix, beyond groundwork for #9370: a snapshot with a deleted file and a live file can report the deleted file's bucket layout through SnapshotReader.partitionEntries(). That result also feeds partition expiration and compact planning. The new same-snapshot regression demonstrates the old behavior and the desired live-file result.

Implementation: CLEAN on 0d853a8. The revised scan first collects delete identifiers, then aggregates only surviving ADD entries per manifest. That removes deleted files from both the count and totalBuckets choice while keeping the per-manifest aggregation bounded; it addresses my earlier concern about retaining all ADD entries from all manifests. I traced the snapshot-reader callers and did not find a new correctness regression. Manifests containing both ADD and DELETE entries are read in both passes, so monitor partition-scan latency on very large tables, but I do not have evidence of a blocking regression here.

Verification: FileStoreScanPartitionBucketEntryTest 10/10 and PartitionLayoutScanTest 1/1 passed on this head; current CI checks are green. No production blocker found.