[core] Speed up row-id manifest sorting

[leaves12138]

What changed

• add manifest-specific primitive Avro reader and writer APIs under paimon-format/.../avro/primitive
• expose only the ordering, filtering, and identity fields required by row-id manifest sorting
• reuse a plain mutable record view instead of materializing InternalRow or ManifestEntry objects
• discover naturally sorted runs and merge them with a fixed selection tree
• copy eligible encoded records and compressed Avro blocks directly into output manifests
• keep the generic format, stats collector, and rolling-writer frameworks unchanged
• retain the spillable external sorter as a bounded fallback for fragmented inputs
• preserve exact DELETE identifier matching and manifest statistics
• add manifest-sort.run-merge-optimize.enabled as a fallback switch

Why

Full row-id manifest compaction currently materializes every ManifestEntry, feeds all entries through the external sorter, and encodes every surviving entry again. Large data-evolution tables therefore spend substantial CPU time, cumulative heap allocation, and temporary-disk I/O in this path even when their manifests already consist of a small number of sorted runs.

The new manifest-only Avro path reads just the primitive fields needed for classification and ordering, interns partition data, and merges already-sorted runs directly. When an encoded record or compressed Avro block can be reused safely, it is copied without materializing a full entry or encoding it again. Fragmented input still falls back to the existing bounded external sorter.

Performance

The benchmark uses the production snapshot 7537 metadata fixture on Linux x86_64:

• 87 input manifests, 666.2 MiB of manifest data
• 26,100,963 input entries
• six-way manifest read parallelism
• 64 MiB external-sort buffer and 128 file handles
• no -Xmx limit
• identical code and configuration except that the existing path forces external sort while the new path enables natural-run merge

Metric	Existing external sort	Final run merge	Improvement
Sort wall time	180.41 s	11.12 s	16.2x faster
Total allocation during sort	150,607.3 MiB (147.08 GiB)	2,536.2 MiB (2.48 GiB)	59.4x lower
Benchmark-command peak RSS	2.25 GiB	1.19 GiB	47.1% lower
Initial sorter spill	1,080,034,577 B (1.006 GiB), 273 files	0 B	eliminated
Intermediate fan-in rewrite	1,514,367,922 B (1.410 GiB)	0 B	eliminated
Total temporary sorter bytes written	2,594,402,499 B (2.416 GiB)	0 B	eliminated

Total allocation during sort is cumulative allocation, calculated by summing JVM per-thread allocation deltas across the measured sort interval. It is GC-reclaimable allocation volume, not live heap or RSS. Peak RSS is the maximum resident set reported by /usr/bin/time -v for the Maven/Surefire benchmark command.

Spill values count compressed temporary files created by the sorter only; they exclude source-manifest reads and final output-manifest writes. External sort first wrote 1,080,034,577 bytes across 273 runs. Because this exceeded the 128-file-handle limit, two fan-in operations wrote another 506,977,657 and 1,007,390,265 bytes. The final run-merge path created no sorter spill files.

Result equivalence

Both paths produced exactly 26,095,685 output entries. Their logical encoded-record SHA-256 was identical:

afe6fcdaff7e8b2e7d047c199164c19457ced5b3cb2adaadd32c48cf11a1cdf5

Validation

• 3 targeted AvroFileFormatTest cases covering primitive projection, nullable/nested fields, reader reuse, and large Zstd blocks
• 13 targeted ManifestFileMetaTest cases covering row-id order, secondary keys, exact DELETE identifiers, raw identity fields, many partitions, fragmented fallback, read amplification, block stats, configured partition fields, missing row-id stats, and minor compaction
• 5 ByteArrayKeyTest cases plus 12 binary-manifest identifier/projection cases
• mvn -pl paimon-format,paimon-core -am -DskipTests compile with checkstyle, Spotless, and enforcer enabled
• snapshot 7537 output-count and logical-digest comparison on Linux x86_64