[python] Align data directories with Java and enable Native IO

[JingsongLi]

Purpose

Enable Native planning, reading, writing, updating, and committing for Paimon tables using data-file.path-directory, and fix Python path resolution to follow Java.

Depends on apache/paimon-rust#974. This PR stays in draft until that change is on paimon-rust@main; the CI dependency remains apache/paimon-rust@main.

• Remove the blanket Native fallbacks for tables with a configured data directory.
• Introduce shared Java-style resolution for data roots, bucket paths, and external paths, supporting relative/normalized/absolute/URI directories and Windows drives/UNC paths.
• Correct URI handling around escaped partition paths, and preserve Python's literal timestamp partition components during resolution.
• Reject empty configured directories and table-copy overrides that would change the persisted data layout.
• Cover append, PK, and Data Evolution tables across Python/Native writers, planners, and readers; include batch/stream writing, updates/upserts, repeated deletion vectors, history, abort cleanup, Blob reads, and escaped partitions under relocated data roots.

Tests

Using a locally built binding from the companion Rust PR and PyArrow 18.1, with all five Native CI flags enabled:

• 1368 passed, 2 skipped, and 62 subtests passed across directory/external-path/deletion-vector tests and Native plan/read/write/update/upsert/sequence regression suites.
• Native execution assertions recorded 1224 plans, 1709 reads, 909 writes, 6 REST commits, and update coverage for row-ID, grouped, predicate, upsert, and incremental paths.
• Shared path resolution checked against the current Java Path implementation: 311 POSIX/Windows oracle cases matched; representative expected outputs are committed as regression tests.
• Flake8 passed for all changed Python files.

No Native fork pin or CI dependency override is introduced. Native CI is expected to require the Rust dependency above before it can pass on this branch.

[leaves12138]

Reviewed commit 03420e9c8b78e7e6d8e63e43e81b6dd0f8252ec6 together with apache/paimon-rust#974.

Fixed and pushed the findings from this review:

• Match Java path normalization for relative leading-colon components and empty paths under scheme/authority-only parents, including explicit Windows-mode tests.
• Preserve literal %, ?, and # characters in configured file: data directories. Previously Python URL decoding and Rust literal-path handling could write/read different physical locations. The fix covers generated data paths, the native split/deletion-vector bridge, and generated external paths without globally changing LocalFileIO URL behavior.
• Added cross-backend tests for literal directories: append/PK/data-evolution, partitioned/unpartitioned reads, batch/stream writes, updates, deletion vectors, historical reads, abort cleanup, and round-robin/entropy-inject/weight-robin external paths.
• Marked the inherited Python-internal streaming test as python_write, so both batch and stream variants still execute their Python instrumentation instead of incorrectly asserting that native execution invokes Python internals.

Validation:

• Paired native suite using Rust 2a46e49fe7433165bf066c1e4c6abac8cf3cec60 and PyArrow 18.1.0: 1,748 passed, 4 skipped, 143 subtests passed; all five native execution families were exercised.
• Python-backend regression suite: 313 passed, 51 skipped, 14 subtests passed.
• Actual Java Path oracle: 960 POSIX/Windows input combinations, of which Java accepted 893; all 893 accepted inputs matched. The 67 inputs rejected by Java were excluded from the string-result comparison.
• Repository-configured flake8, Python 3.6 grammar checks for changed files, and git diff --check: passed.

No remaining blocking code-review findings for these paired revisions. Keep the existing dependency and draft status: merge apache/paimon-rust#974 first, then rerun and require the Python Native CI against upstream Rust main before merging this PR. The successful local validation used the paired Rust revision, not the older upstream-main wheel from the previous CI run. I have not changed the CI dependency or merged either PR. LGTM for the reviewed code.