[PoC] Unify file I/O behind a storage entry point

[Xuanwo]

Architecture at a glance

Register a backend once; route the existing file APIs through the same storage entry point. This diagram illustrates the PoC direction, not a finalized API.

Contracts, SDK dependencies, and a suggested reading path

Start with the Storage contract and FileReader, then follow either the ObjectStore adapter or the OpenDAL adapter. The integration tests show how applications use the common entry point.

Which issue does this PR close?

Related to #14854 and #25144. This PoC does not close either discussion.

Rationale for this change

This is a PoC to demonstrate my initial ideas, and the API is not final. The main goal is to bring file I/O calls currently scattered throughout DataFusion behind one common entry point, so applications can register a storage backend once and use it across the existing file APIs.

The API largely carries over the operations DataFusion already uses. It is a starting point for discussing the dependency boundary and integration model, with substantial room for future optimization and API changes. I would especially appreciate feedback on whether this direction provides the right place to integrate existing object_store clients, OpenDAL operators, and custom storage implementations.

What changes are included in this PR?

• Introduce a backend-independent datafusion-storage crate with Storage for namespace operations, FileReader for reads, and Tokio AsyncWrite for outputs. Shared file metadata and paths no longer depend on SDK types.
• Route listing, schema inference, built-in file scans, COPY TO, and file table INSERT through the same registration and storage binding. Applications use register_storage once and continue using the ordinary format APIs.
• Provide separate object_store and OpenDAL adapters. Existing SDK clients retain their configuration and middleware; backend features control adapter assembly instead of adding conditional branches throughout file operations.
• Allow an OpenDAL-backed file stack without an object_store normal dependency, while preserving native operations such as batched ranges, suffix reads, delimiter listing, and backend writer buffering.
• Update the affected CLI, examples, tests, and serialization call sites, and add integration and dependency checks.

The storage README describes the interfaces, registration model, and current limits. In this PoC, Avro and sequential Arrow IPC buffer complete inputs, including local files, after removing the SDK-specific local payload path.

What is the testing strategy for this PR?

Local validation completed:

• Extended workspace tests: 11,385 Rust tests passed across 76 test targets, and all 513 SQL logic test files completed.
• Storage and adapter tests, plus seven OpenDAL-only integration tests covering the normal format APIs, writes, manifest inputs, custom Parquet readers, registration replacement, and query contexts. The corresponding normal dependency graph contains no object_store package.
• Existing CSV, JSON, and Parquet request snapshots retain their pre-refactor expectations. Additional adapter tests cover native read requests and writer buffer thresholds in datafusion/core/tests/datasource/object_store_access.rs.
• Standalone CLI tests: 70 passed.
• Formatting, whole-workspace/all-target/all-feature Clippy with warnings denied, documentation checks, and the repository lint suite.

These tests cover local and in-memory backends and mocked services; they do not constitute cloud-service qualification. No performance improvement is claimed.

Are there any user-facing changes?

Yes. This PoC makes breaking changes to registration and file I/O extension APIs, including replacing register_object_store with register_storage and removing SDK types from shared file interfaces.

This PoC does not address compatibility or migration. If this direction proves viable, I will separately design a roadmap for migrating the community to the new API. The changes here are intended to make the initial idea concrete and reviewable, not to establish the final API or migration plan.

[Xuanwo]

cc @alamb @AdamGS for a look. As explained in the PR description, I'm just trying to give you a feeling of what datafusion-storage will look like.

TBH, I have many new ideas for this new trait, but I think it's better to establish a good starting point before I go further. Once we have a datafusion-storage crate, it should be easier for the whole community to optimize it based on the real usage in datafusion.