feat: prune Variant inputs using DataFusion UDF field requirements

[peterxcli]

Summary

Use the generic UDF input-field capability from apache/datafusion#25013 to prune unused Parquet fields for variant_get and variant_get_field, while preserving encoded fallback values and the original UDF evaluation.

Variant path interpretation, shredding layouts and fallback decisions stay in this repository. DataFusion receives only physical field paths through required_input_fields().

Changes

• Allow literal variant_get and variant_get_field expressions to move into scan projections, including arguments to aggregates such as SUM. Dynamic arguments keep conservative placement.

• Declare required metadata, encoded values at each traversed level, and the selected typed subtree. Unknown or unsupported traversal keeps conservative inputs.

• Normalize untyped shredded extraction to the declared Variant output field. The integration test exposed this existing output-type mismatch even with the new capability disabled.

• Pin DataFusion and its SQL test dependency to peterxcli/datafusion@1180ae33d9903de607cd8dd5cd04c3efbbfa10b4, the tested head of the capability PR. Use Arrow 59.2.0 to match that checkout. The lockfile contains Git sources and requires no local path overrides.

• Add a Parquet integration test exercising both accessor path conventions, aliases, raw and shredded input, typed and untyped output, nulls and encoded fallback.

The DataFusion API is still proposed upstream. This fork PR demonstrates its downstream use; the Git pin can be replaced after a release includes the capability.

Validation

The implementation was validated against a local checkout of the exact pinned DataFusion commit before publication:

• cargo test --profile ci --test parquet_input_fields: passed all 32 configurations of raw/shredded input, dot/single-field paths, capability on/off and pushdown on/off. Actual UDF inputs exclude unused siblings, and the decoder prunes four of five rows when both controls are enabled. A value of 4,000,000,000 verifies encoded fallback when the selected Int32 shredded field cannot represent it.
• cargo clippy --lib --tests -- -D warnings: passed.
• Changed Rust files pass formatting.

With the portable Git pins, cargo test --profile ci --test parquet_input_fields passes both tests locally. The aggregate coverage checks 32 configurations with placement enabled/disabled, preserves nulls and encoded fallback, and verifies that unused siblings disappear only when both placement and input-field pruning are enabled. The same tests pass in the AWS release harness. Changed-file formatting and cargo clippy --profile ci --lib --tests -- -D warnings pass. Broader all-target/all-feature Clippy also encounters an existing SqlHelper::default() warning in unchanged examples/cli.rs.

Known broader-suite failures: 77 unit tests pass and three integer/Boolean coercion tests fail; the SQL suite stops at the corresponding Boolean case. The original downstream source reproduces these failures on the same DataFusion 55 / Arrow 59.2 stack. Repository-wide formatting also reports existing blank-line violations in 13 untouched files. Those unrelated changes are excluded from this PR.

To run the new integration from this branch:

cargo +1.97.0 test --locked --test parquet_input_fields

Aggregate benchmark

On chia-ping-aws1, the partially shredded aggregate runs in 165.49 ms versus 758.71 ms at previous PR head fc1ef311d1270f9016dce07a48f893155fb06c8e (4.58×, warm cache), and 306.28 ms versus 1367.28 ms with advisory cold-cache reads (4.46×). Reader bytes decrease 80.39%. The same-binary placement-disabled control confirms 4.61× and 4.42× gains. Raw-input aggregates and direct projection/filter controls remain approximately unchanged.

This is a synthetic 1,048,576-row workload with wide unrelated fields and encoded fallback for every tenth age. Timings cover execution and collection, with 12 warm trials and six advisory-cold trials per variant; variant order rotates. Planning is recorded separately. Both revisions use the same DataFusion/Arrow pins and retained files. All 249 executions, including warmups and controls, passed result assertions. The storage cache is uncontrolled, so advisory-cold timings do not establish cold physical-media performance. Results and reproducible artifacts remain under ~/oss/variant-pr1-benchmark-20260908/placement-results on the host.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T08:25:45.503601Z	fc1ef31	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc1ef311d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restore a dependency stack that passes the full test suite

With this pinned DataFusion 55 / Arrow 59 stack, three existing integer/Boolean coercion unit tests fail and the SQL logic suite stops at the corresponding Boolean case. Since the inspected .github/workflows/test.yml runs the full cargo test, this dependency switch leaves every PR and main-branch test job failing; retain compatible versions or resolve the coercion regressions before adopting this fork.

Useful? React with 👍 / 👎.