[thorfour]

Summary

• Tracking Issue: Support file stats on nested fields #6389

Being able to perform file level pruning on nested columns speeds up queries.

Changes

Adds post order stats to the file stats footer for nested columns. It adds a new is_nested: bool = false to the stats set flatbuffer for backward compatibility.

Additionally it modifies the way the stats footer is accumulated to use the same AggRef functions that the zone maps use as well.

Only goofy thing it it also has to track truncatable stats separately because the aggregate partials don't have a way to express exactness.

[codspeed]

Merging this PR will degrade performance by 6.04%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 2 improved benchmarks
❌ 8 regressed benchmarks
✅ 2398 untouched benchmarks
⏩ 12 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Walltime	threshold_neon[8]	1.3 µs	2.7 µs	-49.59%
❌	Walltime	filter_neon[8]	893 ns	1,491 ns	-40.11%
❌	Walltime	threshold_neon[64]	2.6 µs	3.6 µs	-28.43%
❌	Walltime	filter_neon[80]	2.5 µs	3.4 µs	-26.31%
❌	Walltime	filter_neon[64]	1.7 µs	2.2 µs	-21.41%
❌	CPU Simulation	test_scan_filter[int]	3.1 ms	3.7 ms	-18.38%
❌	Walltime	threshold_neon[80]	3.7 µs	4.5 µs	-17.83%
❌	Walltime	bitpack_blocked_compress_avx2	6.7 µs	7.6 µs	-11.66%
⚡	CPU Simulation	bench_compare_sliced_dict_primitive[(2000, 10000)]	67.2 µs	13 µs	×5.1
⚡	CPU Simulation	density_sweep_dense_runs[0.001]	62.6 µs	44.5 µs	+40.52%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing polarsignals:statistices-nested-columns (4341dbe) with develop (d270200)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

I think for maximum compatibility this has to be a new top level field in https://github.com/vortex-data/vortex/pull/9775/changes#diff-9c5a36ce8f55480706984e8996f192cd3abbdd1b97afa6f702f646b3ce42e039L25. The problem you run into is that old readers already shipped code that make assumption about existing footer statistics. One issue is that they validate field count and if the count doesn't add up they will assume it's a corrupted file. The writer could optionally populate both of the segments and the new version will only ever read the new field.

If you go that route you will see all the places that make assumptions about field stats that need fixing, i.e. datafusion, duckdb and jni

[joseph-isaacs]

We also want to use aggregate partials here?

[thorfour]

@joseph-isaacs I've modified the summary stats to use aggregate partials.
@robert3005 I've moved the stats to a new top-level field, and left the legacy stats behavior. The writer can opt out of writing the legacy stats.

[thorfour]

One thing that we have to do here is to ban any file stats for dtypes that have structural nullability so we cannot allow stats for nullable lists and structs all the way through since those stats would be wrong right now. We need to fix propagation of nulls through structural types before they're allowed

I didn't realize stats were broken for nullable nested types. I think to support this we'd need to change the footer to support marking leaves as skipped. Which could open up an api to allow users to specify per-columns stats (or not recording stats at all for certain columns). Or we can leave it as is and just wait for the fix of nulls in structural types to land first. wdyt?

We can implement this by having skipped fields point to empty AggregateSets. Is there an issue I can write in the comments for the code to skip those fields? Looks like #9205 covers it

[robert3005]

Right now we have https://github.com/vortex-data/vortex/blob/develop/vortex-layout/src/layouts/file_stats.rs#L468-L471. For non nullable Structs things are correct but at the moment vortex allows you to have parent nullability different from child which needs either changing or adding handling for

[thorfour]

I'm on PTO next week, will pick this back up after vacation

[thorfour]

One thing that we have to do here is to ban any file stats for dtypes that have structural nullability so we cannot allow stats for nullable lists and structs all the way through since those stats would be wrong right now. We need to fix propagation of nulls through structural types before they're allowed

I've added an Unrecorded type to StatsNode such that these fields don't contain stats, but are recorded in the post order layout.

[robert3005]

I think we should fix the Sum, otherwise this is good to go