feat: add dense union encoding - #9367
Conversation
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
| fn dense_take(bencher: Bencher) { | ||
| bencher | ||
| .with_inputs(|| (dense_union(), indices())) | ||
| .bench_values(|(array, indices)| divan::black_box(array.take(indices).unwrap())); | ||
| } | ||
|
|
||
| #[divan::bench] | ||
| fn sparse_take(bencher: Bencher) { | ||
| bencher | ||
| .with_inputs(|| (sparse_union(), indices())) | ||
| .bench_values(|(array, indices)| divan::black_box(array.take(indices).unwrap())); | ||
| } |
There was a problem hiding this comment.
You need to execute the array here otherwise there is no work that is done
There was a problem hiding this comment.
Ah, that's correct, for now it is lazy-take. Fixed.
There was a problem hiding this comment.
im interested in what the new benchmark results are?
There was a problem hiding this comment.
Already showed in the PR content. DenseUnion is 2.10-2.12x slower.
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
|
It seems like this actually has a similar issue to And if someone inevitably wants that behavior, they actually have no way to express it? I guess technically we can express it by canonicalizing into a sparse encoding by literally rebuilding the whole thing from scratch, but that is very inefficient. Maybe its time we add that garbage collection array that we've talked about for almost a year? |
Rationale for this change
GeoArrow mixed-geometry arrays use dense unions, while the Vortex canonical union representation is sparse. This PR adds a generic external dense physical encoding without moving encoding policy into
vortex-array.What changes are included in this PR?
vortex-dense-unionencoding for logicalDType::Unionvalues.UnionArrayusing dictionary-backed children without copying payload values.What APIs are changed? Are there any user-facing changes?
vortex-dense-unioncrate and itsDenseUnionconstruction/accessor APIs.DType::Unionand canonical sparseUnionArrayremain invortex-array.Performance
takeUnionArray