Skip to content

feat: add dense union encoding - #9367

Open
HarukiMoriarty wants to merge 4 commits into
developfrom
nemo/dense-union
Open

feat: add dense union encoding#9367
HarukiMoriarty wants to merge 4 commits into
developfrom
nemo/dense-union

Conversation

@HarukiMoriarty

@HarukiMoriarty HarukiMoriarty commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

GeoArrow mixed-geometry arrays use dense unions, while the Vortex canonical union representation is sparse. This PR adds a generic external dense physical encoding without moving encoding policy into vortex-array.

What changes are included in this PR?

  • Add the external vortex-dense-union encoding for logical DType::Union values.
  • Store row-aligned type IDs and offsets with compact variant children.
  • Preserve DenseUnion through slice, filter, take, and mask operations.
  • Canonicalize unsupported operations to sparse UnionArray using dictionary-backed children without copying payload values.
  • Register DenseUnion for Vortex file serialization and deserialization.
  • Add behavior, validation, serde, and benchmark coverage.

What APIs are changed? Are there any user-facing changes?

  • Adds the vortex-dense-union crate and its DenseUnion construction/accessor APIs.
  • DType::Union and canonical sparse UnionArray remain in vortex-array.
  • This does not yet add a GeoUnion extension or GeoArrow dense-union conversion.

Performance

Representation Median executed take Relative time
Sparse UnionArray 42.29–42.56 µs 1.00×
DenseUnion 88.91–90.35 µs 2.10–2.12×

Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Comment on lines +86 to +97
fn dense_take(bencher: Bencher) {
bencher
.with_inputs(|| (dense_union(), indices()))
.bench_values(|(array, indices)| divan::black_box(array.take(indices).unwrap()));
}

#[divan::bench]
fn sparse_take(bencher: Bencher) {
bencher
.with_inputs(|| (sparse_union(), indices()))
.bench_values(|(array, indices)| divan::black_box(array.take(indices).unwrap()));
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to execute the array here otherwise there is no work that is done

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that's correct, for now it is lazy-take. Fixed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im interested in what the new benchmark results are?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already showed in the PR content. DenseUnion is 2.10-2.12x slower.

Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
@connortsui20

connortsui20 commented Aug 12, 2026

Copy link
Copy Markdown
Member

It seems like this actually has a similar issue to ListView, where we can filter, take, slice the type IDs and offsets, but then we have no way of garbage collecting the unused data (which kind of defeats the purpose of the Dense encoding)?

And if someone inevitably wants that behavior, they actually have no way to express it? I guess technically we can express it by canonicalizing into a sparse encoding by literally rebuilding the whole thing from scratch, but that is very inefficient.

Maybe its time we add that garbage collection array that we've talked about for almost a year?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants