Skip to content

Add list_transform - #10503

Open
mhk197 wants to merge 3 commits into
developfrom
mk/list-transform-v3
Open

Add list_transform#10503
mhk197 wants to merge 3 commits into
developfrom
mk/list-transform-v3

Conversation

@mhk197

@mhk197 mhk197 commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

list_transform(list, lambda) applies a one-parameter lambda to every element of a List (in any encoding), keeping each list's length and validity.

Changes

  • ListTransform's options are ListTransformOptions: the lambda as written (Lambda) in an expression, or bound to the element's dtype (BoundLambda) in a bound expression or an array. Its only child is the list.
  • bind_options binds the lambda to the element's dtype when the node is bound, so execute only runs it. A FixedSizeList, or a lambda with anything but exactly one parameter, is rejected there and in return_dtype. A lambda already bound to another element dtype is an error.
  • The result keeps the input's shape: a ListArray stays a ListArray and a list view stays a list view, with the same offsets (and sizes) and validity around the new elements.
  • A lambda that can't fail runs once over the whole elements buffer, including elements no valid list shows (under a null list, in a gap of a list view, the rest of a slice).
  • A lambda that can fail runs only on the elements valid lists show:
    • a ListArray whose null lists hold no elements is sliced from its first list to its last, with its offsets shifted down;
    • a list view marked as convertible to a list without copying, whose null lists are empty, is trimmed the same way;
    • anything else is rebuilt: null lists are emptied and list_from_list_view lays the lists out back to back.
  • The function is strict (a null list gives a null result), it can fail only if the lambda can, and it is not serializable yet.
list_transform(col("l"), Lambda::new(["x"], checked_add(var("x"), lit(1_i32)))?)
// vortex.list.transform($.l, (x) -> (@x + 1i32))

Benchmark

100k lists, ~800k i32 elements, median µs, local scratch divan benchmark (not committed). "Before" always made the lists exact.

Input Lambda Before After
Exact list either 126–129 119–129
Null lists holding elements can't fail 1,288 130
List view, lists out of order can't fail 1,324 84
Canonical list view either 130–141 73–84
Sliced list can fail 183 122

Testing

New tests cover a sliced list, a list view with a leftover in front and one with a gap (a lambda that can fail sees only the lists' elements), and a lambda that can't fail running over the elements in place.

@mhk197 mhk197 added the changelog/feature A new feature label Oct 14, 2026
@codspeed

codspeed Bot commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will degrade performance by 22.02%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 15 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 3 improved benchmarks
❌ 6 regressed benchmarks
✅ 2399 untouched benchmarks
⏩ 12 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime filter_neon[8] 936 ns 1,994 ns -53.06%
❌ Walltime threshold_neon[8] 1.3 µs 2.8 µs -53%
❌ Walltime filter_neon[64] 1.8 µs 2.7 µs -33.88%
❌ Walltime filter_neon[80] 2.5 µs 3.6 µs -30.03%
❌ Walltime threshold_neon[64] 2.6 µs 3.5 µs -26.24%
❌ Walltime threshold_neon[80] 3.6 µs 4.8 µs -25.56%
⚡ CPU Simulation decode_primitives[i64, (1000, 8)] 37.6 µs 24.7 µs +52.4%
⚡ CPU Simulation random_i128[0.01] 5.2 µs 4.7 µs +11.93%
⚡ CPU Simulation take[(256, 2, 63)] 109.7 µs 98.4 µs +11.48%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing mk/list-transform-v3 (5ac5458) with develop (e083440)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from b532ee2 to 42ca3cd Compare October 19, 2026 14:13
@mhk197 mhk197 mentioned this pull request Oct 19, 2026
Closed
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from 42ca3cd to fca3b22 Compare October 19, 2026 14:19
@mhk197
mhk197 added this pull request to stack #10614 October 19, 2026 14:32
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch 2 times, most recently from 6d06424 to b9493f8 Compare October 19, 2026 16:40
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch 3 times, most recently from 1c8172e to f687da3 Compare October 19, 2026 20:11
@mhk197 mhk197 mentioned this pull request Oct 19, 2026
Merged
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from f687da3 to 941e01d Compare October 20, 2026 18:45
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from 941e01d to 01f5e6c Compare October 20, 2026 19:55
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from 01f5e6c to c180f45 Compare October 21, 2026 15:30
@mhk197
mhk197 deployed to duckdb-build October 21, 2026 15:30 — with GitHub Actions Active
@mhk197
mhk197 requested a review from joseph-isaacs October 21, 2026 15:50
@mhk197
mhk197 marked this pull request as ready for review October 21, 2026 15:50
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from c180f45 to b672f3c Compare October 21, 2026 18:22
@mhk197 mhk197 mentioned this pull request Oct 22, 2026
Draft
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from b672f3c to 83f89b4 Compare October 22, 2026 19:05
@mhk197 mhk197 mentioned this pull request Oct 22, 2026
Draft
Base automatically changed from mk/lambdas-v3 to develop October 23, 2026 14:02
mhk197 added a commit that referenced this pull request Oct 23, 2026
@mhk197
Add lambdas and variables for higher-order scalar functions (#10502)
ced05c2 
## Summary

Adds lambdas, the function arguments of higher-order scalar functions
such as `list_transform` (#10503). A lambda is a list of parameter names
and a body that reads them through variables. A function that takes a
lambda keeps it in its options, binds it once when its node is bound,
and runs the bound lambda over values it makes from its inputs.

## Variables

- `var("x")` builds `Expression::Variable("x")`: a leaf, like `$`, that
reads the parameter `x` of the enclosing lambda.
- It displays as `@x`, so `@x` (a parameter) and `$.x` (a field of the
input) never look alike.
- A variable exists only in a written body. Binding the lambda turns
`@x` into `$.x`, a field of the struct of the lambda's inputs (see
below), so `BoundExpression` is unchanged.
- Anywhere else it fails to bind, type, apply and serialize with
"variable @x is not a parameter of an enclosing lambda".

## Lambdas

```rust
let lambda = Lambda::new(["x"], checked_add(var("x"), lit(1_i32)))?;
// (x) -> (@x + 1i32)
let bound = lambda.bind(&[DType::Primitive(PType::I32, NonNullable)])?;
// (x) -> ($.x + 1i32): an ordinary bound expression over the struct {x: i32}
```

- `Lambda` is the lambda as written: `params()` and `body()`, an
`Expression`.
- `Lambda::new` rejects a repeated parameter, and a body that reads from
outside the lambda: `$` or a column, another lambda's parameter, or the
scan's `row_idx()` and `row_count()`. Constants are fine, such as the
literal set in `x -> list_contains([2, 3], x)`.
- `bind(param_dtypes)` turns the lambda into an ordinary expression over
a struct of its inputs, one field per parameter: each `@x` becomes
`$.x`, and the body is bound to that struct. It returns a `BoundLambda`.
- `return_dtype(param_dtypes)` gives the body's dtype over the same
struct, without binding it, the way `Expression::return_dtype` does for
a whole expression.
- `BoundLambda` is the lambda bound: `params()` with their dtypes,
`body()`, an ordinary `BoundExpression` over the struct, `dtype()`, the
body's, and `is_bound_to(param_dtypes)`.
- `BoundLambda::evaluate(args, len)` puts one array per parameter in
that struct and applies the bound body to it. The arrays must have the
dtypes the lambda is bound to. It binds nothing, and like any applied
expression, the result is computed later.
- `is_infallible()`, on both, says whether nothing in the body can fail,
so a function knows whether the body may run on values whose results are
never read, such as dictionary values no row uses.

## Using a lambda in a function

A function's options are an enum of the lambda as written and as bound.
Its children are its ordinary inputs.

```rust
enum MyFnOptions {
    Lambda(Lambda),
    Bound(BoundLambda),
}

impl ScalarFnVTable for MyFn {
    type Options = MyFnOptions;

    fn bind_options(&self, options: &MyFnOptions, args: &[DType]) -> VortexResult<Option<MyFnOptions>> {
        match options {
            MyFnOptions::Lambda(lambda) => Ok(Some(MyFnOptions::Bound(lambda.bind(/* param dtypes, from args */)?))),
            MyFnOptions::Bound(lambda) => {
                vortex_ensure!(lambda.is_bound_to(/* param dtypes, from args */), "bound to other dtypes");
                Ok(None)
            }
        }
    }

    fn execute(&self, options: &MyFnOptions, args: &dyn ExecutionArgs, ctx: &mut ExecutionCtx) -> VortexResult<ArrayRef> {
        let MyFnOptions::Bound(lambda) = options else { vortex_bail!("not bound") };
        lambda.evaluate(vec![/* the parameters' arrays, made from args */], len)
    }
}
```

- `ScalarFnVTable::bind_options(options, arg_dtypes)` is new, hidden
from the docs, and does nothing by default. It runs whenever a bound
node is built, and whenever a `ScalarFnArray` is built from its
children, before `return_dtype`. That's where a function binds its
lambda, once, so `execute` finds it bound.
- A rebuilt node calls `bind_options` again, possibly with other dtypes,
so a function checks that options already bound still fit, and errors if
they don't. No rewrite today changes the dtype a lambda is bound to.
- An `Expression` holds the written lambda, and a `BoundExpression` or
an array holds the bound one.
- The body reads only its parameters, so the function's children are
everything it reads. Binding, traversal, `replace`, field pruning and
pushdown see an ordinary node and never look inside the lambda.
- Lambdas nest: a call that takes a lambda is an ordinary node inside
another lambda's body. Its children are part of the outer body, so they
can read the outer parameters, and its own lambda reads only its own.
Binding the outer lambda binds the inner one.

```text
(xs) -> vortex.list.transform(@xs, (y) -> (@y + 1i32))
```

## Not supported yet

- Reading anything outside the lambda, such as `x -> x + $.a`.
- Serializing a lambda: a function that takes one is not serializable
yet.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch 2 times, most recently from a4e9fe2 to 87ed29c Compare October 23, 2026 14:11
mhk197 added 2 commits October 23, 2026 10:48
@mhk197
list_transform scalar fn
9534b7d 
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
skip the list rebuild when lists are already exact
ed77d49 
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 force-pushed the mk/list-transform-v3 branch from 3de16b8 to ed77d49 Compare October 23, 2026 14:49
@mhk197
keep the lists' shape unless a lambda that can fail would see hidden …
5ac5458 
…elements

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
@mhk197
mhk197 removed this pull request from stack #10614 October 23, 2026 18:14
@mhk197 mhk197 mentioned this pull request Oct 23, 2026
Draft

This branch was successfully deployed

1 active (outdated) deployment
duckdb-build — c180f459 Deployed Oct 21, 2026 by mhk197 via Mirror DuckDB to R2 / Mirror DuckDB libduckdb-linux-arm64.zip to R2 #42092
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@joseph-isaacs joseph-isaacs joseph-isaacs approved these changes

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mhk197 @joseph-isaacs