Plain values in grouped expressions are not broadcast since Polars 0.52

[jotaviobiondo]

Explorer passes plain values in expressions to Polars as one-item series. Examples are the branches of if, the 1 in sum(a) + 1, and the fallback of coalesce. Up to Polars 0.49, Polars took any literal of length 1 as a scalar. Since Polars 0.52 (Explorer v0.12.0), only scalar literals are broadcast over groups. A series literal expands to its own values in each group. On grouped dataframes this means:

• mutate/2 raises lengths don't match
• summarise/2 returns lists instead of one value per group

Fix

Explorer.PolarsBackend.Expression now sends a series of size 1 to Polars as a scalar literal. This applies both to a lazy from_list of one value and to an eager series of size 1. The literal is cast to the series dtype when that differs from the inferred dtype. Values that the NIF has no scalar literal for (such as Decimal and Time) are sent as lit(series).first(), which Polars also takes as one value per group. There are no changes on the Rust side.

Reproduction

Mix.install([{:explorer, "0.13.0"}])

defmodule Repro do
  require Explorer.DataFrame, as: DF

  def run do
    df = DF.new(a: [1, 2, 4], c: [1, 1, 2]) |> DF.group_by(:c, stable: true)

    check("mutate: if with plain branches", fn ->
      DF.mutate(df, x: if(a > 1, do: 1, else: 0))
    end)

    check("mutate: coalesce with a series of size 1", fn ->
      DF.mutate(df, x: coalesce(shift(a, 1), Explorer.Series.from_list([0])))
    end)

    check("summarise: arithmetic with a plain value", fn ->
      DF.summarise(df, x: sum(a) + 1)
    end)

    check("summarise: if over aggregations", fn ->
      DF.summarise(df, x: if(sum(a) > 3, do: sum(a), else: nil))
    end)
  end

  defp check(label, fun) do
    result =
      try do
        fun.() |> DF.to_columns(atom_keys: true) |> Map.get(:x) |> inspect()
      rescue
        e -> "** " <> (e |> Exception.message() |> String.split("\n") |> hd())
      end

    IO.puts("#{label}: #{result}")
  end
end

Repro.run()

Before (v0.13.0):

mutate: if with plain branches: ** Polars Error: lengths don't match: shapes of `self`, `mask` and `other` are not suitable for `zip_with` operation
mutate: coalesce with a series of size 1: ** Polars Error: lengths don't match: shapes of `self`, `mask` and `other` are not suitable for `zip_with` operation
summarise: arithmetic with a plain value: [[4], [5]]
summarise: if over aggregations: [[nil], [4]]

After (this branch):

mutate: if with plain branches: [0, 1, 1]
mutate: coalesce with a series of size 1: [0, 1, 0]
summarise: arithmetic with a plain value: [4, 5]
summarise: if over aggregations: [nil, 4]

This PR was done with the help of AI. I'm not fully sure if there is a better solution here, but seemed the right path. I wanted to at least report the problem problem together with a possible solution.