Skip to content

[Python] Fix DataFrame skew() and kurtosis() with a single non-empty partition - #41284

Open
sohailshk wants to merge 2 commits into
apache:masterfrom
sohailshk:fix/dataframe-skew-kurtosis-single-partition
Open

sohailshk wants to merge 2 commits into
apache:masterfrom
sohailshk:fix/dataframe-skew-kurtosis-single-partition

Conversation

@sohailshk

Copy link
Copy Markdown
Contributor

Fixes #41283

Series.skew() and Series.kurtosis() merge per-partition moments in combine_moments. The result used combined_n, which is only assigned when two non-empty partitions are merged, so when only one partition holds data (for example a small input processed in one bundle) the call raised UnboundLocalError: cannot access local variable 'combined_n'. After the merge loop n already holds the total count (it equals combined_n after every merge), so the result now uses n; results for several partitions are unchanged.

Tests: test_series_skew_kurtosis_single_partition evaluates both on one partition (distributed=False); it fails on master with the UnboundLocalError and passes with the change. pytest apache_beam/dataframe/frames_test.py: 460 passed, 19 skipped. With to_dataframe on a beam.Create of 6 floats, both calls failed on master on the default runner and on the FnApiRunner; after the change they return 0.9191843544212499 and -0.25414322711619963, the same as pandas.

…partition

combine_moments used `combined_n`, which is only assigned when two
non-empty partitions are merged. When one partition holds all the data
(for example a small input in one bundle), skew() and kurtosis() raised
UnboundLocalError. The accumulated count `n` equals `combined_n` after
every merge, so use it.
@github-actions github-actions Bot added python size/s S: 10-49 lines changed labels Nov 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves an UnboundLocalError in DeferredSeries.skew() and DeferredSeries.kurtosis() when all non-empty data resides within a single partition. Replacing combined_n with n in the final return expressions of combine_moments is mathematically sound and cleanly handles both single-partition and multi-partition inputs.

🔍 General Feedback

  • The fix in sdks/python/apache_beam/dataframe/frames.py accurately uses n, which holds the total sample count after iterating through all partitions regardless of whether the merge branch (else) was entered.
  • The added unit test in sdks/python/apache_beam/dataframe/frames_test.py with distributed=False directly covers the single-partition execution path.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.
Backup reviewer: tvalentyn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: DataFrame API skew() and kurtosis() raise UnboundLocalError when all the data is in one partition

1 participant