Skip to content

fix percentile reordering data in DescriptiveStatistics and StatUtils - #350

Open
insaf021 wants to merge 2 commits into
apache:masterfrom
insaf021:descriptivestats-percentile-copy
Open

insaf021 wants to merge 2 commits into
apache:masterfrom
insaf021:descriptivestats-percentile-copy

Conversation

@insaf021

@insaf021 insaf021 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

With the default percentile implementation, on master:

DescriptiveStatistics stats = new DescriptiveStatistics(5);   // add 5, 4, 3, 2, 1
stats.getPercentile(50);   // 3.0
stats.getValues();         // [1.0, 2.0, 3.0, 4.0, 5.0]    expected [5.0, 4.0, 3.0, 2.0, 1.0]
stats.addValue(100);
stats.getValues();         // [2.0, 3.0, 4.0, 5.0, 100.0]  expected [4.0, 3.0, 2.0, 1.0, 100.0]

double[] x = {5, 4, 3, 2, 1};
StatUtils.percentile(x, 50);   // x is now [1.0, 2.0, 3.0, 4.0, 5.0]

Statistics.Percentile and StatUtils build their Commons Statistics Quantile with withCopy(false), so it partially sorts the array it is given. DescriptiveStatistics.getPercentile evaluates it through apply, which passes the stored window itself, so a read-only query reorders the data. From then on getValues() and getElement(i) no longer follow insertion order, and a rolling window discards the smallest value on the next addValue rather than the oldest, so later statistics are computed over the wrong values (mean 22.8 instead of 22.0 above). toString() reports the median and reorders the window the same way. The Percentile class replaced in MATH-1671 (fb1c36b) worked on a copy, so both call sites changed behaviour in that commit.

Both quantiles now use withCopy(true); Quantile.withDefaults() is in place, so the flag has to be explicit. DescriptiveStatisticsTest.testPercentileDoesNotReorderValues (also run by SynchronizedDescriptiveStatisticsTest) and StatUtilsTest.testPercentileDoesNotModifyInput fail on master and pass with the fix, and the default mvn build is green.

The Commons Statistics Quantile used by Statistics.Percentile and
StatUtils was built with withCopy(false), so it partially sorts the
array it is given. DescriptiveStatistics passes its stored window, so
getPercentile (and toString) reordered the values: getValues no longer
returned them in insertion order, and a rolling window then discarded
the smallest value instead of the oldest. StatUtils.percentile
reordered the caller's array. The Percentile class replaced in
MATH-1671 worked on a copy.
@aherbert

aherbert commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Multiple calls to getPercentile will have significant performance overhead using a copy.

StatUtils is not documented to be non-destructive to the array order. So an option is to document the current behaviour. If we keep this change to revert to Math3 behaviour the method should be documented to have performance overhead.

However the DescriptiveStatistics is documented to return getValues() in order added. So this is breaking behaviour. The patch is valid but users should be aware of the implications of calling repeatedly (e.g. with different percentile) when the underlying data has not changed. This should be documented in the getPercentile(double) method.

The previous Percentile implementation used a working array copy and maintained pivots to improve performance of the single-pivot quick select method. The current class could be modified to cache a working copy and a reference to input values and the [begin, length) to identify repeat calls. Feel free to examine this idea. This moves the copy operation from Quantile to be managed more efficiently by Percentile class.

The simpler alternative is to document exactly what is happening on each call and provide the user with advice to get the values array once and call Quantile themselves with all desired percentiles in a single call.

@insaf021

Copy link
Copy Markdown
Contributor Author

Kept the copy and documented it. getPercentile(double) now says each call copies and re-selects even when the data hasn't changed, and shows getting the values once and passing all the probabilities to Quantile.evaluate. The example sets EstimationMethod.HF6, since Quantile.withDefaults() is HF8 and would otherwise give different numbers from getPercentile.

For StatUtils I kept the Math3 behaviour, as the rest of StatUtils only reads its input, and documented the per-call copy on both percentile overloads with the same pointer to Quantile. If you'd rather leave it in place and document the reordering instead, that's a one-line revert of the StatUtils hunk plus dropping testPercentileDoesNotModifyInput.

On caching a working copy: keying it on the array reference and [begin, length) isn't safe for DescriptiveStatistics. replaceMostRecentValue goes through substituteMostRecentElement, which writes into the same internal array at the same start and length, and removeMostRecentValue followed by addValue does the same, so a repeat getPercentile would hit the cache and return the old value. It would need DescriptiveStatistics to invalidate the cache on every mutation, so I've left it out of this PR. Happy to look at it separately if you think it's worth it.

@aherbert

Copy link
Copy Markdown
Contributor

I agree with using the javadoc to direct users to the new Quantile method. It simplifies this code and alerts users to a more efficient way to obtain many percentiles.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.23%. Comparing base (f554608) to head (ba1aa9e).
⚠️ Report is 420 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #350      +/-   ##
============================================
+ Coverage     86.54%   87.23%   +0.69%     
+ Complexity     9787       91    -9696     
============================================
  Files           532      499      -33     
  Lines         35516    33542    -1974     
  Branches       6194     5856     -338     
============================================
- Hits          30738    29262    -1476     
+ Misses         3518     3160     -358     
+ Partials       1260     1120     -140     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants