[SYSTEMDS-3863] Add PowerTransformer built-in functions

[WenliangCao]

Summary

This pull request adds PowerTransformer built-ins with separate fit and apply workflows. The implementation supports Yeo-Johnson and Box-Cox transformations, per-column parameter estimation, optional standardization, and reuse of fitted parameters on new data.

Yeo-Johnson is the default method and supports mixed-sign inputs. Box-Cox is available for strictly positive inputs.

API

[Y, lambdas, means, scales] = powerTransform(
    X,
    method="yeo-johnson",
    standardize=TRUE
)

Y = powerTransformApply(
    X,
    lambdas,
    means,
    scales,
    method="yeo-johnson"
)

powerTransform estimates one lambda per column and optionally standardizes the transformed columns. powerTransformApply reuses the fitted lambdas, means, and scales without re-estimation.

Implementation

• Add powerTransform.dml for fitting and transforming feature matrices.
• Add powerTransformApply.dml for applying fitted transformations.
• Support Yeo-Johnson and Box-Cox transformations.
• Keep Yeo-Johnson as the default method.
• Support optional zero-mean, unit-variance standardization.
• Estimate each lambda by minimizing the negative log-likelihood with Brent optimization.
• Treat [-2, 2] as an initial bracket and expand it when necessary, allowing the optimum to lie outside the initial interval.
• Validate method names, fitted-state dimensions, and Box-Cox input requirements.
• Register both functions as SystemDS built-ins.
• Add entries for both functions to the built-in reference documentation.

Validation

mvn -Dtest=BuiltinPowerTransformTest test

Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The tests cover:

• default Yeo-Johnson fitting with mixed-sign and constant columns;
• Box-Cox fitting without standardization;
• Yeo-Johnson and Box-Cox optima outside the initial [-2, 2] interval;
• rejection of non-positive Box-Cox input;
• fitted Yeo-Johnson and Box-Cox apply workflows; and
• standardization state reuse.

Numerical outputs are compared with independent R reference implementations.

Documentation

User-facing signatures, arguments, return values, input requirements, and examples are documented in docs/site/builtins-reference.md.

Experiments

The reproducible evaluation compares no scaling, standard scaling, robust scaling, Yeo-Johnson, and Box-Cox on nine public datasets. Each experiment uses the fixed seeds 13, 37, 73, 101, and 149.

Task	Estimator	Primary metric
Regression	5-nearest-neighbors regressor	RMSE (lower is better)
Classification	5-nearest-neighbors classifier	Macro-F1 (higher is better)
Clustering	K-means (n_init=20)	ARI (higher is better)

Supervised tasks use 80/20 train/test splits; classification is stratified and Parkinsons Telemonitoring uses subject-level group splits. Each transformation is fitted on training data and applied to test data. Clustering uses the full dataset.

The table reports the mean primary metric over five runs. A dash indicates that Box-Cox is not applicable because the feature matrix contains non-positive values.

Task	Dataset	Metric	None	Standard	Robust	Yeo-Johnson	Box-Cox
Regression	California Housing	RMSE	1.0607	0.6486	0.6460	0.6114	—
Regression	Abalone	RMSE	2.2966	2.3575	2.3673	2.3547	2.3580
Regression	Parkinsons Telemonitoring	RMSE	12.8276	13.0932	12.6718	12.7222	—
Classification	Breast Cancer Wisconsin	Macro-F1	0.9303	0.9559	0.9560	0.9618	—
Classification	Wine	Macro-F1	0.6448	0.9734	0.9629	0.9683	0.9683
Classification	HTRU2	Macro-F1	0.9116	0.9312	0.9339	0.9327	—
Clustering	Iris	ARI	0.7302	0.6201	0.5803	0.6410	0.6180
Clustering	Seeds	ARI	0.7166	0.7733	0.7722	0.7759	0.7759
Clustering	Wholesale Customers	ARI	-0.0306	0.1873	-0.0024	0.5264	0.5330

Power transformations rank first or tie for first on four of the nine datasets. Representative results include:

• California Housing: Yeo-Johnson reduces RMSE from 0.6486 with standard scaling to 0.6114.
• Breast Cancer Wisconsin: Yeo-Johnson achieves the highest Macro-F1 at 0.9618.
• Seeds: Yeo-Johnson and Box-Cox tie for the highest ARI at 0.7759.
• Wholesale Customers: Box-Cox achieves an ARI of 0.5330, compared with 0.1873 for standard scaling.

Reproducible artifacts

• Experiment repository
• Experiment report
• Raw results
• Combined summary
• Figures
• English leaderboard workbook
• Dataset sources and citations

Limitations

The downstream effect of preprocessing is dataset- and estimator-dependent. The evaluation uses fixed distance-based estimators, three datasets per task, and no hyperparameter search or statistical significance test. The results therefore demonstrate practical cases where power transformations help, rather than universal superiority over other preprocessing methods.

Related issue

SYSTEMDS-3863

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.44%. Comparing base (e4f0987) to head (130c741).
⚠️ Report is 55 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2499      +/-   ##
============================================
- Coverage     71.47%   71.44%   -0.04%     
+ Complexity    48883    48858      -25     
============================================
  Files          1573     1573              
  Lines        189238   189240       +2     
  Branches      37128    37128              
============================================
- Hits         135261   135200      -61     
- Misses        43530    43581      +51     
- Partials      10447    10459      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.