[SPARK-59247][ML] Optimize tree ensemble classification model transform closures

[zhengruifeng]

What changes were proposed in this pull request?

This PR reduces the transform closure size of RandomForestClassificationModel and
GBTClassificationModel. It is the tree-ensemble counterpart to #58454.

The column-expression prediction hooks snapshot only the root nodes, tree weights, class count,
loss, and thresholds needed by each requested output column. Companion-object helpers perform raw
prediction and probability conversion without retaining the complete model. A shared
TreeEnsembleModel helper similarly detaches leaf prediction from the model.

Why are the changes needed?

The default probabilistic classifier hooks and existing leaf UDF invoke bound model methods. Their
closures therefore retain the complete model and parameter graph even though scoring only needs
the tree nodes and a small amount of immutable prediction state. This adds avoidable driver memory
pressure for long-lived Spark Connect servers.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

The following check passed:

build/sbt mllib/compile

No new tests were added because the existing testPredictMethods coverage in
RandomForestClassifierSuite and GBTClassifierSuite exercises all combinations of raw
prediction, probability, and prediction columns.

A temporary local probe trained each model on the same deterministic dataset with 256 rows and
16 features. Both models used 20 trees; Random Forest used maximum depth 5 and GBT used maximum
depth 3. The probe extracted each ScalaUDF.function from the analyzed transform plan and
serialized it with Spark's closure serializer. "All" is the sum of the raw-prediction,
probability, prediction, and leaf UDF closures.

Model/output	Before	After	Reduction
Random Forest raw prediction	75,171 B	20,951 B	72.1%
Random Forest probability	75,204 B	20,959 B	72.1%
Random Forest prediction	75,108 B	20,925 B	72.1%
Random Forest leaf	75,223 B	20,866 B	72.3%
Random Forest all	300,846 B	46,059 B	84.7%
GBT raw prediction	71,108 B	21,164 B	70.2%
GBT probability	71,141 B	21,357 B	70.0%
GBT prediction	71,045 B	21,138 B	70.2%
GBT leaf	71,142 B	20,985 B	70.5%
GBT all	284,576 B	46,814 B	83.5%

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)