Nested CV is Memory-inefficient

[pgoodling-usgs]

Hello,

I noticed that when using nested_cv (or bringing over a custom nested resampling scheme via caret2rsample(), the memory does scale linearly with the number of folds. This is in contrast to a single resampling (for example the bootstrap resampling example on the rsample documentation landing page), where a feature of the package is that the memory doesn't blow up when resampling a dataset many times.

I adapted that example to show what I'm talking about:

library(rsample)
library(mlbench)

data(LetterRecognition)
lobstr::obj_size(LetterRecognition)
# 2,644,640 B

set.seed(35222)
#For an example, an outer 5 fold cv with each outer fold having an inner 2 fold cv
nested <- rsample::nested_cv(LetterRecognition, outside = vfold_cv(times = 5),inside=vfold_cv(times = 2))
lobstr::obj_size(nested)
#34,434,200 B

# Object size per resample - Actually slightly bigger than what we started with
lobstr::obj_size(nested)/nrow(nested)
#3,443,420 B


# Fold increase is > 10
as.numeric(lobstr::obj_size(nested)/lobstr::obj_size(LetterRecognition))
#13.02037

Unless I'm missing something, nested resampling on a large dataset doesn't seem to be possible, since the memory required will quickly add up.

[juliasilge]

Interesting result here that doing 10 x 5 also results in a fold increase of on the order of 10:

library(rsample)
library(mlbench)

data(LetterRecognition)
lobstr::obj_size(LetterRecognition)
#> 2,644,640 B

set.seed(123)
# nested scheme that is 10 * 5
nested <- nested_cv(LetterRecognition, outside = vfold_cv(v = 10), inside = bootstraps(times = 5))
nested
#> # Nested resampling:
#> #  outer: 10-fold cross-validation
#> #  inner: Bootstrap sampling
#> # A tibble: 10 × 3
#>    splits               id     inner_resamples     
#>    <list>               <chr>  <list>              
#>  1 <split [18000/2000]> Fold01 <bootstraps [5 × 2]>
#>  2 <split [18000/2000]> Fold02 <bootstraps [5 × 2]>
#>  3 <split [18000/2000]> Fold03 <bootstraps [5 × 2]>
#>  4 <split [18000/2000]> Fold04 <bootstraps [5 × 2]>
#>  5 <split [18000/2000]> Fold05 <bootstraps [5 × 2]>
#>  6 <split [18000/2000]> Fold06 <bootstraps [5 × 2]>
#>  7 <split [18000/2000]> Fold07 <bootstraps [5 × 2]>
#>  8 <split [18000/2000]> Fold08 <bootstraps [5 × 2]>
#>  9 <split [18000/2000]> Fold09 <bootstraps [5 × 2]>
#> 10 <split [18000/2000]> Fold10 <bootstraps [5 × 2]>

lobstr::obj_size(nested)
#> 31,517,216 B
as.numeric(lobstr::obj_size(nested) / lobstr::obj_size(LetterRecognition))
#> [1] 11.91739

^{Created on 2022-03-17 by the reprex package (v2.0.1)}

Doing a scheme that is 10 x 100 results in a fold increase of ~40:

library(rsample)
library(mlbench)

data(LetterRecognition)
lobstr::obj_size(LetterRecognition)
#> 2,644,640 B

set.seed(123)
# nested scheme that is 10 * 5
nested <- nested_cv(LetterRecognition, outside = vfold_cv(v = 10), inside = bootstraps(times = 100))
nested
#> # Nested resampling:
#> #  outer: 10-fold cross-validation
#> #  inner: Bootstrap sampling
#> # A tibble: 10 × 3
#>    splits               id     inner_resamples       
#>    <list>               <chr>  <list>                
#>  1 <split [18000/2000]> Fold01 <bootstraps [100 × 2]>
#>  2 <split [18000/2000]> Fold02 <bootstraps [100 × 2]>
#>  3 <split [18000/2000]> Fold03 <bootstraps [100 × 2]>
#>  4 <split [18000/2000]> Fold04 <bootstraps [100 × 2]>
#>  5 <split [18000/2000]> Fold05 <bootstraps [100 × 2]>
#>  6 <split [18000/2000]> Fold06 <bootstraps [100 × 2]>
#>  7 <split [18000/2000]> Fold07 <bootstraps [100 × 2]>
#>  8 <split [18000/2000]> Fold08 <bootstraps [100 × 2]>
#>  9 <split [18000/2000]> Fold09 <bootstraps [100 × 2]>
#> 10 <split [18000/2000]> Fold10 <bootstraps [100 × 2]>

lobstr::obj_size(nested)
#> 100,622,336 B
as.numeric(lobstr::obj_size(nested) / lobstr::obj_size(LetterRecognition))
#> [1] 38.04765

^{Created on 2022-03-17 by the reprex package (v2.0.1)}

Looks like the current approach does scale up in a sense but looks pretty bad when you are dealing with a small number of resamples.

[pgoodling-usgs]

Aha! That is true - I didn't expand my search to the upper end of the data. I did some additional tests and it does seem like there are significant memory efficiencies involved as the number of inner folds increases, not as much for the outer folds.

library(rsample)
#> Warning: package 'rsample' was built under R version 4.1.2
library(mlbench)
#> Warning: package 'mlbench' was built under R version 4.1.3
library(ggplot2)
#> Warning: package 'ggplot2' was built under R version 4.1.2

data(LetterRecognition)

#Data Size in Megabytes
datasize<-as.numeric(lobstr::obj_size(LetterRecognition))/1E6

#Number of outer folds
outer_n<-c(2,5,10,50,100)

#Number of Inner Folds
inner_n<-c(2,5,10,50,100)

#Create output object
output_boostraps<-expand.grid(outer_n=outer_n,inner_n=outer_n)
output_boostraps$total_n<-output_boostraps$outer_n+output_boostraps$inner_n
output_boostraps$MB_tot<-NA
output_boostraps$MB_per_row<-NA

for(i in seq(nrow(output_boostraps))){

set.seed(35222)

nested_bootstraps <- rsample::nested_cv(LetterRecognition,
                             outside = vfold_cv(v = output_boostraps[i,"outer_n"]),
                             inside = bootstraps(times = output_boostraps[i,"inner_n"]))

#Store Sizes
output_boostraps$MB_tot[i]<-as.numeric(lobstr::obj_size(nested_bootstraps))/1E6
output_boostraps$MB_per_row[i]<-as.numeric(lobstr::obj_size(nested_bootstraps)/nrow(nested_bootstraps))/1E6


}

#Total Memory
ggplot(output_boostraps)+geom_line(aes(x=outer_n,
                             y=MB_tot/datasize,
                             color=factor(inner_n)))+
  labs(title = "Total Memory",
       x = "# of outer folds",
       y = "Memory Ratio \n (n-fold increase in size over original data)",
       color = "# of inner folds")

#Memory Per Row
ggplot(output_boostraps)+geom_line(aes(x=outer_n,
                                       y=MB_per_row/datasize,
                                       color=factor(inner_n)))+
  labs(title = "Memory per Row",
       x = "# of outer folds",
       y = "Memory Ratio per row\n (n-fold increase in size over original data)",
       color = "# of inner folds")

^{Created on 2022-03-18 by the reprex package (v2.0.1)}

[juliasilge]

Thank you so much for those visualizations! 🙌

We'll take a look at this and see if we can improve the strategy for nested CV at all (especially for those outer folds). For expectation setting, it isn't going to be absolute top priority for now, but we'll see what we can do.

[jmgirard]

The growth here is real, and it comes down to a single line.

nested_cv() builds each outer fold's inner resamples by materializing that
fold's analysis set, so the object ends up carrying an extra copy of your data
for every outer fold. The line is in inside_resample():

inside_resample <- function(src, cl, env) {
  cl <- rlang::call_modify(cl, data = as.data.frame(src))
  eval(cl, envir = env)
}

as.data.frame() on an rsplit hands back the analysis rows as a brand new
frame, and the inner rset then indexes that copy instead of the original data.
That's why the cost scales with the outer fold count and stays nearly flat in
the inner one — more inner folds only add index vectors. It's the same shape as
the plots above, where the lines separate by outer folds and barely move with
inner ones.

Here's everything the object holds:

term	bytes
one shared copy of the source data	data_bytes
v analysis frames, each holding (v-1)/v of the rows	data_bytes * (v - 1)
outer analysis indices	4 * n * (v - 1)
inner analysis indices	4 * n * (v - 1) * (inner_v - 1)

Only analysis indices show up because out_id is NA throughout — assessment
rows are derived as the complement rather than stored. The terms collapse to:

data_bytes * v + 4 * n * (v - 1) * inner_v

One caveat on the 13× figure

Worth knowing before anyone sizes a fix against it: the reprex measures a 10 × 10
design, not the 5 × 2 its text describes.

vfold_cv() has no times argument. Its fold count is v, and repeats is
what repeats the scheme. times belongs to the Monte Carlo and bootstrap
functions — bootstraps() among them — and the landing-page bootstrap example
the reprex says it adapted is one of those, so it's an easy slip to make. Back in
2022 times fell into ... and was dropped silently, leaving both levels at the
default v = 10.

The reprex's own output shows it: obj_size(nested) / nrow(nested) prints
3,443,420, which is 34,434,200 / 10. Five outer folds would have divided by
five.

Current rsample catches this. vfold_cv() opens with check_dots_empty(), so
the original call errors outright now:

Error in `rsample::vfold_cv()`:
! `...` must be empty.
✖ Problematic argument:
• times = 5

Measurements

Same dataset as the issue — mlbench::LetterRecognition, 20,000 × 17,
2,644,640 B. rsample 1.3.2, R 4.6.1, aarch64-apple-darwin25.4.0, seed 35222. I
rebuilt both schemes explicitly, since the original call no longer runs.

scheme	outer folds	bytes	× data	model	residual
10 × 10 (what the reprex built)	10	33,715,400	12.749	12.722	−0.20%
5 × 2 (what its text describes)	5	13,871,840	5.245	5.242	−0.06%
10 × 5	10	30,078,080	11.373	11.361	−0.11%

The model lands just under each measurement, which is what you'd want — it counts
storage and not the per-object overhead of the rsplit lists and tibbles.

The 12.749 is a shade under the 13.020 reported in 2022, and that gap resolves
too: 34,434,200 − 33,715,400 = 718,800 B, against 720,000 B for ten explicit
integer row-name vectors of 18,000 elements. Those frames carry compact row names
today — .row_names_info() returns −18,000. So it's row-name storage rather than
any change in the behaviour itself.

Could it be leaner?

Yes, and nothing about rsplit requires the copy. An inner split can index the
original data directly, as long as it carries an explicit assessment index.
out_id can't stay NA in that case, because the complement of an index into
the whole dataset would pull in the outer fold's assessment rows. The object
then grows only with index vectors:

data_bytes + 4 * n * (v - 1) * (inner_v + 1)

That models 3.995× at 10 × 10, against the 12.749× measured above. To be clear,
3.995 is modelled and not measured — I only measured the lean form at 10 × 5,
where it came to 2.649× against 2.633× modelled, so the model runs a little under
there as well.

One thing that cost me time, in case it saves you some: building these splits
from scratch with make_splits() and manual_rset() looks like the obvious
route, and it isn't. make_splits() returns a bare rsplit, so you lose the
split subclass and the per-split id tibble that labels() and
add_resample_id() read, and manual_rset() drops id2. Rewriting the data,
in_id and out_id fields of the splits nested_cv() already produced keeps
all of that intact, and splits built that way come out row-identical to the
current ones.

All of this fell out of building a memory-lean nested resampling constructor on
top of rsample (nestedtune). The
11.373× above is a committed test oracle there, and the field-rewriting approach
is what it ships, so everything here is reproducible from a script in that repo
if you'd like to re-run it.

I'm posting this as a diagnosis rather than a PR because whether nested_cv()
should change shape is your call, and there's a backward-compatibility question
in it I can't answer from outside. Happy to put a PR together if that would help.