nestedtune

You tune a model with cross-validation, keep the setting with the best score, and then score that setting once on a held-out test set. That test score is one number from one split of the data, and nothing in it says whether the split is representative. nestedtune scores the whole tune-and-fit procedure on several outer splits instead. The outer scores give the mean across those splits and show how far the score moves from one split to the next. Each outer fold tunes on its own inner resamples with tune or finetune, so no outer score is the score that picked its fold’s winner. The score that picks a winner tends to be optimistic, because the winner was picked for scoring well. nestedtune keeps what every fold chose.

The mean of the outer scores is the number to report for the model you deploy. That model is the same procedure run once more on all the data, so there is no second number to compute for it.

Installation

# install.packages("pak")
pak::pak("tidymodels/nestedtune")

Example

library(tidymodels)
library(nestedtune)

set.seed(1)
folds <- nested_resamples(
  mtcars,
  outside = vfold_cv(v = 5),
  inside = vfold_cv(v = 5)
)

wf <- workflow(
  mpg ~ .,
  rand_forest(mtry = tune(), min_n = tune()) |>
    set_engine("ranger") |>
    set_mode("regression")
)
grid <- expand.grid(mtry = c(2L, 5L, 8L), min_n = c(2L, 10L))

set.seed(2)
res <- nested_tune_grid(wf, folds, grid = grid)

# The number to report for the model you deploy.
collect_metrics(res)
#> # A tibble: 2 × 5
#>   .metric .estimator  mean     n std_err
#>   <chr>   <chr>      <dbl> <int>   <dbl>
#> 1 rmse    standard   2.46      5  0.445 
#> 2 rsq     standard   0.844     5  0.0267

# The model to deploy, the same procedure run once more on all the data.
set.seed(3)
final <- nested_final_fit(wf, res)
predict(final, new_data = mtcars[1:3, ])
#> # A tibble: 3 × 1
#>   .pred
#>   <dbl>
#> 1  20.9
#> 2  20.9
#> 3  23.8

Supported resampling designs

The table covers 15 resampling functions from rsample. It says whether each one works as the outer loop and as the inner loop of a nested design.

Function	Outer loop	Inner loop
vfold_cv()	Yes	Yes
mc_cv()	Yes	Yes
group_vfold_cv()	Yes	Yes
group_mc_cv()	Yes	Yes
clustering_cv()	Yes	Yes
bootstraps()	Refused	Yes
group_bootstraps()	Refused	Yes
loo_cv()	No	No
apparent()	No	No
validation_set()	Yes1	No
permutations()	No	No
rolling_origin()	Yes	Yes
sliding_window()	Yes	Yes
sliding_index()	Yes	Yes
sliding_period()	Yes	Yes

Yes means that the test suite runs the design in that loop through nested_tune_grid() and that every outer fold completes. It holds for the arguments those tests use. For a Yes cell in the first eleven rows, the design in the other loop is a v-fold design. For a Yes cell in the four time-series rows, it is a time-series design. An inner time-series design is tested only with a time-series outer design. The “Time-series designs” section of ?nested_resamples says which other functions are tested on each time-series design.

Refused means that nested_resamples() stops with an error. An outer bootstrap can put the same row in both the inner analysis set and the inner assessment set, which makes the estimate invalid.

No means that the design gives no valid nested estimate in that loop. As the outer loop, loo_cv() holds out one row per fold. R² then cannot be computed, and the average RMSE is the mean absolute error. As the inner loop, tune refuses loo_cv(), so every outer fold fails. As the outer loop, apparent() scores its one fold on the rows it trained on. As the inner loop, tune reports that no results are available, so every outer fold fails. As the inner loop, validation_set() cannot be built, because it takes a split from initial_validation_split() rather than a data frame. As the outer loop, permutations() gives each fold no assessment set, so every outer fold fails. As the inner loop, tune refuses it, so every outer fold fails.

A cell reads Untested when no test shows whether the design works in that loop.

Learn more:

• Nested cross-validation, the path from a design to a write-up.
• What the estimate means, which quantity the nested number is and how far to trust it.
• Choosing the inner tuner, the Bayesian search, the two racing searches, simulated annealing and a set of workflows on one design.
• Reading the results, every column of the results object and every function that reads it.
• Running the outer loop in parallel, the same call on a pool of mirai daemons.

Footnotes

1. Build the validation set before the call, with split <- initial_validation_split(data) and validation_set(split). Then pass rbind(training(split), validation(split)) as data, because those are the rows the validation set refers to. A call such as outside = validation_set() cannot be built, because the function takes a split rather than a data frame. ↩