Automated Machine Learning with tidylearn

A First Run

result <- tl_auto_ml(
  iris, Species ~ .,
  time_budget = 30,
  cv_folds = 3
)

result$leaderboard
#> # A tibble: 8 × 3
#>   model            score evaluation
#>   <chr>            <dbl> <chr>     
#> 1 advanced_svm     0.953 cv        
#> 2 advanced_xgboost 0.953 cv        
#> 3 baseline_forest  0.947 cv        
#> 4 clustered_tree   0.947 cv        
#> 5 clustered_forest 0.947 cv        
#> 6 baseline_tree    0.933 cv        
#> 7 pca_tree         0.913 cv        
#> 8 pca_forest       0.893 cv

Three columns: which model, what it scored, and how the score was obtained.

result$best_model
#> tidylearn Model
#> ===============
#> Paradigm: supervised 
#> Method: svm 
#> Task: Classification 
#> Formula: Species ~ . 
#> 
#> Training observations: 150

result$task
#> [1] "classification"
result$metric
#> [1] "accuracy"
round(as.numeric(result$runtime, units = "secs"), 1)
#> [1] 1.7

Regression is the same call with a numeric response — task = "auto" reads it off the data:

result_reg <- tl_auto_ml(
  mtcars, mpg ~ .,
  time_budget = 30,
  cv_folds = 3
)

result_reg$task
#> [1] "regression"

result_reg$leaderboard
#> # A tibble: 11 × 3
#>    model            score evaluation
#>    <chr>            <dbl> <chr>     
#>  1 pca_linear        2.46 cv        
#>  2 clustered_forest  2.67 cv        
#>  3 baseline_forest   2.83 cv        
#>  4 advanced_ridge    2.89 cv        
#>  5 advanced_lasso    3.28 cv        
#>  6 clustered_linear  3.61 cv        
#>  7 pca_tree          3.79 cv        
#>  8 pca_forest        3.97 cv        
#>  9 baseline_tree     4.31 cv        
#> 10 clustered_tree    4.54 cv        
#> 11 baseline_linear   4.61 cv

How the Search Works

Four phases run in order, each gated on the remaining budget and on the use_reduction and use_clustering toggles.

Phase	What it does	Classification	Regression
1. Baselines	Standard models on the raw features	tree, logistic*, forest	tree, linear, forest
2. PCA variants	PCA preprocessing, then the baselines	pca_tree, pca_logistic*, pca_forest	pca_tree, pca_linear, pca_forest
3. Cluster variants	Cluster assignment added as a feature	clustered_tree, clustered_logistic*, clustered_forest	clustered_tree, clustered_linear, clustered_forest
4. Advanced	Heavier methods	svm, xgboost	ridge, lasso

* Logistic regression is binary only, so it appears only when the response has exactly two classes. On a multiclass problem such as iris the baselines are tree and forest.

You can see the phases in what actually got trained:

names(result$models)
#> [1] "baseline_tree"    "baseline_forest"  "pca_tree"         "pca_forest"      
#> [5] "clustered_tree"   "clustered_forest" "advanced_svm"     "advanced_xgboost"

# The same search on a two-class problem picks up the logistic variants
iris_binary <- iris |>
  filter(Species != "setosa") |>
  mutate(Species = droplevels(Species))

binary_result <- tl_auto_ml(iris_binary, Species ~ ., time_budget = 30,
                            cv_folds = 3)
names(binary_result$models)
#>  [1] "baseline_tree"      "baseline_logistic"  "baseline_forest"   
#>  [4] "pca_tree"           "pca_logistic"       "pca_forest"        
#>  [7] "clustered_tree"     "clustered_logistic" "clustered_forest"  
#> [10] "advanced_svm"       "advanced_xgboost"

That call emits a run of glm.fit: algorithm did not converge warnings, which this chunk hides only because there are several of them. They are worth understanding rather than ignoring: versicolor and virginica overlap only slightly, and a three-fold split of 100 rows will sometimes produce a fold where the two are perfectly separable. Logistic regression has no finite maximum likelihood estimate on separable data, so the coefficients diverge and glm() says so.

The fit on the full data is fine:

tl_model(iris_binary, Species ~ ., method = "logistic")$spec$method
#> [1] "logistic"

It is the folds, not the dataset. Seeing this on your own data means the classes are close to separable at that sample size — a reason to prefer a regularised method, which is what ridge and lasso are doing in phase 4 of the regression search.

The evaluation column

Each model is fitted on the full training data, then cross-validated if the budget allows. When it does not, the leaderboard falls back to training-set metrics.

table(result$leaderboard$evaluation)
#> 
#> cv 
#>  8

Training metrics are optimistically biased, so a leaderboard mixing "cv" and "train" is not ranking like with like. If you see both, raise the budget or lower cv_folds.

The Time Budget

time_budget is in seconds and is checked between model fits, not during them. Once a model starts it runs to completion, because the wrapped packages execute C-level code that R cannot safely interrupt. Wall-clock time can therefore overshoot by the duration of whichever model started last.

The budget controls how much gets tried. The relationship on this data and this machine:

budgets <- c(2, 5, 10, 30)

sweep <- lapply(budgets, function(b) {
  t0 <- Sys.time()
  r <- tl_auto_ml(iris, Species ~ ., time_budget = b, cv_folds = 3)
  data.frame(
    budget = b,
    elapsed = round(as.numeric(difftime(Sys.time(), t0, units = "secs")), 1),
    models = nrow(r$leaderboard),
    cv_scored = sum(r$leaderboard$evaluation == "cv"),
    best = r$leaderboard$model[1]
  )
})

do.call(rbind, sweep)
#>   budget elapsed models cv_scored          best
#> 1      2     0.0      1         1 baseline_tree
#> 2      5     0.0      1         1 baseline_tree
#> 3     10     0.1      3         3 baseline_tree
#> 4     30     1.7      8         8  advanced_svm

iris is 150 rows, so everything here is quick and the budget is barely touched. What changes across the sweep is which of these gates each budget opens:

The forest baseline, its PCA and cluster variants, and the advanced models are only attempted when time_budget >= 30.
The PCA and cluster phases start only while more than 5 seconds and more than 10% of the budget remain, and stop adding variants once fewer than 2 seconds or 5% are left. A budget of 5 seconds or less never reaches them.
The advanced phase starts only while more than 40% of the budget remains.
A model is cross-validated only while more than 30% of the budget remains when it is scored. After that it is scored on the training data, and the evaluation column says "train".

On data where a single forest fit takes ten seconds, the last three gates close early and the same budgets give very different results, so run the sweep on your own data. Cross-validation is the expensive step: tl_cv(folds = 5) fits five models where a plain fit costs one, so cv_folds is the most effective lever.

Controlling the Search

# Baselines only -- no PCA, no cluster features
baseline_only <- tl_auto_ml(
  iris, Species ~ .,
  use_reduction = FALSE,
  use_clustering = FALSE,
  time_budget = 30,
  cv_folds = 3
)

names(baseline_only$models)
#> [1] "baseline_tree"    "baseline_forest"  "advanced_svm"     "advanced_xgboost"

# Keep PCA, drop clustering
no_clustering <- tl_auto_ml(
  iris, Species ~ .,
  use_clustering = FALSE,
  time_budget = 30,
  cv_folds = 3
)

names(no_clustering$models)
#> [1] "baseline_tree"    "baseline_forest"  "pca_tree"         "pca_forest"      
#> [5] "advanced_svm"     "advanced_xgboost"

Naming a metric changes what the leaderboard optimises. Classification accepts accuracy, precision, recall, sensitivity, specificity, f1, auc and pr_auc; regression accepts rmse, mse, mae, mape and rsq.

by_f1 <- tl_auto_ml(iris_binary, Species ~ ., metric = "f1",
                    time_budget = 30, cv_folds = 3)

by_f1$metric
#> [1] "f1"
by_f1$leaderboard
#> # A tibble: 11 × 3
#>    model              score evaluation
#>    <chr>              <dbl> <chr>     
#>  1 advanced_svm       0.935 cv        
#>  2 advanced_xgboost   0.927 cv        
#>  3 pca_forest         0.927 cv        
#>  4 clustered_logistic 0.921 cv        
#>  5 baseline_logistic  0.921 cv        
#>  6 clustered_forest   0.916 cv        
#>  7 baseline_forest    0.908 cv        
#>  8 baseline_tree      0.904 cv        
#>  9 pca_logistic       0.876 cv        
#> 10 clustered_tree     0.848 cv        
#> 11 pca_tree           0.815 cv

Using the Result

split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 123)

automl <- tl_auto_ml(split$train, Species ~ ., time_budget = 30, cv_folds = 3)

test_preds <- predict(automl$best_model, new_data = split$test)
mean(test_preds$.pred == split$test$Species)
#> [1] 0.9333333

Any model on the leaderboard can be pulled out by name:

available <- names(automl$models)
available
#> [1] "baseline_tree"    "baseline_forest"  "pca_tree"         "pca_forest"      
#> [5] "clustered_tree"   "clustered_forest" "advanced_svm"     "advanced_xgboost"

scores <- vapply(available, function(nm) {
  preds <- predict(automl$models[[nm]], new_data = split$test)
  mean(preds$.pred == split$test$Species)
}, numeric(1))

data.frame(model = available, test_accuracy = round(scores, 3),
           row.names = NULL) |>
  arrange(desc(test_accuracy))
#>              model test_accuracy
#> 1    baseline_tree         0.933
#> 2  baseline_forest         0.933
#> 3   clustered_tree         0.933
#> 4     advanced_svm         0.933
#> 5 clustered_forest         0.911
#> 6 advanced_xgboost         0.911
#> 7         pca_tree         0.889
#> 8       pca_forest         0.844

Comparing the leaderboard against held-out accuracy is worth doing: the model that won the search is not always the one that generalises, and with a dataset this small the difference between the top few is noise.

AutoML Against a Single Choice

manual <- tl_model(split$train, Species ~ ., method = "forest")
manual_acc <- mean(
  predict(manual, new_data = split$test)$.pred == split$test$Species
)

automl_acc <- mean(test_preds$.pred == split$test$Species)

data.frame(
  approach = c("forest, chosen by hand", "AutoML best"),
  test_accuracy = round(c(manual_acc, automl_acc), 3)
)
#>                 approach test_accuracy
#> 1 forest, chosen by hand         0.933
#> 2            AutoML best         0.933

On iris a random forest is already the right answer, so the search does not improve on it. That is the expected result on an easy, well-understood dataset — AutoML earns its cost when you do not already know which method suits the data.

Preprocessing First

AutoML does not impute. Handle missing values before the call, and apply the same transformation to any data you later predict on:

processed <- tl_prepare_data(
  split$train, Species ~ .,
  scale_method = "standardize",
  remove_correlated = TRUE
)

automl_processed <- tl_auto_ml(processed$data, Species ~ .,
                               time_budget = 30, cv_folds = 3)

automl_processed$leaderboard$model[1]
#> [1] "baseline_tree"

tl_pipeline() handles that bookkeeping properly — see vignette("tuning-and-pipelines").

When a Model Is Missing or Scores NA

A model that fails to fit, or errors during evaluation, is dropped with a message explaining why and never reaches the leaderboard. Run without message = FALSE to see which and why.

A model that evaluates but produces no value for the chosen metric appears with an NA score. tl_auto_ml() refuses an unrecognised metric name before fitting anything, so an NA means the metric was undefined on the rows scored: auc needs rows of at least two classes, for instance, and is undefined on a fold that holds one. Cross-validation leaves such folds out of its average, so a cross-validated score is NA only when the metric was undefined in every fold. If every score is NA, tl_auto_ml() warns and returns the first model trained rather than pretending to have ranked them.

sum(is.na(result$leaderboard$score))
#> [1] 0

AutoML or Tuning?

They search different axes and compose in one direction:

tl_auto_ml() searches across methods at default hyperparameters.
tl_tune_grid() and tl_tune_random() search within one method.

Run AutoML first to find out which family suits the data, then tune the winner:

winner <- automl$best_model$spec$method
winner
#> [1] "tree"

tuned <- tl_tune_grid(
  split$train, Species ~ .,
  method = winner,
  param_grid = tl_default_param_grid(winner, size = "small"),
  folds = 3,
  verbose = FALSE
)

attr(tuned, "tuning_results")$best_params
#> $cp
#> [1] 0.01
#> 
#> $minsplit
#> [1] 10

mean(predict(tuned, new_data = split$test)$.pred == split$test$Species)
#> [1] 0.9777778

Guidance

Start at time_budget = 10 to confirm the call is well formed, then raise it. Below 30 seconds the forest and the advanced models are never tried, so a 10-second run cannot tell you which model is best.
Lower cv_folds before lowering time_budget. When fits are slow, the 30% gate above leaves the later models with training-set scores. Two folds fit two models per evaluation instead of five and still give out-of-sample estimates, which leaves more of the budget for cross-validating the rest.
Preprocess first. AutoML does not impute or scale.
Score on held-out data. The leaderboard ranks; it does not report final performance.
Read past the first row. When the top few scores are within noise of each other, pick on interpretability or fit time instead.
Check the evaluation column before comparing scores.