| Version: | 2.0.0 |
| Date: | 2026-11-16 |
| Title: | Random Forest Super Greedy Trees |
| Author: | Min Lu [aut], Udaya B. Kogalur [aut, cre], Hemant Ishwaran [aut] |
| Maintainer: | Udaya B. Kogalur <ubk@kogalur.com> |
| BugReports: | https://github.com/kogalur/randomForestSGT/issues/ |
| Depends: | R (≥ 4.3.0) |
| Imports: | randomForestSRC (≥ 3.6.2), varPro (≥ 3.1.0) |
| Suggests: | mlbench, interp, glmnet |
| Description: | Implements random forest Super Greedy Trees (SGTs) for regression. SGTs extend classification and regression tree splitting by fitting lasso-penalized local parametric models at tree nodes, producing sparse univariate and multivariate geometric cuts such as axis-aligned splits, hyperplanes, ellipsoids, hyperboloids, and interaction-based cuts. Trees are grown best-split-first by selecting cuts that reduce empirical risk, and ensembles provide out-of-bag error estimation, prediction on new data, variable filtering, tuning of the hcut complexity parameter, coordinate-descent lasso fitting, variable importance, and local coefficient summaries. For the underlying method, see Ishwaran (2026) <doi:10.1007/s10462-026-11541-6>. |
| License: | GPL (≥ 3) |
| URL: | https://ishwaran.org/ |
| NeedsCompilation: | yes |
| Packaged: | 2026-11-16 14:42:09 UTC; kogalur |
| Repository: | CRAN |
| Date/Publication: | 2026-11-16 16:10:02 UTC |
Coordinate Descent Lasso
Description
Fit lasso for regression using coordinate descent.
Usage
cdlasso(formula,
data,
nfolds = 0,
weights = NULL,
nlambda = 100,
lambda.min.ratio = ifelse(n < n.xvar, 0.01, 1e-04),
lambda = NULL,
threshold = 1e-7,
eps = .0001,
maxit = 5000,
efficiency = ifelse(n.xvar < 500, "covariance", "naive"),
seed = NULL,
do.trace = FALSE)
Arguments
formula |
Formula describing the model to be fit. |
data |
Data frame containing response and features. |
nfolds |
Number of cross-validation folds where default is 0 corresponding to no cross-validation. |
weights |
Observation weights. Default is 1 for each observation. |
nlambda |
The number of |
lambda.min.ratio |
Smallest value for |
lambda |
Lasso |
threshold |
Convergence threshold for coordinate descent. Each
inner coordinate-descent loop continues until the maximum change in
the objective after any coefficient update is less than
|
eps |
Multiplication factor applied to |
maxit |
Maximum number of passes over the data for all
|
efficiency |
Switches the algorithm to efficiency or naive mode
depending on number of variables. Efficiency |
seed |
Negative integer specifying seed for the random number generator. |
do.trace |
Number of seconds between updates to the user on approximate time to completion. |
Details
Use coordinate descent to fit lasso to a regression model.
Value
A list containing the fitted lasso solution path. The list contains:
convgCountConvergence counter returned by the coordinate-descent routine.
lambdaCountNumber of
lambdavalues in the fitted solution path.lambdaThe sequence of
lambdavalues used.betaMatrix of regression coefficients for the lasso solution path. Rows correspond to values in
lambda; columns contain the intercept followed by the encoded predictor variables.xvarNumeric predictor matrix used in the fit, after any internal encoding of the supplied data.
yvarResponse vector used in the fit.
yHatCross-validated fitted values or predictions by
lambdaand observation. Returned only when cross-validation output is available, such as whennfoldsis greater than 1.lambda.min.indxIndex of the
lambdavalue with minimum cross-validation error. Returned only when cross-validation output is available.lambda.1se.min.indxIndex of the minimum
lambdavalue within one standard error of the minimum cross-validation error. Returned only when cross-validation output is available.lambda.1se.max.indxIndex of the maximum
lambdavalue within one standard error of the minimum cross-validation error. Returned only when cross-validation output is available.lambda.cvmMean cross-validation error for each
lambda. Returned only when cross-validation output is available.lambda.cvsdCross-validation standard-error values for each
lambda. Returned only when cross-validation output is available.ctime.internalTiming information reported by the native coordinate-descent routine.
ctime.externalElapsed R-side timing, computed from
proc.time().
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
References
Friedman, J., Hastie, T. and Tibshirani, R. (2010) Regularization paths for generalized linear models via coordinate descent, J. of Statistical Software, 33(1):1-22.
See Also
Examples
## ------------------------------------------------------------
## regression example: boston housing
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
## load the data
data(BostonHousing, package = "mlbench")
## 10-fold validation
o <- cdlasso(medv ~., BostonHousing, nfolds=10)
## lasso solution
bhat <- data.frame(bhat.min=o$beta[o$lambda.min.indx,],
bhat.1se=o$beta[o$lambda.1se.max.indx[1],])
print(bhat)
## compare to results from glmnet
if (library("glmnet", logical.return = TRUE)) {
oo <- cv.glmnet(data.matrix(o$xvar), o$yvar, nfolds=10)
bhat2 <- cbind(data.matrix(coef(oo, s=oo$lambda.min)),
data.matrix(coef(oo, s=oo$lambda.1se)))
rownames(bhat2) <- rownames(bhat)
print(bhat2)
}
}
Cross-Validation for Super Greedy Forests
Description
Estimates the prediction error of a super greedy forest by M-fold
cross-validation. Every step of training, including the variable
filter and, if requested, the tuning of hcut, is repeated within
each training fold. The result is an honest estimate of prediction
error, free of the optimism that the OOB error of rfsgt can
have. make.folds creates folds that can be reused across calls.
Usage
cv.rfsgt(formula, data, M = 10, folds = NULL, tune = FALSE,
verbose = FALSE, fold.seeds = NULL, ...)
make.folds(n, M = 10)
## S3 method for class 'cv.rfsgt'
print(x, ...)
Arguments
formula |
Model formula, as in |
data |
Data frame containing the response and the features. Missing and non-finite values are not allowed. |
M |
Number of folds. Ignored when |
folds |
Optional fold assignment, given as the output of
|
tune |
If |
verbose |
Logical. Print the error of each fold as it is computed? |
fold.seeds |
Optional vector of positive integers, one per fold.
Each fold uses its seed for both the R and the native random number
generators during tuning, training and prediction, which makes
repeated calls on the same folds reproducible. Cannot be combined
with a |
... |
Further arguments passed to |
n |
Sample size, or the data frame itself. |
x |
Object of class |
Details
The OOB error printed by rfsgt is a quick estimate of
prediction error, but it can be optimistic. The variable filter,
which is on by default for hcut > 1, screens the model terms
using all of the data, including the observations that are out-of-bag
for a given tree. In our tests this made the OOB error too small by
about 5 to 10 percent. The OOB error is also optimistic when
bsf = "oob", because the out-of-bag data are then used to
compare candidate splits while each tree is grown.
Cross-validation avoids both problems. The data are split into
M folds. For each fold, rfsgt is run on the other
M - 1 folds with the arguments given in ..., and the
resulting forest predicts the held-out fold. Filtering, and the
tuning of hcut when tune is used, therefore see only the
training folds, and the responses of the held-out fold play no part in
training. When all folds are done, every observation has exactly one
held-out prediction. The cross-validated error err is the mean
squared error of these predictions, and its standard error serr
is computed from the M fold errors.
Because the folds can be fixed in advance, cv.rfsgt is also a
good way to compare settings, for example different values of
hcut, treesize or node.restrict. Running every
setting on the same folds removes the variation that comes from
splitting the data differently (see the examples). Currently only
regression is supported.
Folds and reproducibility
make.folds assigns the observations to M folds at
random and returns a list with the indices of the held-out
observations in each fold. When folds is not given,
cv.rfsgt calls make.folds before doing anything else,
and calling set.seed() first makes the folds reproducible. To
compare settings on identical splits, create the folds once and pass
them to every call.
Within a fold, the results also depend on the random number
generators used by tune.hcut and rfsgt. To make a run
completely reproducible, give fold.seeds. Each fold then sets
both the R and the native random number generators from its own seed
before tuning, training and prediction.
Tuning hcut within the folds
With tune = TRUE, tune.hcut is run on each training
fold, and the forest for that fold is grown with the filter it
returns. What happens next depends on whether hcut is given
in ....
If hcut is not given, each fold uses the hcut chosen by
its own tuner. The choice of hcut is then part of what is
cross-validated, and the folds can end up with different values,
which are returned in hcut.
If hcut is given, every fold uses that value together with the
filter the tuner found for it (see use.tune.hcut), and the
search of the tuner is extended to include that value if necessary.
The value each tuner would have chosen is still returned in
hcut.selected.
To control the tuner, pass a list instead of TRUE. For
example, tune = list(hcut = 5) sets the largest hcut
the tuner considers, and other list elements are passed on to
tune.hcut. Unless they are set in the list, the values of
node.center and bsf given in ... are also used
for tuning.
A tune.hcut object can also be passed directly as
filter, without tune. The same filter is then used in
every fold, and cv.rfsgt issues a warning. If the filter was
tuned on the full data, it has already seen the held-out
observations, and the cross-validated error is optimistic.
Errors in a fold
If a fold fails, cv.rfsgt stops with an error of class
"cv.rfsgt.error". The message names the fold and the stage
that failed (tune.hcut, use.tune.hcut, rfsgt or
predict). The error object also contains the original error,
the training and test rows of the fold, the filter, and the arguments
of the failed call. Catch it with tryCatch to examine or
reproduce the failure.
Value
An object of class cv.rfsgt, a list with the following
components.
err |
Cross-validated mean squared error over all |
serr |
Standard error of |
err.fold |
Mean squared error in each fold. |
hcut |
The |
predicted |
Held-out prediction for each observation. |
yvar |
Response values. |
yvar.names |
Name of the response. |
folds |
List with the indices of the held-out observations in each fold. |
fold.seeds |
The fold seeds used, or |
M |
Number of folds. |
n |
Sample size. |
tuning |
The |
filter |
The |
hcut.selected |
The |
hcut.requested |
The |
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
See Also
rfsgt,
predict.rfsgt,
tune.hcut
Examples
## ------------------------------------------------------------
##
## boston housing: OOB error versus cross-validated error
##
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
data(BostonHousing, package = "mlbench")
## the OOB error is optimistic because the filter used every observation
print(rfsgt(medv ~ ., BostonHousing, hcut = 3))
## cross-validation repeats the filter inside each training fold
print(cv.rfsgt(medv ~ ., BostonHousing, hcut = 3))
## ------------------------------------------------------------
##
## compare settings on identical folds
##
## ------------------------------------------------------------
set.seed(1)
folds <- make.folds(BostonHousing, M = 10)
## cross-validated error for hcut = 0 to 4
err <- sapply(0:4, function(h) {
cv.rfsgt(medv ~ ., BostonHousing, folds = folds, hcut = h)$err
})
print(setNames(err, paste0("hcut=", 0:4)))
## restricted score (the default) versus ordinary evaluation of the
## lasso terminal nodes
print(c(restricted = cv.rfsgt(medv ~ ., BostonHousing, folds = folds, hcut = 3)$err,
ordinary = cv.rfsgt(medv ~ ., BostonHousing, folds = folds, hcut = 3,
node.restrict = FALSE)$err))
## ------------------------------------------------------------
##
## tune hcut inside every fold, using mtcars for speed
##
## ------------------------------------------------------------
## the tuner considers hcut up to 4, and each fold uses the value its
## tuner selects
cvt <- cv.rfsgt(mpg ~ ., mtcars, tune = list(hcut = 4), verbose = TRUE)
print(cvt)
## the hcut selected in each fold
print(cvt$hcut)
}
Local Coefficients and Term Contributions of a Super Greedy Forest
Description
Extracts the local regression coefficients (beta values) that a super
greedy forest uses to predict each observation, and breaks each
prediction down into the contributions of the individual terms. Works
on the training data (a grow object) and on new data (a prediction
object). An optional relaxed estimate replaces the lasso coefficients
by least squares estimates computed from out-of-bag data. The helpers
partial.beta and importance.beta use these coefficients
to plot the partial effect of each variable and to compare it with the
response.
Usage
get.beta(o, bag = c("oob", "inbag", "all")[1], papply = lapply,
relaxed = FALSE, active = c("global", "local"))
partial.beta(b, o, xvar.names = NULL, by = NULL, granule = 5,
same.scale = TRUE, max.plots = 12, ...)
importance.beta(b, o)
Arguments
o |
An object of class |
bag |
Which trees to use for each training observation. The
default |
papply |
Function used to loop over the trees, either
|
relaxed |
Logical. If |
active |
Terms used by the least squares estimate when
|
b |
The value of |
xvar.names |
Names of the variables to plot. By default all
variables with a nonzero partial effect are plotted, ordered by the
standard deviation of the effect, up to |
by |
Optional name of a second variable. The observations are divided into at most three groups of this variable, and each group gets its own curve. See Details. |
granule |
A variable with at most this many distinct values is shown by boxplots instead of a scatter plot. |
same.scale |
Logical. If |
max.plots |
Maximum number of panels when |
... |
Further graphical parameters passed to |
Details
The terminal node of a super greedy tree contains the results from a
lasso fit to the inbag cases (in the case of a lasso failure, standard
CART prediction is used). get.beta collects the polynomials of
these successful lasso models for each observation and averages their
coefficients over the trees. The result, beta, contains one
set of local regression coefficients per observation. These
coefficients describe how the prediction of the forest depends on each
term near that observation, which can change from one region of the
feature space to another.
get.beta also breaks each prediction down by term. The
contribution of a term is its coefficient times the value of the term
for the observation, for example \beta_1 x_1 for a main
effect or \beta_{12} x_1 x_2 for an interaction. The
contributions are averaged over trees and returned in partial,
whose first column is the intercept. Each row of partial adds
up to the prediction, apart from an adjustment for the restricted
score described below. Plotting the contribution of a term against
its variable shows the partial effect of that term, and the column
means of beta summarize which terms the forest uses overall.
Interpreting these coefficients needs some care. The lasso shrinks them
toward zero, especially in deep trees. Growing shallow trees (a small
treesize) reduces the shrinkage, and relaxed = TRUE
further mitigates this problem by re-estimating the coefficients by
least squares on out-of-bag data. Both options are explained below.
Which trees are averaged
The option bag decides which trees are used for the
calculations. With the default bag = "oob", only the trees
for which the observation is out-of-bag are used, and the
reconstructed prediction predicted equals the OOB prediction
of the forest (predicted.oob). With bag = "inbag" it
equals the inbag prediction (predicted), and bag =
"all" uses every tree.
For new data, every tree used by the prediction contributes (all
trees, or the trees chosen with get.tree in
predict.rfsgt), and predicted equals the prediction
stored in the prediction object.
A tree can also contain CART terminal nodes, which the algorithm uses
in place of lasso nodes where the lasso becomes unstable (see
rfsgt). A CART terminal node predicts with the mean response
of its inbag cases. get.beta treats this mean as the
intercept and sets all other coefficients of the node to zero.
Coefficients in the original units
With node.center = TRUE (the default in rfsgt), each
node estimates its polynomial after centering the variables at the
node. Coefficients on this centered scale depend on the node center
and cannot be compared or averaged across nodes. get.beta
therefore rewrites each node polynomial in terms of the original,
uncentered variables before averaging, and all coefficients and
contributions are reported on the scale of the data.
Rewriting a centered square or product produces additional lower
order terms. For example, a product term in a node with centers
c_1 and c_2 expands as
(x_1 - c_1)(x_2 - c_2) = x_1 x_2 - c_2 x_1 - c_1 x_2 + c_1 c_2.
The expansion contains the main effects x_1 and
x_2, and the constant is added to the intercept. For this
reason beta and partial can have more columns than the
forest has retained terms. In this example they include columns for
X1 and X2 even if the forest retained only the product
X1_I_X2.
The columns are labeled by the dictionary terms. The first column
of beta (labeled betaZ) is the intercept. A square
ends in _SQ2_, as in X3_SQ2_ for the square of
X3, and a product joins the two variable names with
_I_, as in X1_I_X2 for the product of X1 and
X2.
Restricted score
When the forest uses the restricted score (node.restrict = TRUE,
the default in rfsgt), out-of-bag and new observations are
evaluated in two steps (see predict.rfsgt). The node
polynomial is evaluated at the nearest point of the box spanned by
the inbag cases of the node, and the result is then confined to the
range of the inbag responses. The contributions in partial
are computed at that nearest point. The second step is not part of
the polynomial, and get.beta reports its effect separately as
response.adjustment. With this adjustment included,
rowSums(partial) + response.adjustment equals
predicted exactly.
Shrinkage of the coefficients
In deep trees lasso coefficients are biased toward zero for two reasons. First, the lasso penalty, chosen by cross-validation within the node, shrinks the coefficients. In a small node a square or a product varies little, and the penalty tends to drop these terms altogether. Second, the splits that formed the node were chosen using the same responses that are then used to estimate its polynomial, which attenuates the coefficients further. Note that this biasing does not degrade prediction, it is merely an artifact of penalization/regularization.
When the true regression function is a polynomial (or can be
approximated by the dictionary terms), the column means of
beta estimate the coefficients. In our runs of the example
below, with about 100 inbag cases per terminal node the column means
were within 15 percent of the true values for the linear and square
terms and within 25 percent for the interaction. With about 30
cases per node (treesize = 10) they fell to about half the
true values, while the prediction error stayed the same. If the
coefficients are of primary interest, grow shallow trees by setting
a small treesize. The default tree size of rfsgt is
chosen for prediction, and its terminal nodes are too small to
estimate the coefficients well.
Relaxed estimation
Setting relaxed = TRUE keeps the terms of each node but
replaces their lasso coefficients by ordinary least squares
estimates, as in the relaxed lasso with relaxation parameter zero.
In each terminal node, least squares uses the out-of-bag cases of
that tree that fall in the node. With the default
bsf = "inbag", these cases played no part in growing the tree,
and the estimates are free of both the lasso shrinkage and the
attenuation caused by the splits. In our runs of the example below,
the relaxed coefficients were within 5 percent of the true values for
trees with 3 and 5 terminal nodes, and the relaxed polynomials
predicted as well as the forest.
The argument active decides which terms enter the least
squares estimate. With active = "global", every node uses all
terms retained by the forest. With active = "local", each node
uses only the terms its own lasso kept. The local version has fewer
coefficients and needs fewer cases, but a term that the node lasso
dropped stays out.
Least squares needs enough data. A node is used only if its number
of out-of-bag cases exceeds its number of coefficients (the intercept
plus the active terms) by at least five, and the other nodes are
skipped. Shallow trees, and the large nodes of deeper trees, meet
this requirement. The component nodes.used reports the
fraction of nodes that were used, and nodes gives the number
of nodes averaged for each observation. An observation that falls
only in skipped nodes gets NA.
A term that is an exact linear combination of the other active terms
within a node, such as the square of a binary variable, cannot be
estimated there. Least squares drops it from that node and its
coefficient in the node is zero. The component aliased
reports for each term the fraction of used nodes in which this
happened.
The relaxed polynomials are evaluated directly, without the
restricted score, and response.adjustment is zero. For a
training observation, predicted is a leave-one-out prediction.
In each node where the observation was one of the out-of-bag cases
used by least squares, its value is computed from the estimate with
that observation left out, and the error of predicted is
therefore an out-of-sample error. The in-sample values are in
fitted, and the contributions in partial add up to
fitted. For new data, predicted and fitted are
the same.
The standard errors in se come from the least squares
estimates. If a coefficient is averaged over B nodes whose
least squares standard errors are s_1, \ldots, s_B,
its reported standard error is
\sqrt{s_1^2 + \cdots + s_B^2} / B,
which is the standard error of an average of B independent
estimates.
To check whether a coefficient varies over the feature space, compare
its spread over the observations with its standard error. If the
standard deviation of a column of beta is no larger than its
typical standard error, the data give no evidence that the
coefficient changes from one region to another.
Partial effect plots
partial.beta plots the partial effect of each original
variable from the value of get.beta. A variable can enter
several terms, namely its linear term, its powers and its products
with other variables, and its partial effect combines all of them.
For an observation with local coefficients \beta, the
partial effect of a variable x is the value of its local
polynomial minus the value of the same polynomial with x
set to its mean, while the other variables keep their observed
values. For example, for the polynomial
\beta_1 x + \beta_{11} x^2 + \beta_{12} x z
the partial effect of x is
\beta_1 (x - m) + \beta_{11} (x^2 - m^2) + \beta_{12} (x - m) z,
where m is the mean of x. Terms that do not
involve x cancel. Unlike a sum of the contributions in
partial, the partial effect does not depend on where the
origin of a variable lies. For a binary variable or a factor, whose
mean is not one of its values, the average of the polynomial over
the values of x in the data replaces the polynomial at the
mean.
Each panel plots the partial effect against the variable and adds a
smooth curve. The points form a band around the curve because the
coefficients are local and because product terms involve other
variables. A variable with at most granule distinct values
is shown by boxplots. A factor is treated as one variable, and its
effect combines the terms of all its levels. Levels that the filter
did not retain are shown together as other. By default the
panels are ordered by the standard deviation of the effect and use
the same vertical scale.
With by, the observations are divided into groups of a second
variable and each group gets its own curve. The groups are the
values of the variable if it has at most three, and otherwise its
low, middle and high thirds. Curves with different slopes or shapes
indicate an interaction between the two variables.
The coefficients can come from the lasso or from the relaxed estimate. Partial effects computed from lasso coefficients inherit their shrinkage.
Importance from the partial effects
importance.beta compares the partial effect of each variable
with the response. Let y be the response, r the
response minus predicted, and e the partial effect
of the variable, with y and e centered at their
means. The marginal value is
\frac{\mathrm{mean}\,(y - e)^2}{\mathrm{mean}\,y^2},
which compares the partial effect of the variable alone with the response. The conditional value is
1 - \frac{\mathrm{mean}\,(r + e)^2 - \mathrm{mean}\,r^2}{\mathrm{mean}\,y^2},
one minus the increase in the standardized prediction error when the partial effect is removed from the prediction. Both values equal 1 for a variable with no term in the forest and are smaller for a larger effect. The marginal value is the smaller of the two when the partial effect is positively correlated with the rest of the prediction, as for correlated variables with effects of the same sign.
Value
A list with the following components.
beta |
Matrix of local coefficients with one row per observation.
Each entry is the coefficient of a term, averaged over the terminal
nodes the observation falls in and expressed on the original scale
of the data. The first column ( |
betaZ |
Matrix with one row per observation and one column per
retained term, describing the CART terminal nodes. When an
observation falls in a CART terminal node, the node mean is recorded
for each term used in the split that created the node, and zero for
the other terms. These values are averaged over the trees in which
the observation falls in a CART terminal node. |
lasso.percent |
Percentage of the terminal nodes in the forest that are lasso nodes. The rest are CART terminal nodes. |
predicted |
Prediction reconstructed from the terminal node
models. It equals the forest prediction for the trees selected by
|
partial |
Matrix of term contributions with the same layout as
|
response.adjustment |
Change in the prediction caused by
confining it to the inbag response range of the node, averaged over
trees. Zero when the restricted score is off or never takes effect,
and when |
When relaxed = TRUE, the list also contains the following
components.
fitted |
In-sample values of the relaxed polynomials, equal to
|
se |
Matrix of standard errors for the coefficients in
|
nodes |
Number of terminal nodes averaged for each observation. |
nodes.used |
Fraction of the terminal nodes with enough out-of-bag cases for least squares. |
aliased |
Fraction of the used nodes in which each term was an exact linear combination of the other active terms and was dropped from the least squares estimate. |
relaxed |
List with the settings used, namely |
partial.beta is called for its plot. It returns, invisibly, a
list with the following components.
effect |
Matrix of partial effects with one row per observation and one column per plotted variable. |
spread |
Standard deviation of each partial effect over the observations, in the units of the response. |
group |
Factor with the group of each observation when |
importance.beta returns a data frame with one row per original
variable and the columns marginal and conditional
defined in Details, ordered by marginal. A variable that the
filter removed has the value 1 in both columns.
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
See Also
rfsgt,
predict.rfsgt,
vimp.rfsgt
Examples
## ------------------------------------------------------------
## local coefficients and partial effects
## ------------------------------------------------------------
## The true model is y = 2 x1 - x2 + 1.5 x1 x2 + x3^2 + noise, and x4 and
## x5 have no effect. With hcut = 3 the model terms include all linear,
## square and pairwise product terms, which lets the terminal node
## polynomials represent the true model.
set.seed(1106)
n <- 500
d <- data.frame(matrix(runif(n * 5, -1, 1), n, 5))
d$y <- with(d, 2 * X1 - X2 + 1.5 * X1 * X2 + X3^2) + rnorm(n, sd = 0.25)
## Keep all 20 terms (filter = FALSE), use lasso terminal nodes
## everywhere (pure.lasso = TRUE), and grow shallow trees with 3 terminal
## nodes. Each node then holds about 100 inbag cases, enough to estimate
## the coefficients well.
o <- rfsgt(y ~ ., d, hcut = 3, filter = FALSE, pure.lasso = TRUE, treesize = 3)
print(o)
## Coefficients and contributions behind the OOB predictions (the
## default bag = "oob"). The reconstructed predictions match the forest.
b <- get.beta(o)
print(all.equal(b$predicted, o$predicted.oob))
## One row per observation and one column per term. betaZ is the
## intercept, X3_SQ2_ is the square of X3, and X1_I_X2 is the product of
## X1 and X2.
print(round(head(b$beta), 2))
## The column means estimate the true coefficients of x1, x2, x1 x2 and
## x3^2 (2, -1, 1.5 and 1), shrunk somewhat toward zero by the lasso, and
## the means of the other terms are near zero. Deeper trees shrink the
## coefficients more (try treesize = 10).
print(round(sort(colMeans(b$beta[, -1], na.rm = TRUE), decreasing = TRUE), 2))
## Each contribution is a coefficient times the value of its term. The
## contributions plus the response adjustment add up to the prediction.
print(round(head(b$partial), 2))
print(all.equal(rowSums(b$partial) + b$response.adjustment, b$predicted))
## Partial effects, shown by plotting the contribution of a term against
## its variable.
oldpar <- par(mfrow = c(1, 3))
plot(d$X1, b$partial[, "X1"], xlab = "x1", ylab = "contribution of x1")
plot(d$X3, b$partial[, "X3_SQ2_"], xlab = "x3", ylab = "contribution of x3^2")
plot(d$X1 * d$X2, b$partial[, "X1_I_X2"], xlab = "x1 x2",
ylab = "contribution of x1 x2")
par(oldpar)
## Prediction example
new <- data.frame(matrix(runif(5 * 5, -1, 1), 5, 5))
p <- predict(o, new)
bn <- get.beta(p, bag = "all")
print(round(cbind(predicted = p$predicted, bn$partial), 2))
## Relaxed estimation by least squares on the OOB cases of each node.
## Without the lasso penalty and the selection effect of the splits, the
## column means are close to the true coefficients.
r <- get.beta(o, relaxed = TRUE)
print(round(sort(colMeans(r$beta[, -1], na.rm = TRUE), decreasing = TRUE), 2))
cat("fraction of nodes with enough OOB cases:", round(r$nodes.used, 2), "\n")
## OOB error of the forest compared with the leave-one-out error of the
## relaxed polynomials.
c(forest = mean((d$y - o$predicted.oob)^2, na.rm = TRUE),
relaxed = mean((d$y - r$predicted)^2, na.rm = TRUE))
## Spread of the x1 coefficient over the observations compared with its
## standard error. The true coefficient is constant and equal to 2.
print(c(sd = sd(r$beta[, "X1"], na.rm = TRUE), se = median(r$se[, "X1"], na.rm = TRUE)))
## Partial effect plots of the variables, here from the relaxed
## coefficients. The effect of a variable combines all terms that
## involve it. The true effects are a line of slope 2 for X1, a line of
## slope -1 for X2 and a parabola for X3, and X4 and X5 have no effect.
pe <- partial.beta(r, o)
print(round(pe$spread, 2))
## Effect of X1 for low, middle and high values of X2. The product term
## 1.5 x1 x2 gives slopes near 1, 2 and 3.
partial.beta(r, o, "X1", by = "X2")
## Partial effect importance of each variable compared with the response, from the
## OOB lasso coefficients. X4 and X5 have no effect and values near 1.
print(round(importance.beta(b, o), 3))
Utility Functions for Random Forest Super Greedy Trees
Description
These are internal utility functions exported for advanced usage. This Rd file is used solely to register aliases.
Value
Return values depend on the helper function used:
filter.rfsgtandfilter.custom.rfsgtCharacter vector of variables or generated base-learner terms retained by the filtering step.
make.baselearnerA data frame or matrix containing the original predictors together with generated polynomial or interaction base-learner terms.
tune.hcutAn object of class
"tune.hcut". This is a character vector of selected terms with attributes includingall,formula,formula.all,term.map,term.map.all,cv,bsf,hcut, andhcutSeq.use.tune.hcutA
"tune.hcut"object restricted to the requestedhcut, with tuning attributes preserved.vimp.rfsgtVariable-importance scores, typically returned as a named numeric vector.
Tree Size Diagnostics for Random Forest Super Greedy Trees
Description
Two diagnostics for choosing the tree size of a random forest of super greedy trees: the out-of-bag (OOB) empirical risk of each tree as it grows, and the OOB error of whole forests grown with different tree sizes.
Usage
## S3 method for class 'rfsgt'
plot(x,
which = c("risk", "forest"),
treesize = c(1, 2, 3, 6, 12, 25),
lag = 8,
smooth = TRUE,
verbose = TRUE,
...)
Arguments
x |
An object of class |
which |
Which plots to draw: |
treesize |
Tree sizes (upper bounds on the number of terminal
nodes) at which forests are grown for the |
lag |
Window of the running average applied to each tree's OOB
empirical risk. Use |
smooth |
Logical. If |
verbose |
Logical. If |
... |
Further arguments passed to |
Details
OOB empirical risk by tree size. Each tree is grown best split
first, one terminal node at a time, and its OOB empirical risk is
recorded at each size (x$oob.empr.risk). Each tree's risk is
shown as a running average over lag sizes (grey) together with
the average over trees (thin dark line) and its smooth (thick line).
The average at a given size is over the trees that reached it. It is
drawn where at least a fifth of the trees with recorded risk contributed,
with a minimum of three contributors when available. If only one or two
trees have recorded risk, all of them must contribute at that size.
Forest OOB error by tree size. The per-tree risk does not
account for the averaging of the forest: deeper trees are individually
worse but vary more from tree to tree, and averaging removes much of
that variation. The forest is therefore often best at a larger tree
size than the per-tree curve suggests. The second plot grows a forest
at each value of treesize and shows its OOB error against the
average number of terminal nodes (log scale); the forest x is
shown in red at its average tree size. Use this plot to choose the
tree size. The forests are grown from common random numbers; hence
the differences between tree sizes are not confounded with run-to-run
variation. Supplying seed in ... turns this off. The
OOB error of a forest still varies from run to run, more for small
data sets; the overall level of the curve can therefore shift between
runs, and a larger ntree gives a more stable curve.
The forest-size comparison requires a finite OOB error for the fitted
forest and each comparison forest. It is unavailable with
bootstrap="none", which provides no OOB observations.
The forests are grown exactly as x was grown, apart from the
tree size: from the response and source predictors stored in x,
including parents of retained polynomial terms, with the basis terms
of x (rebuilt from x$term.map; hence any
filtering applied to x, for example by tune.hcut, is
retained), and with the number of trees, hcut, node size,
node centering, node restriction, best split first setting and sampling scheme of
x. Grown at its own tree size, this reproduces x.
Settings that are not stored in x (pure.lasso,
eps, maxit, fast) take their defaults; pass them
in ... if x used other values.
Value
Invisibly, a list with components risk (the running averages
for each tree, curves; the average and smoothed average by tree
size, average; and the size with the lowest smoothed average,
lowest) and forest (a data frame of tree size, average
number of terminal nodes and OOB error for each forest, with
treesize = NA for x).
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
References
Ishwaran H. (2026). Super greedy trees. Artificial Intelligence Review, 59:142. doi:10.1007/s10462-026-11541-6
See Also
Examples
## ------------------------------------------------------------
##
## ozone
##
## ------------------------------------------------------------
## load the data, calendar variables as integers
data(Ozone, package = "mlbench")
o <- data.frame(data.matrix(na.omit(Ozone)))
## fit the forest and plot both diagnostics
fit <- rfsgt(V4 ~ ., o, hcut = 3)
plot(fit)
## per-tree risk only (no forests are grown)
plot(fit, which = "risk")
## the forest error at more tree sizes, keeping the results
res <- plot(fit, which = "forest", treesize = c(1, 2, 4, 8, 16, 32, 64))
print(res$forest)
## ------------------------------------------------------------
##
## boston housing, grown using a tune.hcut filter
##
## ------------------------------------------------------------
data(BostonHousing, package = "mlbench")
f <- tune.hcut(medv ~ ., BostonHousing, hcut = 3)
fit <- rfsgt(medv ~ ., BostonHousing, filter = use.tune.hcut(f, hcut = 3))
## the filtered basis terms of fit are retained by the forests
plot(fit)
Prediction on Test Data for Super Greedy Forests
Description
Obtain predicted values from a trained super greedy forest. Predictions can be made for new data or, when no new data are given, for the training data, in which case the original forest is restored.
Usage
## S3 method for class 'rfsgt'
predict(object, newdata, get.tree = NULL,
block.size = 10, seed = NULL, do.trace = FALSE,
node.restrict = NULL, ...)
Arguments
object |
An object of class |
newdata |
Test data. If omitted, the training data are used and the original training forest is restored (see Details). |
get.tree |
Vector of integers identifying the trees used for the ensemble. By default all trees in the forest are used. |
block.size |
Controls how the cumulative error rate is computed.
The error is recorded after every |
seed |
Negative integer specifying the seed for the random number generator. |
do.trace |
Number of seconds between progress updates on the approximate time to completion. |
node.restrict |
Logical or |
... |
Additional options. |
Details
To predict an observation, each tree sends the case from the root down
to a terminal node. When hcut > 0, the splits and the terminal
node predictions are both based on scores. A score is a sparse lasso
combination of the dictionary terms, which can include squares,
products and higher powers of the variables. With node.center =
TRUE, the terms are formed from variables centered at the node. At
an internal node, the observation goes to the left daughter if its
score is at or below the threshold of the split, and to the right
daughter otherwise. Once the observation reaches a terminal node, the
node predicts its response using a sparse polynomial estimated by the
lasso from the terminal node's inbag cases. When the tree has
switched to CART (see rfsgt), the split is a standard CART cut,
and the terminal node predicts values using the mean of the inbag
responses. With hcut = 0 every node is a CART node, in which
case the forest is similar to a random forest.
The function works in two modes. Given newdata, it predicts
the new observations. Without newdata, it restores the
training forest and reproduces the inbag and out-of-bag predictions of
the grow object. Restoring is useful for recomputing the OOB
predictions and error rate with a subset of trees (get.tree) or
with a different node.restrict setting.
Restricted score
A lasso terminal node score uses a polynomial estimated from its
inbag cases. An observation that reaches the node can lie outside
the region those cases occupy, and there a square or product can take
values that no inbag case produced. The prediction can then fall far
outside the range of the response, most often in small trees and at
larger hcut.
The restricted score prevents this. With node.restrict = TRUE,
each variable is first clamped to its range among the inbag cases of
the node, which places the observation at the nearest point of the
inbag box. The polynomial is evaluated at that point, and the result
is then confined to the range of the inbag responses. The restriction
applies to every observation in newdata and, when the training
forest is restored, to the out-of-bag observations. Inbag
observations are evaluated as usual. The restriction does not change
which terminal node an observation reaches, because splits are always
evaluated in the ordinary way, and it has no effect on CART terminal
nodes. It works the same way with and without node.center.
The default node.restrict = NULL uses the value originally
used in the rfsgt training forest call (TRUE by default).
Restoring the training forest
When newdata is omitted, the training data are run down the
trees again. For each training observation, predicted
averages the trees whose bootstrap sample contains the observation,
evaluated exactly as during growth, and predicted.oob averages
the remaining trees, using the restricted score when it is on. With
the saved node.restrict setting, these reproduce
predicted and predicted.oob of the grow object, and
err.rate is the OOB error.
Predicting new data
newdata must contain the variables used for training. They
are encoded, filtered and expanded into model terms exactly as in
training. The response is optional; if present, it is used only to
compute err.rate. Every tree in get.tree contributes to
the prediction of every observation.
Passing the training data as newdata is not the same as
restoring the forest. The observations are then treated as new
data. Every tree contributes, including the trees for which an
observation was inbag, and the restricted score is applied to every
observation. The result therefore matches neither predicted
nor predicted.oob of the grow object.
Explaining predictions
get.beta breaks each prediction down into local coefficients
and the contributions of the individual terms. For new data, call
get.beta(pred, bag = "all", grow = fit), where pred is
the prediction object and fit is the grow object.
Value
An object of class c("rfsgt", "predict", family). When
newdata is omitted it describes the restored training forest;
otherwise it describes the predictions for the new data. The object
is a list with the following components.
callThe matched call.
familyModel family of the forest.
nNumber of observations predicted.
samptypeSampling type used to grow the forest.
sampsizeBootstrap sample size of each tree.
ntreeNumber of trees in the forest.
get.treeIndices of the trees used for the prediction.
get.betaalso uses them.hcuthcutvalue of the forest.node.restrictRestricted score setting used for this call.
splitruleSplit rule of the forest.
yvarResponse values used to compute the error rate. These are the training responses when the forest is restored, and the response in
newdataif it is present; otherwiseNULL.yvar.namesName of the response.
xvarVariables of the predicted observations, after the same encoding and filtering as in training.
xvar.augmentValues of the additional model terms (squares, products and higher powers) for the predicted observations, or
NULLwhen the forest uses the variables only.xvar.namesNames of the retained variables.
xvar.augment.namesNames of the additional model terms, or
NULL.xvar.infoEncoding information used to align new data with the training data.
term.mapMap from each model term to the variables and powers that form it.
leaf.countNumber of terminal nodes in each tree.
forestThe forest used to make the predictions.
membershipMatrix giving the terminal node of each observation in each tree, if membership output was requested; otherwise
NULL.inbagMatrix of bootstrap counts for each observation and tree, when the forest is restored and membership output was requested; otherwise
NULL.block.sizeBlock size used for the cumulative error rate.
perf.typeInternal code for the performance measure.
predictedPredicted values. For new data, the average over the trees in
get.tree. For the restored forest, the inbag prediction, which averages the trees whose bootstrap sample contains the observation.predicted.oobOut-of-bag predictions when the forest is restored;
NULLfor new data.ambrOffsetInternal matrix locating the terminal node of each new observation in each tree, used by
get.beta;NULLwhen the forest is restored.err.rateCumulative mean squared error, recorded after every
block.sizetrees, when response values are available; otherwiseNULL.ctime.internalComputing time reported by the native code.
ctime.externalElapsed time measured in R with
proc.time().
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
References
Ishwaran H. (2026). Super greedy trees. Artificial Intelligence Review, 59:142. doi:10.1007/s10462-026-11541-6
See Also
Examples
## ------------------------------------------------------------
##
## mtcars: for CRAN testing
##
## ------------------------------------------------------------
o <- rfsgt(mpg~., mtcars[1:20,], ntree=3, treesize=1)
p <- predict(o, mtcars[-(1:20),])
print(o)
print(p)
## ------------------------------------------------------------
##
## train/test using Friedman 3
##
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
## small forest of three trees
d.trn <- data.frame(mlbench::mlbench.friedman3(100))
o <- rfsgt(y ~ ., d.trn, hcut = 1, ntree = 3, treesize = 1)
print(o)
## test set mean squared error
d.tst <- data.frame(mlbench::mlbench.friedman3(200))
y.tst <- d.tst$y
x.tst <- d.tst[, colnames(d.tst) != "y"]
yhat <- predict(o, x.tst)$predicted
mean((yhat - y.tst)^2)
## full forest with hcut = 3
d.trn <- data.frame(mlbench::mlbench.friedman3(500))
o <- rfsgt(y~.,d.trn, hcut=3)
print(o)
## test set mean squared error on 1000 new observations
d.tst <- data.frame(mlbench::mlbench.friedman3(1000))
y.tst <- d.tst$y
x.tst <- d.tst[, colnames(d.tst)!= "y"]
yhat <- predict(o, x.tst)$predicted
cat("test set mse:", mean((yhat - y.tst)^2), "\n")
## ------------------------------------------------------------
##
## restore a trained forest using Boston housing
##
## ------------------------------------------------------------
## grow the forest
data(BostonHousing, package = "mlbench")
o <- rfsgt(medv~., BostonHousing)
print(o)
## restore the forest; the OOB error matches the grow object
print(predict(o))
## ------------------------------------------------------------
##
## restored versus test predictions, using Boston housing
## with factors
##
## ------------------------------------------------------------
## turn some of the variables into factors
data(BostonHousing, package = "mlbench")
Boston <- BostonHousing[1:40,]
Boston$zn <- factor(Boston$zn)
Boston$chas <- factor(Boston$chas)
Boston$lstat <- factor(round(0.2 * Boston$lstat))
Boston$nox <- factor(round(20 * Boston$nox))
Boston$rm <- factor(round(Boston$rm))
## grow a single tree and keep the inbag information
o <- rfsgt(medv~., Boston, hcut=2, filter=FALSE, ntree=1, membership=TRUE, nodesize=3)
## predictions from the grow object, from the restored forest, and from
## the training data passed as new data, with and without the
## restricted score
pred <- data.frame(
inbag=o$inbag,
pred.inb=o$predicted,
pred.oob=o$predicted.oob,
pred.inb.restore=predict(o)$predicted,
pred.oob.restore=predict(o)$predicted.oob,
pred.test=predict(o,Boston)$predicted,
pred.test.unrestricted=predict(o,Boston,node.restrict=FALSE)$predicted)
print(pred)
## Inbag observations are evaluated without the restriction and OOB
## observations with it. The inbag predictions therefore match the
## unrestricted test predictions, and the OOB predictions match the
## restricted ones.
cat("coherence for inbag data:", sum(pred$pred.inb-pred$pred.test.unrestricted,na.rm=TRUE)==0, "\n")
cat(" coherence for oob data:", sum(pred$pred.oob-pred$pred.test,na.rm=TRUE)==0, "\n")
## train on half of the data and predict the other half
trn <- sample(1:nrow(Boston), nrow(Boston)/2, replace=FALSE)
o.trn <- rfsgt(medv~., Boston[trn,], hcut=2)
print(predict(o.trn,Boston[-trn,]))
## ------------------------------------------------------------
## prediction after tuning hcut and filtering with tune.hcut
## ------------------------------------------------------------
## grow the forest with the tuned hcut and filter
dta <- data.frame(mlbench::mlbench.friedman3(500))
f <- tune.hcut(y~., dta, hcut=5, verbose=TRUE)
o <- rfsgt(y~., dta, filter=f)
print(o)
## test the tuned forest on new data
print(predict(o, data.frame(mlbench::mlbench.friedman3(25000))))
## override the tuned hcut
o2 <- rfsgt(y~., dta, filter=use.tune.hcut(f, hcut=2))
print(o2)
print(predict(o2, data.frame(mlbench::mlbench.friedman3(25000))))
}
Print Output from a Random Forest Super Greedy Tree Analysis
Description
Print summary output from a Random Forest SGT analysis. This is the default print method for the package.
Usage
## S3 method for class 'rfsgt'
print(x, ...)
## S3 method for class 'vimp.rfsgt'
print(x, ...)
Arguments
x |
An object of class |
... |
Further arguments passed to or from other methods. |
Value
Called for its side effect of printing a summary of an rfsgt
grow or predict object. The return value is NULL.
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
Random Forest Super Greedy Trees
Description
Grow a forest of Super Greedy Trees (SGTs) using lasso. In addition to prediction, the fitted forest supports local beta-value and partial-contribution summaries that show how predictions are assembled.
Usage
rfsgt(formula,
data,
ntree = if (hcut == 0) 500 else 100,
hcut = 2,
treesize = NULL,
nodesize = NULL,
filter = (hcut > 1),
bsf = if (hcut > 0 && isFALSE(node.center)) "oob" else "inbag",
keep.only = NULL,
fast = TRUE,
pure.lasso = FALSE,
eps = .005,
maxit = 500,
nfolds = 10,
block.size = 1,
bootstrap = c("by.root", "none", "by.user"),
samptype = c("swor", "swr"), samp = NULL, membership = TRUE,
sampsize = if (samptype == "swor") function(x){x * .632} else function(x){x},
node.center = TRUE,
seed = NULL,
do.trace = FALSE,
node.restrict = TRUE,
...)
Arguments
formula |
Formula describing the model to be fit. |
data |
Data frame containing response and features. |
ntree |
Number of trees to grow. |
hcut |
Integer value indexing type of parametric regression model to use for splitting. The default 2 uses linear and quadratic terms; 3 and 4 add pairwise interactions and cubic terms and were the most accurate in benchmarks. See details below. |
treesize |
Upper bound for the size of a tree (number of
terminal nodes): an integer, or a function of the sample size
|
nodesize |
Minumum size of terminal node. Set internally if not specified. |
filter |
Logical value specifying whether dimension reduction
(filtering) of features should be performed. Can also be specified
using the helper function |
bsf |
Best split first (BSF) empirical risk minimization
strategy. Accepted values are |
keep.only |
Character vector specifying the features of interest.
The data is pre-filtered to keep only these requested variables.
Ignored if filter is specified using |
fast |
Use fast filtering? |
pure.lasso |
Logical value specifying whether lasso splitting should be strictly adhered to. In general, lasso splits are replaced with CART whenever numerical instability occurs (for example, small node sample sizes may make it impossible to obtain the cross-validated lasso parameter). This option will generally produce shallow trees which may not be appropriate in all settings. |
eps |
Parameter used by |
maxit |
Parameter used by |
nfolds |
Number of cross-validation folds to be used for the lasso. |
block.size |
Determines how cumulative error rate is calculated. To
obtain the cumulative error rate on every nth tree, set the value to
an integer between 1 and |
bootstrap |
Bootstrap protocol. Default is |
samptype |
Type of bootstrap used when |
samp |
Bootstrap specification when |
membership |
Should terminal node membership and inbag information be returned? |
sampsize |
Function specifying bootstrap size when |
seed |
Negative integer specifying seed for the random number generator. |
do.trace |
Number of seconds between updates to the user on approximate time to completion. |
node.center |
Logical value specifying whether to center covariates
within each node before forming powers and products for the lasso.
Default is |
node.restrict |
Logical value enabling restricted lasso terminal
scores. Default is |
... |
Further arguments passed to |
Details
Super Greedy Trees (SGTs) are tree-based models that extend ordinary CART-style splitting in a fundamental way. In a standard tree, a split typically tests one variable at a time; hence tilted, curved, or interaction-driven decision boundaries must be approximated by many small axis-aligned cuts. SGTs instead estimate a sparse score inside a node and then split the node using that score. This allows a split to depend on several variables at once; the fitted tree can therefore represent hyperplane, elliptical, hyperboloid, and other higher-order geometric boundaries much more directly.
Operationally, the procedure is organized in stages. First, a family of
candidate score functions is chosen through hcut. Second, within
each node, a lasso model is fit by coordinate descent. Third, the fitted
node-wise score is used to order observations, and an allowable threshold
on that score is searched for the best split. The resulting daughter
nodes are then re-fit and compared using empirical risk. In this way,
the difficult multivariate split problem is converted into a manageable
one-dimensional threshold search along an estimated score.
Tree growth uses a best split first (BSF) strategy. Rather than expanding nodes in strict depth-first or breadth-first order, BSF scans the current terminal nodes, measures the reduction in empirical risk for each candidate split, and grows the node giving the largest gain. This makes the search aggressive but focused: computation is directed to the part of the tree that is most promising at the current stage of growth. In a forest, repeating this over bootstrap samples yields an ensemble of trees with the flexibility of multivariate cuts and the stabilizing effect of aggregation.
The main user control for split geometry is hcut. Smaller
values give simpler cut families; larger values allow richer
polynomial and interaction structure. Thus rfsgt is able to
span random-forest-like axis-aligned splits all the way to
genuinely multivariate geometric partitioning. When the signal is
approximately linear or only mildly nonlinear, smaller hcut
values may be sufficient. When the signal involves curvature or
interactions, larger hcut values can reduce bias and often
achieve the same structural fit with fewer splits.
An equally important feature of SGTs is that the fitted forest is not only a prediction device. Because each split score and each terminal-node predictor is a sparse lasso model, the forest also stores local coefficient information. For a given observation, each tree contributes the coefficient vector from the terminal node containing that observation, and averaging across trees yields forest-level beta summaries that are usually more stable than the coefficients from any single tree. These beta values are local and data-adaptive: they can change from one region of the feature space to another; they should therefore be viewed as coefficient functions rather than a single global regression vector.
This gives SGTs a genuinely hybrid character. The partition of the
feature space is nonparametric, as in a tree or forest, but within each
local region the fitted response is represented by a sparse parametric
expansion. Built-in helpers such as get.beta expose this
structure by returning both beta summaries and corresponding partial
term contributions. For a main effect, the contribution is the local
beta multiplied by the covariate value; for an interaction, it is the
local interaction coefficient multiplied by the associated product term.
In many applications, these summaries can be as informative as the
prediction itself because they show which variables, nonlinear terms,
and interactions are driving the fitted value near the observation of
interest. For centered fits, get.beta expands each node
polynomial in the original coordinates before averaging; hence the
coefficients and contributions are in the units of the data, and the
change made by the restricted score is reported as a separate
response.adjustment. For new-data explanations, pass the
original grow object through get.beta(pred, grow = fit).
Parametric models used for splitting are indexed by hcut
corresponding to the following geometric regions:
-
hcut=1 (hyperplane) linear model using all variables. -
hcut=2 (ellipse) plus all quadratic terms. -
hcut=3 (oblique ellipse) plus all pairwise interactions. -
hcut=4 plus all polynomials of degree 3 of two variables. -
hcut=5 plus all polynomials of degree 4 of three variables. -
hcut=6 plus all three-way interactions. -
hcut=7 plus all four-way interactions.
Setting hcut=0 gives CART splits where cuts are parallel to the
coordinate axis (axis-aligned cuts). Thus, hcut=0 is similar to
random forests and can be viewed as the baseline case of the SGT
framework.
A major part of the implementation is devoted to regularization and
stabilization, because richer cut families can otherwise become unstable
in small nodes or in the presence of collinearity. The first safeguard
is the lasso itself. At each node, the split-defining score is fit with
an \ell_1 penalty, and the penalty is chosen by cross-validation.
This induces sparsity, controls local complexity, and lets the procedure
adapt to the amount of information available in the node. Near the root,
where sample sizes are larger, the fitted score can support richer
geometry. Deeper in the tree, lasso sparsity and smaller node sizes tend
to simplify the split automatically.
A second safeguard is feature filtering. When the predictor dimension is
moderate or large, the candidate dictionary implied by hcut can be
very large. The implementation can therefore pre-filter variables using
shallow pilot fits and retain only variables that appear with nonzero
lasso coefficients. This reduces runtime and variance before the final
forest is grown. The helper tune.hcut is the intended front-end
for this step: it can pre-filter variables and also choose a suitable
hcut value before the final call to rfsgt. In practice,
this is often the safest workflow when interactions or higher-order terms
are being considered.
A third safeguard is automatic simplification of a branch when the lasso modeling is no longer paying off. The algorithm then replaces the lasso-induced split by the best CART coordinate-threshold split at that node and in place of local lasso node estimators, simple CART-style sample-average predictors are used along that branch. In other words, in the presence of numerical instability, the procedure can simplify both the split geometry and the local node model, which prevents unstable or unnecessary parametric structure from being pushed deeper into the tree.
The argument pure.lasso controls whether this simplification is
allowed. With the default behavior, a branch may switch from the
richer lasso-based representation to a simpler CART-style
representation when the latter is more stable or gives better
empirical risk reduction. Setting pure.lasso=TRUE shuts this
switch off. This is useful when a user wants a fully lasso-defined
tree for methodological reasons, but in difficult data settings it can
also lead to shallower trees because branches that would otherwise be
stabilized by CART are instead left unsplit.
The argument bsf, although related, addresses a different
question. It does not turn CART fallback on or off. Instead, it
decides which data drive empirical risk minimization during BSF
search. With bsf="inbag", the split comparison is based on
in-bag data. With bsf="oob", the same comparison uses
out-of-bag (OOB) data as a held-out check. This indirectly has
implications when deciding on CART fallback, however, because it
allows the algorithm to use out of sample data to decide whether the
current SGT candidate really improves generalization relative to the
best CART candidate at the same node. If the CART candidate wins
under the chosen risk criterion, the branch is simplified exactly as
described above: CART split geometry is used and CART-style node
estimators replace the local lasso fits down that branch. This guard
is especially helpful under potentially numerically unstable models
when hcut > 0. As with any model-selection procedure that
reuses held-out data, however, users should remember that the reported
OOB error can then be mildly optimistic.
A fourth safeguard is the restricted score, node.restrict. A
lasso terminal node predicts with a polynomial estimated from its
inbag cases. An observation routed to the node can lie outside the
region those cases occupy, and there a retained square or product
takes values no inbag case produced; the prediction can then leave the
response range by orders of magnitude, most often in small trees and
at larger hcut. With node.restrict=TRUE (the default)
the polynomial is evaluated at the nearest point of the node's inbag
box and the result is confined to the node's inbag response range.
Inbag cases are evaluated as before; growth, routing and inbag
predictions are therefore unchanged, and the restriction acts on the OOB ensemble
and error rate, on the OOB risks of candidate daughters when
bsf="oob", and on new data. Over twelve regression datasets
it removed every failure of small trees and left the error of large
trees unchanged; hence tree size can be chosen for speed. With it,
hcut values of 2 to 4 were more accurate than hcut=1
and than CART on average. node.restrict=FALSE reproduces the
evaluation of earlier versions.
From a practical point of view, the main tuning parameters have clear
roles. Use hcut to control cut richness, treesize and
nodesize to control tree complexity, filter or
tune.hcut when the feature space is large, and
pure.lasso only when you explicitly want to over-ride CART
fallback. Users new to the method can usually start with the
defaults. Users working with interaction-heavy or high-dimensional
data will typically benefit from filtering, tune.hcut, and
OOB-based BSF. Users interested mainly in prediction can simply call
predict.rfsgt. Users interested in the local parametric
structure can query the same fitted forest with get.beta to
recover beta summaries and partial contributions without fitting a
separate surrogate model. For that purpose grow shallow trees, with
nodes of about 100 cases or more: the terminal coefficients are
shrunk by the lasso and attenuated by the selection of the splits in
small nodes, while the prediction error is insensitive to tree size
with the restricted score (see get.beta).
Value
An object of class c("rfsgt", "grow", family) containing the
trained super greedy forest and associated training-data summaries. The
object is a list with the following components:
familyModel family.
nNumber of observations in the training data.
bootstrapBootstrap protocol used to grow the forest.
samptypeSampling type used for root-node bootstrap samples.
sampsizeTree sample size used by the bootstrap protocol.
ntreeNumber of trees grown.
hcutFinal
hcutvalue used to construct the splitting model.node.centerValue of the
node.centeroption.node.restrictValue of the
node.restrictoption, also saved inforest$node.restrict.splitruleSplit rule used by the forest.
splitinfoInternal split-rule metadata, including the processed
hcut, split-rule index, number of random split points, and lasso cross-validation setting.yvarTraining response values.
yvar.namesResponse variable name.
yvar.factorFactor-level metadata for the response.
yvar.typesInternal response type code.
xvarTraining predictor data after hot encoding, optional filtering, and any user-requested variable restriction.
xvar.augmentAugmented base-learner data used for higher-order
hcutterms, orNULLwhen no augmented terms are used.xvar.namesNames of the retained predictor variables.
xvar.typesInternal predictor type codes.
xvar.infoPredictor-encoding metadata used internally; this is
NULLfor standard grow objects.xvar.augment.namesNames of augmented base-learner terms, or
NULLwhen no augmented terms are used.term.mapTerm map describing how generated base-learner terms correspond to original variables and powers.
forestStored forest object used by
predict.rfsgt. This component contains the native node array, per-tree leaf counts, terminal-node offsets, bootstrap membership identifiers, fitting options, and package-version metadata.nodeStatNode statistics returned when empirical-risk output is available; otherwise
NULL.empr.riskInbag empirical-risk values by candidate tree size and tree, or
NULLwhen unavailable.oob.empr.riskOut-of-bag empirical-risk values by candidate tree size and tree, or
NULLwhen unavailable.empr.risk.cartInbag empirical-risk values for CART fallback splits, or
NULLwhen unavailable.oob.empr.risk.cartOut-of-bag empirical-risk values for CART fallback splits, or
NULLwhen unavailable.bsfBest-split-first empirical-risk strategy used.
bsf.orderBest-split-first node expansion order, or
NULLwhen unavailable.predictedTraining-data ensemble predictions.
predicted.oobOut-of-bag training-data predictions when available; otherwise
NULL.membershipMatrix of terminal-node membership by observation and tree when
membership = TRUE; otherwiseNULL.inbagMatrix of bootstrap counts by observation and tree when
membership = TRUE; otherwiseNULL.ensembleInternal ensemble type used for prediction summaries.
err.rateCumulative prediction error, typically the out-of-bag mean squared error for regression when available; otherwise
NULL.ctime.internalTiming information reported by the native grow routine.
ctime.externalElapsed R-side timing, computed from
proc.time().
Author(s)
Hemant Ishwaran and Udaya B. Kogalur
References
Ishwaran H. (2026). Super greedy trees. Artificial Intelligence Review, 59:142. doi:10.1007/s10462-026-11541-6
See Also
cdlasso,
cv.rfsgt,
get.beta,
plot.rfsgt,
predict.rfsgt
Examples
## ------------------------------------------------------------
##
## mtcars: for CRAN testing
##
## ------------------------------------------------------------
print(rfsgt(mpg~., mtcars, ntree=3, treesize=1))
## ------------------------------------------------------------
##
## boston housing
##
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
## load the data
data(BostonHousing, package = "mlbench")
## hcut=4: linear, quadratic, pairwise, plus all degree 3 polynomials
## (the default is hcut=2; both apply the filter and the restricted score)
o <- rfsgt(medv~., BostonHousing, hcut=4)
print(o)
## variable selection
sort(vimp.rfsgt(medv~.,BostonHousing))
## examples of hcut=0 (similar to random forests ... but using BSF)
print(rfsgt(medv~., BostonHousing, hcut=0))
print(rfsgt(medv~., BostonHousing, hcut=0, nodesize=1))
## hcut=3 with smaller nodesize
print(rfsgt(medv~., BostonHousing, hcut=3, nodesize=1))
## the OOB error above is optimistic because the filter saw every case;
## cross-validation refits the filter inside each fold (see cv.rfsgt)
print(cv.rfsgt(medv~., BostonHousing, hcut=3))
## ------------------------------------------------------------
##
## boston housing with factors
##
## ------------------------------------------------------------
## load the data
data(BostonHousing, package = "mlbench")
## make some features into factors
Boston <- BostonHousing
Boston$zn <- factor(Boston$zn)
Boston$chas <- factor(Boston$chas)
Boston$lstat <- factor(round(0.2 * Boston$lstat))
Boston$nox <- factor(round(20 * Boston$nox))
Boston$rm <- factor(round(Boston$rm))
## random forest: hcut=0
print(rfsgt(medv~., Boston, hcut=0, nodesize=1))
## hcut=3
print(rfsgt(medv~., Boston, hcut=3))
## ------------------------------------------------------------
##
## ozone
##
## ------------------------------------------------------------
## load the data
data(Ozone, package = "mlbench")
## remove missing values, convert factors to integers
o <- data.frame(data.matrix(na.omit(Ozone)))
## try different hcut values
## note the shallow trees; but performance is good
print(rfsgt(V4~., o, hcut=0, treesize=6))
print(rfsgt(V4~., o, hcut=1, treesize=6))
print(rfsgt(V4~., o, hcut=2, treesize=6))
print(rfsgt(V4~., o, hcut=3, treesize=6))
print(rfsgt(V4~., o, hcut=4, treesize=6))
print(rfsgt(V4~., o, hcut=5, treesize=6))
}
## ------------------------------------------------------------
##
## non-linear boundary illustrates hcut using single tree
##
## ------------------------------------------------------------
## simulate non-linear boundary
n <- 500
p <- 5
signal <- 10
treesize <- 10
ngrid <- 200
## train
x <- matrix(runif(n * p), ncol = p)
fx <- signal * sin(pi * x[, 1] * x[, 2])
nl2d <- data.frame(y = fx, x)
## truth
x1 <- x2 <- seq(0, 1, length = ngrid)
truth <- signal * sin(outer(pi * x1, x2, "*"))
## test
x.tst <- do.call(rbind, lapply(x1, function(x1j) {
cbind(x1j, x2, matrix(runif(length(x2) * (p-2)), ncol=(p-2)))
}))
colnames(x.tst) <- colnames(x)
fx.tst <- signal * sin(pi * x.tst[, 1] * x.tst[, 2])
nl2d.tst <- data.frame(y = fx.tst, x.tst)
## SGT for different hcut values
## Enforce pure lasso
rO <- lapply(0:4, function(hcut) {
cat("hcut", hcut, "\n")
rfsgt(y~., nl2d, ntree=1, hcut=hcut, treesize=treesize, bootstrap="none",
pure.lasso = TRUE, nodesize=1, filter=FALSE)
})
## nice little wrapper for plotting results
plot.image <- function(x, y, z, nlevels=40, points=FALSE) {
xo <- x; yo <- y
x <- unique(x); y <- unique(y)
## X2 varies fastest within each X1 block of x.tst.
z <- matrix(z, nrow=length(x), ncol=length(y), byrow=TRUE)
xlim <- ylim <- range(c(x, y), na.rm = TRUE, finite = TRUE)
z[is.infinite(z)] <- NA
zlim <- q <- quantile(z, c(.01, .99), na.rm = TRUE)
z[z<=q[1]] <- q[1]
z[z>=q[2]] <- q[2]
levels <- pretty(zlim, nlevels)
col <- hcl.colors(length(levels)-1, "YlOrRd", rev = TRUE)
plot.new()
plot.window(xlim, ylim, "", xaxs = "i", yaxs = "i", asp = NA)
.filled.contour(x, y, z, levels, col)
axis(1);axis(2)
if (points)
points(xo,yo ,pch=16, cex=.25)
box()
invisible()
}
oldpar <- par(mfrow=c(3,2))
image(x1, x2, truth, xlab="", ylab="")
contour(x1, x2, truth, nlevels = 15, add = TRUE, drawlabels = FALSE)
mtext(expression(x[1]),1,line=2)
mtext(expression(x[2]),2,line=2)
mtext(expression("truth"),3,line=1)
pO <- lapply(0:4, function(j) {
plot.image(nl2d.tst[,"X1"],nl2d.tst[,"X2"], predict(rO[[j+1]], nl2d.tst)$predicted)
contour(x1, x2, truth, nlevels = 15, add = TRUE, drawlabels = FALSE)
mtext(expression(x[1]),1,line=2)
mtext(expression(x[2]),2,line=2)
mtext(paste0("hcut=",j),3,line=1)
NULL
})
par(oldpar)
## ------------------------------------------------------------
##
## friedman illustration of OOB empirical risk
##
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
## simulate friedman
n <- 500
dta <- data.frame(mlbench::mlbench.friedman1(n, sd=0))
## rf versus rfsgt
o1 <- rfsgt(y~., dta, hcut=0)
o2 <- rfsgt(y~., dta, hcut=3)
## compute running average of OOB empirical risk
runavg <- function(x, lag = 8) {
x <- c(na.omit(x))
lag <- min(lag, length(x))
cx <- c(0,cumsum(x))
rx <- cx[2:lag] / (1:(lag-1))
c(rx, (cx[(lag+1):length(cx)] - cx[1:(length(cx) - lag)]) / lag)
}
risk1 <- lapply(data.frame(o1$oob.empr.risk), runavg)
leaf1 <- o1$forest$leafCount
risk2 <- lapply(data.frame(o2$oob.empr.risk), runavg)
leaf2 <- o2$forest$leafCount
## compare OOB empirical tree risk to OOB forest error
oldpar <- par(mfrow=c(2,2))
plot(c(1,max(leaf1)), range(c(risk1)), type="n",
xlab="Tree size", ylab="RF OOB empirical risk")
l1 <- do.call(rbind, lapply(risk1, function(rsk){
lines(rsk,col=grey(0.8))
cbind(1:length(rsk), rsk)
}))
lines(tapply(l1[,2], l1[,1], mean), lwd=3)
plot(c(1,max(leaf2)), range(c(risk2)), type="n",
xlab="Tree size", ylab="SGF OOB empirical risk")
l2 <- do.call(rbind, lapply(risk2, function(rsk){
lines(rsk,col=grey(0.8))
cbind(1:length(rsk), rsk)
}))
lines(tapply(l2[,2], l2[,1], mean), lwd=3)
plot(1:o1$ntree, o1$err.rate, type="s", xlab="Trees", ylab="RF OOB error")
plot(1:o2$ntree, o2$err.rate, type="s", xlab="Trees", ylab="SGF OOB error")
par(oldpar)
}
## ------------------------------------------------------------
##
## synthetic regression examples with different hcut
##
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
## simulation functions
sim <- list(
friedman1=function(n){data.frame(mlbench::mlbench.friedman1(n))},
friedman2=function(n){data.frame(mlbench::mlbench.friedman2(n))},
friedman3=function(n){data.frame(mlbench::mlbench.friedman3(n))},
peak=function(n){data.frame(mlbench::mlbench.peak(n, 10))},
linear=function(n, sd=.1){
x=matrix(runif(n*10), n)
y=3*x[,1]^3-2*x[,2]^2+3*x[,3]+rnorm(n,sd=sd)
data.frame(y=y,x)
})
## run rfsgt on the simulations
n <- 500
max.hcut <- 3
results <- setNames(lapply(names(sim), function(nm) {
cat("simulation:", nm, "\n")
d <- sim[[nm]](n=n)
rO <- data.frame(do.call(rbind, lapply(0:max.hcut, function(hcut) {
cat(" hcut:", hcut, "\n")
o <- rfsgt(y~.,d,hcut=hcut)
c(hcut, tail(o$err.rate, 1), tail(o$err.rate, 1) / var(o$yvar))
})))
colnames(rO) <- c("hcut", "mse", "smse")
rO
}), names(sim))
## print results
print(results)
## ------------------------------------------------------------
##
## synthetic regression example showing how to tune hcut
##
## ------------------------------------------------------------
hcut.opt <- setNames(sapply(names(sim), function(nm) {
cat("optimize hcut for simulation:", nm, "\n")
f <- tune.hcut(y~., sim[[nm]](n=n), hcut=4)
attr(f, "hcut")
}), names(sim))
## print the optimal hcut
print(hcut.opt)
}
## ------------------------------------------------------------
##
## iowa housing data
##
## ------------------------------------------------------------
data(housing, package = "randomForestSRC")
## remove PID
housing$PID <- NULL
## rough missing data imputation
d <- randomForestSRC::impute(data = data.frame(data.matrix(housing)))
d$SalePrice <- log(d$SalePrice)
d <- data.frame(data.matrix(d))
print(rfsgt(SalePrice~.,d))
## ------------------------------------------------------------
##
## high-dimensional model with variable selection
##
## ------------------------------------------------------------
## simulate big p small n data
n <- 50
p <- 500
d <- data.frame(y = rnorm(n), x = matrix(rnorm(n * p), n))
## we have a big p small n pure noise setting: let's see how well we do
cat("variables selected by vimp.rfsgt:\n")
vmp <- sort(vimp.rfsgt(y~.,d))
print(vmp[vmp > .05])
## internal filtering function can also be used
cat("variables selected by filter.rfsgt:\n")
print(filter.rfsgt(y~.,d, method="conserve"))
## ------------------------------------------------------------
##
## pre-filtering using keep.only
##
## ------------------------------------------------------------
if (requireNamespace("mlbench", quietly = TRUE)) {
## simulate the data
n <- 100
p <- 50
noise <- matrix(runif(n * p), ncol=p)
dta <- data.frame(mlbench::mlbench.friedman1(n, sd=0), noise=noise)
## filter the variables
f <- filter.rfsgt(y~., dta)
## use keep.only to pre-filter the features
print(rfsgt(y~.,dta, keep.only=f, hcut=1))
print(rfsgt(y~.,dta, keep.only=f, hcut=2))
print(rfsgt(y~.,dta, keep.only=f, hcut=3))
## ------------------------------------------------------------
##
## tuning hcut and pre-filtering using tune.hcut
##
## ------------------------------------------------------------
## simulate the data
n <- 100
p <- 50
noise <- matrix(runif(n * p), ncol=p)
dta <- data.frame(mlbench::mlbench.friedman1(n, sd=0), noise=noise)
## tune hcut
f <- tune.hcut(y~., dta, hcut=3)
## use the optimized hcut
print(rfsgt(y~.,dta, filter=f))
## over-ride the tuned hcut value
print(rfsgt(y~.,dta, filter=use.tune.hcut(f, hcut=1)))
print(rfsgt(y~.,dta, filter=use.tune.hcut(f, hcut=2)))
print(rfsgt(y~.,dta, filter=use.tune.hcut(f, hcut=3)))
## ------------------------------------------------------------
##
## get local beta values and partial contributions
##
## SGTs are not only predictive; the same fit can be queried
## for local parametric summaries. We use friedman 1 for
## illustration. See get.beta for details
##
## ------------------------------------------------------------
n <- 100
dta <- data.frame(mlbench::mlbench.friedman1(n))
o <- rfsgt(y~., dta, hcut=3, pure.lasso=TRUE, treesize=10)
b <- get.beta(o)
## print beta, partial values - remove intercept
print(head(b$beta[, -1]))
print(head(b$partial[, -1]))
partial.beta(b, o)
}
Show the NEWS file
Description
Show the NEWS file of the randomForestSGT package.
Usage
rfsgt.news(...)
Arguments
... |
Further arguments passed to or from other methods. |
Value
None.
Author(s)
Hemant Ishwaran and Udaya B. Kogalur