Tutorial for R package TreatmentSelection
Overview of the package
This package implements basic methodology for evaluating one or more biomarkers for their ability to guide patient treatment recommendations.
The methodology assumes that the data come from a randomized and controlled trial (RCT) comparing two treatment options, which we refer to as “treatment” and “no treatment”. These could be, for example, two different active prophylactic or therapeutic interventions, or an experimental intervention and a standard of care. Subjects are followed for the development of a binary clinical outcome or event within a specified time-frame following treatment/no treatment. The biomarker(s) are assumed to be measured at baseline; they could be anything from patient demographics or clinical characteristics, to traditional biomarker measurments or the results of imaging or genetic or proteomic analyses. The methodology accommodates settings where the biomarker(s) are measured on all RCT participants, and also settings in which participants are retrospectively sub-sampled based on clinical outcome and/or treatment assignment for marker measurement.
Install the package
This tutorial uses the Treatment Selection package (v 2.1.0) to analyze the example data provided in the package.
The current version of the package is on github. Use the devtools package to install the package:
if (!require("devtools")) install.packages("devtools")
devtools::install_github("mdbrown/TreatmentSelection")First, load the example data set for this package called tsdata. Four markers are included in the data example, a ‘’weak’‘and a’‘strong’’ marker (\(Y1\) and \(Y2\) respectively), along with a weak/strong discrete markers.
First steps
Load the example data set for this package called tsdata. These are hypothetical data from an RCT with a binary outcome (outcome) and treatment assignment (trt). Four markers, measured on all RCT participants, are included– a ‘’weak’‘and a’‘strong’’ marker (Y1 and Y2 respectively), along with binary versions of these markers.
library(TreatmentSelection)
set.seed(12321)
data(tsdata)
data(surv_tsdata)
tsdata[1:5, ]## trt event Y1 Y2 Y1_disc Y2_disc
## 1 1 1 39.9120 -0.8535 1 0
## 2 1 0 6.6820 0.2905 0 1
## 3 1 0 6.5820 0.0800 0 1
## 4 0 0 1.3581 1.1925 0 1
## 5 0 0 7.6820 -0.2070 0 0Evaluate performance of user-specified marker-based treatment rule
For a pre-defined marker-based treatment rule, we provide a simple function to estimate point estimates of performance measures. This is done using the trtsel_measures() function. The user must specify a vector of clinical outcomes, a vector of treatment assigments, and a vector of marker-based treatment recommendations based on the pre-specified rule.
Here we let Y1_disc represent a user-specified treatment rule and evaluate its performance.
trtsel_measures(event = tsdata$event, trt = tsdata$trt, trt.rule = tsdata$Y1_disc, default.trt = "trt all" )## Estimates of trt.effect are not provided. Only empirical estimates will be calculated.##
## Summary Measure Estimates
## -----------------------------------------------------------
## Decrease in rate of outcomes under marker-based treatment rule (Theta)
## Empirical: 0.011
## Model Based: NA
##
## Proportion recommended no treatment: 0.57
## Proportion recommended treatment: 0.43
##
## Average benefit of no treatment among those recommended no treatment (B.neg)
## Empirical: 0.019
## Model Based: NA
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: 0.106
## Model Based: NA
##
## Variance in estimated treatment effect: NA
## Total Gain: NA
##
## Event Rates:
## --------------------
## Treat all Treat None Marker-based Treatment
## Empirical: 0.217 0.251 0.206
## trtsel_measures(event = surv_tsdata$di,
time = surv_tsdata$xi,
trt = surv_tsdata$trt,
trt.rule = as.numeric(surv_tsdata$Y > 0),
prediction.time = 1,
default.trt = "trt none" )## Estimates of trt.effect are not provided. Only empirical estimates will be calculated.##
## Summary Measure Estimates
## -----------------------------------------------------------
## Decrease in rate of outcomes under marker-based treatment rule (Theta)
## Empirical: -0.015
## Model Based: NA
##
## Proportion recommended no treatment: 0.484
## Proportion recommended treatment: 0.516
##
## Average benefit of no treatment among those recommended no treatment (B.neg)
## Empirical: -0.037
## Model Based: NA
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: -0.029
## Model Based: NA
##
## Variance in estimated treatment effect: NA
## Total Gain: NA
##
## Event Rates:
## --------------------
## Treat all Treat None Marker-based Treatment
## Empirical: 0.132 0.134 0.149
## We can also fit our own risk model using GLM, use this model to develop a marker-based treatment recommendation, and evaluate its performance. This allows us to obtain model-based estimates of performance:
mod <- glm(event~trt*Y1_disc, data = tsdata, family = binomial())
tsdata.0 <- tsdata;
tsdata.0$trt = 0
tsdata.1 <- tsdata;
tsdata.1$trt = 1
delta.hat <- predict(mod, newdata= tsdata.0, type = "response") - predict(mod, newdata= tsdata.1, type = "response")
trtsel_measures(event = tsdata$event, trt = tsdata$trt, trt.rule = 1- tsdata$Y1_disc, trt.effect = delta.hat )##
## Summary Measure Estimates
## -----------------------------------------------------------
## Decrease in rate of outcomes under marker-based treatment rule (Theta)
## Empirical: -0.046
## Model Based: -0.046
##
## Proportion recommended no treatment: 0.43
## Proportion recommended treatment: 0.57
##
## Average benefit of no treatment among those recommended no treatment (B.neg)
## Empirical: -0.106
## Model Based: -0.106
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: -0.019
## Model Based: -0.019
##
## Variance in estimated treatment effect: 0.004
## Total Gain: 0.061
##
## Event Rates:
## --------------------
## Treat all Treat None Marker-based Treatment
## Empirical: 0.217 0.251 0.263
## Create TrtSel objects
An alternative to providing a pre-specified treatment rule the user can use the data to fit a risk model and to develop a treatment rule based on estimated treatment effect.
The user can do this by first creating a treatment selection R object using the function trtsel. Creating this object entails estimating risk of the clinical outcome given treatment and marker, i.e. fitting a “risk model”. Here, logistic regression is used (family = binomial(link="logit")) and the formula is specified in the first argument. The default treatment strategy– that which would be recommended in the absence of marker measurements– is specified as default.trt="trt all". The methodology considers marker-specific treatment rules or policies that recommend treatment if the marker-specific treatment effect– on the risk difference scale– is above a specified thresh; the default is thresh = 0. (Alternatively, see the tab on “additional features”" to see how an arbitrary user-specified treatment rule can be evaluated.) The study design="RCT" indicates that the markers are measured on all RCT participants.
trtsel.Y1 <- trtsel(event ~ Y1*trt,
treatment.name = "trt",
data = tsdata,
family = binomial(),
study.design = "RCT",
default.trt = "trt all")
trtsel.Y1## Study design: RCT
##
## Model Fit:
## Family: binomial
## Link function: logit
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -2.51814383 0.235642511 -10.686288 1.179991e-26
## Y1 0.04760056 0.006453791 7.375597 1.636104e-13
## trt 0.48938620 0.311762857 1.569739 1.164759e-01
## Y1:trt -0.02318881 0.008324063 -2.785756 5.340300e-03
##
##
## Derived Data: (first ten rows)
##
## event Y1 trt fittedrisk.t0 fittedrisk.t1 trt.effect rec.no.trt
## 1 1 39.9120 1 0.35016583 0.2583742 0.0917916549 0
## 2 0 6.6820 1 0.09974358 0.1340472 -0.0343036269 1
## 3 0 6.5820 1 0.09931697 0.1337641 -0.0344471266 1
## 4 0 1.3581 0 0.07918316 0.1196652 -0.0404820847 1
## 5 0 7.6820 0 0.10410005 0.1369063 -0.0328062456 1
## 6 0 41.1720 0 0.36393311 0.2643117 0.0996213622 0
## 7 0 19.4920 1 0.16933976 0.1746644 -0.0053246137 1
## 8 1 20.8220 1 0.17843231 0.1793943 -0.0009620341 1
## 9 0 6.9620 0 0.10094678 0.1348426 -0.0338958439 1
## 10 0 2.5020 0 0.08324538 0.1226384 -0.0393929781 1As we see above, the trtsel.Y1 object contains information about the study design, the fitted risk model, the fitted risk given each treatment, and the estimated marker-specific treatment effect (on the risk difference scale) for each individual. This object also contains the treatment recommendation for each subject, based on a treatment rule that recommends treatment if the estimated treatment effect is above thresh; here thresh = 0.
This is what a trtsel object looks like for a binary marker.
# Y2_disc = as.numeric(Y2>0)
trtsel.Y2_disc <- trtsel(event ~ trt*(Y2_disc),
treatment.name = "trt",
data = tsdata,
study.design = "RCT",
family = binomial("logit"))See ?trtsel for more information.
Now that we have created trtsel objects, we can assess the calibration of the risk models; and plot, evaluate, and compare markers.
Assess model calibration
It is important to assess the fit of the risk model to the data. The package implements methods for examining calibration.
Here we compare (average) predicted and observed risk values by decile (‘group = 10’) of fitted risk, for each treatment group. The risks are shown on the log scale. The Hosmer-Lemeshow goodness of fit test is used to detect evidence of mis-calibration by treatment group.
calibrate(trtsel.Y1,
groups = 10,
plot.type = "treatment effect",
trt.names = c("chemo.", "no chemo."),
point.color = "lightcoral",
line.color = "black")
##
## Hosmer - Lemeshow test for model calibration
## ----------------------------------------------
##
## Number of Groups: 10
##
## No Treatment (trt = 0):
## Test Statistic = 4.496, DF = 8, p value = 0.8098813
##
## Treated (trt = 1):
## Test Statistic = 4.986, DF = 8, p value = 0.7591213calibrate(trtsel.Y1,
groups = 10,
plot.type = "risk.t0",
trt.names = c("chemo.", "no chemo."),
point.color = "lightcoral",
line.color = "black")
##
## Hosmer - Lemeshow test for model calibration
## ----------------------------------------------
##
## Number of Groups: 10
##
## No Treatment (trt = 0):
## Test Statistic = 4.496, DF = 8, p value = 0.8098813
##
## Treated (trt = 1):
## Test Statistic = 4.986, DF = 8, p value = 0.7591213See ?calibrate.trtsel for more plot options.
Plot
The plot() function allows us to visually assess the ability of the marker or markers to predict the treatment effect.
Risk curves
Risk curves apply to individual markers. They are plots of estimated risk of the clinical outcome as a function of the marker, for each treatment group. The two risk curves are aligned with respect to marker percentile, but the option show.marker.axis=TRUE adds a second x-axis showing the marker values corresponding to each percentile. Pointwise confidence intervals, obtained using the bootstrap, can be added (ci="vertical").
plot(trtsel.Y1,
main = "Y1: Oncotype-DX-like marker",
plot.type = "risk",
ci = "vertical",
conf.bands = TRUE,
bootstraps = 50, #more bootstraps should be run than this in practice!
trt.names = c("chemo.", "no chemo."),
show.marker.axis = FALSE)
This is what the plot looks like for a binary marker:
tmp <- plot(trtsel.Y2_disc,
main = "Discrete version of Y2",
plot.type = "risk",
ci = "vertical",
conf.bands = TRUE,
bootstraps = 50,
trt.names = c("chemo.", "no chemo."))
Note that tmp is a list with elements plot that holds the ggplot output, and ci.bounds which holds the information regarding the confidence bounds.
tmp$ci.bounds## risk trt marker lower upper
## 1 0.35984848 1 0 0.32422707 0.39520792
## 2 0.06198347 1 1 0.03510314 0.09270948
## 4 0.32061069 0 1 0.26155639 0.37440180
## 5 0.17241379 0 0 0.12972671 0.20252033Treatment effect curves
Treatment effect curves apply both to univariate and multivariate markers. They show the distribution of marker-specific treatment effects. Using plot.type = "treatment effect" produces a reverse-CDF-type plot:
plot(trtsel.Y1,
plot.type = "treatment effect",
ci = "horizontal",
conf.bands = TRUE,
bootstraps = 50)
Alternatively, using plot.type = "cdf" produces a traditional CDF of the treatment effects:
plot(trtsel.Y1,
plot.type = "cdf",
ci = "vertical",
conf.bands = TRUE,
bootstraps = 50)
Here is an example of a treatment effect plot for a discrete marker:
plot(trtsel.Y2_disc,
plot.type = "treatment effect",
conf.bands = TRUE,
bootstraps = 50)## [1] "vertical confidence intervals will be plotted."
Selection impact plot
The selection impact plot also applies to both univariate and multivariate markers. They show the population rate of the clinical outcome as a function of the proportion of the population recommended treatment according a marker-specific treatment rule. Rules that recommend treatment based on the marker-specific treatment effect (on the risk difference scale) are considered, and are indexed by the proportion of the population recommended treatment according to the rule. The outcome rates under “treat all” and “treat none” policies are shown for comparison.
plot(trtsel.Y1,
plot.type = "selection impact",
ci = "vertical",
conf.bands = TRUE,
bootstraps = 50)
Evaluate marker performance
The function evaluate() provides estimates of measures of marker performance. A wide variety of performance measures are estimated– some which depend on a specified treatment rule and some which do not. For the former, the rule is inherited from the trtsel object. Confidence intervals are calculated using the quantile bootstrap; in every bootstrap sample the risk model is re-fit and the model performance measures are re-estimated.
tmp <- evaluate(trtsel.Y1, bias.correct = TRUE, bootstraps = 50)
tmp##
## Summary Measure Estimates (with % confidence intervals)
## -----------------------------------------------------------
## Decrease in rate of outcomes under model-based treatment rule (Theta)
## Empirical: 0.006 (-0.018,0.035)
## Model Based: 0.01 (0,0.04)
##
## Proportion recommended no treatment:
## 0.465 (0.004,0.739)
## Proportion recommended treatment:
## 0.535 (0.261,0.996)
##
## Average benefit of no treatment among those recommended no trt (B.neg)
## Empirical: 0.016 (-0.056,0.068)
## Model Based: 0.022 (0,0.059)
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: 0.09 (0.023,0.155)
## Model Based: 0.098 (0.051,0.156)
##
##
## Variance in estimated treatment effect:
## 0.008 (0.002,0.018)
## Total Gain:
## 0.069 (0.035,0.11)
##
## Marker positivity threshold: 21.082
##
## Event Rates:
## --------------------
## Treat all Treat none Marker-based treatment rule
## Empirical: 0.213 0.252 0.208
## (0.188,0.263) (0.219,0.294) (0.176,0.237)
## Model Based: 0.213 0.255 0.203
## (0.186,0.259) (0.231,0.297) (0.179,0.233)The evaluate function used this way both fit and evaluates a risk model using the same data, which is known to bias performance measure estimates to be overly optimistic. To correct for this bias, the evaluate function by default sets bias.correct=TRUE. This implements bootstrap bias correction via the ``refined bootstrap" method described in chapter 17 of An introduction to the Bootstrap by Efron and Tibshirani 1994.
In short, we sample \(B\) bootstrap datasets. For each, obtain a new treatment selection rule based on the re-fit model and calculate the difference in estimated performance of this rule using the bootstrap vs. original data. The average of these differences estimates the bias. We shift naive performance estimates and confidence intervals down by the estimated bias.
Here is what performance evaluation looks like for a discrete marker:
# discrete marker
evaluate(trtsel.Y2_disc, bootstraps = 50)## Bootstrap bias-correction will be implemented to correct for over-optimism bias in estimation.##
## Summary Measure Estimates (with % confidence intervals)
## -----------------------------------------------------------
## Decrease in rate of outcomes under model-based treatment rule (Theta)
## Empirical: 0.095 (0.061,0.118)
## Model Based: 0.093 (0.059,0.116)
##
## Proportion recommended no treatment:
## 0.498 (0.465,0.524)
## Proportion recommended treatment:
## 0.502 (0.476,0.535)
##
## Average benefit of no treatment among those recommended no trt (B.neg)
## Empirical: 0.191 (0.126,0.242)
## Model Based: 0.187 (0.122,0.238)
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: 0.261 (0.196,0.312)
## Model Based: 0.259 (0.193,0.309)
##
##
## Event Rates:
## --------------------
## Treat all Treat none Marker-based treatment rule
## Empirical: 0.219 0.251 0.116
## (0.184,0.254) (0.216,0.285) (0.100,0.140)
## Model Based: 0.210 0.247 0.117
## (0.176,0.239) (0.215,0.284) (0.102,0.141)Compare markers
Two markers– or marker-specific treatment rules– can be compared using the compare() function. Note that the compare() function requires that that the treatment and outcome labels are identical for the two markers/treatment rules.
Here, we compare the continuous markers Y1 and Y2. The first step is to create a trtsel object for Y2.
# trtsel object for markers 1 and 2
trtsel.Y2 <- trtsel(event~trt*(Y2 + Y1),
treatment.name = "trt",
data = tsdata,
default.trt = "trt all")## Binary outcome detected: setting family = binomial(link = 'logit')A suite of performance measures is estimated for each marker, and marker-comparisons are based on Wald hypothesis tests for differences in measures with variances estimated using the bootstrap.
# Compare the markers based on summary measures
mycompare <- compare(x = trtsel.Y1,
x2 = trtsel.Y2,
bias.correct =TRUE,
model.names = c("Y1", "Y1 + Y2"),
bootstraps = 50,
ci = "vertical")
mycompare## Summary Measure Estimates
## (with 95 % confidence intervals)
##
## Y1 | Y1 + Y2 | difference (p-value)
## ------------------------------------------------------------------------
##
## Decrease in rate of outcomes under model-based treatment rules (Theta)
## Empirical: 0.005 | 0.088 | -0.079 (< 0.02)
## (-0.018,0.167) | (0.064,0.277) | (-0.112,-0.086)
## Model Based: 0.010 | 0.096 | -0.087 (< 0.02)
## (0.000,0.042) | (0.073,0.127) | (-0.108,-0.056)
##
## Proportion recommended no treatment:
## 0.459 | 0.410 | 0.047 (0.88)
## (0.051,0.811) | (0.349,0.505) | (-0.299,0.315)
## Proportion recommended treatment:
## 0.541 | 0.589 | -0.047 (0.88)
## (0.189,0.949) | (0.495,0.651) | (-0.315,0.299)
##
## Average benefit of no treatment among those recommended no treatment (B.neg)
## Empirical: 0.021 | 0.218 | -0.194 (< 0.02)
## (-0.131,0.067) | (0.165,0.288) | (-0.367,-0.136)
## Model Based: 0.023 | 0.234 | -0.212 (< 0.02)
## (0.003,0.066) | (0.188,0.296) | (-0.264,-0.158)
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: 0.092 | 0.217 | -0.130 (< 0.02)
## (0.020,0.144) | (0.157,0.262) | (-0.188,-0.071)
## Model Based: 0.098 | 0.234 | -0.137 (< 0.02)
## (0.037,0.167) | (0.188,0.277) | (-0.178,-0.086)
##
##
## Variance in estimated treatment effect :
## 0.008 | 0.094 | -0.086 (< 0.02)
## (0.002,0.021) | (0.074,0.119) | (-0.110,-0.059)
##
## Total Gain:
## 0.071 | 0.231 | -0.165 (< 0.02)
## (0.032,0.118) | (0.204,0.263) | (-0.197,-0.117)## Compare two discrete markers Y1_disc = as.numeric(Y1>mean(Y1))
trtsel.Y1_disc <- trtsel(event~Y1_disc*trt,
treatment.name = "trt",
data = tsdata)## Binary outcome detected: setting family = binomial(link = 'logit')compare(x = trtsel.Y1_disc,
x2 = trtsel.Y2_disc,
bias.correct = TRUE,
ci = "vertical",
offset = 0.2,
bootstraps = 50,
conf.bands = TRUE,
annotate.plot = TRUE)## Warning: Ignoring unknown aesthetics: shape
## Summary Measure Estimates
## (with 95 % confidence intervals)
##
## Model 1 | Model 2 | difference (p-value)
## ------------------------------------------------------------------------
##
## Decrease in rate of outcomes under model-based treatment rules (Theta)
## Empirical: 0.013 | 0.092 | -0.082 (< 0.02)
## (-0.021,0.184) | (0.058,0.322) | (-0.110,-0.098)
## Model Based: 0.011 | 0.093 | -0.082 (< 0.02)
## (-0.023,0.044) | (0.060,0.124) | (-0.113,-0.057)
##
## Proportion recommended no treatment:
## 0.569 | 0.499 | 0.074 (< 0.02)
## (0.543,0.592) | (0.473,0.534) | (0.015,0.116)
## Proportion recommended treatment:
## 0.431 | 0.501 | -0.074 (< 0.02)
## (0.408,0.457) | (0.466,0.527) | (-0.116,-0.015)
##
## Average benefit of no treatment among those recommended no treatment (B.neg)
## Empirical: 0.022 | 0.184 | -0.168 (< 0.02)
## (-0.037,0.077) | (0.123,0.245) | (-0.230,-0.121)
## Model Based: 0.019 | 0.187 | -0.168 (< 0.02)
## (-0.040,0.074) | (0.126,0.249) | (-0.237,-0.128)
##
## Average benefit of treatment among those recommended treatment (B.pos)
## Empirical: 0.108 | 0.254 | -0.153 (< 0.02)
## (0.012,0.187) | (0.194,0.318) | (-0.208,-0.092)
## Model Based: 0.106 | 0.259 | -0.153 (< 0.02)
## (0.009,0.184) | (0.198,0.322) | (-0.214,-0.098)
##
##
## Variance in estimated treatment effect :
## 0.005 | 0.050 | -0.046 (< 0.02)
## (0.001,0.011) | (0.034,0.074) | (-0.070,-0.031)
##
## Total Gain:
## 0.063 | 0.223 | -0.162 (< 0.02)
## (0.017,0.105) | (0.184,0.271) | (-0.220,-0.122)See ?compare.trtsel for more options.
Additional features
Other outcome types
Version 2.1.0 of the package allows for continous and time-to-event outcomes.
Continuous
When using a continuous outcome, it is assumed that higher values are associated with worse outcomes.
tsdata$event.c <- tsdata$event + rnorm( nrow(tsdata), mean = 0, sd = .1 )
trtsel.Yc <- trtsel(event.c ~ Y1*trt,
treatment.name = "trt",
data = tsdata,
study.design = "RCT",
family = gaussian("identity"), #specify model error dist.
default.trt = "trt all")
#plot, calibrate, evaluate and compare all work as usual Time-to-event
When evaluating a time-to-event outcome, a landmark prediction time must be specified. Cox proportional hazards models are used for all model-based estimates. Absolute risk estimates are calculated by pairing the Cox model with the Nelson-Aalen baseline hazard estimate.
library(survival)
data("surv_tsdata") #simulated sample data set with time-to-event outcome
trtsel.Ysurv <- trtsel(Surv(xi, di) ~ Y*trt,
treatment.name = "trt",
data = surv_tsdata,
prediction.time = 4, #landmark prediction time must be set
study.design = "RCT",
default.trt = "trt all")
#notice 'censoring weights' can now be found in trtsel.Ysurv$'derived.data'
#plot, calibrate, evaluate and compare all work as usual Specifying fitted risks
An alternative to specifing one or more markers and fitting a risk model is for the user to specify fitted risks given treatment and no treatment for each subject. Importantly, in this case the inference provided based on bootstrap confidence intervals is conditional on these user-specified fitted risks.
Here, we illustrate this by fitting a risk model for Y2 using the glm() function, and then feed the fitted risks into the trtsel() function:
#calculate model fit
mymod <- glm(event~trt*(Y2), data= tsdata, family = binomial("logit"))
tsdata$fitted.t0 <- predict(mymod, newdata=data.frame(trt = 0,Y1 = tsdata$Y1, Y2 = tsdata$Y2), type = "response")
tsdata$fitted.t1 <- predict(mymod, newdata=data.frame(trt = 1,Y1 = tsdata$Y1, Y2 = tsdata$Y2), type = "response")
myfitted.trtsel <- trtsel( event~trt, treatment.name = "trt",
data = tsdata,
fittedrisk.t0 = "fitted.t0",
fittedrisk.t1 = "fitted.t1",
study.design = "RCT",
default.trt = "trt all")## Binary outcome detected: setting family = binomial(link = 'logit')We can now use this trtsel object to plot and evaluate performance just as before, but confidence intervals are narrower because they are conditional on the fitted risk model.
plot(myfitted.trtsel, bootstraps = 50, plot.type = "risk",
ci = "vertical", show.marker.axis = FALSE)
References
Efron B, Tibshirani R. An Introduction to the Bootstrap. New York: Chapman& Hall, 1994.
Song X, Pepe MS. Evaluating markers for selecting a patient’s treatment. Biometrics. 2004;60(4):874-883.
Huang Y, Sullivan Pepe M, Feng Z. Evaluating the predictiveness of a continuous marker. Biometrics. 2007;63:1181-8.
Janes H, Pepe MS, Bossuyt PM, et al. Measuring the performance of markers for guiding treatment decisions. Ann Intern Med. 2011;154(4):253-259.
Janes H, Brown MD, Huang Y, et al. An approach to evaluating and comparing biomarkers for patient treatment selection. Int J Biostat. 2014;10(1):99- 121.