Model Shootout: Comparing Linear Models, Trees, Forests, Boosting and Neural Networks
Jared P. Lander
Lander Analytics
What are we Modeling?
What Models?
Validation
Penalized Binary Regression
\[ \begin{align} & \min_{\beta \in \mathbb{R}^{p+1}} \left[ \frac{1}{N}\sum_{i=1}^N y_i log(\hat{p}_i) + (1-y_i)log(1-\hat{p}_i) + \lambda \mathbf{P} \right] \\ &\text{where} \\ &\mathbf{P} = \alpha\sum_{j=1}^p|\beta| + \frac{1}{2}(1-\alpha)\sum_{j=1}^p\beta^2 \end{align} \]
Feature Engineering
- Balance Class Size
- Impute Missing Data
- Deal with Unknown Data
- Normalize Numeric Data
- Convert Categorical to Dummies
- Include an Intercept
Feature Engineering
rec_linear <- recipe(Status ~ ., data=train) |>
themis::step_downsample(Status, under_ratio=1.2) |>
step_normalize(all_numeric_predictors()) |>
step_other(all_nominal_predictors(), other='misc') |>
step_novel(all_nominal_predictors()) |>
step_impute_knn(all_numeric_predictors()) |>
step_unknown(all_nominal_predictors()) |>
step_dummy(all_nominal_predictors(), one_hot=TRUE) |>
step_intercept()Tuning Parameters
- Penalty
- Mixture
Tuning Parameters
Decision Tree
\[ \hat{p}_{mk} = \frac{1}{N_m} \sum_{x_i \in R_m} I(y_i = k) \]
Feature Engineering
- Balance Class Size
- Impute Missing Data
Feature Engineering
Tuning Parameters
- Tree Depth
- Minimum Node Size
Tuning Parameters
Random Forest
\[ \hat{f} = \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x) \]
Feature Engineering
- Balance Sizes
- Fill in Missing Data
Feature Engineering
Tuning Parameters
- Number of Trees
- Minimum Node Size
- Number of Randomly Selected Predictors
Tuning Parameters
Boosted Tree
\[ \hat{y}_i^{t} = \sum_{k=1}^t f_k(x_i) = \hat{y}_i^{(t-1)} + f_t(x_i) \]
Feature Engineering
- No Need to Balance Class Size
- No Imputing Missing Data
- Convert Categorical Data to Dummies
Feature Engineering
Tuning Parameters
- Tree Depth
- Number of Trees
- Learning Rate
- Number of Columns to Inspect for Each Tree
- Minimum Node Size
- Minimum Loss Reduction
- Sample Size
- Stopping Iterations
Tuning Parameters
Tuning Parameters with GPU
Boosted Pseudo Random Forest
\[ \hat{f} = \frac{1}{B} \sum_{b=1}^B \hat{f}^{*b}(x) \\ \hat{y}_i^{t} = \sum_{k=1}^t f_k(x_i) = \hat{y}_i^{(t-1)} + f_t(x_i) \]
Feature Engineering
- No Need to Balance Class Size
- No Imputing Missing Data
- Convert Categorical Data to Dummies
Feature Engineering
Tuning Parameters
- Tree Depth
- Number of Trees
- Learning Rate
- Number of columns to inspect for each tree
- Minimum Node Size
- Minimum Loss Reduction
- Sample Size
- Sopping Iterations
- Number Parallel Trees
Tuning Parameters
this_scaler <- train |> dplyr::count(Status) |> dplyr::pull(n) |> purrr::reduce(`/`)
spec_pseudo_forest <- boost_tree(
mode='classification',
trees=tune(),
tree_depth=tune()
) |>
set_engine(
'xgboost',
tree_method='gpu_hist',
num_parallel_tree=tune(),
subsample=0.5,
colsample_bytree=4,
scale_pos_weight=this_scaler
)Neural Network
\[ \begin{align} h^{(1)} &= f_1(XW^{(0)} + b)\\ h^{(2)} &= f_2(h^{(1)}W^{(1)} + b)\\ h^{(3)} &= f_3(h^{(2)}W^{(2)} + b)\\ & \vdots \\ h^{(k)} &= f_k(h^{(k-1)}W^{(k-1)} + b)\\ O &= \sigma(h^{(k)}W^{(k)} + b) \end{align} \]
Feature Engineering
- Balance Class Size
- Impute Missing Data
- Normalize Predictors
- Convert Categorical Data to Dummies
Feature Engineering
rec_mlp <- recipe(Status ~ ., data=train) |>
themis::step_upsample(Status, over_ratio=1.2) |>
step_normalize(all_numeric_predictors()) |>
step_other(all_nominal_predictors(), other='misc') |>
step_novel(all_nominal_predictors()) |>
step_impute_knn(all_numeric_predictors()) |>
step_unknown(all_nominal_predictors()) |>
step_integer(Status, zero_based=TRUE) |>
step_dummy(all_nominal_predictors(), one_hot=TRUE) |>
prep(retain=FALSE)Feature Engineering
mlp_dataset <- dataset(
name='credit_data',
initialize=function(df, recipe){
self$x <- recipe |>
bake(all_predictors(), new_data=df, composition='matrix') |>
torch_tensor(dtype=torch_float())
self$y <- recipe |>
bake(all_outcomes(), new_data=df, composition='matrix') |>
torch_tensor(dtype=torch_float())
},
.getitem=function(i){
list(x=self$x[i, ], y=self$y[i])
},
.length=function(){
length(self$y)
}
)Tuning Parameters
- Number of Layers
- Nodes in Each Layer
- Dropout
- Batch Normalization
- Activation Function
- Row Sampling
- Weight Decay
Tuning Parameters
mlp_mod <- nn_module(
classname='credit_net',
initialize=function(
n_col,
fc1_dim,
fc2_dim,
fc3_dim,
output_dim
){
self$fc1 <- nn_linear(in_features=n_col, out_features=fc1_dim)
self$fc2 <- nn_linear(in_features=fc1_dim, out_features=fc2_dim)
self$fc3 <- nn_linear(in_features=fc2_dim, out_features=fc3_dim)
self$output <- nn_linear(in_features=fc3_dim, out_features=output_dim)
},
forward=function(x){
x |>
nnf_dropout(p=0.2) |>
self$fc1() |> nnf_relu() |> nnf_dropout(p=0.5) |>
self$fc2() |> nnf_relu() |> nnf_dropout(p=0.5) |>
self$fc3() |> nnf_relu() |> nnf_dropout(p=0.5) |>
self$output()
}
)
mlp_set <- mlp_mod |>
setup(
loss=nn_bce_with_logits_loss(pos_weight=.5),
optimizer=optim_adam,
metrics = list(
luz_metric_binary_accuracy_with_logits(),
luz_metric_binary_auroc()
)
) |>
set_hparams(
n_col=summary(rec_mlp) |> dplyr::filter(role == 'predictor') |> nrow(),
fc1_dim=512,
fc2_dim=128,
fc3_dim=64,
output_dim=1
) |>
set_opt_hparams(lr=0.03)Tuning Parameters
mlp_set <- mlp_mod |>
setup(
loss=nn_bce_with_logits_loss(pos_weight=.5),
optimizer=optim_adam,
metrics = list(
luz_metric_binary_accuracy_with_logits(),
luz_metric_binary_auroc()
)
) |>
set_hparams(
n_col=summary(rec_mlp) |> dplyr::filter(role == 'predictor') |> nrow(),
fc1_dim=512,
fc2_dim=128,
fc3_dim=64,
output_dim=1
) |>
set_opt_hparams(lr=0.03)Draw the DAG
Run It
AUC
Speed
AUC vs Speed
Bonus
Ensemble
\[ \hat{y}_i = \sum_{j=1}^q \theta_j \cdot f_j(x_i) \]
Combine the Models
Takeways
- Linear Models are Fast and Performant
- Boosted Trees are Better
- Need to Balance Performance with Speed
- Throw Everything Against the Wall
- Ensemble the Models
Thank You
