07_model_evaluation.Rmd

---
title: "01_intro"
output: html_document
---

layout: false
class: middle, center, inverse

# 4.7 `broom`:<br><br>Convert Statistical Objects into Tidy Tibbles

---

background-image: url(https://www.tidymodels.org/images/broom.png)
background-position: 97.5% 2.5%
background-size: 7%
layout: true

---

## 4.7 `broom`: Tidy Model Outputs

`broom` provides three useful functions for converting `parsnip` objects (e.g., `lm`, `glm`, `rpart`) into tidy `tibbles`:
- `tidy()`: produces a tidy output of model components (e.g., coefficients, weights, clusters)
- `glance()`: produces a tidy output of model summaries (e.g., goodness-of-fit, $F$ - statistics)
- `augment()`: adds additional information about observations (e.g., fitted values, residuals)

```{r, echo=F, out.width='60%', fig.align='center'}
knitr::include_graphics(
  "https://raw.githubusercontent.com/allisonhorst/stats-illustrations/master/rstats-artwork/broom_package.png"
)
```

???
- difference to `tidyr`: these functions tidy model objects, `tidyr` is all about tidying and transforming data frames

---

## 4.7 `broom`: Tidy Model Outputs

In order to illustrate the convenience of the three `broom` functions, let us first extract our optimal model.

```{r, results='hide'}
reg_log_cls_fit <- cls_wf_last_fit %>% extract_fit_parsnip()
```

.panelset[
.panel[.panel-name[tidy()]
**`tidy()`** produces a tidy output of model components (e.g., coefficients, weights, clusters)
```{r}
tidy(reg_log_cls_fit) %>% glimpse
```
]
.panel[.panel-name[glance()]
**`glance()`** produces a tidy output of model diagnostics (e.g., goodness-of-fit, F-statistics)
```{r}
glance(reg_log_cls_fit) %>% glimpse
```
]
.panel[.panel-name[augment()]
**`augment()`** adds additional information about observations (e.g., fitted values, residuals)

Unfortunately, `augment()` is not supported for `glmnet` models (check [available methods](https://broom.tidymodels.org/articles/available-methods.html)).
]
]

.footnote[
*Note: Depending on the class of the model object you are providing to `tidy()`, it offers several advanced features, such as returning odds-ratios for logit-models (`exponentiate = T`) or confidence interval (`conf.int = T`).*
]

???
`tidy`: useful for creating visualizations or preparing model tables for a paper
`glance`:
  - useful for investigating overall model performance
  - identify mis-specifications
  - compare models in general
  
- these functions are implemented with the need of data scientist in mind, i.e. what are the statistics the modeler is most likely interested in?
- they also work with techniques from classical statistics such as t-tests


---

layout: false
class: middle, center, inverse

# 4.8 `yardstick`:<br><br>Tidy Characterizations of Model Performance

---

background-image: url(https://www.tidymodels.org/images/yardstick.png)
background-position: 97.5% 2.5%
background-size: 7%
layout: true

---

## 4.8 `yardstick`: Tidy Model Performance

Similar to `broom`, `yardstick`'s endeavor is to enable model evaluation using *tidy data principles*. It provides various performance metrics for both, classification and regression problems.

- **Class metrics for classification:** Metrics that are computed based on the predicted classes (*hard predictions*) and take two `fct` columns (`truth` and `estimate`) as input.<br><br>
- **Class probability metrics for classification:** Metrics that are computed based on the predicted class probabilities (*soft predictions*) and take one `fct` column (`truth`) and one/multiple `dbl` columns (`estimate`) as input.<br><br>
- **Numeric metrics for regression:** Metrics that are computed based on a numerical prediction and take two `dbl` columns (`truth` and `estimate`) as input.

.footnote[
*Note: Find all available metrics grouped by their type on [tidymodels.org](https://yardstick.tidymodels.org/articles/metric-types.html#metrics) or learn more about the `yardstick` package, e.g., about features for multi-class learning problems, in [Kuhn/Silge (2021)](https://www.tmwr.org/performance.html).*
]

---

## 4.8 `yardstick`: Tidy Model Performance

**Class Metrics**

.panelset[
.panel[.panel-name[Confusion Matrix]
```{r}
collect_predictions(cls_wf_last_fit) %>% 
  conf_mat(died, estimate = .pred_class)
```
]
.panel[.panel-name[Sensitivity]
```{r}
collect_predictions(cls_wf_last_fit) %>% 
  sens(died, estimate = .pred_class)
```
]
.panel[.panel-name[Specificity]
```{r}
collect_predictions(cls_wf_last_fit) %>% 
  spec(died, estimate = .pred_class)
```
]
.panel[.panel-name[Accuracy]
```{r}
collect_predictions(cls_wf_last_fit) %>% 
  accuracy(died, estimate = .pred_class)
```
]
.panel[.panel-name[Metric Set]
```{r}
metrics <- metric_set(accuracy, sens, spec)

collect_predictions(cls_wf_last_fit) %>% 
  metrics(died, estimate = .pred_class)
```
]
]

???
- "binary" indicates that the measures are computed for a binary classification problem
- for multi-class problems `yardstick` has implemented generalizations of the original measures
- you already know the metric_set from `tune_grid()` where we specified the metrics that we want to compute during our resampling approach

---

## 4.8 `yardstick`: Tidy Model Performance

**Class Probability Metrics**

.panelset[
.panel[.panel-name[ROC Curve]
```{r}
collect_predictions(cls_wf_last_fit) %>%
  roc_curve(died, .pred_TRUE, event_level = "second")
```
]
.panel[.panel-name[ROC-AUC]
```{r}
collect_predictions(cls_wf_last_fit) %>%
  roc_auc(died, .pred_TRUE, event_level = "second")
```
<br><br><br>
.footnote[
*Note: By default, `yardstick` views the first factor level as the positive class. If your outcome is one-hot encoded (e.g., `0`/`1` or `FALSE`/`TRUE`) and the event of interest relates to the second factor level, you have to make the `event_level` explicit.*
]
]
.panel[.panel-name[Multiple ROC Curves]
.pull-left[
The individual functions can be easily applied to our tuning results as well.
```{r roc_curves, eval=F}
collect_predictions(cls_wf_fit) %>% 
  group_by(id) %>% 
  roc_curve(
    died, .pred_TRUE,
    event_level = "second"
  ) %>% 
  autoplot()
```
]
.pull-right[
```{r, ref.label='roc_curves', echo=F, fig.width = 10, fig.asp = 0.618, fig.align='center'}
```
]
]
]



???
class probability metrics: we provide the probability column for the event of interest

Multiple Roc-Curves:
- yardsticks functions have a consistent API that allows to easily operate on grouped data
- ideally you would not only want to compare the model performance for different folds but also across models -> this is straightforward with `yardstick` as well

---

layout: false

## Excursus: Alternative Evaluation Dimensions

- **Scalability:** How well does the model scale to larger data sets? Is the speed of model re-training and prediction adversely affected in a real-time scenario?<br><br>
- **Robustness**: Is the model robust against perturbations (e.g., missing values or outliers) in the unseen data? Does the performance deteriorate rapidly in the context of [data drift](https://read.deeplearning.ai/the-batch/developing-ai-products-part-5-data-drift-concept-drift-and-other-maintenance-issues/)?<br><br>
- **Transferability:** Can the model be applied to related tasks without substantial re-training of the model and without a substantial loss of predictive accuracy?<br><br>
- **Interpretability:** Are predictions explainable? Can the relationship between a predictor and the outcome be extracted from the model?<br><br>
- **Fairness & Compliance:** Does the model systematically discriminate against certain sub-populations or ethical groups? Does it comply with prevalent law in a given domain?<br><br>
- **Justifiability:** Are the predictions in line with well-known business rules? Are the most important predictors consistent with prior beliefs?<br><br>
- **Causality:** Does the model enable causal inference? If not, does it enable the user to generate hypotheses that can be tested using alternative statistical approaches?<br><br>