Casual Inference Data analysis and other apocrypha
Getting started with multiple hypotheses - Methods that control the FWER and their costs
Multiple hypothesis testing shows up frequently in practice, but needs to be handled thoughtfully. Common methods like the Bonferroni correction are useful quick fixes, but they have important drawbacks that users should understand. *What do methods like Bonferroni and Bonferroni-Holm do, and why do they work? What alternatives are there? Banne... Read more 22 Jul 2020 - 18 minute read
Picking the model with the lowest cross validation error is not enough
We often pick the model with the lowest CV error, but this leaves out valuable information. Specifically, it ignores the uncertainty around the estimated out-of-sample error. It’s useful to calculate the standard errors of a CV score, and incorporate this uncertainty into our model selection process. Doing so avoids false precision in model sele... Read more 07 Jul 2020 - 11 minute read