Casual Inference Data analysis and other apocrypha
The False-Discovery Rate: An alternative to the FWER for multiple comparisons
We’ve previously explored one common method for dealing with testing multiple simultaneous hypotheses, methods that control the Family-wise error rate. However, we realized that the FWER can be quite conservative. The False-Discovery rate is a powerful alternative to the FWER, which is often used in cases where hundreds or thousands of simultane... Read more 09 Aug 2020 - 9 minute read
Getting started with multiple hypotheses - Methods that control the FWER and their costs
Multiple hypothesis testing shows up frequently in practice, but needs to be handled thoughtfully. Common methods like the Bonferroni correction are useful quick fixes, but they have important drawbacks that users should understand. *What do methods like Bonferroni and Bonferroni-Holm do, and why do they work? What alternatives are there? Banne... Read more 22 Jul 2020 - 18 minute read
Picking the model with the lowest cross validation error is not enough
We often pick the model with the lowest CV error, but this leaves out valuable information. Specifically, it ignores the uncertainty around the estimated out-of-sample error. It’s useful to calculate the standard errors of a CV score, and incorporate this uncertainty into our model selection process. Doing so avoids false precision in model sele... Read more 07 Jul 2020 - 11 minute read