Casual Inference Data analysis and other apocrypha
How to QA your slop and stop embarrassing yourself (using Bayesian statistics)
QA your slop, you’re embarrassing yourself
As far as I can tell, every tech company around has gone absolutely slop wild.
If your brain is still working, you probably realize that you need to have some actual people check the output from time to time. But human attention is expensive. You can take batches of output and sample from them. This is about how to use statistics to figure out if your failure rate is good enough
- What is the failure rate for this batch of outputs?
- Is the failure rate within acceptable bounds?
- How many absolute failures are likely given the test cases you saw?
Bayesian analysis fast
Posterior = prior + data
Bayesian analysis is a little three-step dance that goes like this:
- Pick a prior
- Observe the data
- Update the prior to get the posterior. Use the posterior to calculate the probability $\mu < 5\%$.
🤔 Wait, what's a probability distribution?
|
probability distributions 🤔 more references
|
Prior
Beta(1, 1) prior
$\underbrace{\mu}{\text{Our prior on the rate}} \sim \underbrace{Beta(\alpha_0, \beta_0)}{\text{is a Beta distribution with params } \alpha_0, \beta_0}$
You can interpret the prior parameters as “hypothetical data” that summarizes your beliefs about the rate.
🤔 Why Beta(1, 1)? What about other choices
|
other choices 🤔 Tell me more
|
Update
Update equation
🤔 Why is the posterior still a beta distribution?
|
conjugate prior 🤔 What if I can't use a conjugate prior
|
$\underbrace{\mu \mid y, n}{\text{The posterior of the rate given the data}} \sim \underbrace{Beta(\alpha_0 + y, \beta_0 + N - y)}{\text{is given by this beta distribution}}$
Posterior analysis
Beta(a, b) and 95% CI on fail rate
🤔 What else can I do with the posterior?
|
sampling and differencing from scipy.stats import beta y = 40 n = 1000 a_0 = 1./3 b_0 = 1./3 posterior = beta(a_0 + y, b_0 + n - y) n_simulations = 100000 posterior_samples = posterior.rvs(n_simulations) print('Monte carlo estimate of P(Rate) < 5%: ', sum(posterior_samples < .05) / len(posterior_samples)) print('CDF(5%) = ', posterior.cdf(.05))🤔 What about
|
Posterior predictive
Beta binomial (n, a, b) and failure count
🤔 How could I use this to estimate the cost of failure
|
sampling simulation 🤔 What about
|