Project Examples
Selected projects and simulations
A/B Testing
The gold standard of measuring impact: randomized experimentation aka A/B testing. Here is an example where we want to test ...
Improve statistical power with CUPED
There's a tradeoff between experiment precision and required sample size. Increased sample size usually requires increased runtimes, so requiring more precision typically means longer experiments. The CUPED methodology improves precision by controlling for observable characteristics of experiment participants, reducing required sample size and runtime.
Measuring the impact of negative events
A delivery order is late for the holidays. Or your website is buggy, affecting a portion of your customer base. You may want to invest or insure against these bad outcomes, but don't want to overspend if these bad experiences don't meaningfully reduce lifetime customer value.
Importantly, you do NOT want to run an A/B test and randomly assign a 'bad' treatment to your users. We can still measure the impact of these negative outcomes using natural experiments and causal inference methods
Power Analysis Simulation
This simulator helps you understand the relationship between sample size and statistical power. The user identifies the baseline conversion rate for users (control) and the minimum conversion rate they want to detect for treated users. The program then calculates the required sample size, simulates 1000 experiments, and displays the share with statistically significan results
