# #StackBounty: #experiment-design #power #ab-test AB test sample size calculation by hand

### Bounty: 50

Evan Miller has created a well-known online AB test sample size calculator. For the sake of being able to program and modify this formula, I would like to know how to calculate sample size Evan Miller-style by hand.

Personally, I’ll calculate such a metric by working backwards from how we calculate a 95% confidence interval with the z-test of proportions around the difference in conversion between the two variations ($$hat{d}$$) by setting it zero.

I’ll define/assume:

• $$alpha$$ = .05, $$beta$$ = .2
• a 50/50 split between the control and experiment, i.e. $$n_exp$$ = $$n_control$$
• the control conversion rate, i.e. the base rate before the experiment = $$c$$
• $$p$$ = pooled conversion rate = (number of exp conversions + number of control conversions/ (n_control + n_experiment)) -> in this context -> $$(nc+n(c+hat{d}))/2n$$ = $$(2c+hat{d})/2$$

Now time to solve for $$n$$

$$hat{d} + Z_{(1+alpha)/2} * StandardError = 0$$
$$hat{d} + 1.96 * StandardError = 0$$
$$hat{d} + 1.96 * sqrt{p(1-p)(frac{1}{n_exp} + frac{1}{n_control})} = 0$$
$$hat{d} + 1.96 * sqrt{p(1-p)(frac{2}{n})} = 0$$
$$sqrt{p(1-p)(frac{2}{n})} =frac{-hat{d}}{1.96}$$

with more simplifying we get to:

$$frac{(1.96^2) 2p(1-p)}{hat{d}^2} = n$$
$$frac{(1.96^2) (2c+2chat{d}-2c^2+frac{3}{2}hat{d}^2)}{hat{d}^2} = n$$

At the moment though, my calculation doesn’t incorporate power (1-$$beta$$) but Evan Miller’s does.

What should I think about as next steps to incorporate power into my sample size calculation?

(Feel free to also point out other errors in my calculation or assumptions!)

Get this bounty!!!

This site uses Akismet to reduce spam. Learn how your comment data is processed.