Compute the Probability B Beats A and the Expected Loss for a Checkout Test
A checkout test ends with variant A at 120 purchases from 2,000 visitors and variant B at 141 purchases from 2,000. Use a \text{Beta}(1,1) prior on each arm. The team's pre-registered rule is "ship B if P(B \gt A) \ge 0.95 and the expected loss from shipping B is under 0.05 percentage points".
- Write the two posteriors and explain two ways of computing P(p_B \gt p_A) (one numeric integral, one simulation).
- How many Monte Carlo draws do you need for a standard error of 0.001, and what are the exact and simulated values?
- Define and compute the expected loss of shipping B and of keeping A.
- Apply the rule. What do you recommend, and what one sentence do you give the stakeholder about the 91 percent?
1. Posteriors and two methods. p_A \sim \text{Beta}(121, 1881) (mean 0.0604) and p_B \sim \text{Beta}(142, 1860) (mean 0.0709).
- Numerical integration: P(p_B \gt p_A) = \int_0^1 f_B(x)\,F_A(x)\,dx, with f_B the density of B's posterior and F_A the CDF of A's. Evaluate on a fine grid and apply the trapezoid rule.
- Simulation: draw many pairs (p_A^{(s)}, p_B^{(s)}) from the two Betas and take the fraction with p_B^{(s)} \gt p_A^{(s)}. This generalises: the same draws give the lift distribution, expected loss, and any other functional.
2. Accuracy. The exact value is 0.9103 (verified). Monte Carlo estimates were 0.9130 (1,000 draws, standard error 0.0089), 0.9119 (10,000 draws, se 0.0028) and 0.9099 (100,000 draws, se 0.0009; verified). The standard error is \sqrt{q(1-q)/N}; for q = 0.91 and se 0.001 you need N = 0.91 \times 0.09 / 0.001^2 \approx 82{,}000 draws. Use 100,000 or a million, which costs milliseconds.
3. Expected loss. Shipping B loses when B is actually worse:
From 200,000 draws: L_{\text{ship B}} = 0.000325 (0.033 points) and L_{\text{keep A}} = 0.0108 (1.08 points) (verified). The posterior mean lift is 1.05 points with 95 percent HDI -0.47 to +2.60 points, and P(\text{lift} \gt 0.5\ \text{points}) = 0.76.
4. Decision. The two conditions disagree: expected loss from shipping B (0.033 points) is under the 0.05 tolerance, but P(B \gt A) = 0.910 is below 0.95. The rule was stated as a conjunction, so the answer is do not ship yet, and, if there is room under the sample plan, collect more data (or accept that the pre-registered rule was too strict for a cheap, reversible change and change it for the next test, never mid-test). Sentence for the stakeholder: "B is probably better, about a 91 percent chance, and if we are wrong the expected cost is small (about 0.03 points), but we agreed on 95 percent before looking and we are not there yet." A good answer also reports the prior sensitivity and checks guardrails before any launch.
Share this question