Practice — Bayesian Statistics for Data Scientists (6 questions)

An email team has run five earlier campaigns with an average click rate of about 4 percent. They encode that as a \text{Beta}(8, 192) prior on the click probability p of the next campaign. The campaign is sent to 500 people and 30 click.

1. Derive the posterior from Bayes' rule, showing why the Beta prior is conjugate to the Binomial likelihood.
2. Give the posterior, its mean, and show the mean as a weighted average of the prior mean and the observed rate. What is the weight on the data and what does "worth 200 pseudo-visitors" mean?
3. A manager asks for the number of clicks to expect when the campaign goes to 1,000 more people. Why is plugging in the posterior mean the wrong calculation, and what is the right distribution?
4. How do the answers change with a flat \text{Beta}(1,1) prior, and what does that tell you about prior influence?

A checkout test ends with variant A at 120 purchases from 2,000 visitors and variant B at 141 purchases from 2,000. Use a \text{Beta}(1,1) prior on each arm. The team's pre-registered rule is "ship B if P(B \gt A) \ge 0.95 and the expected loss from shipping B is under 0.05 percentage points".

1. Write the two posteriors and explain two ways of computing P(p_B \gt p_A) (one numeric integral, one simulation).
2. How many Monte Carlo draws do you need for a standard error of 0.001, and what are the exact and simulated values?
3. Define and compute the expected loss of shipping B and of keeping A.
4. Apply the rule. What do you recommend, and what one sentence do you give the stakeholder about the 91 percent?