Intermediate
Open
Pro
Analyse the Same A/B Data with a Bayesian and a Frequentist Test and Reconcile Them
A small test has A at 41 conversions out of 800 visitors and B at 56 out of 800. The frequentist analyst reports a two-sample z-test; the Bayesian analyst reports P(B \gt A) with flat priors.
- Compute the key outputs of both analyses (p-value, interval, P(B \gt A), expected loss).
- Explain precisely why one report says "not significant" and the other says "94 percent chance B is better". Is either wrong?
- How sensitive is the Bayesian answer to the prior?
- Write the two-sentence summary you would give the PM, and say what additional data would settle the question.
Share this question