Independence, Joint Tables, and Zero Correlation
An analytics team studies whether users who enable dark mode are more likely to be paying subscribers. Out of all users, 40% use dark mode. The joint proportions are:
| Paying | Free | |
|---|---|---|
| Dark mode | 0.10 | 0.30 |
| Light mode | 0.15 | 0.45 |
- Compute the marginal probability of paying and the conditional probabilities P(\text{Paying} \mid \text{Dark}) and P(\text{Paying} \mid \text{Light}). Are dark-mode usage and paying status independent?
- A second analyst computes the Pearson correlation between a user's daily session count X and a feature Y = (X - \bar{X})^2 (squared deviation from the mean) and finds it is approximately zero. They conclude the two are unrelated. What is wrong with this conclusion?
- Two microservices each have a 1% chance of failing on a given day. An engineer estimates the chance both fail on the same day as 0.01 × 0.01 = 0.0001. What assumption is being made, and give a concrete reason it could be badly wrong.
1. Marginals, conditionals, and the independence test
P(\text{Paying}) = 0.10 + 0.15 = 0.25. P(\text{Dark}) = 0.40, P(\text{Light}) = 0.60.
P(\text{Paying} \mid \text{Dark}) = 0.10 / 0.40 = 0.25. P(\text{Paying} \mid \text{Light}) = 0.15 / 0.60 = 0.25.
Both conditionals equal the marginal 0.25, and every joint cell equals the product of its marginals (0.40 \times 0.25 = 0.10, 0.40 \times 0.75 = 0.30, 0.60 \times 0.25 = 0.15, 0.60 \times 0.75 = 0.45). Dark mode and paying status are independent in this data. Knowing a user's theme tells you nothing about whether they pay. (Note that the "0.10 of all users are dark-mode payers" figure looks smaller than the 0.15 for light mode only because there are fewer dark-mode users overall — a marginal-vs-conditional confusion the analyst should avoid.)
2. Zero correlation is not independence
Pearson correlation measures linear association only. Y = (X - \bar{X})^2 is a deterministic function of X — maximally dependent — but the relationship is U-shaped: for a distribution roughly symmetric about its mean, large positive and large negative deviations both give large Y, so the positive and negative products in the covariance cancel and \rho \approx 0. Independence implies zero correlation; the converse is false. A scatter plot would show the parabola immediately, which is why correlation matrices should be paired with plots.
3. The hidden independence assumption
Multiplying 0.01 \times 0.01 asserts that the two services fail independently: P(A \cap B) = P(A)P(B). In practice, failures share causes — the same cloud region, the same deployment pipeline, the same upstream database, the same traffic spike, the same on-call engineer pushing a bad config. If both depend on a shared database that is down 1% of days, then P(\text{both fail}) \ge 0.01, one hundred times the naive estimate. Correlated failure is the reason "five nines from two redundant three-nines components" so rarely materialises. Whenever you multiply probabilities, name the independence assumption and look for common causes that break it.
Share this question