Probability Fundamentals

Probability is the language that everything else in data science is written in. A confidence interval is a statement about a random variable. An A/B test is a comparison of two Bernoulli parameters. A logistic regression outputs a conditional probability. A spam filter is Bayes' theorem with a large feature vector. If the vocabulary of sample spaces, conditional probability, expectation, and distributions is shaky, every downstream topic — hypothesis testing, regression, model evaluation, causal inference — becomes memorised recipes instead of things you can reason about.

Interviewers know this, which is why data science loops so reliably contain a probability round: a medical-test Bayes question, a "what distribution models this?" question, a birthday-problem or Monty Hall puzzle, and a CLT question dressed up as an A/B testing scenario. None of these require heavy mathematics. All of them require you to set up the problem correctly, which is the skill this subject builds.

This subject stops at the boundary of inference: how to go from a sample back to a claim about the population (p-values, confidence intervals, power) is covered in the Hypothesis Testing subject. Here we build the machinery that inference stands on.

Sample Spaces, Events, and the Axioms

A random experiment is any process whose outcome is uncertain. Its sample space \Omega is the set of all possible outcomes. An event is a subset of \Omega.

Probability is a function P from events to numbers satisfying three axioms:

1. P(A) \ge 0 for every event A
2. P(\Omega) = 1
3. If A and B are mutually exclusive (disjoint, A \cap B = \emptyset), then P(A \cup B) = P(A) + P(B)

Everything else follows. The most-used consequences:

The complement rule is the workhorse of "at least one" problems: P(\text{at least one}) = 1 - P(\text{none}). The inclusion–exclusion rule corrects for double-counting the overlap.

        Ω
  ┌───────────────────────────┐
  │        A          B       │
  │    ┌──────┐   ┌──────┐    │      P(A ∪ B) counts the shaded
  │    │      │███│      │    │      overlap A ∩ B once, so you
  │    │      │███│      │    │      subtract it after adding
  │    └──────┘   └──────┘    │      P(A) + P(B).
  └───────────────────────────┘

Worked example. A die is rolled once. A = "even" = {2,4,6}, B = "greater than 3" = {4,5,6}. Then P(A) = 3/6, P(B) = 3/6, A \cap B = {4,6} so P(A \cap B) = 2/6, and P(A \cup B) = 3/6 + 3/6 - 2/6 = 4/6. Check by listing: A \cup B = {2,4,5,6}, four outcomes. Correct.

Conditional Probability and Independence

The probability of A given that B has happened is

Conditioning shrinks the sample space to B and renormalises. Rearranged, it gives the multiplication (chain) rule: P(A \cap B) = P(A \mid B)\,P(B), and for many events P(A_1 \cap A_2 \cap A_3) = P(A_1)\,P(A_2 \mid A_1)\,P(A_3 \mid A_1 \cap A_2).

Worked example. From a standard 52-card deck, draw two cards without replacement. P(\text{both aces}) = P(\text{first ace}) \cdot P(\text{second ace} \mid \text{first ace}) = \frac{4}{52} \cdot \frac{3}{51} = \frac{12}{2652} \approx 0.0045.

Independence

A and B are independent if knowing one tells you nothing about the other:

Independence is a modelling assumption, and it is the one most often silently violated. Two page views by the same user are not independent. Two servers behind the same power supply do not fail independently. Sales on consecutive days are not independent. Whenever you multiply probabilities together, you are asserting independence — say so out loud and check that it is plausible.

Mutually exclusive is not independent. Disjoint events with positive probability are maximally dependent: if A happened, B certainly did not. P(A \cap B) = 0 \ne P(A)P(B). Interviewers love this confusion.

Conditional independence. A and B may be dependent overall but independent given C: P(A \cap B \mid C) = P(A \mid C)\,P(B \mid C). This is exactly the assumption the naive Bayes classifier makes: word occurrences are treated as independent given the class label. It is wrong, and it works remarkably well anyway.

Law of Total Probability

If B_1, \dots, B_k partition \Omega (mutually exclusive, jointly exhaustive):

This is how you compute an overall rate from segment rates: overall conversion = mobile conversion × share of mobile users + desktop conversion × share of desktop users. It is also the denominator of Bayes' theorem.

Bayes' Theorem and the Base-Rate Fallacy

Combine the definition of conditional probability with the chain rule and you get the single most important formula in this subject:

Bayes' theorem lets you flip a conditional: you usually know P(E \mid H) (a test's sensitivity, how often spam contains "free") and want P(H \mid E) (does this patient have the disease? is this email spam?). The two are not the same, and confusing them is the base-rate fallacy.

Worked example: the medical test

A disease has a prevalence of 1%. A test has sensitivity 99% (if you have the disease, it is positive 99% of the time) and specificity 95% (if you do not have it, it is negative 95% of the time — so the false-positive rate is 5%). You test positive. What is the probability you actually have the disease?

Most people answer "about 99%". Let's compute it, and then see why the intuition fails.

About 17%. The reason is the base rate: the disease is rare, so the small false-positive rate applied to the huge healthy population produces far more false positives than the high sensitivity applied to the tiny sick population produces true positives. The natural-frequency picture makes it obvious:

10,000 people tested
├── 100 have the disease  (1%)
│     ├── 99 test positive   ← true positives
│     └──  1 tests negative
└── 9,900 are healthy      (99%)
      ├── 495 test positive  ← false positives (5% of 9,900)
      └── 9,405 test negative

Positive tests: 99 + 495 = 594
P(disease | positive) = 99 / 594 ≈ 0.167

If the prevalence were 30% instead of 1%, the same test would give P(D \mid +) = \frac{0.99 \times 0.3}{0.99 \times 0.3 + 0.05 \times 0.7} = \frac{0.297}{0.332} \approx 0.89. Same test, wildly different posterior — because the prior changed. When someone hands you a test accuracy, always ask for the base rate.

Worked example: a one-word spam filter

20% of incoming email is spam. The word "free" appears in 60% of spam and 5% of legitimate mail. An email contains "free":

Now the posterior is high, because the base rate (20%) is not tiny and the likelihood ratio 0.60 / 0.05 = 12 is large. A naive Bayes classifier just repeats this update once per word, multiplying likelihoods under the conditional-independence assumption, and compares the two posteriors.

The odds form

Bayes' theorem is cleanest in odds: \text{posterior odds} = \text{prior odds} \times \text{likelihood ratio}. In the medical example, prior odds are 1:99; the likelihood ratio of a positive test is 0.99 / 0.05 = 19.8; posterior odds are 19.8 : 99 = 1 : 5, i.e. probability 1/6 \approx 0.167. Same answer, no denominators. This is the form used in logistic regression, where the log-odds are linear in the features.

Random Variables, PMF, PDF, CDF

A random variable X is a function that assigns a number to each outcome in \Omega. "Number of heads in 3 flips" turns {HHH, HHT, ...} into {3, 2, ...}. Once you have numbers you can take averages, compute variances, and plot distributions.

Discrete random variables take countable values (counts, categories coded as integers). Their distribution is described by the probability mass function:

Continuous random variables take values in an interval (latency, revenue, temperature). P(X = x) = 0 for any single point; probability lives on intervals and is described by the probability density function f_X:

A density is not a probability. f_X(x) can exceed 1 — a Uniform on [0, 0.5] has density 2 everywhere. Only areas under the density are probabilities.

The cumulative distribution function works for both:

It is non-decreasing, runs from 0 to 1, and for continuous variables f_X = F_X'. Two facts you will use constantly:

• P(a < X \le b) = F_X(b) - F_X(a)
• The quantile function is the inverse CDF: the median is F_X^{-1}(0.5), the 95th percentile is F_X^{-1}(0.95). "p99 latency" is exactly this object.

 PMF (discrete)              PDF (continuous)            CDF (either)
 P(X=x)                      f(x)                        F(x)
  │    ▄                       │     ╭─╮                   1 ┤        ╭────
  │  ▄ █ ▄                     │    ╱   ╲                    │      ╱
  │▄ █ █ █ ▄                   │  ╱       ╲                  │    ╱
  └──────────── x              └──────────── x             0 ┼──╱────────── x
 heights sum to 1              area under curve = 1        rises from 0 to 1

Expectation and Variance

The expectation (mean) is the probability-weighted average of the values X can take:

The variance measures spread around the mean, and its square root, the standard deviation, puts that spread back in the units of X:

Worked example: one fair die. E[X] = \frac{1+2+3+4+5+6}{6} = 3.5. E[X^2] = \frac{1+4+9+16+25+36}{6} = \frac{91}{6} \approx 15.17. So \mathrm{Var}(X) = 15.17 - 3.5^2 = 15.17 - 12.25 = 2.92 and \sigma \approx 1.71.

Linearity of expectation

For any random variables, dependent or not, and constants a, b:

This is the most underrated tool in the subject. Expected sum of two dice is 3.5 + 3.5 = 7 — no need to enumerate 36 outcomes. Expected number of matched pairs, expected number of empty buckets in a hash table, expected number of records that collide: define an indicator variable I_j \in \{0, 1\} per item, note E[I_j] = P(I_j = 1), and add them up. Dependence between the indicators is irrelevant to the expectation.

Worked example. 100 users each convert independently with probability 0.05. Expected conversions = 100 \times 0.05 = 5. Even if the users were not independent, the expectation would still be 5 — only the variance would change.

Variance rules

If X and Y are independent (or merely uncorrelated), the covariance term vanishes and variances add. Note that \mathrm{Var}(X - Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) for independent variables — the variances still add, because (-1)^2 = 1. This is why the standard error of a difference between two A/B arms is \sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}, not a subtraction.

The Distributions a Data Scientist Actually Meets

You do not need dozens of distributions. You need to recognise these six by the story that generates them.

Bernoulli and Binomial

A single conversion, click, churn event, or coin flip is Bernoulli(p). Count n of them and you have Binomial(n, p):

Worked example. 10 fair coin flips, exactly 7 heads: \binom{10}{7} (0.5)^{10} = 120 / 1024 \approx 0.117.

Every conversion-rate metric you will ever report is a Binomial count divided by n; that is why its standard error is \sqrt{p(1-p)/n} — the Binomial variance divided by n^2, square-rooted.

Poisson

Requests per second, support tickets per day, defects per metre of cable, goals per match. If events arrive independently at an average rate \lambda per window, the count X in a window is Poisson:

Worked example. A service averages 3 requests per second. P(\text{zero requests in a second}) = e^{-3} \approx 0.050. P(X \ge 5) = 1 - \sum_{k=0}^{4} P(X=k) = 1 - (0.050 + 0.149 + 0.224 + 0.224 + 0.168) \approx 0.185. Capacity planned for exactly the mean load will be exceeded far more often than intuition suggests.

Poisson is also the limit of Binomial(n, p) when n is large and p small with np = \lambda fixed — rare events among many trials. A diagnostic: if the sample variance of your counts is much larger than the mean, the data are overdispersed and Poisson is a poor model (a Negative Binomial usually fits better).

Uniform

Every value in [a, b] equally likely. Rare in nature, essential in computation: random() returns Uniform(0, 1), and every other distribution is simulated by transforming it (inverse-CDF sampling). Also the right null model for "the hash function should spread keys evenly" and "the p-value under the null hypothesis is Uniform(0,1)".

Normal (Gaussian)

Arises whenever a quantity is the sum or average of many small independent contributions (heights, measurement error, and — thanks to the CLT — every sample mean). Standardise with the z-score z = (x - \mu)/\sigma and use the empirical rule:

             68.3%  of mass within  μ ± 1σ
             95.4%  of mass within  μ ± 2σ
             99.7%  of mass within  μ ± 3σ

              ▁▂▄▆█▇█▆▄▂▁
     ─────────┼──┼──┼──┼──┼──┼─────────
             -3σ -2σ -1σ  μ  +1σ +2σ +3σ

Worked example. Latency is Normal with mean 200 ms and SD 30 ms. P(\text{latency} > 260): z = (260 - 200)/30 = 2, so about 2.3\% (half of the 4.6% outside \pm 2\sigma). Real latency distributions are right-skewed and heavy-tailed, which is exactly why p99 is reported rather than the mean — but the Normal is the right tool for the sample mean of latency, per the CLT.

Exponential

If events are Poisson with rate \lambda, the waiting time T until the next one is Exponential(\lambda):

Its defining property is memorylessness: P(T > s + t \mid T > s) = P(T > t). Having already waited 10 minutes for the bus (or a server having already run 1,000 hours without failure) does not change the distribution of the remaining wait. Time-between-arrivals and constant-hazard failure times are the standard uses; anything with wear-out (hazard increasing with age) is not Exponential.

Law of Large Numbers and the Central Limit Theorem

These two theorems answer different questions about the sample mean \bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i of n i.i.d. draws with mean \mu and variance \sigma^2.

Law of Large Numbers (LLN): where does the sample mean go? As n \to \infty, \bar{X}_n \to \mu. Averages of more data are closer to the true mean. This is why a conversion rate measured on 100,000 users is trustworthy and one measured on 20 is not, and why Monte Carlo simulation works at all.

Central Limit Theorem (CLT): how is the sample mean distributed around \mu for finite n? Regardless of the shape of the individual X_i (skewed, discrete, bounded — as long as the variance is finite),

The quantity \sigma/\sqrt{n} is the standard error of the mean. Two things to internalise:

1. The CLT is a statement about the sample mean (or sum), not about the raw data. Latency does not become Normal because you collected more of it; the average latency does.
2. Precision improves with \sqrt{n}, not n. To halve the standard error you need four times the data. This is the arithmetic behind every "how many users do we need for this test?" conversation.

Population (skewed):        Sample means, n = 5:      Sample means, n = 50:
█▇▅▃▂▁▁▁▁▁                     ▂▅█▇▅▃▂▁                    ▁▃█▃▁
└──────────                    └──────────                  └──────────
                               roughly bell-shaped         tight, Normal
                                                           SE = σ/√50

Why this underpins A/B testing and confidence intervals

An A/B test compares two conversion rates. Each rate is a Binomial count over n — a sample mean of Bernoulli variables — so by the CLT each is approximately Normal with standard error \sqrt{p(1-p)/n}. Their difference is a difference of independent Normals, hence Normal with \mathrm{SE}_{\text{diff}} = \sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}. Every z-test, every "95% CI = estimate ± 1.96 × SE", and every sample-size calculator is this theorem applied.

Worked example. Baseline conversion is 5% with 10,000 users per arm. \mathrm{SE}_A = \mathrm{SE}_B = \sqrt{0.05 \times 0.95 / 10{,}000} \approx 0.00218 (0.218 percentage points). \mathrm{SE}_{\text{diff}} = \sqrt{2} \times 0.00218 \approx 0.0031. So an observed lift of 0.3 percentage points is about one standard error — indistinguishable from noise — while a lift of 0.9 pp is about three. Deciding what counts as "significant", how to control error rates, and how to size the test are covered in the Hypothesis Testing and A/B Testing subjects; this subject explains why the Normal shows up in the first place.

Joint, Marginal, and Conditional Distributions

When you have two random variables, the joint distribution P(X = x, Y = y) describes them together. Summing (or integrating) over one variable gives the marginal of the other; dividing gives the conditional.

Worked example. Site visits by device and conversion outcome:

• Marginal conversion: P(C) = 0.03 + 0.04 = 0.07.
• Conditional: P(C \mid \text{mobile}) = 0.03 / 0.60 = 0.05, P(C \mid \text{desktop}) = 0.04 / 0.40 = 0.10.
• Independence check: P(\text{mobile}) \cdot P(C) = 0.60 \times 0.07 = 0.042 \ne 0.03. Device and conversion are not independent — desktop users convert at twice the rate.

Notice the law of total probability at work: 0.05 \times 0.60 + 0.10 \times 0.40 = 0.07. Also notice that a shift in device mix alone (more mobile traffic) would lower the overall conversion rate with no change in either segment's behaviour — the seed of Simpson's paradox, explored in the Descriptive Statistics & EDA subject.

Covariance and Correlation

Covariance measures how two variables move together:

Its sign is meaningful (positive: move together; negative: move oppositely) but its magnitude depends on the units of X and Y — covariance of height and weight changes if you switch from cm to inches. Pearson correlation normalises it to [-1, 1]:

Worked example. x = (1, 2, 3, 4, 5), y = (2, 4, 5, 4, 5). Means: \bar{x} = 3, \bar{y} = 4. Deviations: x - \bar{x} = (-2, -1, 0, 1, 2), y - \bar{y} = (-2, 0, 1, 0, 1). Products: (4, 0, 0, 0, 2), sum 6, so \mathrm{Cov} = 6/5 = 1.2 (population form). \sigma_x = \sqrt{10/5} = 1.414, \sigma_y = \sqrt{6/5} = 1.095. \rho = 1.2 / (1.414 \times 1.095) \approx 0.77. Strong positive linear association.

Three things correlation is not:

• Not causation. Ice-cream sales and drownings correlate because both depend on temperature. Establishing cause is the subject of the Causal Inference Basics subject.
• Not a measure of non-linear dependence. If X is symmetric around 0 and Y = X^2, then \mathrm{Cov}(X, Y) = 0 but Y is a deterministic function of X. Independence implies zero correlation; zero correlation does not imply independence (except in the special case of jointly Normal variables).
• Not robust. One extreme point can drag Pearson's \rho anywhere. Spearman's rank correlation, discussed in the EDA subject, is the usual defence.

Correlation is also what makes variances of sums interesting: a portfolio (or an ensemble of models) built from positively correlated components has more variance than the independent case predicts, and diversification only helps to the extent that correlations are low.

Classic Interview Puzzles

These are asked because they test whether you can set a problem up, not because the arithmetic is hard.

The birthday problem

How many people must be in a room for the chance that two share a birthday to exceed 50%? Attack the complement: the probability that n people all have different birthdays is

For n = 23 this product is about 0.493, so P(\text{at least one match}) \approx 0.507. Twenty-three people. The intuition-breaker is that you are not comparing each person to you; you are comparing all \binom{23}{2} = 253 pairs. The useful approximation P(\text{match}) \approx 1 - e^{-n(n-1)/(2 \times 365)} gives 1 - e^{-0.693} \approx 0.5 for n = 23.

Same mathematics governs hash collisions: with H buckets, the expected number of keys until the first collision is roughly \sqrt{\pi H / 2} \approx 1.25\sqrt{H}. A 32-bit hash (H = 2^{32}) is expected to collide after about 82,000 keys, not four billion.

Monty Hall

Three doors, one car, two goats. You pick door 1. The host, who knows where the car is, opens a goat door among the other two and offers a switch. Switch or stay?

Switch — it wins 2/3 of the time. Your initial pick is right with probability 1/3, and nothing the host does changes that (he can always open a goat door). The remaining 2/3 of the probability was spread over doors 2 and 3; the host's action concentrates all of it on the unopened door. The key modelling detail is that the host's choice is not random — he never reveals the car. If a host who did not know opened a door and it happened to be a goat, switching would be a coin flip. Interviewers probe exactly this: "what changes if the host picks randomly?"

Expected waits and indicators

Two more shapes worth recognising. "How many coin flips until the first head?" — geometric, expected 1/p, so 2 for a fair coin. "n people leave their hats in a pile and each grabs one at random; how many get their own hat back on average?" — indicators: each person's chance is 1/n, linearity gives n \times 1/n = 1, regardless of n and despite the indicators being dependent.

Common Mistakes and Interview Traps

• Confusing P(A \mid B) with P(B \mid A). "The test is 99% accurate, so a positive means 99% chance of disease." No — that is P(+ \mid D), and you want P(D \mid +), which depends on the base rate. In court this is called the prosecutor's fallacy.
• Treating mutually exclusive as independent. They are opposites: disjoint events with positive probability cannot be independent.
• The gambler's fallacy. After five reds, the roulette wheel is not "due" for black. The LLN says the proportion converges; it does not say past deviations get compensated. Independent trials have no memory.
• Believing a density is a probability. PDF values can exceed 1; P(X = x) = 0 for continuous X. Only integrals of a density are probabilities.
• Applying the CLT to the data instead of the mean. More observations do not make a skewed distribution Normal. They make its sample mean Normal.
• Subtracting variances. \mathrm{Var}(X - Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) for independent variables. Standard errors of a difference add in quadrature.
• Reading zero correlation as independence. Y = X^2 is the canonical counter-example. Always look at the scatter plot.
• Multiplying probabilities without checking independence. Two events on the same user, the same day, or the same machine are rarely independent, and the resulting probability of "both fail" can be off by orders of magnitude.
• Ignoring the host's knowledge in Monty Hall. The 2/3 answer depends on the host deliberately revealing a goat.
• Modelling counts with a Normal when they are small. A Poisson with \lambda = 2 is very skewed and cannot go negative; a Normal approximation is only reasonable once \lambda (or np and n(1-p)) is comfortably large, roughly 10 or more.

Key Takeaways

• Probability is a function on events satisfying three axioms; the complement rule and inclusion–exclusion do most of the day-to-day work.
• P(A \mid B) = P(A \cap B)/P(B); independence means P(A \cap B) = P(A)P(B). Mutually exclusive events are dependent, not independent.
• Bayes' theorem flips a conditional. Always weigh the likelihood against the base rate: a 99%-accurate test for a 1%-prevalence condition gives a posterior of only about 17%.
• A PMF gives point probabilities; a PDF gives densities whose areas are probabilities; the CDF F(x) = P(X \le x) works for both, and quantiles are its inverse.
• E[aX + bY] = aE[X] + bE[Y] always — use indicators. \mathrm{Var}(X \pm Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) when independent.
• Know six stories: Bernoulli (one trial), Binomial (count of n trials), Poisson (events per window), Uniform (equally likely / random numbers), Normal (sums of many small effects), Exponential (memoryless waiting time).
• LLN: the sample mean converges to \mu. CLT: the sample mean is approximately Normal with standard error \sigma/\sqrt{n} — the foundation of confidence intervals and A/B testing, developed in the Hypothesis Testing subject.
• Marginals sum out; conditionals divide; independence means the joint factorises.
• Correlation is scale-free covariance in [-1, 1]; it is not causation and zero correlation does not imply independence.
• Birthday problem: 23 people for a 50% match, because there are 253 pairs. Monty Hall: switch, 2/3 — because the host knows.