Descriptive Statistics & Exploratory Data Analysis

Every modelling failure that is not a modelling failure — the feature that leaked the label, the column that was in cents on one system and dollars on another, the 30% missing values that were not missing at random, the "significant" effect that reverses when you split by segment — is caught, or missed, during exploratory data analysis. EDA is the part of the job where you earn the right to trust the numbers you are about to compute. It is also the part where analysts most often skip straight to model.fit().

Interviewers test this in two ways. The take-home or live-coding round hands you a messy CSV and watches what you look at first. The conceptual round asks "how would you handle outliers?", "what is the difference between MCAR and MAR?", "when is the median better than the mean?", or "what is Simpson's paradox and have you seen it?". Both reward the same thing: a structured, sceptical habit of describing data before drawing conclusions from it.

This subject builds that habit. The probability vocabulary (random variables, expectation, variance, distributions) is assumed from the Probability Fundamentals subject. Transforming and encoding features for a model is the Feature Engineering subject; querying the data out of a warehouse is the SQL for Data Analysis subject. Here the question is simply: what is in this dataset, and can I trust it?

Measures of Centre

Given a sample x_1, \dots, x_n:

The mean is the value that minimises the sum of squared deviations; the median minimises the sum of absolute deviations. That is exactly why the mean chases outliers (squares amplify them) and the median does not.

When mean vs median: skew

Right-skewed (income, latency, order value):    Left-skewed (exam scores near ceiling):

  █                                                                 █
  ██                                                               ██
  ███▄                                                           ▄███
  █████▄▂▁      ▁                                        ▁      ▂▄█████
  ─────────────────────                              ─────────────────────
   mode  median  mean                                   mean  median  mode
         (mean pulled right by the tail)                (mean pulled left)

For a right-skewed distribution the mean is typically greater than the median, because the long right tail contributes large values to the sum. So:

• Report the median for "what does a typical user/order/request look like?" — p50 latency, median household income, median session length.
• Report the mean when the total is what matters, because \text{total} = n \times \bar{x} — average revenue per user drives revenue forecasts even though most users spend below it.
• Report both and let the gap between them tell you how skewed the data is.

Trimmed means (drop the top and bottom 5% and average the rest) and winsorised means (clip rather than drop) are compromises used when you need mean-like additivity with some robustness.

Measures of Spread

The SD is interpretable through the Normal empirical rule (68/95/99.7% within 1/2/3 SD) only when the data are approximately Normal. For skewed or heavy-tailed data, "two standard deviations" means nothing in particular, and the IQR or explicit percentiles should carry the message instead.

Percentiles and quantiles

The p-th percentile is the value below which p\% of the data fall. Quartiles are the 25th, 50th, 75th percentiles; deciles split into tenths. Engineers live on p50 / p95 / p99 latency because tail behaviour is what users notice and what SLOs are written against — a mean latency of 120 ms is compatible with 1% of requests taking 4 seconds.

Two practical facts:

• There is no single definition of a sample quantile. numpy and pandas default to linear interpolation between order statistics; textbooks often use "median of the lower/upper half"; other software offers nine variants. On small samples the numbers differ, so state the method if it matters.
• Percentiles of aggregates are not aggregates of percentiles: you cannot average per-server p99s to get fleet p99. Compute quantiles on the pooled data (or use a mergeable sketch such as t-digest).

Distribution Shape

Beyond centre and spread, three shape features change how you should summarise and model.

Skewness measures asymmetry. The sample skewness is (roughly) the average cubed standardised deviation, \frac{1}{n}\sum \left(\frac{x_i - \bar{x}}{s}\right)^3. Positive = right tail longer (income, prices, counts, durations); negative = left tail longer (scores capped at 100, ages at retirement); around zero = symmetric. As a rough guide, |\text{skew}| > 1 is strongly skewed. When a variable is strongly right-skewed and positive, plotting it on a log scale (or np.log1p for data with zeros) is usually the fastest way to see its structure — whether you also feed the transformed version to a model is a Feature Engineering decision.

Kurtosis measures tail weight — how much probability sits far from the centre relative to a Normal. Reported as excess kurtosis (Normal = 0; pandas.Series.kurt() uses this convention). Positive excess kurtosis means heavy tails: extreme values happen far more often than the Normal predicts. Financial returns, network latencies, insurance claims, and most "amount" columns are heavy-tailed. Consequences: the sample mean and SD are unstable from sample to sample, the empirical 68/95/99.7 rule fails badly, and z-score outlier rules break down.

Modality. A bimodal histogram (two peaks) is almost always two populations mixed together — mobile and desktop, weekday and weekend, two pricing tiers, before and after a change. Summary statistics of a mixture are meaningless; find the segmenting variable.

Symmetric, light tails      Right-skewed, heavy tail       Bimodal (two populations)
      ▂▅█▅▂                     █▇▄▂▁▁ ▁   ▁                  ▂▅█▅▂   ▂▅█▅▂
   ───────────              ────────────────────           ────────────────────
   mean ≈ median            mean > median; kurtosis > 0    which segment is this?

Visualisation: Which Plot Reveals What

The purpose of an EDA plot is to reveal something a summary number hides. Match the plot to the question.

Two habits worth building. First, always plot the raw distribution before trusting a summary — Anscombe's quartet is four datasets with identical means, variances, correlation, and regression line whose scatter plots look nothing alike. Second, plot against time even for "non-time-series" data — a sudden step in a feature's mean usually means a logging change, not a change in the world.

import seaborn as sns
import numpy as np

df["order_value"].hist(bins=50)                       # shape
np.log1p(df["order_value"]).hist(bins=50)             # skewed → look on log scale
sns.boxplot(data=df, x="segment", y="order_value")   # compare groups
sns.ecdfplot(data=df, x="latency_ms", hue="region")  # exact percentiles per group
sns.pairplot(df[["age", "tenure", "spend", "visits"]].sample(2000))
sns.heatmap(df.corr(numeric_only=True), annot=True, cmap="coolwarm", center=0)

Outliers: Detection and What to Do

An outlier is a point far from the bulk of the data. It might be a data-entry error, a unit mismatch, a genuine extreme (a whale customer, a Black Friday day), or the most important row in the file (fraud). Detection is mechanical; the decision about what to do is not.

Detection rules

Z-score. z_i = (x_i - \bar{x}) / s; flag |z| > 3. Simple, but the mean and SD used to compute z are themselves inflated by the outliers — a single huge value can drag s up until its own z-score looks unremarkable (masking). Fails on skewed data, where legitimate right-tail values exceed 3s routinely.

IQR rule (Tukey fences). Flag points below Q_1 - 1.5 \times \text{IQR} or above Q_3 + 1.5 \times \text{IQR} (use 3 \times \text{IQR} for "far out"). Quartiles are robust, so the fences are not corrupted by the outliers themselves. This is what a box plot draws.

Robust z-score. (x_i - \text{median}) / (1.4826 \times \text{MAD}); flag |z| > 3.5. Robust and on a familiar scale.

Isolation Forest (intuition). Build random trees that split on random features at random thresholds. A point that is far from everything else gets isolated in a leaf after very few splits; a point inside a dense cluster needs many. Average path length across trees becomes an anomaly score. Works in many dimensions where "far from the centre" on any single axis is not the issue — the outlier is a combination of values (age 25, tenure 30 years).

Worked example

Ten daily order values (in dollars):

12, 15, 15, 18, 20, 22, 25, 30, 45, 200

• Mean = 402 / 10 = 40.2. Median = (20 + 22)/2 = 21. Mode = 15.
• Sample SD: deviations squared sum to 29,191.6; s^2 = 29{,}191.6 / 9 \approx 3{,}243.5; s \approx 57.0.
• Quartiles (median-of-halves method): lower half {12,15,15,18,20} → Q_1 = 15; upper half {22,25,30,45,200} → Q_3 = 30; IQR = 15. (pandas' default linear interpolation gives Q_1 = 15.75, Q_3 = 28.75 — different numbers, same conclusion.)
• IQR fences: 15 - 22.5 = -7.5 and 30 + 22.5 = 52.5. The value 200 is flagged; 45 is not.
• Z-score of 200: (200 - 40.2)/57.0 \approx 2.8 — below the usual |z| > 3 threshold. The outlier has inflated the SD enough to hide itself. This is masking in one line of arithmetic.
• MAD: absolute deviations from the median 21 are {9, 6, 6, 3, 1, 1, 4, 9, 24, 179}; sorted, the median is 6. Robust z of 200 = (200 - 21)/(1.4826 \times 6) \approx 20. Unmissable.
• Drop the 200 and the mean falls from 40.2 to 22.4 while the median moves from 21 to 20. One point moved the mean by 80%; the median barely noticed.

What to do about them

1. Investigate before touching. Look at the whole row. Is it a unit error (200 was in cents elsewhere)? A test account? A duplicate? A real whale?
2. Fix what is demonstrably wrong (units, typos, sentinel codes). Document it.
3. Keep genuine extremes if the analysis is about totals or tails — removing your top 1% of customers to make the histogram prettier is a lie about revenue.
4. Cap / winsorise at a chosen percentile (e.g. p99) when the model or metric is sensitive to extremes and the exact tail value is not the point.
5. Use robust statistics (median, IQR, MAD, Spearman) or robust models rather than deleting data.
6. Model them separately if they are their own population (fraud, enterprise accounts).

Never silently delete. Every dropped row should be a deliberate, logged decision with a stated rule.

Missing Data: Mechanisms and Handling

The pattern of missingness matters more than the amount. Rubin's taxonomy:

You cannot prove which mechanism holds from the data alone (MNAR is by definition invisible), but you can gather evidence: does the missing indicator for column A correlate with column B, with time, with a segment, with the target? A column that is missing for 40% of mobile users and 2% of desktop users is not MCAR.

Hidden missingness. Missing values are often encoded as real-looking numbers: 0, −1, 999, 9999, 1900-01-01, "N/A", "", "unknown", "null" as a string. A spike in a histogram at exactly such a value is the tell. Convert them to true nulls before computing any statistic.

Handling options

• Drop rows when the fraction is tiny and plausibly MCAR. Report how many.
• Drop the column when it is mostly empty and not central to the question.
• Impute with the median (numeric, robust to skew) or mode (categorical) as a baseline; with a group-wise median (per segment) when MAR is plausible; with a model (regression, KNN, iterative imputation) when the column matters and relationships are strong.
• Add a missing indicator column alongside any imputation. Missingness is often predictive in its own right, and the indicator lets a downstream model use it. (Whether to do this for a model is a Feature Engineering choice; noticing that missingness carries signal is EDA.)
• Leave it and use a model that handles nulls natively — tree-based models such as LightGBM route missing values in a learned direction.

Whatever you choose, impute inside the train/validation split logic, never on the full dataset before splitting — otherwise validation rows influence the imputed values seen by training rows.

df.isna().mean().sort_values(ascending=False)          # missing fraction per column
df.replace({-1: np.nan, 999: np.nan, "": np.nan})       # unmask sentinel codes (per column, once verified)
df.assign(income_missing=df["income"].isna()).groupby("age_band")["income_missing"].mean()  # MAR evidence

Data Quality Checks

A short list of checks that catch most of the disasters. Run them before any statistic you intend to repeat.

Duplicates. Exact duplicate rows (df.duplicated().sum()) and — more importantly — duplicate keys (df.duplicated(subset=["order_id"]).sum()). Duplicated keys usually mean a join fanned out or a pipeline reran. Both silently inflate every count and every sum.

Types. Numbers stored as strings ("1,234", "$12.50"), dates stored as strings in mixed formats, booleans as "Y"/"N"/"yes"/1, IDs stored as floats (which mangles large integers). df.dtypes and df.info() first; pd.to_numeric(errors="coerce") and pd.to_datetime to fix and to surface the rows that would not parse.

Units and scales. The same quantity in cents on one source and dollars on another; seconds vs milliseconds; kilograms vs pounds; percentages as 0–1 vs 0–100. A bimodal histogram with modes 100× apart is the classic symptom. Range checks (describe(), min/max, negative ages, prices of zero) catch the rest.

Categoricals. Cardinality (nunique()), value counts, whitespace and case variants (" Premium" vs "premium"), rare levels, and levels that only appear after a certain date (schema changes).

Timestamps. Time zones (naive vs aware, UTC vs local), events in the future, created_at > updated_at, gaps in the time series (outages), and duplicated periods (backfills).

Leakage of future information. The most expensive one. Any column that would not exist at prediction time — refund_amount when predicting churn, days_to_close when predicting whether a lead closes, last_login computed after the churn date, aggregates that include the row's own outcome. Ask of every column: when is this value known? A feature that is suspiciously predictive (correlation 0.9 with the target) is guilty until proven innocent. Detecting leakage is EDA; designing features that avoid it is Feature Engineering.

Row counts against ground truth. Total orders in your extract vs the number in the finance dashboard. If they differ by more than a rounding error, stop.

df.info()
df.describe(include="all").T
df.duplicated(subset=["order_id"]).sum()
df["country"].str.strip().str.lower().value_counts().head(20)
(df["event_ts"] > pd.Timestamp.now(tz="UTC")).sum()     # future timestamps
df.corr(numeric_only=True)["target"].sort_values()      # suspiciously strong correlations → leakage?

Correlation Matrices and Simpson's Paradox

Pearson correlation measures linear association; Spearman correlates the ranks and therefore captures any monotone relationship and shrugs off outliers. Compute both; where they disagree substantially, the relationship is non-linear or a few points are dominating. Use df.corr(method="spearman", numeric_only=True).

A correlation heatmap is a map of redundancy as much as of signal: blocks of highly correlated features (seven versions of "activity in the last N days") tell you the effective dimensionality is lower than the column count. What to do about that — dropping, combining, regularising — is covered in Feature Engineering and Regularization (L1/L2).

Simpson's paradox

An association that holds in every subgroup can reverse when the groups are pooled, if group membership is unevenly distributed across the thing being compared. It is the single most important reason to always segment before concluding.

Worked example. A new checkout flow is compared to the old one:

The new flow converts better on mobile and better on desktop, yet worse overall. The reason is the mix: the new flow was mostly shown to mobile users (80%), who convert at a lower rate on any flow, while the old flow was mostly shown to desktop users. Device is a confounder — it affects both which flow a user saw and how likely they were to convert. The pooled comparison is answering "who saw more desktop traffic?", not "which flow is better?".

The defence is procedural: compare within segments, or better, make sure the assignment does not depend on the segment in the first place (randomisation — the A/B Testing subject) and, when it does, adjust for it (the Causal Inference Basics subject). In EDA the job is to notice: any time a pooled rate disagrees with the segment rates, look for the lurking variable. The classic textbook instance is a kidney-stone treatment comparison where the treatment that won on both small and large stones lost overall because it had been given the harder cases.

A Structured EDA Checklist

Do these in order; each step feeds the next.

1. Provenance and grain. Where did the data come from, what does one row represent, over what date range, and does the row count match a source of truth?
2. Schema pass. df.info(): dtypes, non-null counts, memory. Fix types before computing anything.
3. Univariate — numeric. describe(), then a histogram (and a log-scale histogram for anything positive and skewed) for every numeric column. Note skew, modality, spikes at sentinel values, impossible ranges.
4. Univariate — categorical. value_counts() for every categorical. Note cardinality, dominant level, rare levels, whitespace/case duplicates.
5. Missingness. Fraction per column, co-occurrence between columns, and relationship to segments and time. Decide MCAR / MAR / MNAR provisionally and choose handling.
6. Duplicates and keys. Exact duplicates and duplicated primary keys.
7. Time. Plot volume and key means per day/week. Look for gaps, steps, seasonality, future dates.
8. Bivariate. Correlation matrix (Pearson and Spearman); scatter or box plots of each candidate feature against the target; box plots of numerics by key categoricals.
9. Segments and Simpson. Recompute the headline metric by the two or three most important segments. Does the pooled story survive?
10. Outliers. IQR/robust-z flags per column; inspect the flagged rows in full; decide and document.
11. Leakage audit. For every column: when is it known relative to the target event? Anything with implausibly high target correlation gets investigated.
12. Write it down. A short findings note — what is dirty, what was fixed, what assumptions were made — travels with the analysis.

Worked Example in pandas

A compact pass over an e-commerce orders table.

import pandas as pd
import numpy as np

df = pd.read_csv("orders.csv")

# 1–2. grain and schema
print(df.shape)                       # (48_211, 9)
df.info()
df["order_ts"] = pd.to_datetime(df["order_ts"], utc=True, errors="coerce")
df["amount"] = pd.to_numeric(df["amount"].str.replace(r"[$,]", "", regex=True), errors="coerce")

# 3. numeric univariate
df["amount"].describe(percentiles=[.5, .9, .95, .99])
#   mean 63.4 | 50% 31.0 | 95% 210.0 | 99% 640.0 | max 48_000.0   ← heavy right tail
df["amount"].skew(), df["amount"].kurt()      # e.g. 21.3, 780.4 → very heavy tail
np.log1p(df["amount"]).hist(bins=60)          # bimodal on log scale? two currencies?

# 4. categorical univariate
df["country"] = df["country"].str.strip().str.upper()
df["country"].value_counts().head()

# 5. missingness
df.isna().mean().sort_values(ascending=False)
#   coupon_code 0.71 | shipping_region 0.06 | amount 0.003
df.assign(m=df["shipping_region"].isna()).groupby("channel")["m"].mean()   # MAR by channel?

# 6. keys
df.duplicated(subset=["order_id"]).sum()      # 312 → a join fanned out; investigate

# 7. time
df.set_index("order_ts").resample("D")["amount"].agg(["count", "mean"]).plot(subplots=True)

# 8–9. bivariate and segments
df.groupby("device")["converted"].mean()
df.pivot_table(index="device", columns="flow", values="converted", aggfunc="mean")

# 10. outliers (IQR rule)
q1, q3 = df["amount"].quantile([.25, .75]); iqr = q3 - q1
flag = (df["amount"] < q1 - 1.5 * iqr) | (df["amount"] > q3 + 1.5 * iqr)
df.loc[flag].sort_values("amount", ascending=False).head(20)   # look at whole rows

# 11. leakage audit
df.corr(numeric_only=True)["converted"].sort_values()
#   refund_amount  0.93  ← known only after the order; drop before modelling

What this run typically surfaces: an amount column that was partly in cents (bimodal on the log scale), 312 duplicated order IDs from a bad join, shipping_region missing far more often for one channel (MAR), a handful of $48,000 test orders from an internal account, and a refund_amount column that would leak the label. None of it needs a model to find; all of it would have wrecked one.

Common Mistakes and Interview Traps

• Reporting the mean of a skewed variable as "typical". Average session length of 14 minutes with a median of 3 means most sessions are short and a few are enormous. Say which one you mean, and why.
• Applying the 68/95/99.7 rule to non-Normal data. "Two SDs above the mean" is meaningless for latency or revenue; use percentiles.
• Using the z-score to find outliers in the presence of outliers. The SD is inflated by the very points you are looking for (masking). Use the IQR fence or MAD-based robust z.
• Deleting outliers to make the model fit. Investigate, fix, cap, or model robustly — and document. Deleting your largest customers is not cleaning.
• Treating 0 / −1 / 999 as data. Sentinel codes masquerade as values and shift every mean. Look for spikes in the histogram.
• Assuming missingness is MCAR because it is inconvenient otherwise. Check whether the missing indicator relates to segments, time, or the target. Median imputation on MNAR data manufactures a false certainty.
• Imputing before splitting. Statistics computed on the whole dataset leak validation information into training.
• Trusting a correlation matrix without a scatter plot. Anscombe's quartet; Pearson misses non-linear and non-monotone relationships; one point can create or destroy a correlation.
• Reading a pooled metric without segmenting. Simpson's paradox is not exotic — any time group mix differs between the things being compared, the pooled number can lie.
• Ignoring the timestamp on "static" data. Level shifts in a feature almost always mean a logging or schema change. Plot everything against time once.
• Missing leakage because the feature is "obviously useful". Suspiciously strong correlation with the target is a red flag, not a win. Ask when the value becomes known.
• Averaging percentiles. Fleet p99 is not the mean of per-host p99s. Pool the data or use a mergeable sketch.

Key Takeaways

• Centre: mean for totals and symmetric data, median for "typical" and skewed data, mode for categoricals and sentinel detection. Report both mean and median; the gap measures skew.
• Spread: SD/variance for symmetric data; IQR and MAD are the robust equivalents. Percentiles (p50/p95/p99) describe tails honestly.
• Shape: skewness (asymmetry), excess kurtosis (tail weight), modality (mixtures). Heavy tails break z-scores and the empirical rule; log-scale plots reveal structure in skewed data.
• Choose plots by question: histogram for shape, box plot to compare groups, ECDF for exact percentiles, scatter for relationships, pair plot for overview, heatmap for redundancy, line-over-time for stability. Always plot before trusting a summary.
• Outliers: detect with the IQR fence or MAD-based robust z (z-scores mask); then investigate, fix, keep, cap, or model robustly — never silently delete.
• Missing data: MCAR (ignorable), MAR (recoverable using observed variables), MNAR (needs modelling or external data). Unmask sentinel codes; consider a missing indicator; impute inside the split.
• Data quality: duplicates and duplicated keys, wrong types, unit mismatches, categorical variants, timestamp sanity, row counts against a source of truth, and — above all — leakage of information not available at prediction time.
• Correlations: compute Pearson and Spearman; disagreement means non-linearity or outliers. Segment before concluding — Simpson's paradox reverses pooled comparisons when group mix is confounded with treatment.
• Follow the checklist every time: provenance → schema → univariate → missingness → keys → time → bivariate → segments → outliers → leakage → write-up.