Advanced Data Science & Experimentation

Product Analytics, Metrics & LTV

Define, compute and sanity-check the numbers a product is steered by: the metric hierarchy (north star, driver metrics, guardrails, the OEC, a worked metric tree for a subscription product, leading versus lagging), what makes a metric good (sensitivity, directionality, actionability, time-to-signal, Goodhart's law, validating proxy metrics), core definitions stated precisely (DAU, WAU, MAU and stickiness, activation, N-day versus unbounded versus bracketed retention, churn, resurrection, cohort triangles, growth accounting with the quick ratio), funnels (strict versus any-order, conversion windows, time-to-convert, attribution caveats), distributions rather than averages (heavy tails, per-user versus per-event, the ratio-metric trap with a delta-method A/A simulation, winsorization), unit economics (LTV by the simple formula, by cohort, with discounting and margin, CAC, payback, LTV/CAC pitfalls, predictive LTV), segmentation and Simpson's paradox with a mix-versus-rate decomposition, instrumentation (event taxonomy, identity stitching, sessionization rules, tracking QA), dashboards and metric ownership, and a worked example that builds a retention triangle and growth accounting from 12 event rows and computes three LTV variants for one cohort. Every number was recomputed by running code.

A/B Testing & Online Experimentation

Learn to design trustworthy online experiments: pick metrics and randomisation units, size a test with power and MDE, catch SRM and peeking, cut variance with CUPED, and read lifts, segments and holdouts correctly.

Advanced Experimentation: Interference, Marketplaces & Platforms

Run experiments when units are not independent: how SUTVA breaks in social networks, marketplaces, ad auctions and shared infrastructure, why a rider-level test overstates a marketplace lift, how to diagnose interference with exposure mappings and neighbour regressions, and the designs that fix it (cluster randomization, ego-clusters, switchback with carryover and burn-in, budget and supply splits, geo experiments, staggered rollouts) with the design effect 1 + (m - 1) ICC, cluster-robust and permutation analysis, long-term effects (universal holdouts and their contamination, the surrogate index), heterogeneity without p-hacking, team-draft interleaving, non-inferiority tests, and the anatomy of an experimentation platform (hashing and salts, layers, triggered analysis). Includes a ride-hailing worked example with simulated numbers for bias, switchback block length and sample size.

Bayesian Statistics for Data Scientists

Do Bayesian inference the way data teams actually use it: Bayes' rule with sequential updating, conjugate Beta-Binomial, Gamma-Poisson and Normal-Normal updates worked by hand (posterior mean as a precision-weighted average, posterior predictive), credible versus confidence intervals (HDI versus equal-tailed, coverage that is not 95 percent everywhere), Bayesian A/B testing with P(B > A), expected loss, ROPE and prior sensitivity, what optional stopping really does to a posterior-threshold rule, hierarchical models with partial pooling and empirical Bayes, the exact link between L2 and a Gaussian prior, computation from grids and Laplace to Metropolis, HMC/NUTS and variational inference with R-hat, ESS and divergence diagnostics, prior and posterior predictive checks with LOO, Bayesian regression, and industry patterns such as Beta lower-bound ranking. Includes a two-variant test and a five-country shrinkage example whose numbers were run in NumPy, SciPy and PyMC.

Causal Inference Basics

Learn potential outcomes, confounding and colliders, and the observational toolkit — regression adjustment, matching, IPW, difference-in-differences, regression discontinuity, instrumental variables and synthetic control — with worked examples and failure modes.

Uplift Modeling & Heterogeneous Treatment Effects

Target the customers an intervention actually changes, not the ones who would have bought anyway: the move from ATE to the conditional effect tau(x), response models versus uplift models, the four quadrants (persuadables, sure things, lost causes, sleeping dogs) worked by hand with dollars, the data a CATE model needs (randomized logs, known propensities, overlap and balance checks), S-, T-, X-, R- and DR-learners with their bias and variance trade-offs, uplift trees and honest causal forests with confidence intervals, double/debiased machine learning with Frisch-Waugh-Lovell orthogonalization and cross-fitting, evaluation with uplift and Qini curves, AUUC, uplift@k, decile calibration and placebo tests (and why AUC fails), turning CATE into a budgeted cost-aware policy with IPS/DR policy value, multi-arm and continuous treatments, and operating the policy with a three-arm validation A/B. Includes a 50,000-user coupon simulation with a known true effect whose numbers were run in NumPy, scikit-learn and EconML.

Uncertainty Quantification & Conformal Prediction

Turn a point prediction into a statement you can act on: aleatoric versus epistemic uncertainty and the decisions that need an interval (a newsvendor order worked by hand), the classical tools and their assumptions (OLS prediction intervals, bootstrap, quantile regression with pinball loss, quantile crossing), why calibration is not coverage, and conformal prediction from first principles: exchangeability, nonconformity scores, split conformal with the quantile index ceil((n+1)(1-alpha)), the finite-sample marginal guarantee and its Beta-distributed spread, conformalized quantile regression, LAC/APS/RAPS prediction sets, class-conditional variants, full versus split versus CV+/jackknife+, and why conditional coverage cannot be guaranteed. Then weighted conformal under covariate shift, adaptive conformal inference for drift, deep ensembles and MC dropout and where they fail out of distribution, evaluating UQ with coverage, width, PICP/MPIW, CRPS and group audits, and using uncertainty for abstention, routing and risk-aware thresholds. Every number comes from a heteroscedastic simulation and a 10-class classifier run in NumPy and scikit-learn, including 500-point split conformal at 90 percent, CQR, a coverage-by-group audit, and a 3,000-split check against the Beta law.

Survival Analysis & Time-to-Event Modeling

Model how long something lasts when many of your observations are not finished yet: right, left and interval censoring versus truncation, what a 'churned by day 30' label throws away (a 20,000-user simulation where the naive label under-states 90-day churn by 11 points and an events-only analysis puts the median at 19 days instead of 46), and immortal-time bias (a feature with no effect that looks like a 76 percent hazard cut). Then the core functions (survival, hazard, cumulative hazard, restricted mean), the Kaplan-Meier estimator with Greenwood variance and log-log intervals, Nelson-Aalen and the log-rank test (including crossing curves where it has no power), Cox proportional hazards (partial likelihood, hazard-ratio interpretation, ties, Schoenfeld residual tests, stratification, time-varying covariates and effects), parametric and accelerated-failure-time models (Weibull, log-normal, log-logistic, and extrapolation that goes badly wrong when the family is wrong), competing risks (1-KM overstating cumulative incidence 61 percent versus 46, cause-specific versus Fine-Gray), recurrent events, machine learning for survival (random survival forests, boosted Cox, DeepSurv and DeepHit, and discrete-time survival as classification on person-period data, the practical workhorse), evaluation under censoring (Harrell and Uno concordance, time-dependent AUC, IPCW Brier score and IBS, calibration at a horizon), and applications from expected lifetime and LTV to credit default and readmission. The worked example computes Kaplan-Meier by hand for 10 users, interprets a Cox table fitted on 4,000 simulated subscribers, and converts survival curves into expected lifetime and LTV. Every number comes from code that was run.

Time Series Fundamentals

Master time series forecasting: trend and seasonal decomposition, stationarity and differencing, ACF/PACF, exponential smoothing and ARIMA/SARIMA, lag-feature gradient boosting, temporal backtesting with MAE, MAPE, sMAPE and MASE, prediction intervals, and the leakage traps interviewers probe.

Forecasting at Scale: Global, Probabilistic & Hierarchical Models

Take forecasting from one series to a million: why one global model trained across series beats per-series models once histories are short (a 100-series simulation where the per-series boosted model scored MASE 0.995 and a pooled model on 1,000 series scored 0.862), global LightGBM panels with lags, calendar, promo and price features, target scaling (raw targets lost 17 points of WAPE on the smallest-scale quintile), and direct versus recursive versus window horizons (a 14-model direct set, one window model and a recursive loop landed within 1.3 percent of each other). Then deep forecasters (N-BEATS and N-HiTS, DeepAR, Temporal Fusion Transformer, PatchTST) and time-series foundation models (TimesFM, Chronos, Moirai) with an honest account of when they beat boosting and why their benchmarks can leak; probabilistic forecasting with pinball loss, CRPS and coverage (quantile GBM covered 79.5 percent of an 80 percent interval, an in-sample residual band only 73.2); intermittent demand with Croston, SBA, TSB and Tweedie, where an all-zeros forecast won MAE by 30 percent and the metric choice flipped the winner three times; hierarchical and grouped forecasting with coherence, bottom-up, top-down, middle-out, OLS, WLS and MinT reconciliation, temporal hierarchies, and a 7-series worked example with the S matrix and the full MinT arithmetic; evaluation at scale with rolling-origin backtests, MASE, RMSSE, WAPE, pooled versus per-series aggregation and forecast value added; known-future inputs and price elasticity (an endogenous price turned a true elasticity of -1.5 into +0.38); and production concerns from retraining cadence to forecast overrides. Every number comes from code that was run.

Anomaly Detection for Tabular Data & Time Series

Learn to detect anomalies when almost nothing is labelled and every alert costs someone's time: point, contextual and collective anomalies, outlier versus novelty detection, and the alert budget as the first design input (20 true anomalies a day and a budget of 10 alerts caps recall at 50 percent before any model is chosen). Statistical baselines with their failure modes (z-scores flagged 1.64 percent of t3 data against 0.23 percent for normal data, the MAD-based robust z masked nothing at 10 percent contamination but failed at 33, generalized ESD found two outliers a single Grubbs pass missed, Mahalanobis separated a point whose coordinates were each ordinary); density and distance methods (kNN versus LOF on clusters of different density, scaling that moved AUROC from 0.809 to 1.000, distance concentration, DBSCAN noise); Isolation Forest worked by hand with the exact path-length arithmetic and sklearn's contamination='auto' flagging 479 rows for 50 real anomalies; PCA residuals and autoencoders that memorised 50 training anomalies down to an error 59 times lower than fresh ones; time series with robust STL residuals, seasonal-hybrid ESD, forecast-then-flag with its baseline lock-out failure (270 false-alarm days), CUSUM, PELT and Bayesian online changepoint detection, and the difference between an anomaly and a changepoint; streaming and drift; evaluation with precision@k from review samples, synthetic injection that scored recall 0.00 on the real detector, PR-AUC, range-based recall and point-adjust inflation (a random detector scoring F1 0.13 instead of 0.036); and detector-to-system design with severity, feature attribution, analyst feedback and grouping. A worked example sets thresholds for 10 alerts a day on a 5-feature transaction model and on 10,000 daily metrics. Every number comes from code that was run.

Pricing Optimization & Marketplace Economics

Set prices and match supply to demand with economics, estimation and optimization rather than guesswork: the demand curve, own- and cross-price elasticity, the Lerner markup rule and why a revenue-maximizing price is almost never a profit-maximizing one (a 4 percent profit gain from a 17 percent revenue loss in the worked example); estimating elasticity when observed prices are endogenous (a naive log-log slope of -1.09 against a true -2.0, repaired by randomization at -2.01 and by a cost instrument at -2.16), price experiments, double machine learning, conjoint and pooling across products; multinomial and nested logit demand with the cross-elasticity matrix and the independence-of-irrelevant-alternatives trap, cannibalization and promotion pull-forward worked by hand; single-product first-order conditions, margin floors, price ladders and MAP, a multi-SKU linear program over price scenarios with inventory and margin constraints whose duals are read as shadow prices (0.47 and 0.38 of profit per extra unit of stock, 7.74 per point of margin floor), markdown optimization, and the legal and ethical limits of personalized pricing; revenue management with Littlewood protection levels (a protection level of 44 seats maximizing simulated revenue), surge pricing and the supply response it needs, and bandits with an off-policy-evaluation gate; two-sided marketplaces (liquidity, take rate, subsidies, network effects, bipartite matching where Hungarian assignment saved 14.7 percent of ETA over greedy, stable matching, dispatch); an operations research primer on LP and MIP duals, solvers and heuristics; and how to measure pricing changes with guardrails, long-term effects and fairness audits. Every number comes from code that was run (NumPy, SciPy HiGHS, statsmodels, fixed seeds).

Responsible ML: Interpretability, Fairness & Privacy

Explain, audit and protect a model before it decides things about people: why regulation (EU AI Act risk tiers, GDPR, fair-lending adverse action) and plain debugging value make this part of the job, the interpretability taxonomy (intrinsic versus post-hoc, global versus local), global methods with their traps (impurity importance ranked a pure-noise feature at 0.103 above a real predictor at 0.053 while permutation importance gave the noise feature -0.001; correlated twins split permutation importance; PDP range 0.12 hiding ICE curves that swing 0.83), local methods (LIME instability across seeds and kernel widths, Shapley axioms, exact brute-force Shapley values matching TreeSHAP to 3e-9 and summing exactly to the prediction, interaction values, counterfactual recourse, Integrated Gradients, Grad-CAM, why attention and LLM self-explanations are not explanations), and turning SHAP into adverse-action reason codes without reading it as causal; fairness (protected attributes and proxies, demographic parity, equal opportunity and equalized odds, predictive parity, calibration within groups, the impossibility theorem demonstrated with numbers, individual versus group fairness, sources of bias, feedback loops), interventions (reweighing, constraints, group thresholds, the four-fifths rule, audits) with a measured trade-off table; privacy (data minimization, k-anonymity limits, differential privacy with the Laplace and Gaussian mechanisms, composition, DP-SGD, federated averaging, memorization in LLMs, synthetic-data caveats, retention); and governance artifacts (model cards, datasheets, risk assessments, human oversight, incident response). The worked example is a simulated credit model with two groups, run in scikit-learn and SHAP, with every number reproduced from code.

Case Study: Marketing Uplift Targeting

Model interview answer for a coupon-targeting problem: define incremental conversions per euro under a budget with unsubscribe and margin guardrails, mine a past randomized campaign (5 percent treated) with a leakage and balance audit, show numerically why the usual response-model baseline targets sure things and loses money, build T-, X-, DR-learners and a causal forest with cross-fitting, evaluate with Qini/AUUC, decile calibration and placebo tests, turn scores into a budgeted and cost-aware policy, estimate its value from logs with IPS and doubly robust estimators, validate it in a three-arm online test with a ghost-treatment holdout and a worked sample-size calculation, and operate it with batch scoring, effect-drift monitoring and an exploration budget. All numbers come from a fixed-seed synthetic simulation with known ground truth.

Case Study: Demand Forecasting for Retail & Logistics

Model interview answer for a retail replenishment forecasting problem (50,000 SKUs by 500 stores, daily, 28 days ahead): frame the work as an inventory decision rather than a point forecast, treat stockouts as censoring of demand, climb a forecast-value-added ladder from seasonal naive and ETS to a global LightGBM with Tweedie loss, compare four ways of handling censored history, produce quantile and lead-time-sum forecasts, convert them to order-up-to levels with the newsvendor critical ratio and its cost asymmetry, reconcile a store/region/total hierarchy with bottom-up, WLS and MinT and check coherence, run a rolling-origin backtest with horizon-wise metrics and leakage checks, size a batch architecture from explicit arithmetic, and monitor bias, FVA and drift. All numbers come from a fixed-seed synthetic panel with known true demand.

Case Study: Dynamic Pricing for a Marketplace

Model interview answer for surge-style dynamic pricing in a ride-hailing or delivery marketplace: a constrained reward with liquidity and trust guardrails, a simulation showing that a user-level A/B recovers only 16 percent of the true profit effect, a switchback design with block length, burn-in, cluster standard errors and a design effect, a supply-response measurement, a policy ladder from a clearing rule to a Thompson-sampling bandit with a measured exploration cost, an off-policy gate (IPS, SNIPS, doubly robust) that catches a candidate breaching the exposure cap, a non-inferiority test on a trust metric, and serving and monitoring plans. All numbers come from a re-runnable synthetic simulation.