Reinforcement Learning & Long-term Optimization

Probability Fundamentals

Master conditional probability, Bayes' theorem and the base-rate fallacy, random variables, expectation and variance, the six distributions data scientists actually meet, the Law of Large Numbers and Central Limit Theorem, and the classic interview puzzles.

RL Foundations: MDPs & Value Functions

Learn how to model a real decision problem — a messaging agent choosing whether to send a notification — as a Markov Decision Process, define returns and discounting with worked numbers, derive the Bellman expectation and optimality equations from first principles, and use policy iteration and value iteration to compute optimal behavior.

Multi-Armed Bandits & Exploration

Learn the multi-armed bandit framework as the simplest form of reinforcement learning used in production: the explore/exploit tradeoff, ε-greedy, UCB1 with a derivation sketch, Thompson sampling with a worked Beta-Bernoulli example, and regret as the metric that makes 'how good is this exploration strategy' precise — contrasted directly with fixed-split A/B testing.

Contextual Bandits for Personalization

Learn how contextual bandits extend multi-armed bandits by conditioning the policy on features of the user and situation: the LinUCB algorithm and why a linear reward model gives closed-form confidence bounds, neural/deep contextual bandits for richer reward functions, action features vs. arm identity for scaling to large or changing catalogues, and position bias in bandit-served lists.

Value-Based Methods: Q-Learning to DQN

Learn temporal-difference learning, tabular Q-learning and its off-policy Bellman update, why tables collapse at scale, function approximation with neural networks, and the experience replay, target network and double-Q fixes that make Deep Q-Networks trainable — framed around the deadly triad of function approximation, bootstrapping and off-policy learning.

Policy Gradients & Actor-Critic

Learn the policy gradient theorem intuitively, REINFORCE as Monte Carlo policy gradient, why it has high variance, how a baseline and the advantage function reduce it without introducing bias, and how Advantage Actor-Critic (A2C) combines a policy (actor) with a learned value function (critic) into a stable, online training loop.

PPO & Modern Policy Optimization

Learn why unconstrained policy gradient steps can destroy a policy in one update, how TRPO's trust region fixes this with a constrained optimization problem, how PPO approximates that trust region cheaply with a clipped surrogate objective, how Generalized Advantage Estimation trades bias for variance in the advantage estimate, and the implementation details — advantage normalization, minibatch epochs, value clipping, KL early stopping — that determine whether a PPO run actually converges.

Model-Based RL & Planning

A survey of model-based reinforcement learning: learning a transition/reward model and planning against it, Monte Carlo Tree Search (selection, expansion, simulation, backpropagation, UCT), Dyna-style architectures that mix real and simulated experience, why sample efficiency is the core motivation, and when model-based approaches win (accurate-model domains like games) versus lose (hard-to-model domains like open-ended user behavior).

Reward Design & Delayed Credit Assignment

Learn potential-based reward shaping and why it provably preserves the optimal policy, how reward hacking and Goodhart's Law wreck real products, and the mechanics of credit assignment over long horizons — n-step returns and eligibility traces (TD(λ), forward and backward views) with worked numeric examples.

Long-Term Value & Delayed Reward Systems

Learn to design product objectives for delayed-reward systems: LTV as an optimization target, why and how to use fast proxy metrics without letting them diverge from the true objective, the end-to-end delayed-feedback problem (attribution windows, censored outcomes, label construction), the explicit short-term vs long-term metric tension as a business tradeoff, and retention-aware objective design that blends immediate reward with a bootstrapped long-term value estimate.

Off-Policy Evaluation & Offline RL

Learn why you cannot safely A/B test an untested RL policy in production, importance sampling and per-decision importance sampling for off-policy evaluation, IPS/SNIPS variance reduction, the doubly-robust estimator, the distribution-shift problem in offline RL and how Conservative Q-Learning addresses it conceptually, and why off-policy evaluation is the mandatory gate before any production RL launch.

RL in Production: Safe Exploration & Serving

Learn the systems discipline that separates a reinforcement learning paper from a reinforcement learning product: why you must log the probability your policy assigned to the action taken (not just the action) so later off-policy evaluation is possible, how to enforce hard guardrails a learned policy may never violate, how to bound exploration traffic and duration, the end-to-end serving architecture from feature fetch to action logging, and how a deployed policy shapes the very data it will be retrained on.

A/B Testing & Online Experimentation

Learn to design trustworthy online experiments: pick metrics and randomisation units, size a test with power and MDE, catch SRM and peeking, cut variance with CUPED, and read lifts, segments and holdouts correctly.

Causal Inference Basics

Learn potential outcomes, confounding and colliders, and the observational toolkit — regression adjustment, matching, IPW, difference-in-differences, regression discontinuity, instrumental variables and synthetic control — with worked examples and failure modes.

Ranking & Recommendation System Architecture

Learn the multi-stage recommendation architecture: candidate generation with two-tower models and ANN search, pointwise/pairwise/listwise rankers, multi-task value formulas, position debiasing, cold start, re-ranking policy, and how offline metrics relate to online A/B results.

Fine-Tuning: SFT, LoRA/QLoRA, RLHF and DPO

A deep, standalone treatment of fine-tuning for AI engineering interviews: the mechanics of SFT, LoRA/QLoRA, RLHF and DPO; realistic data requirements and where the data comes from; the sharp line between behaviour problems (fine-tune) and knowledge problems (RAG); training and serving cost order-of-magnitude; how to evaluate a fine-tune against catastrophic forgetting and a prompted baseline; and a worked decision scenario for a formatting-adherence bug in a support bot.

ML System Design Interview Framework

Learn how ML system design interviews are scored, a 7-step framework from requirements to monitoring, back-of-envelope estimation for QPS, embeddings and GPU cost, common problem framings, and the mistakes that sink strong candidates.

Case Study: Notification Timing & Content Optimization

Model interview answer for designing a reinforcement-learning system that decides what and when to send in a messaging product, optimizing for long-term retention instead of short-term opens: state, action and reward design that avoids reward hacking, a bandit-to-RL model progression, a mandatory off-policy evaluation gate before launch, production guardrails against fatigue, and an experiment design that measures retention rather than open rate.

Case Study: Long-Term Engagement Recommender

Model interview answer for redesigning a recommendation feed to optimize cumulative long-term engagement rather than immediate clicks: surrogate rewards built from a learned retention/session-return predictor, delayed-feedback handling in the training pipeline, exploration inside the retrieval-to-re-ranking funnel, and why a short A/B test on CTR can be actively misleading for a long-term-optimizing model.

Case Study: Designing an RLHF / Preference-Tuning Platform

Model interview answer for the platform-engineering capstone behind every instruction-tuned LLM: prompt sourcing and privacy filtering, comparison-UI and annotator-calibration design, SFT to reward-model to policy-training pipelines (and the DPO variant that collapses two stages into one), reward-hacking-aware evaluation, and a system design where rollout generation on inference-engine workers, not the learner, is the dominant cost. Grounded in InstructGPT, Bradley-Terry, DPO, RLAIF and open-source RLHF-infra patterns.