Reward Design & Delayed Credit Assignment
A notification system sends a push notification, and eleven days later the user cancels their subscription. Somewhere between "notification sent" and "subscription cancelled" is a causal chain the learning algorithm never sees directly — dozens of other notifications, sessions, purchases, and moments of friction, all folded into a single delayed, noisy outcome. The central engineering problem of this subject is: given a reward signal that arrives late, sparse, and entangled with everything else the agent did, how do you get credit (or blame) onto the specific decisions that caused it, without teaching the agent to cheat?
That problem splits cleanly into two questions, and this track answers them in two different subjects on purpose. This subject — Reward Design & Delayed Credit Assignment — is about mechanics: given a reward signal you have already decided to use, how do you shape it so learning stays fast without changing what "optimal" means, and how do you propagate credit backward through a long chain of actions so the agent can actually learn from a delayed signal. The companion subject, Long-Term Value & Delayed Reward Systems, is about product objectives: what should that reward signal even measure — a proxy like predicted watch-time, or a slower ground truth like 90-day retention — and how do you resolve the very real business tension between a metric that moves this week and one that reflects whether the business is healthy in a year. If an interviewer asks "how would you reward this agent?", the mechanics in this subject are what stop your shaped reward from being gamed; the objective design in the companion subject is what stops your reward from optimizing the wrong thing in the first place. You need both, and conflating them is the most common way candidates lose points on this topic.
Everything here builds on the MDP formalism from RL Foundations: MDPs & Value Functions and feeds directly into Value-Based Methods: Q-Learning to DQN and Policy Gradients & Actor-Critic, where these credit-assignment mechanisms are what actually gets implemented inside the learning update.