Advanced
Open
Pro
Compute the Clipped Surrogate Objective for a Batch of Samples
A PPO update is being computed with clip range \epsilon = 0.2. The batch contains four (ratio, advantage) pairs from the current epoch:
| Sample | r_t(\theta) | A_t |
|---|---|---|
| 1 | 1.45 | +1.0 |
| 2 | 0.55 | −1.0 |
| 3 | 1.45 | −1.0 |
| 4 | 0.55 | +1.0 |
- Compute L^{\text{CLIP}} for each sample, showing the unclipped term, the clipped term, and which one the min operator selects.
- For each sample, state whether this sample's gradient contribution pushes r_t(\theta) further from 1, pulls it back toward 1, or is flat (zero gradient) with respect to r_t, and explain why.
- A teammate claims "clipping means PPO ignores samples 1 and 3 because their ratio is outside [0.8, 1.2]." Correct this misunderstanding using your answer to part 2.
Share this question