Debugging a Leaky Gradient Boosting Forecast Pipeline

A colleague built a LightGBM model to forecast daily store revenue 14 days ahead. Backtest MAE looks excellent — far better than seasonal naive — but the model's real-world performance after deployment is much worse. Their feature pipeline, computed once over the full historical dataframe before any train/test split:

df["roll_mean_7"] = df.groupby("store")["revenue"].transform(
    lambda s: s.rolling(7).mean()
)
df["lag_1"] = df.groupby("store")["revenue"].shift(1)
df["target"] = df.groupby("store")["revenue"].shift(-14)
# random 80/20 split:
train, test = train_test_split(df, test_size=0.2, shuffle=True)

1. Identify every leakage bug in this pipeline (there are at least three).
2. Rewrite the pipeline description (in words or pseudocode) so it is safe for a 14-day-ahead forecast.
3. Why does a leaky pipeline often show a great backtest score but fail in production, rather than simply erroring out?