Intermediate
Open
Pro
Fix the Metric-Accumulation Bugs in an Evaluation Loop
A binary classifier (50,000 training samples, 10,010 validation samples) reports a training loss of 0.0050 per epoch, which looks too good, and a validation accuracy that changes by half a point when someone re-evaluates with a different batch size. The code:
train_dl = DataLoader(train_ds, batch_size=128, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=1000, drop_last=True)
def run_epoch(model, opt):
model.train(); running = 0.0
for x, y in train_dl:
loss = F.binary_cross_entropy_with_logits(model(x), y)
opt.zero_grad(); loss.backward(); opt.step()
running += loss.item()
return running / len(train_ds)
def evaluate(model):
accs, aucs = [], []
with torch.no_grad():
for x, y in val_dl:
p = torch.sigmoid(model(x))
accs.append(((p > 0.5) == y).float().mean().item())
aucs.append(roc_auc_score(y.numpy(), p.numpy()))
return sum(accs) / len(accs), sum(aucs) / len(aucs)
- Identify every bug and quantify the effect: the training-loss denominator, and the validation accuracy if the ten full batches each have accuracy 0.90 while the 10 leftover samples have 4 correct.
- What does
drop_last=Truedo to this validation set? - Why are the per-batch AUCs averaged incorrectly, and what is the correct computation?
- Rewrite both functions correctly.
Share this question