Match a job Paths Subjects Questions Quizzes Pricing
Intermediate Open Pro

Fix the Metric-Accumulation Bugs in an Evaluation Loop

A binary classifier (50,000 training samples, 10,010 validation samples) reports a training loss of 0.0050 per epoch, which looks too good, and a validation accuracy that changes by half a point when someone re-evaluates with a different batch size. The code:

train_dl = DataLoader(train_ds, batch_size=128, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=1000, drop_last=True)

def run_epoch(model, opt):
    model.train(); running = 0.0
    for x, y in train_dl:
        loss = F.binary_cross_entropy_with_logits(model(x), y)
        opt.zero_grad(); loss.backward(); opt.step()
        running += loss.item()
    return running / len(train_ds)

def evaluate(model):
    accs, aucs = [], []
    with torch.no_grad():
        for x, y in val_dl:
            p = torch.sigmoid(model(x))
            accs.append(((p > 0.5) == y).float().mean().item())
            aucs.append(roc_auc_score(y.numpy(), p.numpy()))
    return sum(accs) / len(accs), sum(aucs) / len(aucs)
  1. Identify every bug and quantify the effect: the training-loss denominator, and the validation accuracy if the ten full batches each have accuracy 0.90 while the 10 leftover samples have 4 correct.
  2. What does drop_last=True do to this validation set?
  3. Why are the per-batch AUCs averaged incorrectly, and what is the correct computation?
  4. Rewrite both functions correctly.

Share this question

← Back to The PyTorch Training Loop from Scratch practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.