How CLIP Enables Zero-Shot Classification, and Where It Breaks

A resale marketplace has 4M product photos, no category labels for the new "home & garden" vertical, and 300 target categories. A teammate proposes a CLIP-style model and says "it can classify without any training data". Another worries it is a gimmick and wants to label 50,000 images and fine-tune a CNN.

1. Explain how a contrastively trained image-text model was trained and how it classifies images into categories it was never trained on. Write down the exact inference procedure.
2. For an image whose cosine similarities with the prompts for three categories are 0.31, 0.29 and 0.22, compute the class probabilities with logit scale 100 and with scale 14.3. What does this say about calibration?
3. List four failure modes you would expect on this data and a check or mitigation for each.
4. Propose an escalation path from zero-shot to a deployed classifier that uses CLIP embeddings, with its costs and benefits compared with the 50,000-label CNN proposal.