Multimodal LLMs and Vision

"How does GPT-4V know which item in the image I'm asking about?" is a question that trips people up because it imports the wrong mental model. Classical computer vision — YOLO, Faster R-CNN, SSD — was built to answer "where are the objects and what are they," and it answers with coordinates: a bounding box, a class label, a confidence score. A multimodal LLM like GPT-4V or GPT-4o was not built to do that. It was built to answer "what does this image mean, in the context of this conversation," and it answers with tokens — the same next-token-prediction machinery it uses for text. Understanding that difference explains both why these models are so good at open-ended visual reasoning and why they're unreliable at exactly the tasks — precise counting, tight localization, telling apart two visually similar items — that a purpose-built detector handles well.

This subject covers how a vision-language model actually sees an image, how it distinguishes between multiple objects in a scene, where that mechanism breaks down, and the practical techniques used in production systems to work around the gaps.

From Pixels to Tokens: How the Model "Sees"

A multimodal LLM doesn't look at an image the way a detector does. The image is converted into the same kind of sequence the language model already knows how to consume:

1. Patchify. The image is cut into a grid of fixed-size patches (classically 14×14 or 16×16 pixels, following the Vision Transformer / ViT approach). A 224×224 image with 16×16 patches becomes a 14×14 grid — 196 patches.
2. Embed. Each patch is linearly projected into a vector — a "visual token" — the same shape as a text token embedding.
3. Position-encode. Each patch token gets a positional embedding so the model knows patch (3, 7) is to the right of and below patch (2, 6).
4. Project into the LLM's embedding space. A trained adapter (a small MLP or cross-attention layer, depending on architecture) maps the vision encoder's output space into the language model's token space, so visual and text tokens can sit in the same sequence.
5. Feed the sequence to the transformer. From here it's ordinary decoder-only self-attention: the model attends over a mix of image-patch tokens and text tokens (the prompt) to generate the next output token.

The vision encoder itself (commonly a CLIP-style or SigLIP-style ViT) is usually pretrained separately on hundreds of millions of image–text pairs using a contrastive objective: the encoder learns to produce an image embedding that's close to the embedding of its correct caption and far from embeddings of unrelated captions. That pretraining is why the model has any notion of "cat" or "stop sign" in visual form before it ever sees a downstream task — it never had bounding-box labels, only (image, caption) pairs.