Why Patch Tokenization Instead of CNN Feature Maps

Older image models (ResNet-style CNNs) produced a dense feature map where nearby pixels naturally share context through convolution. Modern vision-language models instead cut the image into a flat grid of patches and treat each one as an independent token. Why is this patch-based approach (Vision Transformer / ViT style) preferred for feeding images into an LLM, given it seems to throw away the built-in locality that convolution provides?