How a Vision LLM Works: A Practitioner's Read of the 2026 Multimodal Stack

Vision LLMs don't see images. They tokenize them. The image gets compressed into a few thousand tokens by an encoder before the language model ever touches it, and whatever doesn't survive that compression isn't in the model's input. Once that lands, every other behavior of these models, the cost, the failure modes, the resolution sensitivity, falls out cleanly.

Most explainers skip this. They jump straight to "Claude can read this chart" or "Gemini handles documents." That's the surface. The actual mental model worth carrying around is much more mechanical, and once you have it, every product surface starts making sense.

I've been wiring vision LLMs into agentic pipelines for several months: document parsing, screenshot-driven UI agents, chart reads, multi-image comparisons. Claude with vision, Gemini 3, GPT-5, Qwen3.5 on a self-hosted box. All four labs publish polished marketing pages and almost no architecture documentation. Most published deep-dives are either two years out of date or paper summaries written for ML PhDs. This post is the in-between read I wanted and couldn't find: a practitioner's mental model of what a 2026-era vision LLM is actually doing under the hood.

The cast as of April 2026, per the Artificial Analysis Vision leaderboard: Gemini 3.1 Pro Preview and GPT-5.5 lead on MMMU Pro at 80-82%, with Claude Opus 4.7, Kimi K2.6, GPT-5.4, and Grok 4.3 all within a few points. The open-source side is dominated by the Qwen3.5 family (397B A17B, 122B A10B, 27B), with GLM-4.6V, MiMo-V2-Omni, and Gemma 4 holding their own at smaller sizes. None of the closed labs publish architecture details. This list will be stale within months: vision-model leaderboards turn over fast and the spread between top-10 models is small enough that one release reorders the top half. Spot-check the leaderboard before quoting any specific name. The architecture story below is the part that's stable; the names on top of it rotate.

In this post:

1. The "model just sees pixels" fallacy: why thinking of vision as a black box is the wrong starting point
2. The three building blocks: vision encoder, connector, language model
3. How an image becomes tokens: patches, ViT, projector, side by side with text
4. The connector is where the design space lives: five ways to bolt a vision encoder onto an LLM
5. Native dynamic resolution: why a 2026 model doesn't squish your screenshot into a square
6. The vision token tax: an image is ~1,500 tokens at 1080p
7. What the marketing pages don't tell you: counting, spatial reasoning, hallucinated objects, OCR limits
8. What's actually next: multi-layer injection, reasoning VLMs, agentic UI use

A note before we open the hood: most frontier labs (Anthropic, Google, OpenAI) don't publish architecture details for their vision stacks. The architecture-level claims below are inferred from the published academic lineage (LLaVA, BLIP-2, Flamingo, Qwen-VL), the open-source frontier (Qwen2-VL¹ and Qwen3-VL² are the only ones with detailed write-ups), and what's consistent across product surface area and pricing. I'll flag where the evidence is thin.

---

The "model just sees pixels" fallacy

If you came to vision LLMs from text LLMs, the natural mental model is: "A text LLM reads a string. A vision LLM reads a string and an image."

This is wrong in a way that matters.

The model doesn't read the image. It reads tokens. The image gets converted into a sequence of tokens by a vision encoder before it ever touches the language model. Whatever doesn't survive that conversion is not in the model's input. Anything subtle in the original pixels (small text, a thin object, a fine spatial relationship between two things) has to fit into a few thousand tokens at most. If it doesn't fit, the model literally cannot see it.

This sounds obvious in the abstract. It stops feeling abstract the first time you watch a model count five items in a list at one resolution and four at another. Or read a chart axis correctly at 1024×1024 and miss the units at 512×512.

The "model sees pixels" view: "I gave the model a clear screenshot. It should be able to read everything."

The actual model: "The vision encoder turned that screenshot into ~1,500 tokens. Whatever didn't fit in 1,500 tokens isn't there."

That's the whole shift. The rest of this post is the consequences.

---

The three building blocks

Strip away the marketing and a modern vision LLM has three parts:

1. A vision encoder. Almost always a Vision Transformer (ViT) or a SigLIP/CLIP variant. Takes raw RGB pixels and produces a dense set of visual feature vectors, one per image patch. This is the same family of model that powered ImageNet classifiers, just stripped of its classification head.

2. A connector. The piece that turns visual features into something the language model can read. This is the most varied part of the architecture and the part where the field is still moving. More on this in The connector is where the design space lives.

3. A language model. A standard decoder LLM (Llama, Qwen, Mistral, GPT, Claude, Gemini). In most modern VLMs the LLM is a frozen or lightly-tuned version of an existing text model.

Every current vision LLM fits this template. On the open side: Qwen3.5, GLM-4.6V, MiMo-V2-Omni, Gemma 4, and the LLaVA / InternVL lineage they grew out of. On the closed side: Gemini 3, GPT-5, Claude vision, Grok, Kimi. The differences are in the size of each block, the connector design, the training data mix, and what got post-trained on top. This is the same pattern as the video model architecture: one transformer, swappable input encoders, training data and post-training doing most of the heavy lifting.

The two-stage training recipe is also stable: freeze the vision encoder and the language model, train only the connector to align visual features with text embeddings; then unfreeze the language model and fine-tune end-to-end on instruction data. LLaVA established this in 2023 and most subsequent open VLMs use a variant.³

---

How an image becomes tokens

The vision encoder's job is to take an RGB image and turn it into a dense grid of feature vectors that the connector can convert into LLM tokens.

For text, this is well-understood: a tokenizer splits the string into subword units, and an embedding layer turns each subword into a vector of, say, 4096 dimensions. The LLM never sees the original characters; it sees the embeddings.

Vision is the same idea on a 2D grid. The image gets split into fixed-size patches (typically 14×14 or 16×16 pixels). Each patch is flattened into a vector and run through a linear projection to produce a "patch embedding," exactly analogous to a subword embedding. The Vision Transformer then runs self-attention over these patch embeddings, producing a dense feature vector per patch. A 224×224 image at 14×14 patches gives 256 patches; a 1024×1024 image at 14×14 gives ~5,300.

	Text	Image
Input unit	Character	Pixel
Tokenizer unit	Subword (BPE)	Patch (14×14 or 16×16 px)
Tokens for typical input	~750 tokens / page	~1,500 tokens / 1080p image
Encoder	Embedding lookup	ViT (self-attention over patches)
Output to LLM	Token embeddings	Patch embeddings (after projector)

The third row is the entire economic story of vision LLMs. An image isn't a free add-on to a text prompt. It's roughly the same number of tokens as 1-2 pages of dense text. Once you internalize this, every "why is this expensive?" and "why didn't it see that detail?" question becomes obvious.

Once the patches are encoded, the connector takes them the rest of the way into the LLM. That's where the design space lives.

---

The connector is where the design space lives

The vision encoder is mostly settled (a ViT or SigLIP). The language model is mostly settled (a decoder transformer). The connector, how visual features get into the LLM, is where labs still disagree.

Sebastian Raschka's framing is the cleanest: every VLM lands in one of two camps.⁴

Camp	What it does	Examples
Unified Embedding Decoder ("Method A")	Visual features get projected into the LLM's token space and concatenated with text tokens. The LLM sees one sequence of mixed tokens.	LLaVA, Pixtral, Qwen-VL family, Llama 3.2 11B vision, most open VLMs
Cross-Modality Attention ("Method B")	Visual features stay separate. The LLM reaches them via cross-attention layers inserted into transformer blocks.	Flamingo, Llama 3.2 90B vision, NVLM-X, Idefics

Method A is dominant because it's simpler: no architecture changes to the LLM, just a projector module on the front. Most production VLMs you'll touch in 2026 are Method A.

Within Method A, the projector itself has variants:

1. Linear projector. A single linear layer maps ViT features to LLM-token space. LLaVA original. Cheap. Works.
2. MLP projector. Two-layer MLP. LLaVA-1.5 onward. Marginal quality gain over linear.
3. Q-Former (BLIP-2). A small transformer that learns a fixed number of "query" tokens (typically 32 or 64), regardless of image size. Compresses visual information aggressively. Was popular 2023; falling out of favor because it loses information.
4. Interleaved-token (Fuyu). No separate ViT at all. Image patches go straight into a linear projection inside the LLM. Simplest possible architecture. Less mature in production.
5. Multi-layer injection (DeepStack). Visual features from multiple layers of the ViT get injected into multiple layers of the LLM, not just at the input. The plug-and-play precursor was Dense Connector (May 2024)⁵, which showed multi-layer features beat last-layer-only across MLLM benchmarks. Qwen3-VL productionised the idea as DeepStack and shipped it.² Captures finer-grained visual detail.

Connector design isn't an academic detail. It's the load-bearing decision behind "can this model read a 6-point font in a chart axis." Aggressive compression (Q-Former) is cheap and lossy. Per-patch projection (linear/MLP) preserves more. Multi-layer injection (DeepStack) preserves finer features at cost.

What looks like model capability is often connector design. The Qwen team's switch from single-layer injection to DeepStack in Qwen3-VL is a connector change, not a vision-encoder change or an LLM change. The reported capability lift on fine-grained visual tasks comes from that one decision.

---

Native dynamic resolution

By 2026, every frontier vision LLM accepts arbitrary input shapes. The pre-2024 generation didn't: CLIP was 224×224 fixed. LLaVA-1.5 was 336×336 fixed. CogVLM was 490×490 fixed. A 1920×1080 screenshot lost its aspect ratio and most of its detail before the model ever saw it. That constraint is gone now, but knowing how it went away tells you why your current model behaves the way it does.

Two architectures handle variable-shape inputs in 2026:

Tiling (also called AnyRes). Split a high-res image into multiple fixed-size tiles, run the ViT on each, concatenate the resulting tokens. Conceptually simple. Bolts onto an existing fixed-size ViT without retraining. Introduces seams between tiles and loses some global context. Originated with LLaVA-NeXT in 2024 and is still the default for any model whose underlying ViT is fixed-resolution.

Native dynamic resolution. Modify the ViT to accept variable-shape inputs directly. A 1920×1080 image produces ~1,200 tokens; a 4K image produces more; a thumbnail produces fewer. No tiling, no resizing to fit. Named in Qwen2-VL alongside M-RoPE (Multimodal Rotary Position Embedding), which encodes positional information across text, image, and video on the same axis system.¹ Now standard in Qwen3.5, GLM-4.6V, and most native-multimodal models from the major labs.

The Qwen2-VL paper put the architectural shift plainly: "M-RoPE… facilitating the effective fusion of positional information across text, images, and videos. We employ a unified paradigm for processing both images and videos." That unified paradigm is the part that mattered: once your positional encoding works on a 3D grid (height, width, time), an image is just a 1-frame video and a video is just an N-frame image.

Approach	How it handles a 1920×1080 screenshot	Tokens	Strengths	Weaknesses
Fixed square (CLIP, LLaVA-1.5)	Resize to 336×336, aspect ratio destroyed	~576	Cheap, simple	Loses detail, distorts geometry
Tiled / AnyRes (LLaVA-NeXT)	Cut into ~6 tiles at 336×336 + global thumbnail	~3,500	Preserves detail, simple to bolt onto existing models	Tile seams, loses cross-tile context
Native dynamic (Qwen2-VL)	Process at native shape, ~1,200 tokens	~1,200	No seams, true aspect ratio	Requires retraining ViT

AnyRes is the bolt-on retrofit for ViTs that were already trained at a fixed size; native dynamic is the green-field default for any model trained from scratch in the last 18 months. Both produce the same outcome for the practitioner: you can stop worrying about whether your input is square.

---

The vision token tax

The single most important number in this post is the per-image token count.

Anthropic publishes the formula directly in the Claude vision docs: "An image uses approximately width * height / 750 tokens." Caps: Claude Opus 4.7 tops out at 4,784 tokens per image; other Claude models cap at 1,568 tokens.

What that means in practice, at Sonnet 4.6's $3-per-million-input-token rate:

Image size	Tokens	Cost / image	Cost / 1k images
200×200 (0.04 MP)	~54	~$0.00016	~$0.16
1000×1000 (1 MP)	~1,334	~$0.004	~$4.00
1920×1080 (HD)	~1,568 (capped)	~$0.0047	~$4.70
2000×1500 (3 MP)	~1,568 (capped)	~$0.0047	~$4.70

Three things follow from this table that don't get said out loud often enough:

1. A 1080p screenshot costs about as much as 4-5 pages of text. Document-AI pipelines that treat images as "free" because they're not prose are wrong by an order of magnitude.
2. Resizing before upload is real money. A 4K screenshot downsampled to 1080p cuts the per-image cost roughly 4× before the cap kicks in. For a screenshot agent doing 100K screenshots a day, that's the difference between $470 and $1,800 per day.
3. Cap behavior matters. Above ~1.19 MP on most Claude models, you're paying for a 1,568-token image whether your input is 1.2 MP or 50 MP. The model has resized internally. Coordinates returned by the model (bounding boxes, point detections) are in the resized image's coordinate system, and you have to rescale them client-side.⁶

The cost story is concrete because Anthropic publishes the formula. Most other labs don't. The open-source models with native dynamic resolution (Qwen2-VL, Qwen3-VL) sit in roughly the same ballpark from what I've measured: a 1080p image lands somewhere around a thousand to a few thousand visual tokens, depending on aspect ratio and the model's patch size. Gemini and GPT publish per-image input pricing but not the underlying token math, so treat their numbers as opaque.

---

What the marketing pages don't tell you

The capability lists on Claude, Gemini, GPT-5, and Qwen pages are real. They also paper over the failure modes that anyone using these models daily runs into.

Counting breaks at boundaries. In my testing, counting holds up for small sets of clearly separated objects and degrades quickly once the count climbs into the dozens. The "Vision Language Models are Biased" study⁷ formalised this: frontier VLMs averaged 17% accuracy on counting tasks where the answer contradicts a memorised prior (the canonical example: count the stripes on an Adidas logo modified to have 4 stripes instead of 3). The reason isn't that the model "can't count": counting requires attending to each object as a discrete unit, and the patch-based tokenization deliberately blurs object boundaries. The model literally doesn't have one token per object, and when its prior says "Adidas has 3 stripes" the prior wins.

Fine-grained spatial reasoning is fragile. "Is the cup to the left of the plate?" works. "Is the third button from the top aligned with the second tab?" doesn't, reliably. Spatial relations between small UI elements rely on attention across tokens that may have been compressed differently.

Sub-pixel text is gone. OCR works for text that's clearly readable to a human at the model's working resolution. Text that requires zoom or that sits below ~8px in the input image is below the patch grid. Resize-before-upload is your OCR optimizer.

Bounding boxes are approximate, not exact. Models trained for grounding (Qwen-VL, Molmo, PaliGemma) output coordinates, but the coordinates are in the resized-and-padded image space and the boxes drift by 10-30 pixels even on clean inputs. Treat them as "regions of interest," not as detections.

Hallucinated objects. VLMs hallucinate the same way text LLMs hallucinate: they fill in plausible details that aren't in the input. A model asked "describe everything in this office photo" will reliably name a coffee mug whether or not one is in frame, because office photos have coffee mugs in the training distribution.

Resolution sensitivity is asymmetric. The opening of this post: same screenshot at two resolutions, two different counts. This is the single most disorienting failure mode for new users, because the input looks the same to a human. Always normalize image resolution at the pipeline boundary.

The vocabulary of image prompts hasn't matured. What works for "describe this chart" doesn't work for "extract a structured table from this chart." Prompting for vision tasks is a context engineering problem wearing a vision coat.

---

What's actually next

The capability list above describes today, not the frontier. Three things vision LLMs genuinely don't yet do well:

Robust agentic UI use. Models like Qwen3-VL² and UI-TARS market computer-use capability and demo cleanly. In production, screenshot-driven agents still drift past 20-30 steps because each screenshot is independently tokenized: the model has no cheap way to know that the button it clicked at step 12 is the same button at step 13. State persistence across screenshots is the next wall.

Connector innovation beyond DeepStack. Multi-layer injection helps, but it's still an additive trick. The deeper question is whether vision tokens should be the same shape and live in the same context window as text tokens at all. A few labs are exploring "vision sub-context" architectures where image tokens have a separate cheap memory and get pulled into the main context only when relevant. Nothing is productized.

Native multimodal training. Most current VLMs glue a vision encoder onto a pretrained LLM. The next generation is being trained jointly from the start, with image and text tokens in the same pretraining mixture. Qwen3-VL claims early-stage joint pretraining; Gemini has claimed "natively multimodal" since the 1.0 release. The gap between "bolted-on" and "native" is real and the natively-trained models handle vision-text interleaving meaningfully better.

What's not changing: the three-block template is stable. The work for the next two years is in connector design, joint pretraining, and inference cost reduction. Not in replacing ViT or rebuilding from scratch. Vision LLMs are in their post-LLaVA phase. The plumbing is settled. The product surface is the action.

---

Related: How a Video Model Works · Context Engineering Patterns

This is the second in a planned three-part series on the modern multimodal model stack. The companion explainer on image diffusion is next.

Footnotes

1. Wang et al., Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, arXiv:2409.12191 (link). ↩ ↩²

2. Qwen team, Qwen3-VL: Sharper Vision, Deeper Thought, Broader Action, September 2025 (link). ↩ ↩² ↩³

3. Noyan & Beeching, Vision Language Models Explained, Hugging Face, April 2024 (link). See also the May 2025 update: Vision Language Models (Better, faster, stronger) (link). ↩

4. Raschka, Understanding Multimodal LLMs, November 2024 (link). The Method A / Method B framing is his. ↩

5. Yao et al., Dense Connector for MLLMs, arXiv:2405.13800, May 2024 (link). The canonical pre-DeepStack reference: a plug-and-play connector showing that integrating multi-layer ViT features (rather than just the final layer) consistently improves MLLM performance. ↩

6. Anthropic, Vision (link). Source for the width * height / 750 token formula and the resize/pad-to-multiple-of-28 detail. ↩

7. Vision Language Models are Biased, ICLR 2026 (link). Demonstrates that frontier VLMs hit ~17% accuracy when counting tasks contradict memorised priors, evidence that the patch-tokenization architecture relies heavily on prior knowledge rather than per-object visual attention. ↩