Why Visual Multimodal Models Launch Late: The Real Cost of Vision Tokens
A clear explanation of why cost-focused LLM vendors tend to ship vision capabilities after their text flagships. It breaks down why vision tokens are expensive, the three technical levers for compressing them, the engineering trade-off between plugging in a vision encoder and rebuilding around visual primitives, and how vision ultimately serves the Agent execution loop. It also offers practical guidance for enterprises on controlling visual inference costs.
Recently, a number of major LLM vendors have added native vision capabilities, giving document parsing, chart understanding, and screenshot Q&A a unified entry point. Some vendors released vision understanding several months after their text flagship and rolled it out first under an “experimental” suffix, usually covering only static images while video and open weights remained pending.
So “when will it launch” is no longer the core question. The more worthwhile question is: for vendors known for low prices, why does vision always seem to arrive last?
A popular interpretation is “insufficient technical accumulation.” But static image understanding is already a mature research area: splitting an image into patches, feeding them through a vision encoder, then aligning them to the language model’s embedding space via a projection layer is now standard industry practice. What actually determines the launch cadence is a different problem: vision tokens are expensive, which conflicts with the low-price foundation these vendors are built on.
This article unpacks the problem across three levels—token cost, technical routes, and product positioning—and offers a reference for enterprises to control visual inference costs.
1. Why vision tokens are so expensive
1.1 One image can be thousands to tens of thousands of tokens
Vision-language models typically divide an image into fixed-size patches and encode each patch as a token. With a 16×16 patch, a 1024×1024 image becomes 64×64 = 4,096 tokens—before counting the text. For information-dense long screenshots or scanned documents, exceeding 10,000 tokens is common.
1.2 The quadratic cost of attention
Attention is the core of the Transformer: every token computes relevance against every other token, so the compute grows quadratically with sequence length. Language models can ease this with sparse attention and sliding windows, but the “higher resolution → more tokens” mapping is unavoidable, and the prefill stage feels it most.
def attention_pairs(n: int) -> int:"""Approximate number of attention pairs for n tokens (illustrative; no parallelism optimizations)"""return n * (n + 1) // 2print(attention_pairs(4096)) # 8388608print(attention_pairs(1024)) # 524800print(attention_pairs(64)) # 2080
Cut the token count to one quarter and attention compute drops to about one sixteenth; cut it to one sixty-fourth and only about one four-thousandth remains.
1.3 Context fills up fast
“Tens of thousands of text tokens plus thousands of image tokens” is a common combination in vision conversations. Images noticeably compress the available window, so long conversations or high document-page counts quickly hit the limit—ending in either truncation or repeated calls.
1.4 The KV cache keeps growing
During decoding, the model looks back at the full history for every token it generates, so the inference framework must cache each layer’s Key/Value matrices (the KV cache), whose size is proportional to sequence length. Once vision tokens enter the context, every round of a long session pays this memory and bandwidth cost again—especially tight under high concurrency.
Together these four points mean the input-token scale of a vision scenario is often several times—or even an order of magnitude—that of pure text, while inference price is roughly linear in token usage. For vendors with monthly call volumes in the hundreds of millions, this directly determines the cost structure. That is precisely why vendors holding back on vision are usually waiting for a way to make “vision cheap too.”
2. How to compress vision tokens: three main lines
The industry compresses vision tokens along three main lines: representation, attention, and systems.
1. Representation layer: fewer, more precise visual primitives
Mainstream models “flatten” an image into a regular grid, where many patches carry only low-information content such as texture or background. A differentiated route explicitly represents structured elements—points, bounding boxes, text lines—as tokens the model can reference, so inference revolves around these “visual primitives.” Its ceiling depends on how reliable the upstream detection, recognition, and structured-parsing capabilities are.
2. Attention layer: compression and sparsification
- Token merging and pooling: cluster and merge adjacent patches, or apply spatial pooling, to approximate a region with few tokens.
- Learnable compression and query mechanisms: use a small set of learnable queries or tokens to compress the image representation a second time.
- Sparse attention: let vision tokens interact only with local neighborhoods or a global summary, avoiding full pairwise computation.
3. System layer: caching and reuse
- Prefix caching: system prompts and system reference images stay unchanged across requests, so their vision tokens can be encoded once and reused across turns.
- Image deduplication: when the same image is called repeatedly, prefer a hit on the cached encoding result.
- Dynamic resolution: a preprocessing step such as thumbnail generation or layout analysis judges whether an image contains key regions like text or charts, then decides whether to process it at full resolution.
from dataclasses import dataclass@dataclassclass SparseImageEncoder:n_tokens: int = 4096 # Tokens for a 1024×1024 image with patch=16group_size: int = 16 # Merge: compress every 16 adjacent patches into 1n_queries: int = 64 # Learnable queries: fixed output of 64 semantic tokensdef encode(self):grouped = (self.n_tokens + self.group_size - 1) // self.group_sizereturn self.n_queries # Text and vision both use the same sparse-attention computedef compression_ratio(self) -> float:return self.n_tokens / len(self.encode())encoder = SparseImageEncoder()print(encoder.compression_ratio()) # 64.0
This is illustrative code; a real system’s compression ratio, accuracy loss, and implementation complexity depend on the specific design.
3. Why it’s still slow: two routes and the cost of “self-built data”
If compression techniques aren’t scarce, why does launch still drag? The answer is in the engineering route. Adding vision capability roughly splits into two paths:
| Dimension | Route A: plug in a vision encoder | Route B: visual primitives + compressed sparse attention |
|---|---|---|
| Implementation | Reuse a mature vision encoder, add a projection layer, and jointly train with the language model | Rebuild across three layers: representation, attention, and the data pipeline |
| Context overhead | Vision tokens grow with resolution; needs system-level optimization to ease it | Few, precise vision tokens; little long-context pressure |
| Inference cost | Relatively higher | More competitive |
| Preprocessing dependency | Weak; encoding is end-to-end | Strong; depends on detection, recognition, and other structured capabilities |
| Engineering scale | Weeks to months | Several months or more |
| Fit | General visual Q&A, document and chart understanding | Agent observation, cost-sensitive, large-scale calls |
Route A wins on speed: both the encoder and the training paradigm are mature, so a model can quickly gain the ability to “see images”—at the cost of a cost structure constrained by the number of vision tokens. Route B builds the cost advantage into the base layer but requires dual investment in engineering and data. This is the technical reason behind the “slowness.”
Notably, the data for this kind of structural compression can’t be obtained “for free” from general image-text corpora. It requires systematic detection annotation and multiple rounds of screening and cleaning before enough usable training samples accumulate. The process includes:
- Establish annotation guidelines consistent with the business scenario;
- Initial screening to remove blurry, duplicate, and mislabeled samples;
- A second screening pass to keep samples of moderate difficulty with sufficient information;
- Balance and augmentation by task type.
A raw candidate set of around 100,000 leaving only about 30,000 usable samples after two rounds of filtering is not unusual. Add the fact that compression and sparse attention themselves need to be redesigned and validated, and the training and evaluation cycles stretch further. In one sentence: this isn’t “can’t connect”—it’s “unwilling to settle.”
4. The strategic layer: vision first serves the Agent loop
The route choice is only the technical-side explanation; one level deeper lies product positioning.
In recent years, top vendors have generally concentrated their efforts on language models and Agent capabilities: the flagship model’s post-training stage focuses on tool use, planning, and multi-step execution, while a companion execution framework turns model output into real task actions. For such a system, vision isn’t about being first to ship “native omni-modal”—it’s about filling the most-missing piece of the execution loop: seeing.
A typical Agent task flow is “execute–observe–correct”:
User task: organize last quarter's sales report and generate a chart│├─ Agent: locate the data source → open the spreadsheet tool → filter quarterly data│├─ Screenshot ──▶ Vision model: identify headers/units/empty anomalies, extract key values│├─ Agent: correct parameters based on observations → generate the chart → screenshot again to verify│└─ Return the result (with verification records)
In this loop, vision isn’t a “read-image-and-chat” toy but a verifier of execution results: whether table headers align, whether value units are consistent, whether chart titles are complete—all confirmed by “looking.” This yields three implications:
- Structured output matters more than “sounding human”: giving the Agent a definite conclusion like “the table has 37 rows and row 12’s amount is empty” beats a vague description.
- The usability bar is higher than general leaderboards: misreading a single decimal point can throw the entire task chain off course.
- Call frequency grows linearly with the session: each step may trigger a vision call, multiplying the token cost discussed earlier.
This also explains why Route B fits Agent scenarios better: the fewer the vision tokens, the cheaper each observation step, and the more viable long, multi-step tasks become.
At the same time, many vendors choose “flagship = native omni-modal,” making image and video understanding a core selling point of the flagship model—an equally reasonable commercial choice. The two aren’t a matter of higher or lower, but of different paths: for one, multimodality is the face of the flagship; for the other, vision is a patch in the Agent system.
5. Back to the original debate: on the “not technically capable” claim
Taken together, there are three more plausible explanations for a vendor shipping vision late:
- Cost constraint: the bloat of vision tokens conflicts with the low-price positioning, so a structural compression solution must come first;
- Route cost: not taking the “plug in an encoder” shortcut means bearing the cycle of rebuilding and self-built data;
- Priority ordering: resources go to the Agent main line first, with vision deferred as a loop component.
Equating “late” with “not technically capable” confuses release cadence with technical strength. What’s actually worth watching are three things: how far vision tokens can be compressed, how accurate structured output is, and whether unit task cost can be amortized as call volume scales. For teams preparing to adopt vision capability, rather than waiting for the “most complete” model, it’s better to first figure out which category your scenario falls into:
| Scenario type | Typical task | Focus | Suggested strategy |
|---|---|---|---|
| Exploratory | Image Q&A, content understanding | Semantic accuracy | Medium resolution + hybrid sampling to control context |
| Execution | Agent screenshot verification, form-filling checks | Structured accuracy, per-call cost | Few, precise vision tokens + cache reuse |
| Throughput | Batch document/screenshot parsing | Unit cost, stability | Preprocessing triage + high compression ratio + caching |
6. The enterprise side: caging vision cost
Regardless of the route chosen, enterprises should build cost-control mechanisms as they roll out:
- Establish a “per-task cost” metric: convert vision token count, tool-call count, and retry count into the cost of a single task—not just the unit price. A low unit price with repeated retries can still mean a higher total cost.
- Manage context: cache and reuse system prompts and reference images, archive historical screenshots promptly, and send only the current frame to the model.
- Prioritize structured output: replace free text with programmatically verifiable results—object counts, coordinates, field values—to reduce re-parsing and manual checks.
- Build a scenario evaluation set: cover boundary samples at least—blurry images, skewed images, dense small text, complex charts—and evaluate accuracy and token usage together.
- Layered routing: use preprocessing such as thumbnails and layout analysis to judge complexity, sending simple tasks to a lightweight model or rules and only hard tasks to a large vision model.
- Make full use of infrastructure: switch models on demand through a model service platform; archive images in object storage and reference them in later requests; use logging and monitoring/alerting to continuously track token consumption and task success rate.
Conclusion
From token cost to attention complexity, from the data pipeline to the Agent loop, the late launch of vision capability follows a clear cost-and-route logic rather than an absence of technical ability. For users, a more valuable judgment than “whose vision is stronger” is: does your business need vision to “say more,” or to “see accurately and cheaply”? Once that’s clear, model selection stops being driven by release timelines.
