Why Vision Understanding Models Arrived Late: Cost, Compression Paths, and the Agent Loop
Vision APIs often lag not because teams cannot encode images, but because visual tokens blow up unit economics. This article unpacks input, attention/KV-cache, and multi-step agent costs, the primitive-based compression path, and how to evaluate task cost, localization, and closed-loop stability.
Why Vision Understanding Models Arrived Late: Cost, Compression Paths, and the Agent Loop
1. The bottleneck is not “can it see,” but “can you afford to look”
When a vision-understanding model or API ships late, the first guess is usually “not enough technical readiness.” That is often too simple. For a team that already has a mature language model, inference stack, and engineering system, wiring in a vision encoder is rarely an impossible barrier. The hard part is whether unit-task cost stays acceptable after you plug it in.
Visual and text inputs have different cost structures. Text tokens scale roughly with character count. Image tokens balloon with resolution, tiling, number of regions, and multi-turn context. A high-resolution image can be split into many patches, then projected into hundreds to tens of thousands of visual tokens. If an agent also screenshots and re-observes results every step, those tokens keep accumulating in context and the KV cache. Inference cost is no longer a one-shot “look at the image” fee; it is multiplied by “look at every step.”
So vision is often late not because it cannot be done, but because it cannot be done cheaply. For a product line built on low price, high throughput, and unit economics, visual tokens can become a cost sink that breaks the existing price model and user expectations. Teams typically need to prove that visual understanding still works at very low cost before they open a public API.
2. Why visual tokens are expensive: three cost layers
Visual-token cost can be split into at least three layers.
Layer 1: input tokens. Higher resolution means more patches and more visual tokens in the language model. A naive “encode the whole high-res image” path can produce thousands or tens of thousands of tokens per image, making a single call several times more expensive than a text-only job.
Layer 2: attention and KV cache. Visual tokens lengthen the sequence and enter attention and KV cache. In multi-turn chat or agent loops, historical visual tokens that are not compressed or dropped keep consuming memory bandwidth and context window. More images and longer traces make cache pressure worse.
Layer 3: multi-step task cost. Agents often run a plan–execute–screenshot–observe–correct loop. Screenshotting and sending every step to a vision model makes cost grow with step count. Without resolution tiers, region crops, cache reuse, and a token budget, vision goes from “usable” to “unaffordable.”
A simplified budget looks like this:
function estimate_visual_tokens(width, height, patch_size, merge_ratio):raw_patches = (width / patch_size) * (height / patch_size)effective_tokens = raw_patches / merge_ratioreturn effective_tokensfunction estimate_task_cost(steps, tokens_per_step, price_per_token):return steps * tokens_per_step * price_per_token
Visual cost is not a single knob. It is the joint result of resolution, tiling, compression ratio, number of turns, and unit price. Any one of those out of control can break a low-price strategy.
3. Hard constraint of a low-price line: vision must not burn unit economics
Low price is a product promise, not just a marketing tactic. Users pick cheap models because unit-task cost is predictable, batchable, and sustainable. If a vision API is billed like a full high-res image encode, single-task cost can jump several times, and high-volume workloads stop making sense.
Teams that ship vision later often already have encoders. What they lacked was an engineering answer for cheap vision. They need three things at once: large compression of visual tokens; enough semantics and localization after compression; and compatibility with the reasoning chain, tool calls, and agent loop. Only then is a public vision API viable.
That is why some models first lock in a price advantage on text and code, then add vision. The slower cadence is often defending a cost floor before expanding capability.
4. Compression: let visual primitives enter reasoning, not just long captions
There are two typical ways to cut visual cost. One is encoder → natural-language caption → reason. It is faster to ship, but spatial detail is lost in language, and the caption itself costs tokens. The more aggressive path treats points, boxes, and regions as tokens that participate in reasoning, so the chain of thought cites “which point, which box, which region” instead of restating the image in prose.
Primitives are denser. A box can encode an object’s location and extent with a few coordinates and a label; a point can mark a control or an anomaly. That fits agent work such as click, select, locate, and compare better than a long caption of the whole frame.
The cost is also clear. The model must learn coordinate reference, spatial relations, and composition of primitives. The chain of thought must support “localize first, then reason.” Training data needs detection, localization, and region description. Add sparse attention and KV-cache compression that keep only salient visual tokens, and engineering complexity rises further.
A sketch:
visual_primitive = {type: "point" | "box" | "region",coords: [...],label: "...",confidence: 0.0}reasoning_step = {observation: [visual_primitive, ...],conclusion: "...",next_action: "..."}
The point is not merely shrinking the image; it is structuring vision enough to reason over. Compression ratios can be high only if the training pipeline and inference chain are rebuilt together. Publicly described recipes often filter from hundreds of thousands of candidate detections down to tens of thousands of high-quality samples, then build localization-and-reasoning data. Cleaning, label consistency, negative samples, and eval sets stretch the timeline. That is why this path takes longer than “just attach an encoder.”
5. Strategy: vision as a patch on the agent loop, not a launch headline
Cost is not the only reason for delay. Priority also matters. Some teams treat multimodal as a flagship feature and try to ship image, text, and video together. Others put language models and agent execution first; vision only lets the agent see the result of its own work.
Those plays differ. The first optimizes for capability showcase and launch noise. The second optimizes for task closure and stable tool use. For agents, vision’s value is not pretty captions. It is execution feedback: did the UI navigate, was the form filled, is the chart anomalous, did the code output match expectations. Once vision is in place, the agent can move from blind actions to observe–correct–execute again.
A typical loop:
while task_not_done:plan = planner(task, history)action = tools.execute(plan)observation = environment.capture()visual_feedback = vision_model.observe(observation,primitives = ["point", "box", "region"])task = updater(task, visual_feedback, action)
In this loop, vision does not need to be a full-modality first-release. It does need to be stable, cheap, localizable, and reusable across turns. It is a patch in the agent system, not a standalone demo. Shipping it later is not automatically a technical lag; it can be a priority order.
6. How to evaluate this class of vision model
Developers and enterprises should not judge only caption quality on a single image. They should judge unit-task cost and closed-loop ability. Checklist:
- Cost per task: not just per-call price, but how many vision rounds, how many tokens, and how much cache reuse a finished task needs.
- Token efficiency: whether visual tokens can be compressed at the same resolution; support for region crop, resolution tiers, and keeping salient regions.
- Localization and reference: whether the model can emit points, boxes, and regions and cite them stably in later reasoning.
- Multi-turn stability: whether historical visual information can be compressed, dropped, or reused across screenshots.
- Debuggability: whether localization errors, recognition errors, and execution errors can be separated, with logs that can be replayed.
- Data pipeline maturity: high-quality localization data, negatives, hard-case eval sets, and room to iterate.
7. Practical advice for developers
Prefer a layered vision strategy: cheap detection or crops to shrink the field of view, then high-precision understanding on the critical region. Cache visual features for static UIs or documents so you do not re-encode every turn. Cap agent screenshots and alarm on token budget. Prefer structured visual outputs over free-text descriptions that waste tokens.
Evaluate “understood” and “actionable” separately. A good caption is not accurate localization, and accurate localization is not an executable action. For agents, build task-level evals that record each screenshot, observation, action, and correction, and score both success rate and cost per task.
8. Conclusion
A late vision or multimodal API looks like a release-calendar issue. Underneath it is visual-token cost, the engineering cycle of compression, and agent priorities. Visual tokens are expensive because of resolution, attention, KV cache, and multi-step loops. A low-price line cannot let vision destroy unit economics. Primitives plus sparse compression can cut cost, but they require a rebuilt chain of thought and a self-owned data pipeline. Vision then lands as execution-feedback in the agent loop, not as a launch gimmick. Judge it less on “does it exist” and more on whether per-task cost is controllable, localization is executable, and the loop is stable.
