Why Vision Models Arrive Late: Visual Token Costs and the Agent Loop
Vision capabilities are becoming a baseline feature, but low-cost deployment remains difficult because visual tokens can multiply inference cost. This article explains why some vision models are released experimentally, how visual-token compression can reduce cache and inference overhead, why Agent workflows may be a more practical entry point than full multimodal deployment, and what developers should consider when building cost-controlled vision pipelines.
1. The Question Is Not Whether Models Can Do It, but When They Can Open It Up
Multimodal vision capabilities are moving from a differentiating feature to a baseline capability. Three common approaches have emerged across the industry:
- Unifying text, images, and video in a flagship model with native multimodal support
- Maintaining a separate vision branch alongside the language model and adding capabilities for specific use cases
- Treating vision as the perception layer of an Agent system, using it to complete tasks first and gradually exposing it as a standalone capability
Recently released experimental vision models have followed a cautious pattern: they support images but not yet video, do not provide open weights, and carry an experimental label. This suggests that they are intended more as capability and cost validation than as full-scale flagship launches.
The more important question, therefore, is not whether a particular model family can support multimodal input. It is why vision is being opened up experimentally only at this stage.
The core reason can be summarized simply: visual tokens are expensive, and low-price positioning is highly sensitive to cost. Text-token costs are relatively controllable, while visual input can increase the token count of a single request by an order of magnitude. For a model service built around affordability, an expensive vision feature could put both the existing pricing structure and user expectations under pressure. The provider must first demonstrate that vision can operate at very low cost before making it broadly available.
2. Why Visual Tokens Amplify Cost
The cost of visual input does not come only from the vision encoder. It comes from the entire inference pipeline.
After an image enters the model, it is typically divided into multiple image patches, which are then mapped to visual tokens. The token count often scales approximately with the square of the resolution. If the resolution doubles, the number of image patches may approach four times the original count.
The sequence can expand even more rapidly when the request includes multiple images, video frames, screenshots from long documents, or user-interface captures. During autoregressive inference, the cache grows with sequence length. Memory usage, bandwidth pressure, and latency increase accordingly. In scenarios priced by tokens or amortized across calls, the cost of a single request can rise substantially.
A simplified cost model looks like this:
Visual token count ≈ image patch count + special tokens + task-specific supplemental tokensImage patch count ≈ (image height / patch size) × (image width / patch size)Single-inference cost ≈ visual token count × unit token cost × call volume+ cache and bandwidth overhead
The issue is especially sensitive for low-cost services. In text-based scenarios, users are already accustomed to relatively low unit prices. If a vision system applies a high-resolution patching strategy without further optimization, a single image may generate thousands or even tens of thousands of tokens, significantly increasing inference cost.
Without compression, cropping, cache reuse, and task routing, it is difficult to maintain an affordability-focused position. Developers also face hidden costs: high-resolution images, multi-turn follow-up questions, and batch document parsing can quickly consume the available token budget.
3. The Compression Path: Turning Visual Primitives into Reasoning Tokens
Solving the cost problem requires more than reducing prices on the billing side. The model itself must reduce unnecessary visual tokens.
One representative approach is to avoid sending every pixel patch into inference with equal weight. Instead, the system extracts visual primitives such as points, bounding boxes, and region masks. These primitives are then represented as tokens or structured conditions that the model can use for reasoning. Combined with compressed and sparse attention, this approach can reduce the visual cache to a very small size.
Public technical explorations have included approaches that compress the visual cache for a single medium- to high-resolution image to roughly hundreds of entries, with overall compression ratios reaching the order of thousands. The value of this approach is that it shifts visual representation from dense pixel descriptions toward task-relevant primitives, reducing the token cost of redundant backgrounds and repeated textures.
Key technical elements include:
Visual primitive extraction. Detection boxes, keypoints, and region boundaries are used as intermediate representations. The model can reason about where objects are and how they relate to one another instead of describing the entire image patch by patch.
Token compression and sparsification. The visual cache is compressed, sparsely selected, or aggregated hierarchically so that only regions relevant to the current question are retained. Simple tasks can use lower-resolution or coarser representations.
Task-oriented encoding. Classification, counting, chart understanding, and user-interface element localization require different levels of visual precision. Resolution and primitive granularity can be selected dynamically according to the task.
Inference-chain redesign. If visual primitives are expected to participate in the model’s reasoning process, the training objectives and alignment methods must be redesigned. The model must learn to reference regions, compare positions, and verify relationships.
A simplified example is:
function visual_reason(task, image, budget):if task.needs_fine_detail:resolution = choose_resolution(task, image)else:resolution = coarse_resolution(image)primitives = extract_points_boxes_masks(image, resolution)visual_tokens = encode_primitives(primitives)visual_tokens = compress_and_sparsify(visual_tokens, max_cache=budget.cache)if estimate_cost(visual_tokens) > budget.max_cost:visual_tokens = crop_or_reduce(image, visual_tokens, budget)return reason_with_visual_tokens(task, visual_tokens)
This approach also has clear trade-offs. Compression causes information loss, which can affect complex scenes, small objects, dense text, and fine-grained charts. Visual primitive extraction depends on high-quality detection and annotation. The training data pipeline must cover object detection, regional relationships, image-text alignment, and multi-turn follow-up questions.
The engineering cycle is therefore much longer than simply connecting a vision encoder. This is another reason vision models may appear “late”: the work involves redesigning the representation, training process, and inference pipeline rather than adding a single module.
4. Strategic Choice: Agent First, Multimodal Support for the Perception Loop
There is also a strategic explanation. Language models and Agent capabilities are often the primary focus: tool calling, task planning, execution feedback, and result verification form the main loop.
Within this loop, vision allows an Agent to see the outcome of its own actions. It can read interface states, inspect document rendering, understand changes in charts, or verify whether a tool’s output matches expectations. Vision does not need to launch first as a standalone multimodal headline feature. It can instead serve as a perception layer that completes the Agent system.
This differs from the strategy of launching a flagship model with native full multimodal support from the beginning. The former treats multimodality as a flagship selling point; the latter treats vision as a sensor for task execution.
For developers, this means that evaluating a vision model should involve more than single-image question answering. They should also examine whether the model can be embedded reliably into a workflow:
- Does it support a fast, low-resolution path?
- Can it reuse visual caches across multiple turns?
- Can it verify the results of tool execution?
- Can it fall back to text or structured intermediate representations when the cost exceeds the budget?
5. Practical Guidance for Developers
1. Set a visual-token budget
Define a maximum resolution, per-image token limit, and batch-call limit for each task. This prevents high-resolution images from increasing costs unintentionally.
2. Use vision capabilities in layers
Simple classification and presence checks can follow a coarse-grained path. Fine-grained localization and document parsing can use a higher-precision path only when necessary.
3. Prioritize caching and reuse
In multi-turn conversations, repeated images and screenshots should reuse their visual representations whenever possible. This reduces the cost of repeated encoding.
4. Design fallback mechanisms
When visual confidence is low or the cost exceeds the budget, the system should fall back to text descriptions, structured fields, or human review.
5. Evaluate real operational cost
In addition to accuracy, track visual-token consumption per task, latency, failed retries, and the cost of manual correction.
Conclusion
The fact that vision models arrive “late” does not necessarily indicate a lack of technical capability. It is more likely the result of cost constraints, engineering redesign, and strategic prioritization working together.
Visual tokens amplify cost. Compressing visual primitives is a viable but expensive engineering path. Agent workflows provide a more practical driver for initial deployment.
For developers, understanding this cost curve is more valuable than chasing a single experimental release. Vision capabilities can become practical at scale only when resolution, caching, task routing, and evaluation are designed as part of a complete engineering loop.
