Vision Token Costs and the Engineering Path of Multimodal Inference

    Why do some vendors ship vision capabilities late even when the underlying techniques are widely available? This article argues that the real constraint is cost structure rather than model capability. Vision tokens grow roughly with the square of image resolution, driving up prefill compute, KV-cache memory, and billing all at once. We break down where that cost comes from, walk through three combinable techniques for reducing it—token compression, attention sparsification, and structured coordinate representations—and explain why self-built multimodal routes take longer than bolting a vision encoder onto an existing language model. The article closes with the strategic role of vision inside agent systems and a practical engineering checklist covering inference optimization, platform operations, evaluation, and architecture selection.

    1. Introduction: The Real Reason Vision Capabilities Arrive Late

    In 2025, the release cadence of vision capabilities across major large models clearly diverged. Some vendors position native all-modal support as the headline feature of their flagship models. Others close the gap first with a separate vision-model branch. Still others opened image-understanding endpoints only in the second half of the year, and even then as experimental versions that often support images but not video, keep their weights closed, and ship with disclaimers attached.

    Attributing that last pattern to weak technology misses the point. The technical barrier to multimodal work has fallen sharply over the past few years thanks to the open-source community; vision encoders, projection layers, and training recipes all have mature, public reference designs. A team that can iterate steadily on a language model rarely gets stuck precisely at the vision step. A more plausible explanation sits in the business structure: vision input raises inference cost significantly, and once a model has built its reputation on low prices, it must first prove that it can “do vision for almost nothing” before it dares to launch officially.

    The rest of this article breaks down where the difficulty really lies across four layers: the cost math, the technical routes, the data pipeline, and integration with agents.

    2. Why Vision Tokens Are the Real Cost Drivers

    2.1 Token count grows roughly with the square of resolution

    Text tokens are determined by the tokenizer, so their count is roughly proportional to semantic length. Vision tokens instead depend on how many patches an image is cut into. Mainstream vision encoders typically divide an image into patches of about 16×16 and map each patch to a vector fed into the language model. As a result, the number of vision tokens is roughly proportional to width × height ÷ patch side².

    Input size Approx. tokens at 16px patches Roughly how many Chinese characters (1 char ≈ 1.5 tokens)
    448×448 784 ~500
    768×768 2,304 ~1,500
    1344×1344 7,056 ~4,700

    Both the “16px patch” assumption and the “≈1.5 tokens” figure here are illustrative conversion parameters; real implementations differ widely in how they split patches and which tokenizer they use. The conclusion still holds: dropping a screenshot or a high-resolution photo into the context easily costs several thousand tokens. Every time resolution doubles, the token count grows by about 4×.

    This growth is quadratic, yet the useful information carried by an image does not grow at anywhere near the same rate. Large patches of sky, solid-color backgrounds, and blurry decorative patterns all consume just as much compute and bandwidth as text carrying real meaning.

    2.2 Cost comes from three stacked factors

    1. Prefill compute. Attention cost grows quadratically with context length. Prefilling a long vision prefix is a clearly compute-intensive step that directly raises time-to-first-token (TTFT) and per-request GPU occupancy time.
    2. KV-cache memory. Every vision token must cache its Key and Value at every layer and every attention head. In multi-turn conversations, images reappear repeatedly and the cache keeps accumulating. Once memory is exhausted, the only options are to lower concurrency or trigger preemption and recomputation.
    3. Inference billing. Vision tokens are billed as input tokens, while the output is usually just a short description or a single action command. This severe asymmetry between input and output makes the unit cost of a vision request inherently high.

    2.3 Why low-price models are especially sensitive to cost

    A simplified model makes this clear. Let the text unit price be p and the output unit price be q (usually several times p), with V vision tokens, T text tokens, and O output tokens:

    1. single-request cost ≈ (V + T) × p + O × q
    2. vision token share = V / (V + T)

    With V = 4,096 and T = 300, vision tokens account for more than 90% of the input. At that point, launching any vision capability is roughly equivalent to raising the product’s average cost by that same proportion.

    If a model makes “lowest price in its generation” its core selling point and users arrive for exactly that reason, then once vision starts burning money, the price advantage is erased quickly. Conversely, if pricing tracks cost and rises to the top of the market, the “cheap” persona collapses just as fast.

    That is the crux: the timing of a vision launch is not decided by technical maturity alone, but by whether the team can prove that “vision can also be done at extremely low cost.” Cost structure is the real constraint on vision capabilities for flagship models.

    2.4 Quantitative assessment: run the numbers before launch

    Rather than debating “can it be done,” use data to answer “should it ship, and how.” Measure at least three metrics before launch:

    • Vision token share: what fraction of the input is vision tokens on real business samples.
    • Cost sensitivity: how elastic the business gross margin is to reductions in vision token count.
    • Latency budget: how TTFT relates to vision token count.

    Illustrative code:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class CallSample:
    4. image_pixels: int = 0 # image pixel count (width × height)
    5. text_tokens: int = 0
    6. output_tokens: int = 0
    7. # The parameters below are illustrative; actual values should follow the service documentation and benchmarks
    8. PATCH_PIXELS = 16 * 16 # pixels covered by a single vision encoder patch
    9. TEXT_PRICE = 1.0 # text input unit price (any currency unit / 1K tokens)
    10. OUTPUT_FACTOR = 3.0 # output price as a multiple of the input price
    11. MARGIN_FLOOR = 0.5 # minimum gross margin required by the business
    12. def estimate(samples, unit_price):
    13. image_tokens = sum(s.image_pixels / PATCH_PIXELS for s in samples)
    14. text_tokens = sum(s.text_tokens for s in samples)
    15. output_tokens = sum(s.output_tokens for s in samples)
    16. cost = unit_price * (image_tokens + text_tokens + OUTPUT_FACTOR * output_tokens) / 1000
    17. price = unit_price * text_tokens / (1000 * (1 - MARGIN_FLOOR))
    18. share = image_tokens / (image_tokens + text_tokens)
    19. print(f"Vision token share of input: {share:.1%}")
    20. print(f"Minimum billable text tokens to meet {MARGIN_FLOOR:.0%} margin: {price / unit_price * 1000:.1f} / call")
    21. return share, price
    22. # Real images are often 1080p: 1920×1080 ≈ 2 million pixels, about 8,100 vision tokens per image
    23. estimate([CallSample(image_pixels=1920 * 1080, text_tokens=300, output_tokens=200)] * 1000,
    24. unit_price=0.001)

    Running this illustrative code shows that:

    • A single 1080p image produces roughly 27× as many vision tokens as text tokens.
    • Under this distribution, the margin floor (illustratively set to 50%) essentially determines the billable amount of each vision call.

    This is why the key to launching vision often lies not in “can it recognize what is in the image,” but in “can the token count be pushed down.”

    3. Driving Down Vision Tokens: Three Combinable Routes

    3.1 Compression: let a few tokens carry the whole image

    First, distinguish two kinds of vision tokens, which call for different optimizations:

    • Vision prefix: the token sequence produced by the image encoder. It appears only in the turn where vision is input, and is the primary target of compression.
    • Vision output: the coordinates, boxes, and text the model emits. These are an output-side cost and can only be reduced through structured output and precise instructions.

    For vision, the mainstream idea behind prefix compression is “few governs many”: a fixed set of learnable queries, far fewer than the original patch count, uses cross-attention to distill information from all vision features and emit a compact set of tokens. The principle matches the query-based feature aggregation widely used in the vision community. A typical implementation:

    1. import torch
    2. import torch.nn as nn
    3. class VisualTokenCompressor(nn.Module):
    4. """Aggregate all vision patch features with a fixed number of learnable queries (illustrative)"""
    5. def __init__(self, embed_dim: int, num_queries: int = 64, num_heads: int = 8):
    6. super().__init__()
    7. self.queries = nn.Parameter(torch.randn(num_queries, embed_dim) * 0.02)
    8. self.attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
    9. self.norm_q = nn.LayerNorm(embed_dim)
    10. self.norm_kv = nn.LayerNorm(embed_dim)
    11. def forward(self, patch_feats: torch.Tensor) -> torch.Tensor:
    12. # patch_feats: [B, K, D], K is the original patch count and varies with image size
    13. q = self.norm_q(self.queries).unsqueeze(0).expand(patch_feats.size(0), -1, -1)
    14. kv = self.norm_kv(patch_feats)
    15. out, _ = self.attn(q, kv, kv)
    16. return out # [B, num_queries, D]: regardless of original size, vision token count stays num_queries
    17. # 768×768, 16px patch → 2,304 patches; 64 queries → 36x compression
    18. compressor = VisualTokenCompressor(embed_dim=2048, num_queries=64)
    19. feats = torch.randn(2, 2304, 2048)
    20. print(compressor(feats).shape) # torch.Size([2, 64, 2048])

    Its value shows up in two places:

    • Compression happens before the language model, so every subsequent layer processes only a small number of vision tokens.
    • The vision cost of a single image no longer inflates linearly with resolution, so high resolution can be turned on demand.

    The tradeoff is between the compression ratio and fine-grained information. Push the ratio too high and small text, edge details, and dense text are easily lost. A sensible approach is hierarchical compression: first use a few thumbnail tokens to judge “what is in this image,” then load local high-resolution fragments on demand for the regions that matter, such as a particular button or a specific line of text.

    3.2 Sparsification: reduce what attention has to “read” from the cache

    Compression reduces the number of vision tokens; sparsification reduces how much of the cache attention must access. When computing attention, each text token by default has to “read” every vision KV entry. Sparsification instead lets different tokens attend only to the vision information relevant to them:

    • Local attention: focus on neighboring regions.
    • Within-block attention: vision tokens compute attention only among themselves within the corresponding block.
    • Cross-modal sparse selection: a text token retrieves only the few most relevant vision tokens.

    The reason is that the waste in attention is structural:

    1. In any given sentence, the content actually related to the image is usually a minority.
    2. Vision tokens are highly redundant with one another.
    3. Large chunks of irrelevant computation do not improve answer quality; they only drive up cost and latency.

    For this reason, the gains from sparsification tend to stack with compression rather than replace it.

    3.3 Structured representation: coordinates are language too

    If you treat vision tokens as a bag of unordered vectors, the model can only “describe” the image. A different idea is to write positional structure into the tokens themselves, so that coordinates take part in generation and reasoning like ordinary text. A coordinate can be expressed as a combination of contiguous intervals:

    1. bin_00 bin_01 bin_02 ... bin_13
    2. bin_10 bin_11 bin_12 ... bin_23
    3. ... ... ... ... ...
    4. bin_30 bin_31 bin_32 ... bin_43

    Each bin covers a small region of the image, and a cell’s coordinates are written directly into the token sequence:

    1. <bin_12> <bin_27>

    This changes three things:

    • Fine-grained localization: to answer “where is the submit button in this screenshot,” the model outputs a coordinate rather than a passage of text.
    • Region focus: referencing a coordinate during reasoning is equivalent to explicitly directing attention to a local area of the image.
    • Direct supervision signal: training data carries precise coordinates, so no separate localization module is required.

    The design philosophy behind this is to turn “looking at the image” into “reasoning over the image”: the model does not first see the image clearly and then answer, but looks at specific positions on demand during reasoning. This is in the same spirit as chain-of-thought, at the cost of having to redesign the reasoning format and produce supporting data.

    3.4 Choose resolution by task: don’t make one image do everything

    Vision token count is determined by resolution, so preprocessing on the input side is the best cost-control point. The input form can be chosen dynamically per task:

    Task type Input form Vision token magnitude (illustrative)
    Image-content classification, scene categorization 384×384 thumbnail ~600
    Document / screenshot OCR Tile as needed + a global thumbnail ~1,000 per tile, plus ~200
    UI element localization Original resolution + coordinate-interval output Depends on resolution; loaded locally
    Video understanding Key-frame sampling Frames × per-frame tokens

    Illustrative code:

    1. PATCH = 16 # illustrative: vision encoder patch side length
    2. def count_tokens(width: int, height: int, patch: int = PATCH) -> int:
    3. return (width // patch) * (height // patch)
    4. def build_visual_input(width: int, height: int, task: str):
    5. """Build vision input by task (illustrative); returns the vision token count"""
    6. if task == "caption":
    7. return {"scale": 0.5, "tokens": count_tokens(int(width * 0.5), int(height * 0.5))}
    8. if task == "ocr":
    9. # Tile a long screenshot for OCR, each tile no wider than 1024
    10. tiles = max(1, -(-height // 1024))
    11. overview = count_tokens(512, int(512 * height / width))
    12. return {"tiles": tiles, "tokens": tiles * count_tokens(width, 1024) + overview}
    13. if task == "ui_grounding":
    14. # Encode at original resolution while allowing coordinate-interval tokens for a second-pass localization
    15. return {"scale": 1.0, "tokens": count_tokens(width, height), "coord_tokens": True}
    16. raise ValueError(task)
    17. # 1920×1080 screenshot: ~8,100 tokens when encoded directly; ~2,025 tokens when halved
    18. print(build_visual_input(1920, 1080, "caption"))
    19. print(build_visual_input(1920, 1080, "ocr"))

    For the same underlying “read the image” need, the choice of input form across different tasks can make the token count differ by several times. Baking this step into the inference pipeline already opens up most of the room for cost optimization.

    4. Costs and Bottlenecks: Why the Self-Built Route Is Slower

    None of the routes above is free, and their costs are exactly what explain why vision launches lag behind.

    4.1 Rebuilding the reasoning format

    The coordinate-interval-token approach requires the model to learn during training to “emit interval sequences and interpret them correctly.” This touches vocabulary expansion, position-encoding adaptation, training-objective design, and adjustments to how results are evaluated. Every step ripples through the rest of the system. Compared with attaching a vision encoder to an existing language model, the workload and risk are far greater.

    4.2 The high bar of the data pipeline

    General visual-question-answering data can teach a model to “describe” an image, but not to “localize” or “reason by region.” To gain those two abilities, the training data must carry structured region annotations:

    • Data sources: public object-detection, layout-analysis, and similar datasets.
    • Annotation spec: define the interval format per task and unify coordinate normalization.
    • Quality filtering: drop mislabeled, unlabeled, duplicate, too-small targets, and low-clarity samples.
    • Ratio control: balance the distribution across categories and target sizes.

    Illustrative pipeline pseudocode:

    1. # Illustrative pseudocode: shows the pipeline idea, not a real implementation
    2. raw_samples = collect_detection_datasets() # gather public detection/layout datasets
    3. stage1 = [s for s in raw_samples if s.bbox_valid and s.area_ratio >= 0.001]
    4. stage2 = [s for s in stage1 if s.labels_complete and s.image_quality_ok]
    5. deduped = semantic_dedup(stage2) # dual dedup: perceptual hash + embedding similarity
    6. samples = rebalance(deduped, size_bins=("small", "medium", "large"))
    7. # In practice, filtering from hundreds of thousands of raw samples down to tens of thousands of usable ones is not uncommon
    8. print(f"raw={len(raw_samples)}, final={len(samples)}")

    This kind of filtering often takes two or three rounds, and the usable data left at the end is only a fraction of the raw input. Compared with fine-tuning on public multimodal data directly, a “self-built pipeline plus multiple rounds of filtering” takes a much longer engineering cycle.

    4.3 The evaluation system has to be rebuilt too

    Evaluating vision capabilities cannot stop at “does the description look accurate.” Localization accuracy, the legality rate of interval outputs, and performance under occlusion and dense small targets all require dedicated evaluation sets and metrics—another asset that has to be accumulated over the long term.

    5. The Deeper Reason: Where Vision Sits in an Agent System

    Even once cost is solved, the priority of vision is not determined by model capability alone, but also by tradeoffs in the product roadmap.

    5.1 Two approaches

    • The “flagship as selling point” route: natively integrate text, image, audio, and video on the flagship model, where multimodality itself is the headline of the launch.
    • The “agent-first” route: prioritize resources toward the language model and agent capabilities (tool calling, planning and execution, environment interaction). Vision is just one more piece completing the capability map, with the goal of letting the agent “see the scene where it works.”

    5.2 Vision is the agent’s “senses”

    Take a GUI-operating agent as an example; its work loop cannot function without vision:

    1. Screenshot the current interface.
    2. Identify the positions of clickable elements.
    3. Decide the next action.
    4. Execute the action.
    5. Compare the interface before and after execution to judge whether the goal was reached.

    Without vision, an agent can rely only on structured information, such as a page’s accessibility tree or an API’s return values. That information is often incomplete, and dynamically rendered interfaces are especially hard to cover. Vision is the key link that actually closes the execution loop.

    Illustrative code:

    1. # Illustrative pseudocode: an agent treats the interface screenshot as "sensory input" in its loop
    2. def gui_agent_loop(goal: str, env, model, max_steps: int = 20):
    3. history = []
    4. for _ in range(max_steps):
    5. obs = env.screenshot() # vision input
    6. state = compress(obs, keep_regions=goal) # vision token compression
    7. action = model.plan(goal, state, history) # output a structured action
    8. env.execute(action) # execute
    9. history.append(action)
    10. if env.is_goal_reached():
    11. return "done"
    12. return "max_steps_reached"

    This also explains why “experimental,” “images only,” and “not open-source” become common launch forms:

    • Vision only needs to support the agent’s internal perception needs, not cover every scenario the way a general-purpose vision assistant would.
    • Opening the endpoint early is a way to collect real-world feedback.
    • Not committing to long-term support leaves room for later iteration.

    6. Selection Comparison: The Tradeoffs of the Three Routes

    Dimension Native multimodal (vision encoder + LLM jointly trained) Vision token compression / sparsification Structured vision primitives / coordinates
    Core idea Attach a vision encoder directly and jointly pre-train and align Reduce the number of vision tokens entering the LLM Encode position as tokens the model can reason over
    Vision cost High; grows ~quadratically with resolution Medium to low; prefix is constant or near-constant Medium; depends on output-token volume
    Implementation complexity Medium Medium High
    Typical strength Detail fidelity; mature ecosystem Controllable cost; high resolution usable Fine-grained localization and region reasoning
    Main weakness Resolution and cost are tightly coupled Too high a compression ratio drops detail Requires rebuilding the training format and data pipeline
    Best-fit scenarios General VQA, content generation Cost-sensitive, large-scale online services UI localization, document-structure parsing, agents

    The three routes are not mutually exclusive, and real systems commonly combine them:

    • Compression first, structure second: compress the vision prefix to a fixed length and use coordinate-interval tokens on the output side.
    • Task-level routing: send simple classification through thumbnails and complex localization through original resolution plus structured output.
    • Specialized-model fallback: hand mature tasks like OCR to dedicated small models, leaving the vision-language model to handle decisions and integration.

    7. Engineering Implementation Checklist

    7.1 Inference-side optimization

    • Cache reuse: when the image fingerprint matches, reuse the encoding result directly.
    • Batching and queuing: merge requests to raise GPU utilization and lower per-token cost.
    • Length control: cap the token budget per image, and automatically downsample or tile when it is exceeded.
    • Distillation and small models: for high-frequency, low-complexity vision tasks, serve them with dedicated small models distilled from a large one.

    7.2 Platform and operations

    Vision services typically involve a lot of image read/write, so a general-purpose cloud platform can provide the building blocks:

    • Object storage: hold original images and intermediate artifacts.
    • Message queue: smooth out traffic peaks and valleys, and support asynchronous inference.
    • Container platform: host elastically scaling inference services.
    • Logging and monitoring/alerting: track token usage, latency, and failure rates.

    On Baidu AI Cloud, for example, this pipeline can be assembled directly from general-purpose capabilities such as object storage, message queues, a container platform, and monitoring/alerting, without any additional dedicated components.

    7.3 Evaluation and iteration

    • Build a vision-token cost dashboard that tracks the vision token share per business line.
    • Plot a “compression ratio vs. accuracy” curve for each compression scheme to find the point of diminishing returns.
    • Use a regression evaluation set to hold localization accuracy and prevent compression from causing capability regressions.
    • Keep a canary-release and rollback path for launched versions; experimental versions in particular need a clearly defined exit strategy.

    7.4 Architecture selection principles

    1. Treat cost as a constraint, not an optimization target: first decide the acceptable vision token budget per call, then choose the scheme.
    2. Let the task decide the form: classification, OCR, localization, and video understanding each call for a different input construction and model combination.
    3. Close the loop before going all-modal: in agent scenarios, prioritize getting the “see–do–verify” loop working end to end, then expand modality coverage gradually.

    8. Summary

    Delivering multimodal capability looks on the surface like a model-capability problem, but it is fundamentally a cost-structure problem. Vision token count grows with the square of resolution, which raises prefill compute, memory usage, and the billing bill all at once. On models whose foundation is low pricing, the prerequisite for launching vision is often doing the groundwork on token count first.

    Across technical routes, compression, sparsification, and structuring each have their own emphasis and can be combined. In terms of engineering cost, the structured route demands rebuilding the training format and standing up a data pipeline, which takes a noticeably longer cycle. In terms of product strategy, whether vision is an external flagship selling point or the agent’s “senses for seeing where it works” determines both its priority and its launch form.

    For developers, rather than agonizing over when a given vendor will open up its vision endpoint, it is more useful to focus on three practical things:

    • The cost model for vision tokens.
    • The tradeoffs among different technical routes.
    • How to implement the vision loop inside an agent.

    Once these three are clear, whichever vendor’s vision capability you choose, you can keep cost and quality in your own hands.