Visual Token Cost Estimation: A Practical Starting Point for Low-Cost Multimodal LLMs

    This guide explains how to estimate visual token counts and KV cache memory for multimodal large language models before deploying vision features. It includes a Python estimation function, a worked example, and a comparison of cost-reduction approaches, including adaptive resolution, region cropping, prefix caching, and structured text conversion.

    1. Estimate Costs First, Then Choose an Approach

    Whether vision capabilities can go live depends on three questions: whether the cost per call is acceptable, whether latency meets business requirements, and whether the engineering effort fits a manageable timeline. The rollout should therefore follow this order: first estimate the cost of visual tokens and the KV cache, then choose a compression approach, and finally build the feedback loop that lets agents use visual results.

    Multimodal capability is now a standard feature of large language models. What separates products is often how much it costs to process one image and how long that takes. For services built around low cost and high throughput, unchecked visual input overhead affects both pricing and serving capacity.

    2. Where the Cost of Visual Tokens Comes From

    When a language model processes text, each token incurs a fixed amount of computation and storage. For image input, the image is first split into small patches, and each patch maps to one or more visual tokens. Higher resolution means more patches, so the token count grows quickly.

    The function below estimates the number of visual tokens based on common ViT-style encoding. It assumes each patch is patch pixels on a side, every merge × merge group of adjacent patches becomes one token, and any incomplete group is rounded up.

    1. import math
    2. def estimate_visual_tokens(width: int, height: int,
    3. patch: int = 14, merge: int = 2) -> int:
    4. """Estimate the number of visual tokens produced by one image (illustrative;
    5. values vary across models).
    6. All parameters are in pixels. Overhead such as the CLS token, positional
    7. encoding, and separator tokens is not included.
    8. """
    9. for name, value in (("width", width), ("height", height),
    10. ("patch", patch), ("merge", merge)):
    11. if not isinstance(value, int) or value <= 0:
    12. raise ValueError(f"{name} must be a positive integer, got {value!r}")
    13. cols = math.ceil(width / patch)
    14. rows = math.ceil(height / patch)
    15. return math.ceil(cols / merge) * math.ceil(rows / merge)
    16. print(estimate_visual_tokens(1024, 1024)) # Output: 1369

    The output of 1369 comes from the following calculation. Assume a 1024×1024 image, a patch size of 14, and a merge factor of 2. The image splits into 74 patches horizontally and 74 vertically, for 5,476 patches in total. Each 2×2 group of patches merges into one token, giving 37×37 = 1,369 tokens. If the patch size is changed to 16, the same image becomes 64×64 patches, which merge into 32×32 = 1,024 tokens. Patch and merge values differ between models, so these estimates are only useful for comparing options. Before launch, calibrate against the usage field returned by the provider to get the actual billed token count.

    Cost appears mainly in two stages.

    The first is prefill. Standard self-attention compute grows with the square of the sequence length, so longer inputs increase the time to first token. Linear layer compute grows in proportion to length. Actual latency also depends on the hardware and batching strategy, so measure it on your own hardware.

    The second is decoding. The model caches a key vector and a value vector for each layer, known as the KV cache. You can estimate the cache size with this formula:

    KV cache bytes ≈ 2 × number of layers × number of KV heads × head dimension × number of tokens × bytes per element

    The factor of 2 accounts for the K and V caches. Take the number of layers, KV heads, and head dimension from the model’s configuration file. Bytes per element depends on precision: 2 for FP16 or BF16, 4 for FP32, and 1 for INT8.

    Here is an illustrative calculation. Assume 32 layers, 8 KV heads, a head dimension of 128, and FP16 precision. The KV cache for one token is then 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB. An image that produces 1,369 visual tokens needs about 1,369 × 128 KiB, or roughly 171 MiB, of KV cache per request. Switching to FP32 doubles this to about 342 MiB. These figures only show the order of magnitude. Use the actual model configuration for real values.

    Visual tokens use the KV cache the same way text tokens do. Larger images increase memory pressure, so the same hardware can serve fewer concurrent requests.

    3. Three Approaches to Reducing Cost

    Common cost-reduction approaches fall into three groups: reducing the number of visual tokens, restructuring the inference pipeline, and preparing training data. The table below compares specific options.

    Approach Cost target Quality risk Engineering cost Suitable scenarios
    Adaptive resolution and tiling Visual token count, prefill and KV cache Loss of small text, table lines, and fine details Low; mainly routing rules and evaluation Images vary widely in size, and task types can be distinguished
    Region cropping and two-stage detailed reading Number and scope of high-resolution calls Detailed reading is not triggered if the coarse pass misses content Medium; requires two calls and scheduling logic Large images where only local information matters
    Visual encoding and prefix caching Repeated encoding and prefill of the same image Limited benefit when the cache hit rate is low Medium; requires managing cache keys and expiration policies Multi-turn question answering over the same document
    Converting visual content to structured text Replacing all visual tokens with text tokens OCR or layout parsing errors pass through to the model Medium to high; requires maintaining the parsing pipeline