Evaluating Multimodal LLMs: A Practical Framework for Image Understanding, Code Generation, and Cost Control

    This article presents a repeatable approach to evaluating multimodal large language models across three capability dimensions: image understanding, code generation, and complex structure reconstruction. It shows how to design business-relevant test cases, compare models across fixed scoring criteria, and account for reasoning strength, prompt caching, and time-based pricing when calculating total cost of ownership. The goal is a sustainable model-selection process that stays valid despite rapid model iteration.

    1. Models Iterate Fast, So Evaluation Must Keep Up

    The iteration cycle for large models keeps shrinking. A model that led the field a few months ago can be overtaken by a new release, and the gap between a speed-focused “lite” version and a capability-focused “flagship” version shifts over time. For developers and enterprises, this means model selection cannot rest on a one-time conclusion. You need an evaluation method that is repeatable, comparable, and continuously updatable.

    Relying on public leaderboard rankings alone is often unreliable: the benchmark sets are limited, the question types are fixed, and their data distribution diverges from real business workloads. A more robust approach is to build test cases that reflect the capabilities you actually care about, then compare multiple candidate models head to head. The sections below use a few typical capability dimensions to show how to design such evaluations.

    2. Multimodal Understanding: From Entity Recognition to Semantic Extraction

    One of the core strengths of multimodal models is their ability to understand text and images at the same time. When evaluating image understanding, design test cases in layers:

    1. Entity recognition layer: Ask the model to identify people, objects, and scenes in an image and describe their identity or background. This layer measures how rich the visual knowledge in the model’s training data is. One point worth noting: a model’s accuracy on “well-known entities” is closely tied to the freshness of its training corpus and whether it uses a native multimodal architecture. A model pretrained on multimodal data from the ground up usually performs more consistently than one that bolts on a vision module later.

    2. Scene understanding layer: Ask the model to describe the overall layout, actions, and semantic relationships in an image to assess how well it integrates context.

    3. Fine-grained understanding layer: This covers chart reading, text recognition, and judging spatial and occlusion relationships. This is the layer that best separates model quality, because it requires the model to reason about 3D structure, layer ordering, and front-to-back occlusion within a 2D frame.

    When evaluating these capabilities, compare models horizontally using “the same image + multiple models + fixed scoring dimensions,” and record how each model performs at each layer rather than relying on a single aggregate score.

    3. Code Generation: A Browser-Based 3D Scene Example

    Code generation is one of the most practically valuable use cases for large models today. A great stress test is: ask the model to generate a single-file, full-screen animated web page that loads a mainstream 3D rendering library via CDN.

    For example, you could issue a request like this:

    1. Generate a single-file HTML page:
    2. - Load a mainstream JavaScript 3D rendering library via CDN
    3. - Scene set to bright daytime, with ground, sky, and lighting
    4. - Implement a complete rocket launch sequence:
    5. idle -> engine ignition -> burst of thick smoke and dust ->
    6. clamp release -> airframe shakes, lifts off, and accelerates ->
    7. camera follows the ascent up into the clouds
    8. - Drive the animation loop with requestAnimationFrame

    Although this request looks specific, it tests several capabilities at once: the model’s grasp of a third-party library’s API, its ability to model physical motion (acceleration, vibration, particle effects), its understanding of the render loop and camera-follow logic, and whether the final code runs without errors on the first try. When evaluating, you can set a few checkpoints:

    • Does the code run directly, or are there obvious syntax or dependency errors?
    • Does the animation cover every stage specified in the requirements?
    • Is the visual result (particles, lighting, vibration) natural?
    • Is the camera follow sensible?

    Tasks like this, where the output must “just run,” are an effective way to distinguish stronger code-generation models from weaker ones.

    4. Complex Structure Reconstruction: A Higher-Dimensional Challenge

    A harder class of task than generating an animation is reconstructing a 3D structure from a 2D image, or translating a graphic into vector drawing code.

    A typical example is reconstructing the 3D model of a well-known car model: the model must first understand the structure in the reference image, then output a parameterizable 3D geometric description. This kind of task is highly demanding and often exposes the gap between flagship models and ordinary ones.

    Another example is translating a graphic into a vector drawing language. The tools used here are not any single specific product but rather various “drawing description languages,” which describe an image through coordinates, paths, solid and dashed lines, fill regions, and draw order. Asking a model to convert a 2D diagram into this kind of code tests:

    • Understanding of 3D structure, occlusion, and layer relationships in a 2D image;
    • The abstraction ability to map visual information into precise coordinates and paths;
    • Whether the generated code is well structured and has a sensible draw order.

    When evaluating this capability, you can break “fidelity” into sub-items such as geometric accuracy, occlusion correctness, and line-style fidelity, and score each separately for a more granular comparison.

    5. The Impact of Reasoning Modes and Reasoning Strength

    Many models today offer adjustable reasoning strength / reasoning modes (for example, a direct mode, a fast mode, and a deep-reasoning mode). In practice, enabling higher reasoning strength often noticeably improves quality on complex tasks, especially those requiring multi-step thinking such as code generation and structure reconstruction.

    A pragmatic strategy is:

    • Use low reasoning strength for simple tasks (entity recognition, short Q&A) to prioritize speed and cost;
    • Switch to high reasoning strength for complex tasks (3D generation, structure reconstruction, complex logical reasoning) in exchange for quality.

    During evaluation, it helps to run every case at different reasoning strengths, quantify the quality gains and latency costs that reasoning strength brings, and make dynamic trade-offs in production.

    6. Balancing Cost and Performance: Caching and Scheduling

    Model usage cost is an unavoidable part of engineering adoption. The industry currently uses two common mechanisms to reduce costs, and understanding them helps you adopt a more economical calling strategy:

    1. Context / Prefix Caching (Prompt Caching)

    When multiple requests share the same system prompt, long document, or fixed prefix, the model can reuse the already-computed intermediate state (KV Cache). The cached portion of the input is typically billed at a significantly lower price, while the non-cached portion is billed at the standard rate. So placing stable, unchanging content at the front of the request and frequently changing content at the end can significantly raise the cache hit rate and lower cost.

    An illustrative call structure:

    1. [
    2. { "role": "system", "content": "fixed long system prompt / large reference material (cache-friendly)" },
    3. { "role": "user", "content": "this request's specific question (variable part placed at the end)" }
    4. ]

    2. Peak / Off-Peak Pricing

    Many platforms use differentiated pricing for peak and idle hours, with peak prices sometimes several times the off-peak rate. For offline batch processing and non-real-time tasks, schedule runs during idle hours to reduce cost; for latency-sensitive online tasks, you must trade off between cost and response speed.

    3. Controlling Output Token Cost

    The length of generated content directly affects cost. You can reduce unnecessary output tokens by constraining the output format, limiting redundant explanations, and processing long tasks in segments.

    7. A Reusable Evaluation Framework

    Combining the dimensions above, the recommended evaluation workflow is:

    1. Define a capability matrix: Break business requirements into dimensions such as image understanding, code generation, structure reconstruction, and reasoning, and assign a weight to each.
    2. Build a test set: Prepare 5-10 business-relevant cases per dimension, ideally with a gradient of difficulty.
    3. Fix the scoring criteria: Define quantifiable scoring items for each case (such as whether it runs, fidelity, time taken, and token consumption).
    4. Compare multiple models x multiple reasoning strengths: Run a matrix comparison across candidate models and reasoning strengths.
    5. Layer in cost accounting: Factor token consumption, cache hit rate, and time-based pricing into a total cost of ownership (TCO) calculation.
    6. Re-test periodically: Because models iterate quickly, establish a regular regression-testing mechanism to keep selection conclusions up to date.

    8. Summary

    Faced with rapidly iterating multimodal large models, rather than chasing the ranking of a particular leaderboard, it is better to build an evaluation system that fits your own business. Starting from three capability dimensions, image understanding, code generation, and complex structure reconstruction, and combining general engineering measures such as reasoning-strength selection, context caching, and time-based scheduling, this article offers a reusable framework for evaluation and optimization. The core idea remains: quantify capability with business-relevant test cases, constrain investment with a clear cost model, and keep conclusions valid through periodic regression testing. Only this way can you make rational, sustainable model-selection decisions in an environment of frequent model updates.