Evaluating Multimodal LLMs for Image Understanding and 3D Structure Reconstruction
This article explains why multimodal capabilities require separate evaluation and presents three task types: visual semantic recognition, 3D structure inference, and structured drawing-code generation. It then describes a reproducible evaluation framework covering sample tiers, metrics, human review, reasoning-cost control, implementation patterns, risks, and a production checklist.
Multimodal models place text and images in the same reasoning process. For developers, a single API can process screenshots, photos, diagrams, and text instructions together. However, “being able to understand an image” and “being able to turn an image into structured output that can be used directly” are two different things. The former focuses on recognition accuracy; the latter depends on whether the generated content can be called directly by downstream systems.
During project initiation, teams often make the mistake of judging whether a model is usable based on a few demo images. Demo images are usually clear, simple, and feature typical examples, so they cannot reflect occlusion, low resolution, multiple layers, and specialized terminology in real business scenarios. A reproducible evaluation method is therefore needed.
1. Why Multimodal Capabilities Need Separate Evaluation
Multimodal models place text and images in the same reasoning process. For developers, a single API can process screenshots, photos, diagrams, and text instructions together. However, “being able to understand an image” and “being able to turn an image into structured output that can be used directly” are two different things. The former focuses on recognition accuracy; the latter depends on whether the generated content can be called directly by downstream systems.
During project initiation, teams often make the mistake of judging whether a model is usable based on a few demo images. Demo images are usually clear, simple, and feature typical examples, so they cannot reflect occlusion, low resolution, multiple layers, and specialized terminology in real business scenarios. A reproducible evaluation method is therefore needed.
2. Three Typical Tasks and Their Challenges
2.1 Visual Semantic Recognition
The goal of this task is to determine the objects, scenes, and relationships in an image. The challenge lies in fine-grained distinction and contextual inference. For example, when there are multiple objects in the frame, the model needs to answer according to the scope of the question rather than guessing based on appearance. Scenarios involving real people also require additional attention to privacy and compliance boundaries, which will be discussed separately later.
2.2 3D Structure Inference
Reconstructing a 3D shape from a single 2D image requires the model to understand perspective, occlusion, depth order, and the connections between parts. The output is usually a mesh, point cloud, or parametric description. In testing, it can be observed that the overall outline is often more stable than local details, while small parts and symmetry are more prone to errors. Evaluation should therefore use layered scoring.
2.3 Structured Drawing-Code Generation
Converting a diagram into drawing description code requires the model to output coordinates, paths, line types, fill areas, and drawing order. Drawing order determines whether occlusion relationships are correct, and it is the most easily overlooked metric. The generated code must be able to compile or render; otherwise, even accurate visual understanding cannot be delivered.
3. Building a Reproducible Evaluation Framework
3.1 Sample Tiers
The evaluation set should cover the following three tiers:
- Basic samples: clear, single-object, standard viewpoint;
- Medium samples: mild occlusion, multiple objects, common professional charts;
- Difficult samples: low resolution, multiple layers, hand-drawn or scanned documents.
Keep the number of samples in each tier stable, and record the source and annotation time to facilitate cross-version comparison.
3.2 Metric Design
Different tasks require different metrics:
- Recognition tasks: accuracy, recall, and error-type distribution (missed detection, false detection, attribute error);
- Structure tasks: outline similarity, presence of key parts, and correctness of occlusion relationships;
- Code tasks: executable rate, difference between the rendered result and the reference image, and the cost of one manual revision.
Metrics should be calculated under the same prompts and parameters. Otherwise, differences between versions may simply be noise caused by prompt changes.
3.3 Human Review
Automated metrics cannot judge results that “look reasonable overall but contain detail errors.” It is recommended to sample and review difficult samples, with two annotators scoring independently and calculating consistency. If disagreements are large, revise the scoring criteria first, then continue the evaluation.
4. Reasoning Parameters and Cost Control
Most services offer different reasoning-depth options. Higher depth usually improves performance on complex reasoning tasks, but it also increases latency and cost. In practice, a tiered strategy can be used: low tier for simple recognition, high tier for structure inference and code generation, and logging the latency and usage of each request type.
Cost optimization can start from the following aspects:
- Fix the prefix of the system prompt to leverage server-side prefix caching mechanisms, if the service supports them;
- Control image resolution to avoid oversized images consuming meaningless input quota;
- Limit output length and require the model to return only structured results;
- Schedule non-real-time tasks during periods with lower load. Some services have different billing during peak hours.
Specific billing rules vary widely and should be based on actual bills and stress-test data.
5. Implementation Example
The pseudocode below shows a general calling pattern, with the focus on input control and output validation.
def analyze_diagram(image_path: str, task: str) -> dict:image_bytes = load_and_resize(image_path, max_side=1536) # Control input sizeprompt = build_prompt(task) # Fixed template for caching and comparisonreasoning = "high" if task == "structure" else "low" # Tier reasoning by taskraw = call_multimodal_model(image=image_bytes,prompt=prompt,reasoning_effort=reasoning,max_output_tokens=2048,)result = parse_json(raw)validate_schema(result, schema_for(task)) # Validate structurereturn result
This code has three key points. First, the image size is controlled before sending. Second, the task type determines the reasoning depth and prompt template. Third, the output must pass structural validation before entering subsequent processes. If validation fails, retry or degrade rather than writing directly into business data.
For drawing code, a rendering test should also be added: compile or render the output in an isolated environment and compare the result with the annotated image. Compilation failures and rendering timeouts should both be logged to measure model stability across sample types.
6. Risks and Compliance Boundaries
Multimodal models have three common risk categories.
The first is hallucination. The model may provide attributes or values that look plausible but do not match the image. Key information should be cross-checked against the original image or other data sources.
The second is privacy. People, license plates, ID documents, and addresses in images are all sensitive data. System design should clearly define which fields are allowed to be recognized and avoid identity inference about real people. If a business truly needs face-related capabilities, it should first complete a compliance assessment and obtain authorization from the data subject.
The third is intellectual property. When using third-party images for training or testing, confirm the scope of authorization and retain records of data sources.
7. Production Checklist
Before integrating into a production environment, confirm each item:
- Whether the evaluation set covers basic, medium, and difficult tiers;
- Whether metrics can be reproduced under fixed prompts and parameters;
- Whether outputs have undergone structural validation and rendering tests;
- Whether costs are tiered by task and equipped with logs and alerts;
- Whether sensitive information has filtering and compliance review;
- Whether there is a corresponding regression testing process after model version upgrades.
By following the process above, teams can quickly determine whether a multimodal capability is suitable for a specific business scenario and turn evaluation results into engineering metrics that can be tracked continuously.
