Multimodal Model Evaluation in Internal Beta: From Impressive Cases to Reliable Assessment
Internal-beta multimodal reasoning models can change quickly, so evaluation must be reproducible and repeatable. This article presents a practical assessment framework: defining the test object and boundaries, using complex images to expose spatial-reasoning gaps, separating four levels of output quality, applying a weighted scoring rubric, measuring stability across repeated runs, following a five-step test workflow, and documenting common pitfalls. It also offers general team recommendations for isolated validation, data handling, and version comparison. The examples are hypothetical and illustrate judgment methods; they are not measured results for any specific model.
This article discusses multimodal reasoning models in the internal-beta stage. The source material did not provide a specific model name, version number, or test samples. Therefore, the methods described here apply to similar internal-beta versions but do not represent measured conclusions for any particular model. The examples are hypothetical cases used to illustrate judgment methods, not measured data.
1. Define the Test Object and Boundaries First
Internal-beta versions often update within days, and capabilities under the same name may be inconsistent before and after an update. Before testing, record the following information; otherwise, results cannot be reproduced or compared with the official release.
| Item | Information to record | Example |
|---|---|---|
| Model identifier | Vendor, version number, access method | Web UI, API, canary channel |
| Test time | Specific date and time window | A morning in November 2026 |
| Input type | Image resolution, whether text annotations are included | 1200x800 pixels, with dimension annotations |
| Resource consumption | Response time, token count, single-call cost | First response 18 seconds, about 2,400 tokens |
| Control group | Official version or another approach for the same task | Previous generation of the official version |
Without a control group, you can only judge whether the model meets the bar on that task; you cannot judge whether it has improved over previous versions.
2. Why Complex Images Are More Likely to Reveal Differences
Simple image recognition usually requires only answering what is in the image, such as recognizing text, objects, or basic scenes. Most models can complete such tasks, so discrimination is low. Tasks that require building a spatial structure are more likely to expose differences.
Suppose a 2D diagram shows a rectangular block occluding the lower-left part of a cylinder. The bottom edge of the rectangle is dashed to indicate the occluded support structure, and on the right there is a dimension annotation “h=20”. The model is asked to generate drawing code and must make the following judgments at the same time:
- The rectangle is in front of the cylinder, and the occluded part of the cylinder should not be shown.
- The bottom edge is dashed, indicating a hidden contour, and must not be drawn as a solid line.
- The annotation “h=20” corresponds to the height of the cylinder, not the height of the rectangle.
- The drawing order should be cylinder, rectangle, dashed line, annotation.
This task follows the process of “visual input, structural reasoning, program expression.” The model must first understand the image correctly, then convert visual relationships into executable instructions. If the first step is wrong, subsequent code edits are often ineffective. Suppose a generated version has correct syntax and the figure displays, but the cylinder is drawn on top of the rectangle. The problem then lies in the understanding of the occlusion relationship, not in the code itself. At this point, ask the model to list the objects and front-to-back relationships first, then decide whether to modify the code.
3. Four Levels from Generatable to Usable
Evaluation should distinguish four levels, each with different criteria.
The first level is syntactically executable. The code can be parsed and run without errors. This is the minimum requirement and says nothing about graphical quality.
The second level is basically correct. The number of main objects, positional relationships, and connection relationships in the figure match the original, with a small amount of proportional deviation allowed. In the example above, this level is reached if the relative positions of the cylinder and rectangle are correct.
The third level is sufficient detail restoration. Occlusion relationships, solid and dashed lines, annotation positions, and local layout all correspond to the original. In the example, whether the dashed line is correct and whether the annotation points to the cylinder must be checked item by item.
The fourth level is iterable and maintainable. When the user requests a modification such as “increase the rectangle width by 10%,” the model should change only the relevant parameter. The way to judge this is to compare the differences between the two code versions; parts not involved should remain unchanged. If the model rewrites the entire code, even if the result is correct, it can only be considered non-compliant.
For a one-off demonstration, the second level may be sufficient. For technical documentation, teaching materials, or automated production, third- and fourth-level capabilities are required.
4. Dimension Scoring Table and Judgment Rules
Judging model quality from one or two successful cases makes results vulnerable to sample randomness. A more reliable method is to build a task set and score each dimension separately. The weights and scores in the table below are suggested starting points; teams can adjust them for their own business needs.
| Evaluation dimension | Weight | Observation focus |
|---|---|---|
| Visual recognition | 15 | Whether objects, text, colors, and boundaries are recognized accurately |
| Spatial reasoning | 20 | Whether occlusion, front-to-back, connection, and layering relationships are correct |
| Code generation | 15 | Whether syntax, coordinates, and drawing order are reasonable |
| Detail restoration | 20 | Whether dashed/solid lines, fills, annotations, and local layout are close to the target |
| Multi-turn modification | 15 | Whether the original structure is preserved after modification |
| Stability | 10 | Whether repeated calls produce consistent results |
| Engineering usability | 5 | Whether it is easy to execute, debug, and maintain |
The judgment levels and conversion coefficients for each dimension are as follows:
- Fully correct: 100% of the full score for the dimension;
- Basically correct: 70% of the full score;
- Partially wrong: 40% of the full score;
- Unable to complete: 0 points.
When judging, state the specific error location, for example, “the dashed line was drawn as a solid line, which is a partial error in detail restoration.” Writing only “the effect is average” cannot be used for comparison.
5. Use Statistics to Judge Stability
A single run cannot represent model capability. For each task, run at least 5 times at different times and record the following two metrics:
- Pass rate: the proportion of runs that reach “basically correct” or above out of the total number of runs;
- Score fluctuation: the standard deviation of the 5 total scores.
Suppose a task reaches basically correct in 4 out of 5 runs, for a pass rate of 80%, but one run has a dashed/solid line error. The stability of that task should then be judged as medium; it should not be considered stable simply because the pass rate is high. Teams can set their own thresholds; for example, if the pass rate is below 60% or the standard deviation exceeds 15 points, the task does not enter the production workflow.
6. Test Workflow and Record Template
The recommended test workflow has five steps:
Step 1: Ask the model to describe the image structure, without requiring it to generate code immediately.
Step 2: Manually check whether the objects, spatial relationships, occlusion relationships, and annotations are understood correctly.
Step 3: Ask the model to provide a drawing plan explaining the layer order and coordinate strategy.
Step 4: Generate the complete code and require key parameters to be defined centrally.
Step 5: Request local modifications based on the rendered result and check whether the original structure is preserved.
Fill in the following record for each test to facilitate later comparison:
Task ID:Full prompt:First response time:Follow-up needed (count):Code directly executable:Original structure preserved after local modification:Pass rate across repeated runs:Score standard deviation:Main failure types:
7. Common Issues and Check Actions
Over-trusting a single success. Check action: run the same task at least 5 times and compare pass rate and fluctuation.
Prompt information is too vague. “Convert this image into code” provides too little constraint on the output. Check action: specify the coordinate system, layer order, dashed/solid line rules, and whether auxiliary lines should be retained in the prompt.
Mistaking visual errors for code errors. Check action: first ask the model to list objects and relationships, manually verify them, and then ask it to modify the code.
Output is hard to maintain. If all coordinates are written directly into drawing commands, later modification will be difficult. Check action: require the model to define key parameters centrally in a variable section and group them by object.
Missing human review. Outputs involving dimensions, structures, and safety requirements cannot be used directly as final conclusions. Check action: assign a dedicated person to verify them and keep verification records.
8. General Recommendations for Teams
The following recommendations are based on general engineering practice, not measured conclusions about this internal-beta model.
First, validate internal-beta versions in an isolated environment first, rather than connecting them directly to production workflows, and keep a plan for rolling back to the previous version.
Second, do not upload sensitive images, customer data, or unpublished engineering drawings to the internal-beta endpoint. Before use, confirm the data usage terms of that channel and obtain authorization from the data owner.
Third, save the version number, samples, and raw output of each test. After the official release, retest with the same sample set to determine what substantive changes the iteration has actually brought.
