Evaluating Diagram-to-Drawing Generation in Multimodal Models

    This article presents a practical method for evaluating how well multimodal models turn a diagram into structured drawing instructions. It explains why seeing a diagram differs from reproducing it, how to split the task into four semantic layers, how to design graded test sets and output contracts, which quantitative metrics and stress tests to use, and how to fit the evaluation into a production workflow. Readers get a reusable framework for judging whether generated drawing output is executable, reproducible, and auditable.

    1. Background: Why Understanding a Diagram Does Not Mean Being Able to Draw It

    In many engineering scenarios, a model has to do more than image understanding. It must translate visual structure into a drawing description that can be explained, reproduced, and compiled. The input is usually a single schematic. The expected output includes coordinate points, paths, closed regions, line styles, fill styles, and a strict drawing order. At its core, this is cross-modal alignment: the vision component extracts spatial relationships, the language component generates precise instructions, and both must hold at the geometric, topological, and semantic levels at the same time.

    The difficulty is that a schematic is a two-dimensional projection that expresses three-dimensional meaning. For example, a bracket in a diagram may show only its front surface, while a rear support rod is hidden behind it. A model that only describes what it sees may treat the occluded rod as absent and leave out the support. Layering creates a subtler problem. When several semi-transparent regions overlap, the drawing order changes the blended colors. If a base block is placed after the foreground, the visual difference may look small, but in code semantics it is an error. When assessing this capability, the question is not “does it look similar?” but “is it executable, reproducible, and auditable?”

    For this reason, evaluation should move from subjective impressions to structured metrics. These include whether geometric positions are stable, whether occlusion relationships are recovered, whether solid and dashed lines distinguish visible and hidden edges, whether fill regions are closed, and whether the drawing order matches the visual cause. The sections below describe a general acceptance method that does not depend on any particular vendor or tool. Teams can reuse it for model integration, prompt optimization, and regression testing.

    2. From Vision to Drawing Instructions: Four Semantic Layers

    Converting a complex diagram into drawing code works best when the task is broken into four semantic layers.

    The first layer is coordinate semantics. Image coordinates must be mapped to canvas coordinates, so the model needs to know the origin, axis directions, scale, and unit conversions. If the input has no scale reference, the system should at least assume normalized coordinates and state the coordinate convention in the output. It should not silently mix pixels, centimeters, and percentages.

    The second layer is geometric semantics. Lines, polylines, arcs, rectangles, polygons, and Bézier curves must be recognized as basic primitives, with their connections stated explicitly. The most common engineering error is not failing to detect a straight line. It is merging adjacent nodes into one “approximate region,” which makes later paths impossible to close. For structures with slopes, holes, or stepped edges, the model should output endpoints, intersections, and control points, not only a natural language description.

    The third layer is occlusion and depth semantics. Schematics often use dashed lines for hidden edges, solid lines for visible outlines, and transparent layers for stacking order. The model must infer which outlines belong to foreground objects, which belong to background objects truncated by the foreground, which dashed lines mark hidden edges rather than decoration, and which regions look alike in color but belong to different layers. If this layer fails, extra styling in later steps only produces something that looks right but has the wrong meaning.

    The fourth layer is drawing semantics, meaning the operation sequence. Vector drawing is not a static snapshot. It is an ordered set of commands. A base color comes first, then the bottom outlines, then fills, and finally dimensions and dashed lines. An evaluation should check that the command order satisfies dependencies. For example, a fill must run after its path is closed, a clipping region must be declared before the lines it clips, and the target of a transparency blend must be explicit. Only when all four layers are complete can the output render reliably and match the structure of the original diagram.

    3. Test Set Design: Using a Difficulty Ladder to Expose Capability Limits

    Subjective impressions are unstable when samples are random and cannot be reproduced. It helps to build a test set arranged as a difficulty ladder.

    Level one contains single-object flat drawings with only rectangles, circles, and simple connecting lines. It checks coordinate mapping and basic paths. Level two contains multi-object structures without occlusion. The model must distinguish adjacent boundaries and shared nodes. Level three introduces occlusion, where part of a component is hidden by a foreground element, and the model must represent the hidden edges with dashed or broken lines. Level four adds three-dimensional cues such as isometric projection, stepped faces, hole depth, and inclined supports. The model has to recover spatial relationships from projection clues. Level five combines layer blending, semi-transparent overlays, clipping, and annotation arrows. It tests instruction order and the scope of styles.

    Each sample should include four items: the original image, the expected output, an executable description file, and a scoring script. The original image should record its source type and resolution. The expected output should not be only the final rendered image. It should also include a structured inventory, such as a node table, an edge table, a layer table, and drawing steps. The scoring script should cover deterministic rules where possible. These include whether boundaries are closed, whether undeclared colors appear, whether dashed lines are used only for hidden edges, whether coordinates fall out of bounds, whether the layer count exceeds the limit, and the IoU or contour matching error between the rendered result and the original image.

    Perturbation variables can make the test more informative. The same structure can be drawn with different line widths, color schemes, viewpoints, added noise, lower resolution, or a higher occlusion ratio. If a model performs well only on clean samples, it may depend on matching visual templates. If it keeps topology consistent after perturbation, its structural understanding is more likely to transfer. An evaluation report should clearly separate “a high score because the task was easy” from “a high score because the method is stable.”

    4. Prompts and Output Contracts: Turning an Open Task into Constrained Generation

    A multimodal generation task should not simply say “turn this picture into code.” A better approach is an output contract. The contract first declares the coordinate system and units. It then lists layers from background to foreground. For each layer, it gives the visible outlines, hidden outlines, and fill regions. Dashed lines are allowed only for occluded edges. All regions must be closed. Finally, the contract provides drawing steps sorted by dependency. The purpose of the contract is to turn aesthetic judgment into constraints that can be verified.

    A reusable template asks the model to produce a “structure inventory” before the “drawing instructions.” The inventory includes object names, levels, relative positions, occlusion notes, and uncertain areas. The drawing instructions are then generated from this inventory. If generation fails, the inventory makes it possible to locate the cause. If the inventory omits a rear support, the error occurred in visual reasoning. If the inventory is correct but the instruction order is wrong, the error occurred in sequence planning. Engineering teams should also require the model to flag low-confidence areas. For example, it might state, “Hole depth cannot be determined from a single image, so it is assumed to be a through hole.” Allowing uncertainty declarations is safer than forcing a definite answer.

    A skeleton for the output might look like this:

    1. coordinate_system: origin=top_left, x=right, y=down, unit=normalized
    2. layers:
    3. - id: L0; role: background; z_order: 0
    4. - id: L1; role: main_body; z_order: 1
    5. occlusion:
    6. - target: rear_support; occluded_by: front_plate; render_as: dashed
    7. regions:
    8. - id: R1; layer: L1; closed: true; fill: solid
    9. draw_order: [L0_grid, R1_base, hidden_edges, labels]
    10. uncertainty: [hole_depth_assumed_through]

    This skeleton does not bind to a specific format. It only emphasizes that the output should be machine-readable, verifiable, and able to fall back safely. In production systems, teams should also set timeouts, length limits, syntax allowlists, and retry policies. When output does not meet the contract, it should not be re-rendered and shown to the user. Instead, the system should return a structured error, so failed samples can be collected for analysis.

    5. Quantitative Metrics: Do Not Rely Only on Final Screenshot Similarity

    Similarity between the final render and the original image matters, but it is not enough. At least four categories of metrics are useful.

    The first is geometric accuracy. This includes the average error of key points, edge direction error, area ratio error, and aspect ratio deviation. The second is topological correctness. This covers whether connection relationships match, whether the number of closed regions matches, whether parent-child layer relationships match, and whether hidden-edge markers match. The third is sequence validity. It checks whether commands can be parsed, whether forward references exist, whether styles exceed their scope, and whether the drawing order satisfies dependencies. The fourth is stability. This includes the variance across repeated generations of the same sample, the consistency of results before and after perturbation, and the normalized coordinate error across different input resolutions.

    For occlusion and three-dimensional structures, teams can add counterfactual probes. For example, remove the occluded support from the original image and see whether the model still draws it. Change a dashed line to a solid line and see whether the model wrongly infers that the object is visible. Mirror the image left to right and check whether the axis directions get confused. Increase the fill transparency and check whether the layer order is misjudged. Counterfactual probes help distinguish a model that has memorized common patterns from one that has understood projection relationships.

    An evaluation should also report the distribution of failure types. Common categories include coordinate drift, missed hidden parts, extra connections that do not exist, unclosed regions, reversed layers, mixed line semantics, and annotations contaminating the geometry layer. Categorized failures are more useful than a total score because they guide the next optimization step. Coordinate drift points to input normalization problems. Reversed layers point to output contract problems. Missed hidden parts point to insufficient depth reasoning. A single summary score can hide these weaknesses.

    6. Engineering Deployment: From a One-Time Demo to a Maintainable Pipeline

    What matters for a business is integrating the capability into a pipeline, not a single impressive demonstration. Evaluation can be divided into three stages: pre-release admission, gray-scale operation, and online regression.

    In pre-release admission, a fixed test set is run offline to compute metrics. The thresholds should not rely only on the total score. They should set hard limits for occlusion recovery, layer order, and command parsability. In the gray-scale stage, outputs are rendered in a sandbox. They go first to a draft canvas and enter the formal document only after passing structural validation. Every generated artifact should keep its source image, a model version placeholder, the prompt version, the random seed or decoding parameters, and the output with its validation results. These records can be kept in generic categories such as logging services, object storage, and monitoring and alerting.

    Regression testing is essential. Whenever the test set is expanded, the prompt template changes, or the decoding strategy is adjusted, an automatic baseline comparison should run and produce a difference report. The report should not only show images. It should locate differences at the level of layers, paths, and nodes. Examples include “the second rear post changed from dashed to solid,” “closure failed in the lower-right region,” or “the label layer was drawn too early.” These items can go directly into a defect tracker. For high-risk scenarios, a human-in-the-loop process is safer than full automation. The model generates the structure inventory, a user confirms the depth assumptions, and only then are the final instructions generated.

    Security and compliance also matter. Schematics may contain internal structures, cabinet layouts, process flows, or user interface sketches. Even an output without text can reveal topology information. A common practice is to define data classification, restrict external sharing, keep audit logs, and blur or abstract sensitive areas. Generated results should avoid repeating possibly sensitive coordinates. These principles are independent of any cloud vendor. They are standard engineering governance.

    7. Conclusion: Strong on Explainable Reconstruction, Weak on Unconstrained Imagination

    Consider the original scenario. A user provides a fairly complex schematic and asks the model to understand the three-dimensional structure, occlusion, and layer relationships in a two-dimensional picture. The model must then convert that understanding into coordinates, paths, solid and dashed lines, fill regions, and drawing order. If a newer model performs better than earlier versions, the improvement usually shows up not in better color matching but in more complete recovery of hidden edges, fewer contradictions in layering, and a command order closer to how a person would draw by hand. In other words, a real improvement should show both higher structural fidelity and higher executability.

    Still, optimism should be limited. Inferring depth from a single image is an ill-posed problem: one projection can correspond to several three-dimensional interpretations. A model can offer the most plausible assumption, but it cannot guarantee physical correctness. For serious engineering work, multimodal output should be treated as a candidate solution, not established fact. A deterministic validator should enforce syntax and closure, counterfactual probes should test depth reasoning, and human confirmation should resolve ambiguity. If an evaluation is to be objective, it should be built around this kind of evidence chain.

    Whether a model is usable can be reduced to three questions. Can it reliably recover hidden parts? Can it explain its own uncertainty? Can the instructions it generates reproduce the original structure without manual repair? Only if all three answers are yes does “generating code from an image” move from demonstration to engineering practice. Otherwise, even a striking screenshot is suitable only for low-risk drafts, not for documentation, manufacturing instructions, or automated delivery pipelines.