Image-to-Vector Drawing Code: Evaluating and Implementing Multimodal Model Restoration
This article examines the task of converting diagram images into renderable vector drawing code using multimodal large models. It breaks down what the task requires, categorizes the main sources of error, and proposes a quantifiable evaluation framework built on a layered test set and a multi-dimensional metric system. It then covers the engineering practices that make the output stable and production-ready, including structured prompts, a verifiable intermediate representation, a render-driven revision loop, input preprocessing, post-processing, and regression testing, before addressing production infrastructure, capability boundaries, and risk control. Readers will learn how to move from subjective "looks better" judgments to a measurable, verifiable, and continuously improving pipeline.
Introduction: From “Describing an Image” to “Replicating an Image”
Diagrams are everywhere in technical design reviews, architecture explanations, textbook writing, and data visualization. But most diagrams are just “a picture”:
- They blur when scaled up;
- Changing the color scheme or line width means redrawing from scratch;
- They cannot be included in version control or automated workflows.
Converting these diagrams into vector drawing code is an effective way to improve diagram maintainability. The result can be parameterized, versioned, evolved alongside text, and batch-rendered by scripts with a consistent style.
In recent years, the maturation of multimodal large models (models that can understand both images and text) has turned “paste an image, get back drawing code” from an idea into reality. To complete such a conversion, a model needs three capabilities:
- Understanding the image: identifying geometric elements, text labels, connection directions, and spatial relationships;
- Reasoning clearly: reconstructing 3D structure, judging occlusion relationships, and inferring layer order;
- Writing correct code: following the syntax rules of a drawing language to express the understanding as coordinates, paths, solid/dashed lines, filled regions, and drawing order.
However, there is often a long gap between “the effect looks good” and “deliverable in engineering.” This article focuses on such “image-to-vector-drawing-code” tasks, analyzes their difficulties and sources of error, and presents a practical evaluation scheme and engineering practices.
2. Task Breakdown: What Does “Image to Vector Drawing Code” Require?
2.1 Input and Output
- Input: one or more diagrams; optional supplementary instructions such as coordinate-system conventions, target color scheme, and font requirements; and constraints such as canvas size and line-width specifications.
- Output: a piece (or set) of drawing code that, once compiled or rendered, produces a vector image visually highly consistent with the input.
The whole process can be broken into five consecutive stages:
- Perception: identifying primitive types, including points, line segments, polygons, circles, arrows, and curves, as well as text labels;
- Structural understanding: recovering geometric relationships such as parallel, perpendicular, tangent, intersecting, and containment;
- Semantic judgment: distinguishing which lines are visible contours and which are occluded hidden edges; which elements are the main body and which are annotations and auxiliary lines;
- Code generation: mapping the above understanding to coordinates, paths, style attributes (solid, dashed, dash-dot, fill color, line width, arrow style), and drawing order;
- Render verification: rendering the code into an image, comparing it with the original, and correcting accordingly.
This is not a simple “image description” but a constrained structured reconstruction.
2.2 Typical Structure of Complex Diagrams
Take a relatively complex diagram as an example—a 3D structure drawn in “axonometric projection.” Axonometric projection uses parallel projection to present all three dimensions of an object at once and is common in engineering drawings and technical documentation. The input of such diagrams usually has the following characteristics:
- 2D pixels, 3D semantics: the graphic is a flat pixel image, but each shape corresponds to a face, edge, or vertex in 3D space;
- Occlusion relationships: along the viewing direction, nearer faces block farther faces. By drawing convention, visible contours use solid lines and occluded contours use dashed lines;
- Layer relationships: annotation text, leader lines, auxiliary lines, and background grids are often stacked above or below the graphic and need to be handled separately;
- Drawing order: filled faces must be drawn in “far-to-near” order to produce correct occlusion.
The model needs to first “reconstruct” the 2D information in the image into 3D semantics and layer structure, then convert it into concrete coordinates, paths, line types, fills, and drawing order. An error in any step will produce a noticeable deviation from the original in the render.
3. Difficulty Analysis: Where Do Errors Come From?
Around this task chain, errors can be grouped into five categories. This classification is both an analytical framework and the source of the evaluation metrics used later.
| Error type | Typical manifestation | Intuitive consequence |
|---|---|---|
| Recognition error | treating annotation text as a graphic element; missing small elements; confusing arrows with line segments | extra elements, missing elements, misplaced positions |
| Geometric error | coordinate approximation making parallel lines no longer parallel; distorted circles, ellipses, and arcs | distorted proportions, misaligned lines |
| Structural/semantic error | failing to distinguish visible contours from hidden edges; confusing front/back relationships; wrong layer assignment | mixed solid/dashed lines, disordered occlusion |
| Style error | inaccurate line width, color, fill pattern, arrow style; confusing dashed and dash-dot | style inconsistent with the original |
| Order/completeness error | reversed fill order; incomplete code fragment; code that will not compile | render failure, or wrong occlusion |
3.1 Ambiguity from 2D Pixels to 3D Structure
A single axonometric image does not map one-to-one to a real 3D shape. Common projections such as cabinet oblique and isometric projection inherently involve compression and tilt. The same shape may correspond to a rectangular box, a wedge, or a grooved block.
This means the model must perform “constrained inference,” with constraints from three sources:
- implicit line relationships in the image, such as parallel, equal-length, and right angles;
- hidden edges implied by dashed lines;
- dimension information given by text annotations.
The weaker the constraints, the more likely the model is to generate a shape based on “visual impression” that “looks right” but does not match the true structure.
3.2 Layers and Drawing Order
Vector graphics are drawn in an “overwriting” manner: later elements cover earlier ones. Once the order is wrong, even if each primitive is correct, the final effect will be distorted. Common order errors include:
- drawing the outline first and then filling it, causing the fill to cover the outline;
- drawing text first and then the face that occludes it, making the text invisible;
- drawing a background grid on the top layer.
3.3 Strict Syntax of the Output Language
Drawing code usually has stricter syntax requirements than general-purpose programming languages:
- coordinate notation, macro definitions, path closure, and so on all have fixed rules;
- one unmatched bracket or one undefined macro can cause the whole compilation to fail;
- embedding computational expressions in coordinates will also raise errors if the parser does not support the corresponding mechanism.
So “generating code that looks correct” and “generating code that compiles” are two separate hurdles, and must be counted separately in evaluation.
4. Capability Evaluation: Turning “Feels Stronger” into Quantifiable Conclusions
After the model’s capability iterates, “the restoration effect is clearly improved” is common feedback. This subjective feeling is valuable but not enough to support selection, acceptance, or continuous integration. It is recommended to build an evaluation system from the following four aspects.
4.1 Layered Evaluation Set Design
| Layer | Graphic characteristics | Focus | Difficulty |
|---|---|---|---|
| L1 | geometric figures, simple charts | basic coordinate and line restoration | low |
| L2 | flowcharts, block diagrams, timelines | nodes, connections, arrows, text layout | medium |
| L3 | axonometric, perspective wireframe diagrams | 3D structure inference, hidden edges, occlusion, drawing order | high |
| L4 | multi-layer engineering diagrams with dimension annotations and local enlargements | layer assignment, style details, completeness | very high |
Each layer should have no fewer than a few dozen samples, covering different drawing styles, line types, and color schemes. The L3 layer should deliberately include a contrast between regular and irregular shapes—for example, a regular rectangular box versus a block with a beveled face—to test whether the model is merely applying a “default shape template.”
4.2 Multi-dimensional Metric System
| Metric | Meaning | Calculation or judgment |
|---|---|---|
| Render success rate | whether the code renders successfully | number of successfully rendered samples / total samples |
| Structural similarity | degree of overall composition and line restoration | align the rendered image and the reference image by size, then compute grayscale structural similarity |
| Element coverage | whether key elements are restored | manually annotate a list of key elements and compare item by item |
| Style accuracy | whether line type, width, and fill are accurate | sampled statistics by attribute such as line type, fill, and arrow |
| Drawing order accuracy | whether occlusion and layer relationships are correct | manual judgment of fill order and layer assignment |
| Robustness | whether results are stable under input perturbations | repeat tests after applying rotation, scaling, contrast adjustment, and light compression to the input |
| Human subjective score | overall restoration impression | multiple reviewers rate restoration, readability, and editability on a 1–5 scale, then take the mean |
Note that structural similarity can only reflect “how similar it is at the pixel level” and cannot identify “which face was wrongly occluded.” Therefore it must be combined with element coverage and manual spot checks to avoid false positives where “similarity is high but semantics are wrong.”
4.3 Evaluation Execution Suggestions
- Fix sampling parameters: generate multiple times for the same input and count the variance of results to judge stability;
- Unify the rendering environment: fix the renderer version and font environment to avoid mistaking environmental issues for model issues;
- Record the full chain: save the input image, prompt, raw output, and render logs for root-cause analysis;
- Double-blind scoring: reviewers do not know which model version the output came from, reducing subjective bias.
5. Engineering Practice: Making Restoration Results Stable and Usable
Beyond evaluation, it is more important to embed this capability into a reusable engineering workflow. The following six steps, when reasonably combined, can significantly improve output stability.
5.1 Constrain Output with Structured Prompts
The prompt should turn “implicit expectations” into “explicit rules,” especially the coordinate system, layer order, style definitions, and prohibited items. Example:
[Task] Convert the attached image into renderable vector drawing code, strictly following these rules:1. Canvas size is fixed at 800x600; the coordinate origin is at the top-left corner, the x-axis points right, the y-axis points down;2. All coordinates and line-segment lengths are rounded to multiples of 10;3. Primitives are limited to: line segment, rectangle, polygon, circle, text, arrow;4. Styles explicitly specify line width, color, and line type (solid, dashed, dash-dot);5. Occluded contours are always drawn with dashed lines;6. First output a JSON intermediate representation, then output the drawing code; the two must correspond one-to-one;7. Do not output any explanatory text or Markdown markup;8. If information is insufficient, list the assumptions in an "uncertain" field in the intermediate representation.
Rule 6 turns “structure first, then code” into an output contract; rule 8 explicitly exposes the model’s “guesses” for manual review.
5.2 Introduce a Verifiable Intermediate Representation
When generating drawing code directly, the model must do both “understanding” and “encoding” at once, making errors hard to locate. A more robust approach is to first describe the scene graph in JSON, then generate code from it.
{"canvas": { "width": 800, "height": 600 },"layers": [{ "id": "grid", "z": 0, "elements": [{ "type": "line", "from": [0, 300], "to": [800, 300],"style": { "dash": "dotted", "width": 0.5 } }]},{ "id": "body", "z": 1, "elements": [{ "type": "polygon", "points": [[200, 400], [400, 250], [600, 400]],"style": { "fill": "#e8f0fe", "width": 1.2 } },{ "type": "line", "from": [400, 250], "to": [400, 500],"style": { "dash": "dashed", "width": 0.8 },"note": "hidden edge" },{ "type": "text", "at": [400, 220], "content": "support column" }]}]}
The intermediate representation brings three benefits:
- Structural validation: before code generation, check whether layer numbers are continuous, whether coordinates are out of bounds, and whether elements are duplicated;
- Local patching: when an element is wrong, only modify the corresponding entry, without regenerating all the code;
- Reproducibility: the intermediate representation itself can serve as a cache key and version snapshot, so the history of results for the same input can be traced.
5.3 A Render-driven Automatic Revision Loop
Making “generate—render—compare—revise” an automatic loop is the most direct lever for improving restoration. Illustrative code:
def refine(image, prompt, renderer, max_rounds=3):state, history = generate(image, prompt), []for i in range(max_rounds):result = renderer.render(state.code)if not result.ok:feedback = f"render failed: {result.error}"else:score = structural_similarity(result.image, image)if score >= TARGET:return state.code, {"rounds": i + 1, "score": score}feedback = build_feedback(result.image, image) # difference location and typestate = revise(state, feedback)history.append({"round": i + 1, "feedback": feedback})return state.code, {"rounds": max_rounds, "history": history}
Revision feedback should be as specific as possible—for example, “the annotation text in the top-left is occluded by a filled face” or “the start/end points of a dashed edge are offset by about 20 pixels”—rather than a vague “optimize it further.” The more specific the feedback, the more targeted the next revision.
5.4 Input-side Preprocessing
Restoration quality depends largely on input quality. Engineering recommendations:
- remove noise and compression artifacts, crop excess whitespace, and ensure clear lines;
- when the original contains dimension annotations, prioritize extracting the text information as geometric constraints;
- when the graphic area is too small or the resolution is insufficient, upscale before input;
- for very complex diagrams, split them into multiple sub-images by region, restore them separately, then merge the layers.
5.5 Post-processing and Style Unification
Model output often needs post-processing to meet team standards:
- unify color variables, replacing hardcoded colors with design tokens;
- unify line-width tiers—for example, three tiers of thin, main, and outline lines;
- auto-format the code and fill in missing style declarations;
- generate preview images so reviewers can annotate directly on the render.
5.6 Regression Testing and Continuous Integration
Solidify the evaluation set from Chapter 4 into regression cases, run automatically after each prompt adjustment, model change, or post-processing rule modification, and output change curves for metrics such as render success rate and structural similarity. Only a quantified comparison can avoid subjective “feels better” judgments.
6. Production Engineering: From Single Call to Production Pipeline
When this capability needs to serve a team or external customers, a simple “call—return” is no longer enough, and the following infrastructure must be added:
- Elastic inference resources: visual understanding is a high-load task; containerized deployment and auto-scaling can handle traffic fluctuations and avoid peak-time queuing;
- Task scheduling and artifact management: orchestrate “generate—render—validate” as asynchronous tasks, use message queues to smooth peaks and fill valleys, and store render artifacts centrally in object storage for version traceability and access control;
- Observability: set up monitoring and alerts for key metrics such as request volume, failure rate, per-stage latency, and render success rate, to promptly detect quality regressions caused by prompt changes or environment changes;
- Cost control: image inputs significantly increase compute overhead. A layered strategy of “low-resolution screening first, then high-resolution refinement” can be used, and results for identical inputs and configurations can be cached and reused;
- Platform support: enterprises that need to carry the above capabilities can rely on the general-purpose capabilities of cloud platforms such as Baidu AI Cloud—model services, object storage, container services, message queues, and monitoring/alerting—to build according to their own needs, using their own stress-test and evaluation results as the basis for selection.
7. Capability Boundaries and Risk Prevention
Rational use of this capability requires understanding its boundaries:
- Unsuitable scenarios: hand-drawn sketches, severe perspective distortion, too much occlusion, too low resolution, or graphics with complex mathematical formulas will have noticeably larger restoration errors and are better suited to manual drawing;
- Nature of errors: model output is “probabilistic generation” that may contain structural errors without self-awareness, and must be verified by render checks and manual spot checks;
- Compliance requirements: input images should have legitimate usage rights, and images involving internal materials must follow the relevant data security management norms.
8. Conclusion
“Image-to-vector drawing code” is an excellent testbed for evaluating the comprehensive capability of multimodal models: it simultaneously tests visual perception, spatial reasoning, semantic understanding, and constrained code generation. From simple charts to 3D structure diagrams with occlusion relationships, the restoration improvements brought by model iteration are worth attention, but what truly determines success in practice is the measurable, verifiable, evolvable engineering system behind it.
Developers are advised to proceed in the following order:
- First define the output contract, writing the coordinate system, layer order, and style conventions into the prompt;
- Then build a layered evaluation set and multi-dimensional metrics to establish a quantifiable baseline;
- Then introduce the intermediate representation and the render-driven automatic revision loop;
- Finally integrate monitoring, caching, and regression testing, turning a one-time successful demo into a sustainable, continuously evolving graphics production pipeline.
