How Multimodal LLMs Convert Images into Vector Graphics Code

    When people evaluate the capabilities of multimodal large language models, "turning an image into a piece of vector drawing code" is one of the most revealing test cases. This article explains why the task is difficult, which core capabilities a model must have, how a complete engineering pipeline is structured, how to evaluate reconstruction quality, and how to balance quality, latency, and cost in production. It is written for developers who want to set realistic expectations, design proper evaluation systems, and put this capability to work in real engineering scenarios.

    1. Why Image-to-Vector Code Is a Hard Problem

    When discussing the capabilities of multimodal large language models, “turning an image into a piece of vector drawing code” is a highly representative test case. It is neither simple image recognition nor ordinary code completion. Instead, it demands that the model do two things at once: first understand the geometric and semantic structure inside an image, then reconstruct it precisely using a declarative drawing language.

    The difficulty of this task concentrates in several layers:

    • Inferring 3D structure from a 2D projection: The input is often a 2D bitmap, yet the scene itself carries 3D information. How the front, side, and top faces of an object are arranged in the image, which faces are visible, and which are occluded, all have to be inferred without any depth sensor.
    • Determining occlusion relationships: When multiple primitives overlap, deciding what sits in front and what sits behind directly determines whether the reconstruction “looks right.” Get the occlusion wrong, and the scene shows misplaced overlaps or holes.
    • Recovering layers and drawing order: In vector drawing, the drawing order (z-order) is meaningful. Primitives drawn later cover those drawn earlier, so the model must reproduce not only what was drawn but also in what order.
    • Precisely mapping pixel coordinates to vector coordinates: A bitmap is a discrete pixel grid, whereas a vector image consists of continuous coordinates and paths. The model has to convert pixel positions into reasonable vector coordinates while handling scaling, alignment, and proportions.
    • Line styles, fills, and path fitting: Solid and dashed lines, filling closed regions, and curve fitting are all key to reconstruction fidelity.

    In other words, the task requires the model to combine “spatial understanding + code generation.” The tighter the coupling between the two, the better the reconstruction quality.

    2. The Four Core Capabilities a Model Needs

    To complete the tasks above reliably, a model typically needs four capabilities working together:

    1. Visual grounding and spatial reasoning: The model must align natural-language descriptions, image regions, and geometric relationships, and understand relative position, proportion, and perspective.
    2. Command of the target language’s syntax: Whether it is a declarative vector drawing language or a general graphics description format, the model must master its conventions for coordinates, paths, styles, and layers, and output code that can be compiled and rendered.
    3. Stability of structured output: The generated result must be directly parseable by a downstream renderer. This calls for strictly structured output rather than loose, free-form natural-language description.
    4. Self-verification and iterative correction: A high-performing model can do more than generate once; it can progressively approach the original image through a “render–compare–correct” loop.

    3. The Full Technical Pipeline

    Turning these capabilities into an engineered solution can usually be broken into a clear pipeline:

    • Step 1: Image parsing and feature extraction. Preprocess the input image and identify edges, color distribution, and texture regions as a foundation for later segmentation.
    • Step 2: Semantic segmentation and hierarchy recovery. Cut the image into meaningful primitives and regions, and infer the front-to-back occlusion and layer membership among them.
    • Step 3: Geometric vectorization. Fit each region into straight lines, polylines, or curve paths, and determine fill and stroke attributes.
    • Step 4: Code generation. Following the syntax of the target drawing language, organize the geometric results into declarative code, handling coordinate transforms and drawing order.
    • Step 5: Render verification and iteration. Render the generated code into an image, compare it against the original input, and if the difference is significant, go back to earlier steps to fix parameters and order.

    Step 5 is especially important. A robust engineering approach turns “render–compare” into a closed loop, using visual similarity as a feedback signal to drive model iteration, rather than hoping a single generation will reproduce the image perfectly.

    4. Illustrative Code: From Pixels to a Vector Description

    The snippet below uses neutral, illustrative vector drawing pseudocode to show the shape of a reconstruction result (it is not the syntax of any specific tool, and is only meant to convey the structure):

    1. canvas(width=800, height=600)
    2. # Base layer: background rectangle (drawn first)
    3. rect(x=0, y=0, w=800, h=600, fill="#FFFFFF")
    4. # Middle layer: closed polygon with fill and dashed stroke
    5. path(
    6. points=[(100,150), (400,120), (430,420), (120,450)],
    7. closed=true,
    8. fill="#DDEEFF",
    9. stroke="#3366CC",
    10. stroke_width=2,
    11. dash_pattern=[6,4] # dashed line
    12. )
    13. # Upper layer: solid rectangle occluding the layer below (drawn later, covers it)
    14. rect(x=260, y=220, w=220, h=160,
    15. fill="#F5A623",
    16. stroke="#B36B00",
    17. stroke_width=1.5)
    18. # Top layer: text label
    19. text(content="Sample", x=300, y=300, size=18, fill="#222222")

    This illustrative code captures several key points: parameterized coordinates and dimensions, the representation of paths and closed regions, the distinction between solid and dashed line styles, fill-color control, and drawing order (z-order)—later elements cover earlier ones, which correctly expresses occlusion. In real scenarios, models face far more complex combinations of layers and occlusion.

    5. How to Scientifically Evaluate Reconstruction Quality

    Evaluating the quality of “image-to-vector code” cannot rely on “does it look similar” alone. It is better to build a multi-dimensional set of metrics:

    • Visual similarity: Use perceptual similarity measures (such as structural similarity, per-pixel or perceptual differences) to quantify how close the generated image is to the original, optionally combined with manual visual comparison.
    • Compilation/rendering success rate: Whether the generated code can be parsed by the target renderer without errors is the first threshold for usability.
    • Structural fidelity: Whether the number of primitives, the number of layers, and the region division are logically consistent with the original image—avoiding results that are “pixel-close but structurally scrambled.”
    • Editability: Whether the output vector primitives are clean, groupable, and individually modifiable, which determines the practical value of the result.
    • Iterative convergence: In the “render–compare–correct” loop, whether the error decreases steadily with the number of iterations.

    A common mistake is to focus only on visual similarity while ignoring structural correctness. A result that is close at the pixel level but has a jumbled layer structure is nearly worthless in engineering scenarios that require further editing.

    6. The Trade-off Triangle of Quality, Latency, and Cost

    In practice, model size, response latency, and invocation cost often trade off against one another. Larger models usually bring stronger spatial understanding and code generation, but each call is slower and more expensive per unit; as call volume rises, the cumulative overhead grows quickly and can become an unsustainable cost pressure.

    A rational engineering strategy, therefore, is not “use the strongest model for every task,” but to introduce tiering and optimization:

    • Tiered routing: Send simple images to lightweight models and reserve high-capability models for complex scenes, balancing cost and quality through a routing strategy.
    • Cache reuse: Cache results for identical or highly similar inputs to avoid redundant computation.
    • Batching and asynchrony: Merge non-real-time tasks to amortize per-unit cost.
    • Feedback-driven convergence: Through the “render–compare” closed loop, replace blind large-scale repeated generation with a smaller number of high-quality iterations.

    This way, you can achieve results close to those of a top-tier model while keeping cost within an acceptable range.

    7. Best Practices for Production

    Overall, deploying this kind of capability can follow these principles:

    1. Constrain the output format first: Use strict schema validation to guarantee the generated code is parseable, treating “can it compile” as a hard constraint imposed up front.
    2. Break down complex images: For structurally complex images, reconstruct them step by step by layer or semantic component, reducing the cognitive load of any single task.
    3. Build a render feedback loop: Explicitly feed visual comparison results back to the model to drive iteration, rather than relying on a single generation.
    4. Structured prompting: Clearly state the coordinate system, units, and drawing-order conventions in the prompt to reduce ambiguity.
    5. Keep a human-in-the-loop fallback: For high-precision scenarios, retain a manual fine-tuning step so that the model’s output serves as a high-quality draft rather than the final product.

    8. Summary

    Reconstructing a 2D image into editable vector drawing code is a concentrated test of a multimodal LLM’s spatial understanding, occlusion and layer inference, geometric vectorization, and declarative code generation. It is not a display of a single capability but the result of many coordinated stages: seeing, reasoning, drawing, and verifying. Understanding this technical chain helps developers set realistic expectations in actual projects, design proper evaluation systems, and make sound trade-offs between quality and cost—so that this capability can truly land in usable engineering scenarios.