Converting Diagrams to Vector Drawing Code with Multimodal Models: Beta Testing Notes

    Some multimodal models in internal beta testing can convert complex diagrams into vector drawing code. Unlike image captioning, this task requires a model to infer 3D structure, occlusion, and layer order from a 2D image, then express that understanding as coordinates, paths, line styles, fills, and drawing order. This article takes a general technical view rather than focusing on one product. It covers how to evaluate this capability, design test sets and output protocols, build a render-and-verify loop, and apply these methods in engineering practice.

    1. The Core Problem: Scene Reconstruction and Code Generation, Not Image Recognition

    Diagrams usually contain geometric shapes, connecting lines, labels, arrows, text, and style information. The model must complete several conversion steps:

    1. Visual parsing: Identify element types, boundaries, connections, and text content.
    2. Spatial reasoning: Infer front-back occlusion, perspective direction, stacking order, and depth cues from a 2D projection.
    3. Structural abstraction: Organize pixel-level information into a scene graph, including objects, groups, parent-child relationships, and layer order.
    4. Code generation: Map the scene graph to a vector drawing script that outputs coordinates, paths, line styles, fills, and drawing order.
    5. Render verification: Compare the rendered result with the reference image to find geometric offsets, occlusion errors, and missing style details.

    Occlusion and layer order are the most error-prone parts. When one rectangle partly covers another, the model must draw both shapes, decide which one is in front, and decide whether the hidden part should stay as an editable path. If the output only looks correct, the code may be hard to edit. If it only has correct structure, it may lose style details. Evaluation must therefore check both visual fidelity and structural editability.

    2. Evaluation Dimensions: From “Looks Right” to “Verifiable”

    Define the following dimensions and set a checkable metric for each:

    • Semantic correctness: Element types, counts, and connections match the reference.
    • Geometric consistency: Key point coordinates, path closure, and proportions are reasonable.
    • Occlusion and layer reasoning: Front-back relationships, drawing order, and group hierarchy are correct.
    • Style fidelity: Solid and dashed lines, fill colors, line widths, and arrow styles match the reference.
    • Editability: Objects have semantic names, are grouped, use parameterized paths, and keep text separate.
    • Robustness: Output stays stable under scaling, rotation, low resolution, and slight noise.
    • Reproducibility: Repeated runs on the same input produce consistent results, which makes regression testing practical.

    Human visual judgment alone is not enough. In engineering, combine render differences with structural rule checks. Render differences measure visual deviation, and rule checks measure code quality. For occlusion, use metrics such as a layer-order score and boundary continuity in overlapping regions. For editability, track the share of ungrouped objects, unnamed objects, and abnormal path point counts.

    3. Test Set Design: Cover Real Difficulties from Simple to Complex

    Build the test set in layers:

    1. Basic geometry: Rectangles, circles, polygons, arrows, and dashed connectors.
    2. Occlusion: Front-back overlap, partial occlusion, transparent fills, and nested containers.
    3. 3D diagrams: Cubes, pipes, cross-sections, and perspective projections.
    4. Dense annotation: Text, leader lines, numbering, legends, and zoomed-in callouts.
    5. Edge cases: Low resolution, hand-drawn styles, scanning noise, and partially missing content.

    Each sample should keep a reference vector script or a manually annotated scene graph so results can be compared automatically. For enterprise use, add business diagrams as well, but anonymize them so sensitive information stays out of test assets.

    4. Output Protocol: Use an Intermediate Scene Graph to Constrain Generation

    Asking the model to output drawing code directly often causes coordinate drift, layer confusion, and uncontrolled styling. A safer approach is to output an intermediate scene graph first, then convert it into a drawing script. Here is an example protocol:

    1. {
    2. "canvas": {"width": 1000, "height": 700, "unit": "normalized"},
    3. "layers": [
    4. {"id": "bg", "z": 0, "shapes": [
    5. {"type": "rect", "x": 0, "y": 0, "w": 1, "h": 1, "fill": "#f5f5f5"}
    6. ]},
    7. {"id": "body", "z": 10, "shapes": [
    8. {"type": "path", "points": [[0.2,0.3],[0.5,0.2],[0.6,0.6],[0.3,0.7]],
    9. "closed": true, "fill": "#d9e6f2", "stroke": "solid"}
    10. ]},
    11. {"id": "front_panel", "z": 30, "shapes": [
    12. {"type": "path", "points": [[0.35,0.4],[0.55,0.35],[0.58,0.55],[0.38,0.6]],
    13. "closed": true, "fill": "#ffffff", "stroke": "dashed"}
    14. ]}
    15. ],
    16. "draw_order": ["bg", "body", "front_panel"]
    17. }

    This protocol uses normalized coordinates so the output does not depend on resolution. Layers use explicit z values and a defined draw order. Line styles use enumerated values, and filled regions must be closed paths. Objects need semantic names for later editing. The prompt should state that the model must analyze occlusion first, then output the scene graph, and then generate the drawing script. It should also ask the model to attach confidence levels to uncertain regions.

    5. Render Verification Loop: Validate After Generation

    Generated code is only an intermediate artifact, so it must be rendered and validated. The following pseudocode shows one possible flow:

    1. scene = model.generate(image, protocol)
    2. script = compile(scene)
    3. rendered = render(script)
    4. report = compare(rendered, reference)
    5. rules = validate(scene)
    6. if report.pixel_diff > threshold or rules.has_error:
    7. scene = repair(scene, report, rules)
    8. script = compile(scene)

    Validation rules include checking whether paths are closed, whether coordinates fall outside the canvas, whether z values are duplicated or missing, whether dash styles belong to the enumerated set, whether text overlaps severely, and whether groups are empty. Visual comparison can combine pixel difference, edge difference, and key-region difference. Send low-confidence samples to manual spot checks. Feed frequent errors back into prompt templates and post-processing rules.

    6. Engineering Recommendations

    On general-purpose cloud infrastructure, such as Baidu AI Cloud’s object storage, log service, and monitoring and alerting services, teams can centrally manage image assets, task logs, and quality metrics. We recommend the following practices:

    • Task-based batch processing: Split image parsing, scene graph generation, script compilation, and render verification into retryable tasks.
    • Versioning: Store prompt versions, scene graph versions, and render results for regression comparison.
    • Caching and incremental repair: Reuse results for unchanged regions and recompute only the regions that differ.
    • Human review: Route low-confidence or rule-failing samples to a review queue so errors do not reach production directly.
    • Security and compliance: Anonymize test and business images, control access permissions, and keep audit logs.

    Multimodal models already show practical potential for converting diagrams into vector drawing code. Output quality, however, depends on the output protocol, the validation loop, and engineering constraints. Evaluating both visual fidelity and structural editability, generating the scene graph before the code, and repairing output after rendering are the steps that move this capability from demos to stable production use.