SVG as a Probe: Evaluating New Models' Visual Construction Capabilities

1. Introduction: After faster release cycles, evaluation is the real dividing line

Model vendors have clearly accelerated their release cadence. New checkpoints appear briefly in test environments, canary entry points surface early, and benchmark platforms post higher scores ahead of time. This kind of information is common in developer communities. What actually affects engineering decisions is whether a new version can reliably and reproducibly complete tasks within your own business scenarios. Information coming first and versions following is the norm, not the exception.

Judging capability from official release notes and a single screenshot alone carries at least three risks:

• Sampling randomness: For generative models, running the same prompt multiple times can yield very different results. A single success may indicate insufficient capability, or it may just be luck.
• Prompt contamination: Problems that have circulated in public communities for a long time may already be in the training data of later models. A high score on such problems does not necessarily mean stronger generalization.
• Presentation differences: The same vector output can look different across renderers and resolutions. An “obvious improvement” in a screenshot sometimes comes from the display environment rather than the model itself.

A more reliable approach is to build an evaluation method that is reproducible, quantifiable, and sustainable for regression. In this kind of evaluation, vector graphics generation is a high-value probe, and this article uses it as the main thread.

2. Why SVG is an excellent probe for testing visual construction capability

Here, “visual construction capability” means the model’s ability to construct an image using structured code without pixel-level visual feedback. Bitmap generation relies on pixel-level modeling methods such as diffusion, which generally cannot directly output precise vector structures. So using vector graphics generation as a probe tests the language model’s level of structured reasoning.

By comparison, SVG (Scalable Vector Graphics, an XML-based image format) has three characteristics:

1. Pure text, machine-parsable: The graphic is described by geometric elements, style definitions, and transformation relationships. Colors, opacity, and transformation matrices are all ultimately assembled into attribute strings and handed to the rendering engine to composite during rasterization.
2. Many capability dimensions: A single graphic must simultaneously handle layout (coordinate system and canvas proportions), proportion (path geometry and transforms), and lighting (gradients, blur, and opacity layering).
3. Limited length budget: In real business scenarios, icons and illustrations need to hold complete semantics within a few KB, which requires the model to have an awareness of information density.

Now look at the type of problem commonly used in the community: drawing a scene of some animal riding a bike with SVG. The differences between the two model versions show up mainly in the details:

• The older version achieves a “likeness”: complete outlines, reasonable limb posture, and correct positional relationships among wheels, frame, and figure.
• The new version goes further in detail density: the scene gains an extra prop, the ground shows a trail extending with the wheels, and the wheels carry gradients consistent with the light-source direction and a blurred cast shadow.

These increments may look small, but technically they cross a threshold: the new elements must be embedded into existing geometry, and their occlusion relationship with the subject must be handled. More importantly, gradients and shadows can be entirely “drawn” by assembling attribute strings, without any real rendering composition. So the conclusion is that it “looks more like bitmap rendering,” not that “what was generated is a bitmap.” To judge whether this kind of performance can be reproduced, you cannot rely on the eyes alone; you must break the output apart and check it item by item.

3. Breaking SVG output into checkable dimensions

3.1 Structural validity: a prerequisite for all analysis

First, confirm that the output can be parsed by a standard XML parser. Common problems include unclosed tags, missing quotes on attribute values, and missing namespaces. There is another kind of subtle problem: coordinates or radii written as illegal values such as NaN or Infinity. These usually do not cause a parse failure, yet they make an element disappear outright during rendering. This kind of problem is only exposed after actual rasterization, so the structural validity check should be the first gate.

3.2 Semantic completeness: are all elements required by the prompt present?

Break the prompt into an element checklist (subject, accessories, environment, light source, ground-contact shadow), then verify item by item whether each exists, whether its size is reasonable, and whether it falls within the canvas. The example SVG below already covers the light source, sky, ground, subject, and shadow:

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 320 240" role="img"
     aria-labelledby="ttl dsc">
  <title id="ttl">Cyclist on the beach</title>
  <desc id="dsc">A cyclist and bicycle rest on the beach, with soft shadows cast by the wheels.</desc>
  <defs>
    <linearGradient id="sky" x1="0" y1="0" x2="0" y2="1">
      <stop offset="0" stop-color="#bfe3ff"/>
      <stop offset="1" stop-color="#eaf6ff"/>
    </linearGradient>
    <radialGradient id="sunGlow">
      <stop offset="0" stop-color="#fff3c4" stop-opacity="0.9"/>
      <stop offset="1" stop-color="#fff3c4" stop-opacity="0"/>
    </radialGradient>
    <filter id="blur6" x="-30%" y="-30%" width="160%" height="160%">
      <feGaussianBlur stdDeviation="6"/>
    </filter>
    <g id="wheel">
      <circle r="26" fill="none" stroke="#2b2b2b" stroke-width="3"/>
      <circle r="4" fill="#2b2b2b"/>
      <path d="M0-24V24M-24 0H24M-17-17 17 17M-17 17 17-17"
            stroke="#9a9a9a" stroke-width="1"/>
    </g>
  </defs>
  <!-- Background layer: drawn first in document order -->
  <rect width="320" height="240" fill="url(#sky)"/>
  <circle cx="256" cy="48" r="40" fill="url(#sunGlow)"/>
  <path d="M0 186 Q160 168 320 186 V240 H0Z" fill="#e7cf9f"/>
  <!-- Subject layer: shadow first, then body, head drawn last for occlusion -->
  <ellipse cx="160" cy="193" rx="92" ry="8" fill="#000" opacity="0.15" filter="url(#blur6)"/>
  <use href="#wheel" x="112" y="166"/>
  <use href="#wheel" x="208" y="166"/>
  <path d="M112 166 148 126 194 130 208 166" fill="none" stroke="#c0392b" stroke-width="4"/>
  <g id="rider">
    <path d="M150 128 146 96" stroke="#34495e" stroke-width="6" stroke-linecap="round"/>
    <circle cx="170" cy="90" r="14" fill="#f0c090"/>
  </g>
</svg>

3.3 Grouping and reuse: direct evidence of structure

SVG uses <g> to organize elements and <use> to reference reusable units defined in <defs>. Writing one wheel twice means two independent elements; defining it once and referencing it twice reflects an awareness of abstraction and reuse. The latter has clear advantages when bulk edits are needed, and it also makes it easier to see whether the model truly understands the graphic’s compositional structure rather than piecing elements together one by one.

3.4 Lighting layers: gradients, filters, and layering order

Lighting consists of three parts:

• Gradient definitions (linear or radial) and the number of their color stops;
• Filters (such as Gaussian blur) and their parameters;
• Light-and-dark relationships produced by opacity layering.

Here you must distinguish “written” from “used”: defining many gradients that no element references is a common case of “surface-level abundance.” You also need to check whether the light-source direction is self-consistent. The shadow direction should be opposite the light-source position and aligned with where the wheels touch the ground. This is the key to distinguishing “stacked effects” from “understanding of lighting.”

3.5 Occlusion and z-order: SVG has no z-index

SVG has no CSS-like z-index property; the draw order is determined by the order in which elements appear in the document, with later elements covering earlier ones. So the foreground must be drawn first, and occluded parts must be drawn later. In the example, the rider’s head is written after the frame and wheels, so it naturally covers the frame. Once the order is reversed, even correct modeling cannot escape a visual error. An evaluation that only checks whether elements exist cannot catch this kind of problem; it must fall through to the rasterization stage.

3.6 Detail density and ineffective elements

More detail does not equal higher quality. In evaluation, elements should be divided into effective and ineffective:

• Effective elements: those rendered in the final output and making a visible contribution to the picture;
• Ineffective elements: repeatedly stacked shapes of the same color, rectangles with zero width or height, objects with 0 opacity, and resources that are defined but never referenced.

Defining an “effectiveness ratio = effective elements / total output elements” prevents a model from gaming “detail points” by piling up elements.

3.7 Maintainability and accessibility

The generated code may ultimately enter a code repository, so readability matters just as much. Whether coordinates are grouped, whether there are comments, and whether ID naming is semantic all affect subsequent maintenance cost. For accessibility, check whether <title> and <desc> are provided and whether they are associated with the root element through aria-labelledby. For product teams whose main deliverable is icons and illustrations, this item should be part of acceptance.

4. How to build a reproducible evaluation set

4.1 Specifying problems: prompts must be judgeable as right or wrong

A prompt should explicitly state the canvas proportions, the required element list, prohibitions (such as no external resource references), and a judging checklist. An illustration:

Draw a "cyclist on the beach" on a canvas with viewBox="0 0 400 300".
Requirements:
1) Must include the cyclist, a bicycle (two wheels, a frame), the beach ground, and the sky;
2) Use at least 1 linear gradient and 1 Gaussian blur filter;
3) Wheel shadows must align with where the wheels touch the ground;
4) Do not reference any external resources; do not include scripts;
5) Output a single valid SVG with no more than 150 elements total.
Judging: each item above is a binary decision (pass / fail).

4.2 Contamination-resistant design: variants, private problems, and perturbations

After a few rounds of model iteration, scores on fixed problems tend to saturate noticeably. Common ways to maintain discriminative power:

• Variant generation: Replace the subject, light-source direction, and canvas proportions within the same scene template, keeping the difficulty unchanged while making the surface different;
• Private problem pool: Keep part of the problems internal-only, not participating in any public discussion;
• Parameter perturbation: Change the canvas aspect ratio, require a fixed palette, or cap the element count to test whether the model truly understands the structure rather than having memorized a “standard answer”;
• Hidden constraints: When a problem states that an object is “side-on to the light source,” the shadow direction should automatically be consistent. Whether such implicit constraints can be satisfied reflects depth of understanding better than element count does.

4.3 Sampling and stability

Sample the same problem multiple times, and record the pass rate, result variance, and best result. Evaluation should distinguish two abilities: the “average capability” of completing the task consistently, and the “upper-bound capability” of the best single sample among multiple runs. Production environments care more about the former. For generative tasks, you can borrow the idea of taking multiple independent samples, selecting the best result, and then judging, but you must report the number of samples as well; otherwise fair comparison across models is impossible.

5. Automated judging: from static checks to rasterization regression

5.1 Static check script

The following script is based on the standard library and performs quick checks on structural validity, definition references, effectiveness, and accessibility. It relies on standard XML parsing, so it is not affected by browser parsing tolerance policies:

import re
import xml.etree.ElementTree as ET
from collections import Counter
PAINT_TAGS = {"linearGradient", "radialGradient", "pattern", "filter"}
REF_ATTRS = ("fill", "stroke", "filter", "clip-path", "mask")
def local(tag: str) -> str:
    return tag.rsplit("}", 1)[-1]
def analyze(svg_text: str) -> dict:
    root = ET.fromstring(svg_text)  # Invalid XML raises an exception here
    elements = list(root.iter())
    counts = Counter(local(el.tag) for el in elements)
    ids = {el.get("id") for el in elements if el.get("id")}
    # Collect all references: url(#id), <use href="#id">, and ARIA associations
    referenced = set(re.findall(r"url\(#([\w-]+)\)", svg_text))
    referenced |= set(re.findall(r"href=\"#([\w-]+)\"", svg_text))
    for attr in ("aria-labelledby", "aria-describedby"):
        referenced |= set(re.findall(rf"{attr}=\"([^\"]+)\"", svg_text))
    dangling = sorted(ref for ref in referenced if ref not in ids)
    unused_defs = sorted(i for i in ids if i not in referenced)
    defs = {el.get("id"): local(el.tag) for el in elements
            if el.get("id") and local(el.tag) in PAINT_TAGS}
    effective_defs = sum(1 for i in defs if i in referenced)
    uses = counts["use"]
    return {
        "viewBox": root.get("viewBox"),
        "bytes": len(svg_text.encode("utf-8")),
        "elements": sum(counts.values()),
        "groups": counts["g"],
        "reused_via_use": uses >= 2,               # Whether real reuse exists
        "gradients": counts["linearGradient"] + counts["radialGradient"],
        "filters": counts["filter"],
        "effective_defs_ratio": round(effective_defs / max(len(defs), 1), 2),
        "dangling_refs": dangling,                  # References to non-existent definitions
        "unused_defs": unused_defs,                 # Defined but never used
        "a11y_ok": counts["title"] >= 1 and counts["desc"] >= 1,
        "has_script": counts["script"] > 0,
    }

After the static check passes, do a rasterization check:

1. Use an independent rendering engine to convert the SVG into a bitmap;
2. Compare the bitmap against the reference image at the pixel level or the perceptual level to obtain a similarity score;
3. Use an image classification or object detection model to identify whether the subject, props, and shadow exist, replacing manual verification of the element checklist;
4. Render at multiple resolutions to check whether elements go out of bounds or are lost due to illegal values.

Step 4 can directly expose the hidden defect of “statically valid, disappears on rendering.”

6. Scoring model: from a single score to a multi-dimensional report

A weighted score is recommended, with each dimension scored from 0 to 1, and the weights adjusted according to business focus:

Dimension	How it’s judged	Example weight
Structural validity	XML parses, no illegal values, no elements lost on rasterization	0.15
Semantic completeness	Required element checklist matched item by item	0.20
Proportion and pose	Whether the position and angle of key parts deviate from the reference	0.20
Occlusion and z-order	Whether front/back relationships are correct after rasterization	0.15
Lighting layers	Whether gradients/filters are referenced and light-source direction is self-consistent	0.15
Detail effectiveness ratio	Effective elements / total output elements	0.10
Maintainability	Whether grouping/reuse exists, coordinates are grouped, IDs are named consistently	0.05

The report should not contain only a single total score. Dimensions below the threshold should be listed separately, making it easy to locate whether the issue is layout, lighting, or render order. The several dimensions prioritized in the example (structure, semantics, proportion, occlusion) belong to “pass-line” capabilities and are best set as hard gates; lighting and maintainability are more suited to spreading the score differences.

7. Integrating into continuous integration to form a regression baseline

Evaluation only has long-term value once it enters the daily workflow. The recommended process is as follows:

1. Fix the environment and seed: Fix the sampling parameters and random seed, and record the model version identifier, prompt version, and renderer version;
2. Baseline comparison: Run the same set of problems against the current online version for each candidate version, producing per-dimension differences;
3. Set regression gates: If any hard dimension drops by more than a set threshold, block the upgrade;
4. Canary release: A version that passes the gates first runs under a small traffic share, with output quality, latency, and error rate continuously tracked using online task logs and monitoring alerts;
5. Artifact retention: Store the SVG source code and rasterization results in object storage and archive them by version for easy backtracking and comparison.

The evaluation task itself can be containerized and run on a schedule. An example configuration:

# ci/model-probe.yaml (illustrative)
probe:
  cases: cases/*.spec.json      # Each case includes prompt, judging checklist, and reference image
  samples_per_case: 5           # Samples per case
  params: { temperature: 0.7, seed: 42 }
gates:                          # Hard gates: maximum allowed drop relative to baseline
  structure_valid: 0.00
  element_coverage: 0.03
  occlusion_correct: 0.03
report:
  dimensions: [structure, coverage, proportion, occlusion, lighting, density, maintainability]
  artifacts: object-storage://eval-reports/{run_id}/   # Archive source code and raster images

8. Common pitfalls and how to avoid them

• Leaderboard-only thinking: After public problems saturate, scores reflect memorization rather than capability, so a private problem pool should be the source of truth.
• Screenshot-only thinking: Screenshots are affected by the renderer and display size, so rasterization metrics must be the source of truth.
• Ignoring maintainability: Output that “draws correctly” but has messy code carries high maintenance costs once it enters a code repository.
• Security boundaries: SVG can embed scripts and may also initiate resource requests through external references. Model-generated graphics should be rendered in an isolated environment with scripts and external resource loading disabled. If a team builds its evaluation and inference pipeline on Baidu AI Cloud, this kind of isolation can be achieved with general capabilities such as container platform network policies and security groups.
• Accessibility omissions: Graphics missing <title>, <desc>, or ARIA associations are unusable for screen-reader users and should be a fixed check item before delivery.

9. Conclusion: turning “looks stronger” into “provably stronger”

Vector graphics generation is a small entry point, yet it can simultaneously test a model’s overall level in structured output, spatial reasoning, detail control, and code readability. Turning it into a versioned evaluation set, combined with static checks, rasterization regression, and continuous integration, lets a team quickly answer three questions whenever a new version appears: which capabilities have truly improved, which dimensions have regressed, and whether an upgrade is worthwhile.

Nor is it necessary to bet on a single candidate during selection. Applying the same evaluation set to multiple candidate models or vendors, scoring by weight according to business scenarios, and then deciding the primary-and-backup combination is a sustainable engineering decision. What is truly reliable is not leaks and screenshots, but a reproducible, regression-testable evaluation system in your own hands.