Evaluating New LLMs: Using SVG Rendering as a Reproducible Yardstick

Recently, a brief appearance of a suspected new-version entry in a cloud vendor’s model catalog reignited discussion in the tech community about when the next-generation model will ship and how much it improves. For casual readers, this kind of leak is just water-cooler talk. For developers and architects, the more important question is: when a new version actually goes live, what lets you decide it’s better than the last one?

The benchmark scores released by model vendors are one reference, but there’s often a clear gap between those scores and real business performance. A more reliable judgment comes from a self-built, reproducible evaluation system aligned with your business goals. For that system to actually work, the first prerequisite is finding the right thing to observe.

1. Why structured generation tasks make a good observation window

The SVG drawing tasks that have become popular in the community recently — “draw a person riding a bicycle with an animal in SVG,” or “draw a game controller with code” — happen to offer an almost microscope-like way to observe.

1. Output is machine-parsable. SVG is an XML-based vector graphics markup format, and every graphic element is explicit structured text. A gradient definition, a set of coordinate paths, a layer of filter effects — all of these can be read out and counted item by item by a parser, without relying on the human eye.
2. Multiple capabilities are compressed at once. Within a few KB of markup, the model has to do three kinds of work at the same time: layout and proportion control, geometric path construction, and expressing light and shadow layers. The task density is far higher than generating a normal piece of text.
3. Details are easy to expose. Just adding one element or adjusting one path layer leaves a trace in the file structure. These traces make subtle differences between adjacent versions trackable.

However, SVG isn’t a cure-all. It’s better suited to observing a model’s “structured output and spatial organization,” which is not the same as other dimensions such as logical reasoning, long-document understanding, or tool calling. Positioning it as one of many observation windows — rather than the only yardstick — is the first step in building an evaluation system.

2. Turning “does it look good” into countable metrics

Comparing two images purely on visual impression makes conclusions easily swayed by subjective preference. A more robust approach is to first extract objective metrics from the output, then build a scorecard on top of them.

1. Basic parsing and element statistics

The following example script shows how to parse SVG output with the Python standard library and count the key elements in it:

import re
import xml.etree.ElementTree as ET
from collections import Counter
SVG_NS = "{http://www.w3.org/2000/svg}"
def parse_svg(text: str) -> dict:
    """Parse SVG text and return basic structure statistics; raise on parse failure."""
    root = ET.fromstring(text)
    tags = Counter(el.tag.replace(SVG_NS, "") for el in root.iter())
    raw_defs = re.search(r"<defs\b[^>]*>(.*?)</defs>", text, re.S)
    defs = raw_defs.group(1) if raw_defs else ""
    ids = set(re.findall(r'\bid="([^"]+)"', defs))
    return {
        "viewBox": root.get("viewBox"),
        "groups": tags["g"],
        "paths": tags["path"],
        # Referencable resources: gradients, filters, reusable shapes, clip paths
        "defs": sum(tags[k] for k in ("linearGradient", "radialGradient",
                                      "filter", "symbol", "clipPath")),
        # Resources not referenced by any element, which usually means redundancy
        "orphan_defs": [i for i in ids
                        if f'url(#{i})' not in text and f'href="#{i}"' not in text],
        "bytes": len(text.encode("utf-8")),
        "elements_per_kb": round(sum(tags.values()) / max(len(text) / 1024, 1e-6), 2),
    }
# Known legal graphics elements and attributes: tags or attributes not on the whitelist can be treated as hallucinated elements
KNOWN_TAGS = {"svg", "g", "path", "rect", "circle", "ellipse", "line",
              "polyline", "polygon", "defs", "linearGradient", "radialGradient",
              "stop", "filter", "feGaussianBlur", "feOffset", "feBlend",
              "feDropShadow", "clipPath", "symbol", "use", "title", "desc"}
def find_unknown(text: str) -> list:
    """List tags and attributes outside the whitelist, to detect hallucinated elements."""
    bad = []
    for m in re.finditer(r"<\s*([A-Za-z][\w:.-]*)((?:\s+[\w:.-]+\s*=\s*\"[^\"]*\")*)\s*/?>",
                         text):
        tag, attrs = m.group(1).lower(), m.group(2)
        if tag not in KNOWN_TAGS:
            bad.append(("tag", tag))
        for name in re.findall(r'([\w:.-]+)\s*=\s*"', attrs):
            if name.lower() not in KNOWN_TAGS:
                bad.append(("attr", f"{tag}@{name}"))
    return sorted(set(bad))

A parse failure is itself worth noting. Unclosed tags, missing quotes around attribute values, duplicate IDs — all of these make downstream rendering or re-editing pay a cost at the integration stage.

2. From statistics to a scorecard

On top of the statistics above, you can further define a set of metrics:

Metric	How it’s observed	Capability it reflects
Structural compliance	Parse success rate, and whether the viewBox and root element are complete	Format following and basic rigor
Detail density	Elements per KB, and the count of gradients, filters, and groups	Richness of the visual hierarchy
Occlusion handling	Whether elements with front/back relationships are grouped separately, and whether layer order is reasonable	Spatial relationships and object management
Semantic grouping	Whether wheels, frame, creature body, etc. are grouped by part	Code maintainability and intent expression
Hallucinated elements	Tags and attributes outside the whitelist	Whether the model is “making up syntax”

Take the classic drawing task as an example: in the new model’s output, the pelican has an extra fish in its beak, and the tire tracks and wheels come with light and shadow changes. Mapped to the metrics, this shows up as higher detail density, occlusion relationships being expressed explicitly, and more use of gradients and filters.

The game controller task shows another typical phenomenon. The new output is more detailed in surface reflections and connector rendering, but its overall completion is about the same as the previous generation. The reason is that this task’s core examination points — symmetric structure, button layout, proportion coordination — are already close to saturated, and the room for incremental improvement falls mainly on decorative details. This is a reminder to evaluation designers: when a task’s discrimination power drops, piling on more only wastes budget.

3. Putting a “quality gate” on detail density

More details doesn’t mean better. If you put no constraints on the model’s SVG output, a response can “boost” its statistics purely by piling up large numbers of decorative paths and unconditionally stacking filters. The result is bloat, slower rendering, and harder maintenance. When scoring, it’s a good idea to examine three dimensions at the same time:

• Necessity: whether the decoration serves the main expression, rather than being simple copy-paste.
• Resource usage: whether the final size and rendering time are within an acceptable range.
• Editability: whether element grouping matches human editing habits and can be picked up directly by design tools.

3. Test-set contamination: a bigger worry than model capability

Once a drawing task circulates in public communities for more than a year, it turns from an “exam question” into a “practice book.” Teams repeatedly use it as a sample demo, and the model repeatedly encounters similar inputs during training and inference optimization, so the score naturally climbs. But it’s hard to tell whether that gain comes from improved capability or from “familiarity” with a specific task.

An anti-contamination test set can be approached from three levels:

1. Separate public and private questions. Public questions are for comparability of results and external exchange; private questions are for internal version gating, with neither questions nor answers shown externally.
2. Rotate regularly and archive versions. Add a certain proportion of new questions each quarter, while keeping the historical results of old ones, to make long-term longitudinal comparison easy.
3. Focus on discrimination, not absolute scores. If the same question is near full marks across all candidate models, its information value is already very low, and it should be retired promptly.

When selecting questions, you can also actively spread out the difficulty gradient:

• Basic questions: verify format compliance and basic structure restoration.
• Advanced questions: add constraints such as multi-object occlusion and gradient stacking.
• Open questions: no preset “correct answer,” scored by reviewers against a unified standard.

4. Excluding interference: making version comparison reproducible

Even with the same question and the same version, results can differ noticeably under different invocation methods. To do a trustworthy version comparison, you should fix at least the following variables:

• Sampling parameters: unify the random seed (if the interface supports it) or the temperature parameter, and fix the maximum output length.
• Prompts and system instructions: keep them word-for-word identical, and record the version number.
• Whether extended reasoning is enabled: for models that support extended reasoning, record the results with it on and off separately, rather than mixing the two together.
• Sampling count: sample multiple times per question, and report both the mean and the variance, to avoid a “single lucky output” creating a false improvement.

Reproducibility also depends on a complete engineering chain:

Test-set management → batch inference → automated validation and metric statistics → manual spot checks → version baseline comparison → decision

• Test-set management: centralize task definitions, metadata, and historical versions, so questions can be composed by tag combinations.
• Batch inference: use a message queue to smooth out peaks and troughs, scale up and down on demand with a container platform, and store raw outputs in object storage.
• Automated validation: plug into the pipeline, and record failure reasons separately at the parsing, rendering, and metric-statistics stages.
• Manual spot checks: automated metrics can’t cover aesthetic judgment, so sample a certain proportion and have multiple people score independently.
• Monitoring and alerts: consolidate key metric changes into the monitoring and logging system, and automatically compare before and after a version switch.

5. From “watching models” back to “watching scenarios”

The biggest value of drawing tasks is that they force a team to think through its evaluation method clearly — and that same method applies to business scenarios. Concretely: pull 50–200 typical tasks from real business, layer them following the “basic–advanced–open” idea above, and then define an automatically judgeable pass criterion for each task. For example:

• Structured code generation: whether it passes syntax checks and unit tests.
• Document structuring: whether field-extraction accuracy meets the threshold.
• Interface call orchestration: whether parameter selection is correct given a fixed tool set.

Scoring can also bring in a cost dimension. Unit price, average output length, and per-task time together determine whether an “effect improvement is worth it.” A capability improvement only has engineering value for real deployment when the per-unit cost is affordable.

6. Conclusion

The discussion around when a new model will launch and which capability improved by how much won’t stop anytime soon. For developers, rather than chasing every unverified rumor, the better move is to put energy into building a transparent, reproducible evaluation system: use structured observation windows in place of subjective impressions, countable metrics in place of scattered screenshots, a private and dynamically updated test set in place of public fixed exam questions, and business-scenario pass rates in place of unrealistic leaderboard rankings.

When the version switch actually arrives, a team with an evaluation system only needs to run the pipeline once to produce a data-backed conclusion. That’s far more reliable than waiting for leaks.