Testing LLM Generation Capabilities with Structured Probes

Structured output is becoming a hard metric for evaluating the comprehensive capabilities of Large Language Models (LLMs). Text-based Q&A can mask defects through phrasing, while bitmaps hide errors in pixels, making stable visual comparisons difficult. Vector graphics are different: every shape, gradient, group, and filter exists as explicit markup. The file can be parsed, counted, and diffed. When the same composition prompt is given to two model checkpoints, the comparison is not about which looks “cooler,” but whether the newly added constraints in the markup hold simultaneously.

Why Fixed Test Questions Fail

In the community, there are often long-standing public drawing challenges, such as requiring models to draw specific subjects and simple mechanical structures using vector graphics. The prompt is stable and easy to spread, which quickly turns it into a hidden sample for training and alignment. As models gradually align on postures and component counts, rising scores do not necessarily indicate better generalization; they only show that the model is more familiar with this “syllabus.”

Evaluation must distinguish three things. First is prompt memorization: whether the subject’s outline and common color schemes have been memorized. Second is structural constraints: whether the wheels are attached to the frame, and whether the grip matches the subject’s scale. Third is the geometric validity of new elements: when drawing an additional object, whether it is embedded in the correct parent outline and handles occlusion, rather than floating outside the canvas. Only the third category of differences approaches the ability to “manage multiple layers of constraints simultaneously.”

Highly structured objects like handles and icons are often flattened too early across model generations. They test part topology, and once the gap is filled, subsequent versions show little visible difference. Open scenes, soft contours, ground tracks, and local lighting are harder to exhaust, making them suitable probes that still offer discriminative power.

Vector Markup as a Microscope

The texture of bitmap generation relies on perception, and the randomness of two sampling iterations can interfere with comparison. Vector markup lays out capabilities as countable objects:

• Layout layer: Canvas coordinates, grouping hierarchy, and whether the subject crosses boundaries.
• Proportion layer: Relative sizes of components, and whether joint connections are continuous.
• Lighting layer: The number of linear or radial gradients, transparency, and whether shadows align with the light source direction.
• Occlusion layer: Whether sub-paths are clipped by parent outlines, and whether the overlapping order aligns with depth.

In practice, you can use standard text tools to count gradient definitions, filters, path groups, and nesting depth. An extra gradient or group layer leaves explicit traces in the file, eliminating the need for subjective scoring. When comparing two checkpoints, first normalize whitespace and attribute order, then perform a structured diff to prevent format noise from drowning out real differences.

Detail density must be evaluated alongside correctness. Stuffing in more paths while breaking proportions only indicates the model is more “verbose.” A valid improvement is when new elements obey existing geometry—for example, nesting a small object inside a mouth outline and correctly handling the front-back relationship, or ensuring ground tracks align with the wheel’s trajectory rather than being random textures.

How to Implement the Evaluation Process

It is recommended to build small, repeatable probe suites rather than relying on a single viral image.

1. Fix prompts and output constraints: Specify canvas size, allow only vector markup, prohibit external resource links, and limit the maximum markup length. The length limit forces the model to make trade-offs within a finite markup space, avoiding meaningless stacking.
2. Prepare reference dimensions: Manually list required parts, allowed bonus items, and clear failure cases (clipping, disconnection, missing subject).
3. Machine pre-screening: Parse whether the markup is valid; count elements, nesting depth, gradients, and filters; check if coordinates fall within the canvas.
4. Rule-based scoring: Write deterministic checks for occlusion, connections, and directional consistency. For instance, wheel tracks should generally extend along the direction of motion, and the light source should not contradict the highlight position.
5. Manual review: Only look at samples flagged by the machine as different or failed, reducing fatigue-induced bias.
6. Record checkpoints: Store the model ID, temperature, random seed, system prompt, and post-processing in a database. Without this metadata, screenshots cannot be reproduced.

Temperature and sampling counts significantly alter local decorations. When comparing versions, fix the sampling strategy at a minimum, and use the median of multiple generations rather than a single best result. Publicly shared “highlight images” are often the upper bound of multiple draws and should not be treated as stable capability.

Illustrative statistics can be lightweight:

parse(markup) ->
  count(gradients), count(groups), max(depth),
  bbox(subject), overlap(child, parent),
  direction(track) vs direction(motion)

If a check fails, record the reason code instead of just providing a total score. Reason codes facilitate regression testing: did the new checkpoint fix occlusion, or did it just add decorative gradients?

Don’t Tie Capability Dimensions to Release Schedules

Fleeting entries in model directories or undisclosed internal codenames only indicate the existence of early checkpoints; they do not define capability boundaries, pricing, or launch windows. The more robust engineering approach is to treat external rumors as a signal that a new checkpoint might exist, re-test using your own probes via authorized APIs, and let the measured diff decide whether to switch.

Version management should slice capabilities, not rumored commercial milestones:

• Whether long context changes instruction following, rather than just looking at the window number.
• Whether the maximum output length is sufficient to contain the complete markup; when output is truncated, the graphic will show unclosed tags, which is an engineering failure, not an aesthetic issue.
• If a reasoning or thinking mode is enabled by default, latency and cost structures will change. Acceptance should look at latency percentiles and failure rates simultaneously.
• Whether the same prompt yields consistent results across different regions and accounts, avoiding mistaking gray traffic for a capability leap.

These comparisons use generic API concepts: input, output, temperature, maximum generation length, timeout, and retries. Do not write unverified unit prices, context limits, or exclusive parameters into your conclusions.

Common Misjudgments

Treating a single retouched image as a version conclusion is the most common mistake. Next is using a question that has been repeatedly practiced across the industry to declare a “widening gap.” Third is only looking at element counts without verifying if constraints hold. Fourth is ignoring post-processing: external beautification or manual markup edits will make the file look more refined.

To determine whether a model acts more like a “rendering engine” or just “stacking primitives,” check if gradients and shadows are linked to geometry. Hand-stacked elements usually consist of massive independent color blocks; more coherent results will share the same set of groups and coordinates for highlights, contact shadows, and occlusions. Even so, you should not claim the model possesses some undisclosed image architecture based solely on this; markup style can only serve as behavioral evidence.

Acceptance Criteria Teams Can Adopt Immediately

Before internal release or switching models, use a small set of vector probes as a gate: validity pass rate, key constraint pass rate, median markup length, end-to-end latency, and truncation rate. List visual bonus items separately, without weighting them with correctness into an unexplainable total score. Rotate a portion of the prompts quarterly, keeping a few anchor questions to observe regression, while avoiding long-term public exposure for the rest to reduce the benefits of memorization.

Object storage can save raw markup and rendered images, logging services can record request IDs and parameters, and monitoring alerts can track truncation and parse failure rates. These are general engineering capabilities used to ensure evaluation traceability, not to replace the analysis of model behavior.

The conclusion is straightforward: progress in generation quality must be reflected in verifiable constraints. Visual improvements on public old questions often saturate quickly; nesting, occlusion, lighting consistency, and stable reproducibility are more informative benchmarks for the next stage. By using structured probes to write differences into reason codes, teams can decide whether switching checkpoints is worthwhile without being led astray by a single screenshot.