Using Structured Vector Markup to Test Composition in Generative Models

Why “Looks Better” Is Not Enough

When a generative model is upgraded, the most visible changes are often the ones that are easiest to notice: smoother lines, fuller shadows, and more elements. For engineering teams, these visual impressions cannot serve as acceptance criteria. A new checkpoint that briefly appears in a model catalog and then disappears, a slightly higher benchmark score, or a sample image with more fine detail only shows that someone is testing the new version. None of these shows that the version is ready for production.

A more reliable approach is to break model capability into layers that can be checked repeatedly: instruction following, structural fidelity, proportion and occlusion, lighting and materials, stability over long outputs, and regression against the previous version. Each layer needs a repeatable probe. Keeping a single “best-looking” image is not enough.

Using Explicit Markup as a Microscope

A bitmap only provides final pixels, so judging texture quality depends almost entirely on the eye. Vector markup works differently. Shapes, groups, gradients, and filters are written as plain text, so every extra group or gradient remains visible in the file. This makes vector markup suitable as a composition probe. The format is not more advanced than bitmaps. It is useful because it exposes the decisions the model made.

A valid vector probe should constrain at least three layers at the same time:

1. Layout: Whether the relative positions of the subject, props, and background stay stable, without elements leaving the frame or overlapping until they become unreadable.
2. Proportion: Whether component sizes match common sense, and whether connected parts stay joined.
3. Lighting: Whether highlights, shadows, and ground traces serve the forms, rather than being randomly added as decoration.

Detail density should be scored separately. A model can add more gradients and filters to the markup, which makes the file longer and the image look “fuller.” But if the subject’s pose is wrong or occlusion relationships are reversed, the added density is noise. Conversely, a structurally correct image with restrained decoration does not mean the model has stopped improving. The question is whether new elements are integrated into existing forms, rather than floating beside them.

A simple check can be written without relying on any specific evaluation platform:

Count: number of path / group elements
Count: whether gradient and filter definitions are actually referenced
Check: whether key parts fall inside the expected bounding boxes
Check: whether occluded objects are modeled as sub-paths or proper stacking, not pasted on afterward
Compare: markup differences between the new checkpoint and the baseline under the same prompt

Asking the model to add one extra object is a particularly revealing test. Adding a shape to a corner of the image is easy. Integrating a new object into an existing outline, handling the parts that are hidden, and placing its shadow on the correct plane shows that the model controls both geometry and layering. Such constraints should be written into the prompt and verified in the markup, not judged only from a thumbnail.

Why Fixed Test Sets Lose Their Value

Once a community treats a particular task as a shared exam, models and data pipelines may repeatedly encounter similar samples. A rising score does not necessarily mean the model has improved on open-ended tasks. It may only mean the model has become more familiar with fixed compositions and fixed wording. Structural tasks saturate first. Objects such as handles, building facades, and simple mechanical parts have limited topology, so each model generation quickly learns the overall shapes. The remaining differences move to local details such as reflections, connectors, scales, and labels.

For this reason, evaluation sets should be layered, rather than built around one well-known test:

• Saturation probes: Confirm that there is no obvious regression. Their weight should decrease over time.
• Composition probes: Add new constraints to familiar subjects, such as occlusion, ground traces, or local highlights, and check whether the added elements are reasonable.
• Transfer probes: Change the object category, aspect ratio, and style constraints, then check whether the model can do more than reproduce one “standard answer.”
• Failure probes: Deliberately give vague or conflicting requirements, and observe whether the model asks for clarification, conservatively omits details, or fills the image with unnecessary content.

A slightly higher benchmark score should only be treated as a lead. If the codename of a test checkpoint does not match its official release name, the early numbers should not be presented as a capability guarantee. Early checkpoints often differ in safety policies, output length, and reasoning settings. When comparing results, these variables must be aligned. Otherwise, score differences cannot be explained.

Evaluating Structural Fidelity and Detail Density Separately

Split the results of a single generation into two tables.

The structure table asks whether the output is usable. It checks whether all parts are present, whether connections hold, whether text or labels are misaligned, and whether repeated elements are mirrored incorrectly. Structural failures are blocking issues. A more elaborate shadow cannot offset them.

The density table asks whether the output is good enough to use. It checks whether gradients are defined and actually referenced, whether shadow directions are consistent, whether highlights fall on plausible surfaces, and whether background traces match the direction of motion. If higher density comes with bloated markup, unused filters, or deeply nested structures that are hard to maintain, it becomes a burden for downstream editing. One of the main advantages of vector graphics is that they can be edited. If people cannot read or partially replace the generated output, it no longer serves its purpose as structured output.

When comparing two versions, look at the markup differences as well as the images side by side. Check which groups were added, which paths were split, and which effects were simply defined more than once. This helps avoid the bias of thinking that a new version is better everywhere. A common pattern is that local materials become more detailed while the subject’s topology stays the same. Another is that decoration increases, but a key constraint, such as an object being partly hidden behind another, still fails.

How Enterprises Should Handle “Upcoming” Checkpoints

A new entry in a model catalog or an internal test list only indicates that the provider is moving a version through its release process. It is normal for an entry to be withdrawn, for an internal codename to differ from the public name, or for a release timeline to shift from “soon” to “in one or two weeks.” Procurement and engineering teams can use this information to prepare regression testing. They should not use it to change roadmaps or make external commitments.

A practical process can look like this:

1. Freeze the baseline: Select the current production model, and fix the prompts, randomness settings, and output limits.
2. Run a small smoke test: Check the blocking structural criteria first. If too many samples fail, stop before expanding the test.
3. Expand in layers: After the structure checks pass, evaluate density, latency, truncation in long outputs, and format validity.
4. Archive the differences: Store the markup or structured text, not only screenshots, so results can be reviewed later.
5. Use a gradual switch: Do not replace the production alias with a new checkpoint by default. Run controlled comparisons using explicit version identifiers.
6. Define exit conditions: Specify which regressions require a rollback, so that a general impression of “better overall” does not delay decisions.

If extended reasoning is enabled throughout, cost, latency, and log structure all change. Evaluation records should note this separately, so that the cost of reasoning is not counted as evidence that the model is stronger. The same applies to context windows and maximum output lengths. A larger window does not automatically make a task more stable. Long outputs are more likely to lose constraints in the later sections. Applications should still limit output size, validate results in segments, and avoid requesting the full maximum output in one pass.

Applying Probes to Real Tasks

Vector composition is only one example of structured generation. The same principle applies to interface descriptions, tables, flow markup, and hierarchical documents. Whenever the output can be parsed, do not rely only on manual scoring.

Assertions can be designed for each task, rather than relying on a single composite leaderboard:

• Whether required fields exist, and whether their types are correct.
• Whether every referenced identifier is defined, with no dangling styles or broken links.
• Whether the hierarchy exceeds the agreed depth, or contains meaningless repetition.
• Whether new details break structural assertions that already passed.

The validation logic does not need to be complicated:

parse(output)
assert required_parts present
assert references resolve
assert occlusion_rules hold
score density only if structure passes
compare(score, baseline) within agreed tolerance

If a structural assertion fails, the density score should be treated as invalid. This prevents a visually striking sample from dominating the review.

Boundaries of Writing and Use

Evaluation articles are well suited to discussing methods: how to design probes, how to break down metrics, and how to run version regression. They should not present unconfirmed checkpoints, prices, context lengths, or release windows as facts. Screenshots and secondhand timelines from public discussions can serve only as signals that a result needs to be retested.

For developers, the benefit is less dependence on single test scores. Parseable markup can show whether layout, proportion, and lighting hold together, and detail density can be treated as a bonus only after structure passes. For enterprise users, the benefit is that model switching becomes a reversible experiment: a frozen baseline, assertions first, explicit versions, and traceable failures. With this approach, the team evaluates a capability it can maintain, regardless of which new names appear in upstream catalogs, rather than judging a single image that cannot be verified.