Evaluating Code-Driven Visual Creation: A Practical Benchmarking Methodology
Code generation is usually judged by function completion and unit test pass rates, but these metrics say little about how well a model understands spatial relationships, motion, and visual detail. This article presents a practical methodology for evaluating code-driven visual creation using vector animation tasks and rendering tasks. It covers how to design verifiable prompts, how to assess lighting quality in rendered scenes, how to convert subjective aesthetic judgments into comparable scores, and how to control variables for reproducible results. A scoring rubric with weighted dimensions, sample code for soft shadow estimation and score aggregation, and step-by-step guidance for rolling out the evaluation inside a team are also included.
Why Use Visual Tasks to Evaluate Code Generation
Code generation capability is typically measured with metrics like function completion and unit test pass rates. These metrics are effective for checking logical correctness, but they reveal little about how well a model understands spatial relationships, temporal changes, and visual details. Visual tasks require the model to translate natural language descriptions into graphical programs that can run directly. The output is visible to everyone, and errors are easy to spot.
These tasks fall into two broad categories:
- Vector animation tasks: the model outputs a text-format vector graphic, testing geometric structure, animation timing, and interaction control.
- Rendering tasks: the model writes its own drawing or rendering program, testing its ability to model lighting, perspective, and materials.
The two categories answer two separate questions: can the model get the structure right, and can it get the visual effect right.
Vector Animation Tasks: From Prompt to Verifiable Output
Take “generate an animated vector graphic of a riding animal” as an example. The prompt must specify constraints up front; otherwise, outputs from different models cannot be compared. At minimum, specify:
- The output must be a single self-contained file with no external dependencies;
- The animation loop duration, direction of motion, and key elements;
- Whether playback controls, speed adjustment, or other interactions are required.
Here is a minimal runnable example showing how to drive a positional animation with declarative tags:
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 200 120" width="200" height="120"><rect width="200" height="120" fill="#cfe8ff"/><circle cx="30" cy="90" r="12" fill="#333"><animate attributeName="cx" from="30" to="170" dur="3s" repeatCount="indefinite"/></circle></svg>
To verify the result, open the file in an offline browser and check three things: whether it renders correctly, whether the animation plays continuously, and whether it still works after disconnecting from the network. The last check distinguishes truly self-contained output from output that relies on external scripts.
Rendering Tasks: How to Assess Lighting Quality
Rendering tasks generally follow two technical approaches.
Rasterization: geometry is projected onto the screen, and shading models approximate lighting. It is fast enough for real-time use in browsers and games, but shadows and indirect light require extra techniques to simulate.
Path tracing: rays are cast from the viewpoint into the scene, and light transport is estimated through repeated sampling. The result is closer to physical reality, but the computation cost is high—rendering can take hours.
When the same scene is implemented with both approaches, the differences usually show up in the following areas:
- Whether the edges of window shadows are soft, and whether light-to-dark transitions are continuous;
- Whether shadowed areas retain detail, or show harsh black bands;
- Whether objects have plausible contact shadows and reflections;
- Whether composition and lighting match the mood described in the prompt.
Soft shadows are the key discriminator here. The occlusion ratio of an area light can be estimated through multiple samples, as shown below:
def soft_shadow_factor(point, light_samples, is_occluded):"""Estimate visibility using multiple light source sample points.point: the surface point being shadedlight_samples: a list of sample points on the area lightis_occluded(point, light_pos): whether the two points are occludedReturns visibility between 0 and 1, where 1 means fully lit."""if not light_samples:raise ValueError("light_samples must not be empty")visible = sum(1 for p in light_samples if not is_occluded(point, p))return visible / len(light_samples)
With fewer samples, shadow edges tend to show graininess or hard borders. Reviewers can watch for this artifact to judge whether the model understood the physical size of the light source.
Turning Aesthetic Judgment into Comparable Scores
Subjective aesthetics cannot be ranked directly, so they need to be broken down into dimensions with explicit criteria. The following three dimensions are recommended.
| Dimension | Judgment question | Scoring method |
|---|---|---|
| Functional completeness | Are all required elements, animations, and interactions present | Check item by item; deduct points for each missing item |
| Visual quality | Are lighting, perspective, and color coordinated | Blind review by multiple reviewers; take the average score |
| Code quality | Is the structure clean; does it depend on external resources | Manual review plus static checks |
Weighted aggregation can be done with the following function:
WEIGHTS = {"functional_completeness": 0.4, "visual_quality": 0.35, "code_quality": 0.25}def weighted_score(scores: dict[str, float]) -> float:missing = set(WEIGHTS) - set(scores)if missing:raise KeyError(f"Missing score dimensions: {missing}")total = sum(WEIGHTS[k] * scores[k] for k in WEIGHTS)return round(total, 2)
Weights should be fixed before the evaluation begins, never adjusted after seeing the results. Otherwise, scores will drift toward the expected conclusion.
Controlling Variables and Ensuring Reproducibility
Large model outputs are stochastic, so a single run is not enough to support any conclusion. In practice, keep the following points in mind:
- Fix the prompt text and record its full content and version;
- Record inference parameters such as temperature, maximum output length, and reasoning effort level;
- Run each task multiple times at minimum, and report the median and variance rather than keeping only the best run;
- Save the raw output files and run logs so others can verify the results;
- Conduct reviews under blind conditions, where reviewers do not know the source of each output.
Limits of Interpretation
A set of samples demonstrates performance on specific tasks; it cannot be extrapolated to a model’s overall capability. When interpreting results, keep the following in mind:
- Prompt wording significantly affects results—a different phrasing may produce a different ranking;
- Visual evaluation involves subjectivity, so reports should state the number of reviewers and inter-rater agreement metrics;
- A single task type has limited coverage—animation, rendering, and UI generation should be tested separately;
- Results change after model updates, so conclusions should include the test date.
Implementation Recommendations
To establish this evaluation within a team, proceed as follows:
- Collect ten to twenty visual prompts covering a range of difficulty levels;
- Fix the evaluation environment and write automated scripts to generate and save outputs in batch;
- Build an offline viewing page to make blind review easy;
- Aggregate results using the scoring rubric and keep raw samples as a regression test set;
- Re-run the evaluation after every model or parameter change, comparing trends rather than single data points.
Visual tasks look straightforward, but conclusions are only meaningful when the judgment criteria are written down clearly and the process is fully documented.
