Evaluating Multimodal LLMs with Runnable Artifacts: SVG Animation and Code-Only Rendering Practice
As preview versions of multimodal large language models roll out, benchmark scores alone are no longer enough to judge whether a model is genuinely useful. This article proposes evaluating models through reproducible, executable, and verifiable tasks: parameterized animated SVGs that test code correctness, interactivity, and offline runnability, and code-based light-and-shadow rendering that tests graphics understanding, pipeline organization, and visual aesthetics. It covers an evaluation matrix, technical breakdowns of both tasks, a capability profile of current models, and practical engineering guidelines for prompt design, sandboxed execution, and screenshot-based regression testing.
Introduction: From “Writing Code” to “Delivering Runnable Work”
Recently, preview versions of multimodal large models have been released in quick succession. Public announcements generally emphasize improvements in underlying world knowledge, multimodal perception, coding, and agentic capabilities, but reveal little about parameter counts, benchmark results, or training data. For developers, benchmarks alone are no longer sufficient to judge whether a model is good to use. A more effective approach is to design reproducible, executable, and verifiable tasks that push model output from “text answers” to “runnable artifacts.”
This article does not recommend or compare any specific product. Instead, it abstracts two common task types from public demos into general evaluation samples: parameterized animated SVGs, and rendering light-and-shadow scenes purely with code. The former tests code correctness, interaction design, and offline runnability; the latter tests graphics understanding, rendering pipeline organization, and visual aesthetics. Together, they provide a fairly comprehensive view of a model’s limits in coding, spatial perception, and aesthetic control.
1. Why Evaluate Code with Runnable Artifacts
Traditional code evaluation relies mostly on unit tests or algorithm puzzles. These are objective, but they tend to ignore the engineering constraints of real development. Runnable-artifact evaluation emphasizes three points. First, output must be executable—for example, an SVG that opens directly in a browser, or an interactive WebGL page. Second, output must be maintainable, with clear code structure, state management, and parameter naming. Third, output must be verifiable, judged through screenshot regression, performance budgets, and cross-environment checks.
Take an animated SVG as an example. The model must not only draw shapes but also organize timelines, state machines, event bindings, and export logic. A seemingly simple “animal riding a bicycle” animation actually involves wheel rotation, a fluttering scarf, speed adjustment, day-night switching, and pause/resume. If the model only produces static graphics, its code ability stays at the snippet level; if it produces a complete, interactive, offline-runnable file, the result is closer to an engineering-grade deliverable.
A suggested evaluation matrix:
| Dimension | What to Observe | How to Verify |
|---|---|---|
| Correctness | Syntax, element hierarchy, animation references | Opens in browser without errors; key elements visible |
| Interactivity | Pause, speed control, theme switching work | Manual operation plus automated script checks |
| Maintainability | Centralized state, decoupled functions | Code review and cost of secondary modification |
| Performance | Frame rate, DOM count, repaint scope | Performance panel and spot checks on low-end devices |
| Portability | Runs offline after export | Open with network disconnected; cross-browser validation |
| Aesthetics | Composition, color, lighting, detail consistency | Screenshot review and image regression |
2. Task One: Engineering Breakdown of a Parameterized Animated SVG
For animated SVG tasks, ideal output includes a complete scene, adjustable parameters, and offline runnability. In a seaside cycling animation, for instance, the wheels keep spinning, the scarf flutters backward with speed, and the UI offers pause, pedaling-speed control, and day-night switching. If you raise the speed multiplier and export, the SVG should still animate when opened standalone—evidence that the animation logic is embedded in the file itself, not dependent on external scripts or network resources.
Key technical points for this task: use standard animation elements or script-driven timelines; consolidate parameters like speed and day/night mode into a single state object; handle time-basis logic for pause and resume; limit the number of filters, gradients, and paths to avoid frame drops on low-end devices; and add semantic labels to graphics for accessibility. A simplified parameterized structure can be abstracted as:
const state = {speed: 1.0,paused: false,theme: "day"};function updateScene(dt) {if (state.paused) return;const t = dt * state.speed;wheelAngle = (wheelAngle + t * 360) % 360;scarfOffset = Math.sin(t * 2) * 4 + state.speed * 2;applyTheme(state.theme);}
From an evaluation standpoint, the animated SVG task has a key advantage: it is interpretable. Correctness is easy to verify, interactivity can be tested directly, and performance issues can be diagnosed with browser tools. If a model reliably produces such artifacts, it demonstrates strong structured code generation, basic spatial reasoning, and some aesthetic control. Still, note that a finished animation does not guarantee engineering quality—check whether the code is excessively verbose, whether there are cross-browser compatibility issues, and whether the exported file contains external dependencies.
3. Task Two: The Difficulty and Aesthetic Gap in Pure-Code Rendering
Pure-code light-and-shadow rendering is a harder sample. The prompt typically asks to “use known coding techniques to draw an afternoon room with interlacing light and shadow, pursuing a tranquil atmosphere and extreme detail.” In public examples, some solutions use a systems-level language to write a path tracer, rendering for hours to achieve near-photographic results; others build the room in WebGL and then render a CPU-based version. A common highlight of the latter is a shaft of window light slanting in, passing over a small round stool and reaching the bottom edge of the frame—showing that the model has a basic grasp of light direction and scene composition.
But the gaps show up most clearly in aesthetics and physical correctness: corner shadows are too harsh, potted plants and floors look like freshly arranged sets, and light-to-dark transitions lack naturalness. Path-traced solutions have the advantage of global illumination, soft shadows, indirect light, and material reflections that better follow physical laws. Real-time WebGL solutions, lacking high-quality lighting models, tone mapping, and post-processing, often produce “geometrically correct but visually unconvincing” images. Being able to write a rendering loop does not mean being able to tune material roughness, light intensity, exposure, and composition ratios.
Therefore, code-rendering tasks should be verified in layers: the first layer is runnable code that produces an image; the second is graphical correctness, with coordinates, camera, and light direction free of obvious errors; the third is visual quality—natural light transitions, appropriate shadow softness, material noise, and color relationships; the fourth is aesthetic consistency with the atmosphere described in the prompt. Judging model capability on “it renders an image” alone easily overestimates actual delivery quality.
4. Capability Profile: Coding, Agents, and Multimodal Perception
Across both tasks, preview models show clear progress in structured code generation, interactive parameterization, and basic spatial composition—translating natural-language requirements into scene elements, animation states, and rendering loops. Agentic ability shows up in task decomposition and iterative refinement: build the scene first, then add animation, then tune parameters. However, public information is insufficient to determine how much gain the underlying architectural changes actually bring; more black-box tasks and reproducible samples are needed for verification.
Improved multimodal perception helps models understand abstract descriptions like “seaside,” “afternoon,” “tranquil,” and “interlacing light and shadow,” but aesthetics remain a weakness. Aesthetics is not a single metric—it is the combined result of composition, color, material, lighting, and narrative consistency. For developers, the practical approach is to break aesthetics into checkable items: is the subject prominent, are the light-dark levels rich, are shadows too harsh, are materials consistent, is there noise or clipping? Only by combining screenshot regression with human review can “looks good” be turned into an iterative engineering standard.
5. Engineering Recommendations
First, structure your prompts. Break tasks into output format, technical constraints, interaction requirements, performance budgets, and acceptance criteria to reduce uncontrollable improvisation by the model. Second, sandbox execution. Run all generated code in an isolated environment with restricted network and file access to avoid unexpected dependencies. Third, screenshot regression. Capture screenshots of key interaction states, build baseline images, and detect layout or lighting degradation. Fourth, performance budgets. Limit DOM nodes, texture sizes, and animation complexity, with spot checks on low-end devices. Fifth, fallback plans. When real-time rendering is unstable, switch to static export or simplified lighting to preserve usability.
Conclusion
Overall, multimodal LLMs are already quite usable for code generation, animation interaction, and basic 3D scene construction, but physically realistic, aesthetically controlled pure-code rendering remains weaker, with a gap to leading flagship solutions. Catching up is not just about parameter scale—it depends on graphics knowledge, visual aesthetics, and engineering verification systems improving together. For teams, building an evaluation process that is runnable, quantifiable, and regression-testable is far more useful for model selection and deployment than any single demo.
