Testing LLM Coding Ability: SVG Animation and Pure-Code Ray Tracing
This article presents a practical, reproducible approach to evaluating how well flagship large language models handle coding tasks. Using a family of "generate a runnable front-end graphics artifact" tasks—structured SVG animation, pure-code light-and-shadow rendering, and complex 3D/agentic loops—it walks through controlled variables, task tiers, prompt contracts, and automated verification. The findings: structured visual artifacts are already deliverable at scale, aesthetic-driven artifacts still need human review, and long-chain engineering loops require external guardrails. Includes GLSL and pseudocode examples for soft shadows, color management, and self-repair loops.
Why Coding Ability Has Become the Main Battleground for Flagship Models
New preview versions of flagship LLMs are rolling out one after another. Many vendors first release an “anonymous model” into coding tools for gradual rollout, collect real feedback, and only then reveal its identity. This release cadence itself signals something important: programmability is replacing traditional benchmark scores as the primary measure of model quality.
According to official statements, the public information for this generation of models typically stays at the level of capability directions: more complete world knowledge, stronger multimodal perception, improved coding and agentic capabilities, and adjustments to the underlying architecture. But parameter counts, benchmark scores, and training data composition are usually not disclosed. This means external evaluators cannot rely on official metrics for technical selection—they can only depend on reproducible, hands-on testing.
The timing is dense as well: flagship models from multiple frontier labs are being unlocked almost simultaneously, leaving a very short selection window. For developers, rather than arguing over whose leaderboard is higher, it is more valuable to build a testing method of your own that can answer “can it deliver?” This article uses a task family of “generating runnable front-end graphics artifacts” as a sample to walk through the entire method.
Testing Methodology: Turning “Usability” into Reproducible Conclusions
Controlling Variables
LLM output is highly stochastic; without controlled variables, conclusions have no reuse value. This test fixed four variables:
- Same inference entry and tier: In the coding tool, uniformly select the highest reasoning tier to avoid misreading “poor low-tier performance” as a model capability problem.
- Same prompt: No multi-turn follow-up “coddling”; the first-round output is the scoring object, because in production the first-round success rate directly determines rework cost.
- Same context budget: No extra reference material, to assess the model’s own level of knowledge internalization.
- Same acceptance criteria: Runnable, exportable, openable offline, interactive.
Task Selection
Tasks progress through three tiers of difficulty, covering “deterministic output—aesthetic output—engineering closure”:
- Structured animation generation: The classic water-bird riding animation, testing SVG structure, timeline, and interaction controls;
- Pure-code light-and-shadow scene: Testing rendering path selection, global illumination, and color management;
- Complex interaction and agentic closure: Testing 3D scene construction, performance budgets, and self-repair ability.
Prompt Template
Open-ended prompts (such as “make it perfect, pursue the ultimate”) cannot constrain engineering details. It is better to write the delivery contract into the prompt:
[Task] Generate a self-contained single-file SVG depicting a water bird riding a bicycle along a coastline.[Technical Constraints]1. Only SVG + inline CSS/JS allowed; no external resources or network requests;2. Wheel rotation and crank cadence share the same phase, and the period must be parameterized;3. Provide three controls: pause/play, speed multiplier 0.5x–2.0x, day/night toggle;4. Theme switching is implemented via CSS variables; rebuilding the entire node tree is forbidden;5. Self-check before delivery: valid XML structure, explicit transform-origin declarations, no NaN coordinates.
The more specific the constraints, the more concentrated the model’s output distribution, and the higher the comparability of the evaluation results.
Test 1: Self-Contained Delivery of SVG Animation
Conclusion first: this tier of task is basically “passed.” The first-round output produced a character wearing a hat, riding along the coastline with a scarf fluttering in the wind; the wheels are linked to the crank, and it comes with three types of controls—pause, speed multiplier, and day/night toggle. After cranking the speed to a higher multiplier and exporting the SVG separately, opening it offline with the network disconnected still plays correctly.
The real capability threshold behind this is “structural self-consistency,” which can be verified from four checkpoints:
| Checkpoint | Common Failure Mode | Verification Method |
|---|---|---|
| Structural validity | Unclosed tags, missing attribute quotes | XML parser validation |
| Transform origin | Omitting transform-origin, wheels rotate around the wrong origin | Keyframe screenshot comparison |
| Timeline synchronization | Phase drift between CSS animation and JS timers | Long-duration screen recording |
| Interaction state machine | Pause fails to freeze all animations | Click-through regression on each control |
It is advisable to script the verification rather than rely on the naked eye:
# Pseudocode: structural validation + keyframe visual regressionfrom xml.etree import ElementTreetree = ElementTree.parse("output.svg") # structural validityfor t in [0.0, 0.5, 1.0]: # keyframesshot = headless_screenshot("output.svg", seek=t)assert diff_ratio(shot, baseline[t]) < 0.02 # compare against baseline
The engineering implication is clear: this kind of “deterministic visualization” task can already be confidently handed to models for batch production, and the real cost has shifted to the verification stage—which can be covered by automation.
Test 2: Pure-Code Light-and-Shadow Scene—The Hardest Tier
Given the same open-ended prompt (render a scene with interlocking light and shadow, highlighting the tranquil atmosphere of an afternoon), the outputs of different models vary enormously. The best-performing approach chose offline path tracing: implementing a renderer from scratch, accumulating samples over a long time, and finally obtaining near-photographic window-pane projections, wood-floor reflections, and wall brightness gradients. Most models, by contrast, took the route of real-time rasterization—building a WebGL scene in the browser, sometimes with an additional CPU render.
Where Exactly the Gap Appears
The typical weakness of real-time rasterization is not “can’t render it” but “physically wrong”:
- Shadows too hard: Single-source hard shadows leave abrupt black lines in corners, lacking penumbra transitions;
- Missing indirect light: No energy bounce between walls and floor; dark areas are patched with ambient light, creating a break in the bright-to-dark transition;
- Materials look like a stage set: Diffuse and specular highlights are mixed together; plant leaves and floor inlays lack anisotropic detail; the overall look is a “built studio” rather than a “lived-in room.”
These points correspond exactly to the strengths of path tracing: area-light sampling, multiple bounces, and physically based bidirectional reflectance distribution functions. In other words, the root of the gap is a choice of rendering paradigm, not code volume or syntax ability.
Key Implementation: Soft Shadows and Color Management
To bring a real-time pipeline close to offline quality, at least two things must be added. The first is jittered sampling of the area light, turning hard shadows into a penumbra:
// Jitter-sample the light disk N times to obtain a soft penumbrafloat softShadow(vec3 p, vec3 n, vec3 lightPos, float radius, int N) {float lit = 0.0;for (int i = 0; i < N; ++i) {vec3 jitter = diskSample(i, N) * radius;vec3 dir = normalize(lightPos + jitter - p);lit += step(0.0, dot(n, dir)) * rayOccluded(p, n, dir);}return lit / float(N);}
The second is the color pipeline. A lot of the “plastic feel” actually comes from doing lighting calculations directly in sRGB space, and from missing tone mapping that causes highlights to clip hard:
// Accumulate lighting in linear space, then apply filmic tonemapping + linear-to-sRGB conversionvec3 filmicTonemap(vec3 x) {return clamp((x * (2.51 * x + 0.03)) / (x * (2.43 * x + 0.59) + 0.14), 0.0, 1.0);}vec3 linearToSRGB(vec3 c) {return mix(12.92 * c, 1.055 * pow(c, vec3(1.0 / 2.4)) - 0.055, step(0.0031308, c));}
In this test, the real-time scenes generated by mainstream preview models got the direction of the window-pane light spots and the spatial relationship of the light beam crossing the stool toward the lower edge of the frame correct, showing that geometric and camera-composition capabilities are already usable; what is missing is the three layers of “physical correctness” above—penumbra, indirect light, and the color pipeline. At the same time, models generally underestimate rendering time, stuffing high-sample offline rendering into the browser main thread and freezing the page—which suggests the prompt must specify performance budgets and degradation strategies.
Test 3: 3D and Agentic Closure
The third tier examines long-chain self-correction. An effective evaluation loop should hand the “generate—verify—feed back” closure to the model itself to run:
const maxRounds = 3;for (let round = 0; round < maxRounds; round++) {const report = await runChecks(result); // syntax/type/runtime errors + screenshot comparisonif (report.pass) break;prompt += `\n[Previous round failure info]\n${report.errors}\nPlease only fix the reported errors; do not rewrite the overall structure.`;result = await model.generate(prompt);}
The key design is constraining the scope of modification. “Only fix errors, don’t rewrite structure” can significantly suppress capability degradation across multi-round iterations—otherwise the model would tear everything down and start over each round, losing all the previously tuned composition and lighting. In practice, this affects final delivery rate more than improving single-round generation quality.
Attributing the Gap: Why Pure-Code Rendering Is So Hard
- Skewed corpus distribution: Front-end web code occupies a very high proportion of public corpora, while high-quality code related to offline renderers and spectral and color science is relatively scarce; the density of “correct patterns” the model has seen differs.
- Extremely sparse feedback: Rendering errors don’t throw exceptions. A gray image, hard shadows, or plastic-looking materials cannot be caught by compilers or unit tests—only visual judgment can, and the model can hardly obtain this gradient during training.
- Strongly coupled parameters: Geometry, lighting, materials, and color management are intertwined; changing one exposure value affects shadow contrast and highlight clipping, and the combinatorial space grows explosively.
- Long-context consistency: A self-consistent renderer can easily run to thousands of lines, and maintaining consistent naming, coordinate systems, and energy units across files demands a lot from the model’s long-term memory and global constraint ability.
The conclusion is not pessimistic: the ordering of these three tiers happens to correspond to the usability boundary of current capabilities—structured artifacts can already be delivered in batch, aesthetic-driven artifacts still require human review, and long-chain engineering closure needs external guardrails.
Practical Recommendations: Put Models into a Guarded Pipeline
- Templatized prompts: Solidify the delivery contract (file self-containment, interaction checklist, performance budget, self-check items) into a template to reduce room for improvisation.
- Front-load automated verification: Static checks, type checks, structural validation, headless-browser keyframe screenshots, and pixel-level diff comparison—all brought into CI, with visual regression run like unit tests.
- Clear division of labor: Hand interactive motion, form visualization, and single-page data dashboards to the model; hand high-precision offline rendering to a dedicated rendering pipeline, with the model only responsible for writing shaders and scene descriptions.
- Controlled cost: For rendering tasks, be sure to specify resolution caps, sample-count caps, and timeout degradation paths to avoid prolonged main-thread blocking.
- Observability: Collect generation calls, verification results, and human review comments uniformly into a logging service and monitoring alerts, forming a time series of model capabilities for version regression and selection decisions; the artifacts themselves can be archived to object storage for retrospective comparison.
Conclusion
The conclusions of a single test become outdated quickly with version iterations; what really holds long-term value is the evaluation method itself: fixed variables, defined artifacts, automated verification, and closed-loop repair. When you have an evaluation pipeline that can run in an hour and output comparable reports, no new model release is “hearing what the vendor says” but “letting the data speak”—and that is the most reliable way to cope with a dense release cadence.
