# LLM Code Generation Tested: From SVG Animation to Physically Rendered Images

    With a new wave of flagship model releases arriving in a short window, launch-stage numbers say little about real coding ability. This article takes a preview build of a new-generation model running at a very high reasoning setting, tests it on two code-generation tasks — an animated SVG of a pelican riding a bicycle, and a light-and-shadow image rendered entirely in code — and compares the results against capabilities widely regarded as state of the art. It introduces a reusable four-layer evaluation framework (executability, requirement fidelity, physics and aesthetics, engineering quality), breaks down what the model got right and where it lost points, explains why real-time rasterization falls short of offline path tracing for photographic output, and turns "aesthetics" into executable prompt constraints plus an iterative workflow. You will learn how to benchmark code generation yourself, how to choose between real-time and offline rendering, and how to turn these tests into an engineering practice.

    1. Background: With Models Releasing Rapidly, Hands-On Testing Beats Leaderboards

    The LLM field has entered another round of dense releases. Multiple organizations launched new flagship models within a narrow window, each emphasizing different talking points — parameter scale, benchmark scores, context length — and some vendors put out previews anonymously to gather feedback from real users first.

    For developers, chasing the numbers announced at launch events is less useful than going back to concrete tasks and checking whether a model can turn a requirement into code that runs, can be maintained, and can be reused.

    Among such tasks, “produce a runnable artifact with code” is especially representative. Vector animations in the spirit of the pelican-riding-a-bicycle test, an image with convincing light and shadow rendered with nothing but code, and an interactive 3D scene all require the model to do four things at once:

    • understand the intent behind a natural language description;
    • design a sound code structure;
    • master graphics, physics, and geometry;
    • exercise basic aesthetic judgment.

    Publicly available information indicates that some new-generation models adjusted their underlying architecture. Common industry directions include mixture-of-experts structures, sparse activation, attention mechanism optimizations, and larger compute budgets in the reasoning phase — each with its own trade-offs. However the architecture changed, task performance is what answers the question. Vendors also generally emphasize strengthened world knowledge and multimodal perception, along with improved coding and agentic capabilities, which raises expectations for spatial understanding and engineering execution.

    This article takes a preview build of a new-generation model, run in an online coding environment at its “very high” reasoning setting, as the subject under test, and compares it horizontally against coding capabilities widely regarded as state of the art in the industry. The point is not to score a particular model, but to build a reusable evaluation method: turning “is this code good?” into a question that can be described, compared, and improved.

    2. Evaluation Method: Four Layers of Metrics and a Reproducible Process

    2.1 Four-Layer Metric System

    Each generation task is broken into four progressive layers. How models differ from layer to layer is often more revealing than the total score.

    Layer Core question Key checks
    L1 Executability Does it run on the first attempt? Syntax and dependency completeness, console errors, cross-environment compatibility
    L2 Requirement fidelity Was everything implemented? Explicit features, implicit interactions, error states, parameter configurability
    L3 Physics and aesthetics Does it look real, does it look good? Geometric plausibility, lighting and materials, color and composition
    L4 Engineering quality Can it be maintained over time? Structural clarity, performance overhead, extensibility, offline availability

    Most evaluations stop at L1 and L2 — “it runs” and “the features are all there.” What actually separates models is L3 and L4: the former depends on how well a model has internalized how the world works and what things look like, the latter on its grasp of software engineering conventions.

    2.2 A Reproducible Process

    1. Fix the prompt and the environment. Same prompt, same reasoning setting, same runtime. If the interface exposes randomness parameters, set them low and repeat each task several times so a single result does not mislead you.
    2. Compare against a reference. Pick a widely circulated high-quality sample, or a capability generally accepted as state of the art, so “good” has a concrete definition.
    3. Double acceptance check. First run an automated pass in headless browser mode to catch errors, test interactions, and measure resource use; then have reviewers score manually from three angles: visual quality, physics, and code structure.
    4. Root-cause analysis. Trace every lost point back to its origin and decide whether the problem sits in knowledge, planning, or expression. This is more valuable than recording scores alone.

    3. Case One: SVG Vector Animation — the Baseline for Basic Ability

    3.1 Task and Prompt

    The classic test is an animation of a pelican riding a bicycle:

    1. Generate an animated SVG of a pelican riding a bicycle

    The prompt is barely a dozen words long, yet it carries five kinds of information: subject (pelican), action (riding), vehicle (bicycle), format (SVG), and form (animation). The model has to fill in a large number of details on its own — proportions, posture, setting.

    3.2 What the Test Showed

    The model under test produced a pelican wearing a red hat, riding along the coast:

    • the wheels keep turning and the scarf streams backward;
    • the scene includes interactive controls to pause, adjust pedaling speed, and switch between day and night;
    • after export at 1.75x speed, the SVG file plays smoothly when opened on its own with no network connection.

    From the days when this test was “nearly impossible to pass” to today, where “most models deliver a decent result,” it has gradually become the pass line for basic ability. But passing is not a perfect score, and the most interesting lost points usually hide in the details.

    3.3 Technical Breakdown: What the Model Got Right

    • Structured grouping. Wheels, pedals, pelican body, and scarf are separated into their own layers or groups and transformed independently. This is the prerequisite for making the animation move at all.
    • Correct transform origin. Wheel rotation must happen around each wheel’s own center, achieved either through transform-origin on the group or through a composite transform: translate to the center, rotate, translate back. Get the origin wrong and the wheels orbit the top-left corner of the canvas.
    • Animation decoupled from logic. requestAnimationFrame drives the main loop, and pedaling speed is abstracted into a single state parameter. Wheel rotation, pedal rhythm, and scarf sway are all derived from the same variable, so changing speed never leaves one part out of sync.
    • Self-contained and offline-capable. Graphics, styles, and scripts are all inlined with no external resources. Once exported as a single SVG file, double-clicking it is enough to run it.
    1. <svg id="scene" viewBox="0 0 800 400">
    2. <!-- Background: day/night palettes switched via class -->
    3. <rect class="sky" width="800" height="400"/>
    4. <!-- Bike: wheels rotate about their own centers -->
    5. <g id="bike">
    6. <g class="wheel" transform="translate(230,300)">
    7. <circle r="38" fill="none" stroke="#333" stroke-width="4"/>
    8. <line x1="-38" x2="38" stroke="#333" stroke-width="2"/>
    9. <line y1="-38" y2="38" stroke="#333" stroke-width="2"/>
    10. </g>
    11. <g class="wheel" transform="translate(430,300)">
    12. <circle r="38" fill="none" stroke="#333" stroke-width="4"/>
    13. <line x1="-38" x2="38" stroke="#333" stroke-width="2"/>
    14. <line y1="-38" y2="38" stroke="#333" stroke-width="2"/>
    15. </g>
    16. </g>
    17. <!-- Pelican: beak / body / wing / scarf ... -->
    18. <g id="pelican"> <!-- beak / body / wing / scarf ... --> </g>
    19. </svg>
    20. <script>
    21. (function () {
    22. // Single source of state: speed, pause, day/night
    23. const state = { speed: 1, paused: false, night: false };
    24. const wheels = document.querySelectorAll('.wheel');
    25. let angle = 0, last = performance.now();
    26. function tick(now) {
    27. const dt = (now - last) / 1000;
    28. last = now;
    29. if (!state.paused) {
    30. // Uniform angular velocity: scaling speed adjusts everything at once
    31. angle = (angle + 200 * state.speed * dt) % 360;
    32. wheels.forEach((w) =>
    33. w.setAttribute('transform', `translate(${w.dataset.x},${w.dataset.y}) rotate(${angle})`)
    34. );
    35. }
    36. requestAnimationFrame(tick);
    37. }
    38. requestAnimationFrame(tick);
    39. // Pause / speed / day-night control logic ...
    40. })();
    41. </script>

    3.4 Common Failure Modes

    • Geometric distortion: connection points between wings, legs, handlebars, and pedals drift, so parts intersect while moving.
    • Out-of-sync animation: the wheels turn while the pedals stay still, or wheel speed does not match the pedaling rhythm.
    • Rigid decorative elements: scarves, ribbons, and clouds lack delay and inertia, making the whole scene feel mechanical.
    • Feature piles with messy structure: all the controls are there, but styles and scripts are intermixed, making later edits painful.

    4. Case Two: Pure-Code Light and Shadow Rendering — Where the Real Gap Appears

    4.1 Task and Prompt

    If SVG animation tests the basics, then “draw a photo-like image with light and shadow using pure code” is the watershed:

    1. Using any coding technique you know, render an image of interlacing light and shadow
    2. that brings out the quiet atmosphere of afternoon light and shadow. Make the image as
    3. close to perfect as possible — pursue the extreme.

    The prompt deliberately names no tech stack, no renderer, and no compositional details, leaving “how to draw it” entirely to the model. The model under test chose a browser-side real-time rasterization route, built an afternoon room with a graphics API, and rendered a separate version on the CPU. Light from the window grid slants in, passes over a small round stool, and reaches the bottom edge of the frame. The overall impression was decent.

    Measured against a photographic standard, however, the problems showed up: a few black lines in the corner of the wall are harsh and razor-sharp, and the potted plant and floor look more like a temporary set. For comparison, a recently released frontier model given the same prompt implemented its own path tracer in a systems programming language and rendered for a long time; its output has visibly more natural transitions. Another developer’s result, produced with a frontier code execution environment, was also clearly better than the model under test.

    4.2 Technical Comparison: Two Rendering Routes

    Dimension Real-time rasterization Offline path tracing
    Principle Project the scene onto the screen and shade pixel by pixel, approximating reflections and shadows Cast rays randomly from the camera into the scene, bounce multiple times, and accumulate contributions
    Indirect lighting Relies on approximations such as ambient light and screen-space effects Naturally models multiple bounces and color bleeding
    Shadows Shadow-map sampling, hard edges Area-light sampling, naturally soft edges
    Materials Physically based shading models plus parameter tuning Reflection models based on micro-surface behavior
    Time cost Milliseconds, interactive in real time Minutes to hours, converging with sample count
    Strength Smooth interaction, easy cross-platform deployment High quality ceiling, close to a real photograph

    The gap is not about whether the image is neatly drawn. It is about whether light transport in the real world is actually simulated. What makes an afternoon room compelling is precisely the soft gradation of light and shade created after sunlight bounces off walls and floors several times — exactly where the rasterization route is structurally weak.

    4.3 Root-Cause Analysis

    1. No indirect light, so shadows go dead black. The harsh black lines in the corner usually mean only direct lighting was computed, with almost nothing handling the soft falloff produced by light bouncing off walls and floors. In a real room, even the darkest corner is still lit by light reflected from surrounding surfaces.
    2. Materials lack micro detail. The plant and floor look like set dressing because the surfaces have no roughness variation, no normal detail, and no contact shadows. The texture of a real object comes from irregularity at the microscopic scale — the hardest thing to express in a few lines of code.
    3. Composition is “correct but flat.” The light does land at the bottom of the frame, but the distribution of subject, highlights, and shadows has no hierarchy or rhythm, so the intended “tranquility” never forms.
    4. Engineering trade-offs were never weighed. The real-time route is smooth to interact with but cannot reach photographic quality; the offline route delivers quality at the cost of long render times. The model defaulted to the former without explaining why.

    4.4 Actionable Improvements

    • Rendering technique: if the goal is a still image, there is no reason to be bound to a real-time graphics API. In environments that support native compilation, a simplified path tracer or ray tracer written in code, combined with multisampling and denoising, raises quality substantially.
    • Lighting model: make light sources area lights rather than point lights and apply Monte Carlo sampling over the emitter surface, so shadow edges soften from hard to soft naturally. Add a hemisphere-sampled ambient approximation to supply indirect light.
    • Material expression: use procedural noise for roughness, normals, and texture variation; give the floor reflections and a clear-coat layer; add contact shadows where objects meet the floor.
    • Tone control: accumulate lighting in high dynamic range first, then apply a single tone-mapping pass, so highlights do not clip and shadows do not go black. This is one of the keys to the natural gradation in that photographic reference output.
    • Composition guidance: state the camera position, focal length, and eye height explicitly in the prompt, and ask for rule-of-thirds or leading-line composition.
    1. // Illustrative: area light sampling to approximate soft shadows with Monte Carlo
    2. // A simplified path-tracing idea: average many sample points; more samples, smoother result
    3. float softShadow(vec3 p, vec3 n, vec3 areaCenter, vec3 areaU, vec3 areaV) {
    4. float sum = 0.0;
    5. for (int i = 0; i < SAMPLES; ++i) {
    6. // Pick a uniform point on the rectangular light surface
    7. vec3 lp = areaCenter + (rand() - 0.5) * areaU + (rand() - 0.5) * areaV;
    8. vec3 l = lp - p;
    9. float dist2 = dot(l, l);
    10. l = normalize(l);
    11. float ndl = max(dot(n, l), 0.0);
    12. // Intersect along the ray; contribute 0 if occluded
    13. float vis = occluded(p + n * EPS, l, sqrt(dist2)) ? 0.0 : 1.0;
    14. // Cosine term + inverse-square falloff approximate physical lighting
    15. sum += vis * ndl / (dist2 + EPS);
    16. }
    17. return sum / float(SAMPLES);
    18. }

    5. What This Means for 3D Understanding and Agentic Ability

    Case one and case two correspond to 2D and 3D visual expression, and together they test whether the vendor’s claimed gains in “world knowledge and multimodal perception” actually landed. From these results:

    • On the 2D side: the pelican’s proportions, riding posture, and choice of scene elements are basically sound, indicating solid visual common sense about everyday things.
    • On the 3D side: the room’s spatial layout and the direction of light through the window grid are correct, so basic spatial reasoning is present. But physical realism in lighting and materials is clearly insufficient — “knowing what a room looks like” and “understanding how light travels” are still two different things.

    Agentic ability deserves separate testing. If these tasks were handed to an agent, three points matter most:

    1. Task decomposition: can it break the work into stages such as blocking out the scene, building lighting, adding materials, and post-processing?
    2. Tool use: can it discover problems on its own using feedback like browser screenshots and console logs?
    3. Sustained iteration: given qualitative feedback such as “the corner shadows are too hard,” can it locate the specific code responsible and fix it?

    This is also the key dimension for judging whether a model can handle real engineering work.

    6. Prompts and Workflow: Making “Aesthetics” an Executable Instruction

    Aesthetics is weak in models largely because it is implicit knowledge that resists quantification. Models are better at completing a feature checklist than at judging whether an atmosphere has been achieved. The fix is to decompose aesthetics into executable constraints.

    1. [Goal] Output a single-file HTML page that renders a still image of an "afternoon room" in pure code.
    2. [Scene] A west-facing old room with parquet wood flooring, a four-pane wooden window,
    3. a small round stool, and a potted green plant.
    4. [Light] Low-angle afternoon sun around 4 p.m. passing through the window grid and casting
    5. parallelogram light patches on the wall and floor; the light source is a rectangular
    6. area light with soft shadow edges; shadow areas keep bounced ambient light and must
    7. not go dead black.
    8. [Materials] Floor: rough lacquered finish with slight reflection and wood grain; wall: matte
    9. gray plaster with subtle bumps; metal: high reflectivity.
    10. [Rendering] An offline ray tracer or path tracer is acceptable, at least 256 samples per pixel,
    11. followed by tone mapping.
    12. [Composition] Low camera position, rule of thirds, light patches as the visual subject,
    13. the small stool placed at the boundary between light and shade.
    14. [Acceptance] Highlights not blown out, detail visible in shadows, window-grid projection edges
    15. showing gradual transitions.

    A three-step iteration workflow pairs well with this:

    1. Generate a runnable first version and confirm scene layout and camera angle.
    2. Ask the model to optimize only lighting and shadows, without touching geometry.
    3. Finish with material detail and tone mapping.

    Staged constraints sharply reduce the “change one thing, break everything” loop and make it easier to localize problems to a specific stage.

    7. From Evaluation Method to Engineering Practice

    Individual developers should start with low-friction tasks — info cards, form validation, vector icons, lightweight interactive animation — then move up to chart visualization and 3D presentation. Build a personal library of prompt-and-output pairs and keep accumulating experience.

    Enterprise teams should build scenario-based evaluation sets covering business code generation, data visualization, 3D visualization, and agentic workflows, and should focus on stability and consistency across batches of tasks rather than a handful of impressive samples. Metrics worth tracking:

    • first-pass success rate;
    • average number of debugging rounds;
    • generation time and resource cost per task.

    In the cloud, a closed loop can be assembled from general-purpose capabilities: model services for inference, a container platform to orchestrate batch evaluation runs, object storage to archive artifacts and screenshots, and log plus monitoring services to track failure rates and latency shifts. Baidu AI Cloud provides general-category support for this kind of model inference and evaluation engineering work, which teams can use as the basis for their own model evaluation pipeline.

    One point deserves emphasis: real-time and offline rendering are not either/or. The real-time route suits product demos and online previews that need user interaction; the offline route suits posters and promotional material where image quality is the goal. A mature approach runs both in parallel: “real-time preview plus offline final render.” Let the model produce an interactive real-time version to validate composition quickly, then switch to offline rendering for the deliverable once the approach is settled.

    8. Conclusion

    From “a pelican riding a bicycle” to “draw afternoon light and shadow in pure code,” code generation evaluation is shifting from “does it run” to “does it look good, look real, and hold up.” These tests show that the new generation of models is already solid on basic animation, spatial common sense, and engineering conventions — in some cases delivering a single-pass result with full interaction and offline capability. But in the deep water of physical realism and aesthetic expression, a clear gap to the industry’s frontier remains.

    That is not surprising. Aesthetic and physical intuition are implicit capabilities built over a long time, and no single-dimension optimization closes them. For developers, rather than waiting for one omnipotent model, the better move is to turn evaluation methods, prompt engineering, and iteration workflows into team assets, so human and model capabilities complement each other: people define the aesthetic and the standard, and the model lands that standard efficiently.