Evaluating LLM Code Generation for Animated Vector Graphics and Lighting Rendering

    This article evaluates large language model (LLM) code generation through two visual coding tasks: a self-contained animated vector graphic with interaction, and a code-only interior lighting scene. It describes a consistent evaluation setup, including identical prompts, a fixed runtime, multi-dimensional scoring, and repeated sampling; examines task requirements, common failure modes, and rendering trade-offs; and separates functional correctness from visual realism. The findings show that models can already deliver usable rule-based animation and interaction, but still face gaps in physically consistent, nuanced lighting. The article concludes with engineering practices for regression testing, prompt layering, security review, and human aesthetic review.

    1. Why Use Visual Tasks to Evaluate Coding Ability

    Code generation is often measured with algorithm problems or function-completion tasks, but these metrics do not fully reflect a model’s ability in graphics, motion, and interface presentation. Visual tasks place two questions in the same test: whether the code runs, and whether the result matches expectations. This is closer to real work in front-end development and graphics engineering.

    This article uses two common task types: generating an animated vector graphic, and drawing an interior scene with lighting and shadows using code only.

    2. Evaluation Method Design

    1. Unified prompts: Each model receives the same task description, with no extra prompts added, to avoid giving any model an artificial advantage.
    2. Fixed runtime environment: Record the browser version, rendering method, and export format so results can be reproduced.
    3. Multi-dimensional scoring: Score each output across functional completeness, visual quality, interaction correctness, and engineering usability.
    4. Repeated sampling: Generate each prompt at least several times and observe stability rather than looking only at a single best output.

    3. Task 1: Generate a Self-Contained Animated Vector Graphic

    The task asks the model to output an animated vector graphic file showing a character riding a bicycle, with basic interactive controls. Evaluation focuses on the following areas:

    1. Structural correctness: Are the graphic elements complete, and is the hierarchy reasonable?
    2. Animation logic: Does wheel rotation direction match the body’s movement speed, and do flowing elements have reasonable delay and easing?
    3. Interaction: Do controls such as pause, speed adjustment, and day/night toggle actually work?
    4. Portability: Does the file play correctly without a network connection?

    In one set of tests, the evaluated model generated a character wearing a red hat, with continuously rotating wheels, a scarf swaying in the direction of motion, and controls for pause, speed adjustment, and day/night switching. The exported file opened on its own and still played with the network disconnected. This shows that the model could integrate graphic description, animation timing, and interaction logic into a single self-contained file.

    The core wheel rotation can be written as follows, using standard vector animation syntax:

    1. <g>
    2. <circle cx="60" cy="150" r="40" fill="none" stroke="#333" stroke-width="6"/>
    3. <line x1="60" y1="150" x2="60" y2="115" stroke="#333" stroke-width="4"/>
    4. <animateTransform attributeName="transform" type="rotate"
    5. from="0 60 150" to="360 60 150" dur="1s" repeatCount="indefinite"/>
    6. </g>

    There are three common problems in this type of animation. First, wheel speed and displacement can become decoupled: the wheels spin quickly while the body barely moves. Second, repeated computation can cause jank, such as rebuilding the entire graphic tree on every frame. Third, file size can expand, as large amounts of repeated path data slow loading. These details should be checked item by item during evaluation.

    4. Task 2: Generate a Lighting Scene Using Code Only

    This task asks the model to choose any code-based technique and render an afternoon interior lighting scene. The difficulty is lighting. Light entering through a window creates clear light patches and shadows, while indirect light also exists among the walls, furniture, and floor. The transitions between light and dark must match physical intuition.

    There are two common implementation paths.

    1. Path tracing: Cast many rays from the camera and simulate multiple reflections between surfaces. The result can approach real photography, but the computation is heavy; offline rendering often takes minutes to hours.
    2. Rasterization plus approximated lighting: Use the graphics pipeline for real-time drawing and approximate lighting with methods such as shadow maps and ambient occlusion. This is fast and suitable for interaction, but shadow edges and indirect light often look artificial.

    In the test, the evaluated model chose the second path, built a room scene, and also output a separate CPU-rendered result. The observed behavior was as follows:

    • Beams from the window-pane projection entered from an upper angle and extended to the floor, with a clear sense of direction.
    • Hard black lines appeared at wall corners, indicating that contact shadows were not handled with enough nuance.
    • The potted plant and floor textures were too regular in scale and lacked signs of real-world use or wear.

    Sharp light-patch edges usually occur when the light source is treated as a point light or a directional light. If the implementation instead uses an area light and averages multiple sample points, the shadow edges will show a natural penumbra. The following pseudocode shows the core loop of path tracing to illustrate the idea of sample accumulation:

    1. for pixel in image:
    2. color = 0
    3. for s in range(samples_per_pixel):
    4. ray = camera.generate_ray(pixel)
    5. color += trace(ray, depth=0) # Recursively trace indirect light
    6. image[pixel] = color / samples_per_pixel

    The more samples taken, the less noise there is, and the longer rendering takes. Understanding this point makes it possible to judge the trade-off between an image’s quality and its computational cost.

    5. Capability Boundaries of the Two Tasks

    The two tasks examine different capabilities.

    • Animated vector graphics lean toward structured description and temporal logic. They depend on syntax correctness and state management, have clear rules, and are easier for models to bring to a usable level.
    • Lighting rendering leans toward physical simulation and aesthetic judgment. It requires overall trade-offs among light transport, materials, and composition; correct code alone is not enough to guarantee credible visuals.

    Therefore, “it runs” and “it looks realistic” are independent metrics. They should be recorded separately during evaluation, rather than allowing one composite score to hide weaknesses in either area.

    6. Engineering Practice Recommendations

    1. Include visual outputs in regression testing. Save reference images for rendered results and compare them using pixel differences or structural similarity metrics to catch quality regressions introduced by changes.
    2. Layer prompts. Split requirements into structure, constraints, and acceptance criteria. For example, specify image dimensions, light-source positions, and frame-rate limits to reduce ambiguity.
    3. Perform security reviews on generated code. Front-end animation code may include external resource references or dynamic scripts. Check sources and permissions before merging.
    4. Keep human aesthetic review. Model output can serve as a first draft, but composition, color tone, and detail still require review by design staff.

    7. Conclusion

    On visual tasks with clear rules and verifiable outputs, LLMs can already produce complete deliverables including animation, interaction, and offline playback. On tasks that require physical consistency and nuanced lighting, output quality still has gaps. Developers should choose models according to task type and use dimension-specific evaluation and engineering validation to integrate generated results reliably into project workflows.