How to Benchmark LLM Code and 3D Rendering Capabilities: A Repeatable Evaluation Method
This article lays out a practical way to test how well large language models generate code and 3D graphics. It covers a four-layer task design (syntax and structure, interaction, visual and physical plausibility, performance), five rules that keep comparisons credible and reproducible, and two verification points that are easy to miss: hidden CDN dependencies and interaction logic that only looks right. Two hands-on cases anchor the method — SVG animation generation using the well-known "pelican riding a bicycle" benchmark, and an open-ended "draw a photograph with code" task — with a comparison of rasterization, CPU path tracing and offline rendering, a minimal cosine-sampled path-tracing skeleton, and notes on where models still fall short: long-range code coherence, physics and optics intuition, and composition. It closes with advice for developers on routing tasks to model tiers by complexity, building an internal benchmark set, and running the evaluation pipeline in the cloud.
1. Why “code + 3D” has become the dividing line for flagship models
Over the past couple of years, major vendors have released preview flagship models in quick succession, some rolled out anonymously on coding platforms. Guessing which model was behind a given result became a community pastime. More interesting is that the focus of evaluation has moved well past “can it write a sorting algorithm” to the far more demanding combination of code plus graphics:
- Correctness is verifiable. Whether the code runs and whether the image respects basic physical common sense can be judged without relying on subjective impressions.
- The bar is high overall. A single task exercises syntax, math, graphics fundamentals, aesthetic judgment, and long-range planning at the same time.
- Results are durable. Generated SVG, HTML and similar files can be kept indefinitely, which makes side-by-side comparison and version management straightforward.
Preview release cadence has a similar flavor: many vendors open up capacity in roughly the same time window. Deliberate or coincidental, it has made head-to-head comparison between models unusually convenient. Still, when a fresh preview lands, it is more useful to run your own tests than to be led along by official phrasing about “expanded world knowledge,” “stronger multimodal perception,” or “better coding and agentic capabilities.” The approach below applies equally to AI coding assistants and to LLM services that compete primarily on code generation.
2. The evaluation method: making comparisons credible and reproducible
2.1 Design tasks across four layers
| Layer | What it tests | Typical task |
|---|---|---|
| L1 Syntax and structure | Tag closure, valid attributes, complete logic | Generate a standalone SVG/HTML file that runs on its own |
| L2 Interaction and dynamics | Animation, events, state management | Adjustable parameters, play/pause, day/night toggle |
| L3 Visual and physical plausibility | Lighting, materials, geometric sanity | Static render of light and shadow in an afternoon room |
| L4 Performance and engineering | Frame rate, payload size, resource use | Real-time WebGL scene, animation performance budget |
2.2 Five rules for reproducibility
- Fix the environment. On the same testing platform, select the highest reasoning tier for every model (for example, the “extremely high” setting), and record version numbers and timestamps.
- Use one prompt. Give every model the exact same original prompt with no model-specific tuning. Otherwise you end up feeding answers.
- Allow only one round of follow-up. Iteration is normal in real development, but for fairness the first-round output is what gets scored.
- Validate exports outside the environment. Copy the exported files to a browser with no network access and no toolchain, and run them there. This confirms the result depends on nothing external.
- Keep the process record. Generation time, line count, error count, and render duration all get saved so results can be reviewed later.
2.3 Two verification points people tend to miss
- Resource loading. Models routinely pull a script or a web font off a CDN almost as an afterthought, which quietly breaks offline behavior.
- Interaction logic. Many reviewers only ask “does it look right” and never check whether the pause button actually pauses or whether the speed multiplier takes effect. Interaction quality matters as much as visual quality.
3. Case 1: SVG animation generation (the “pelican on a bicycle”)
“Generate an SVG animation of a pelican riding a bicycle” is one of the most widely circulated generative-UI benchmarks. Early on, only a few models could produce something passable; today mainstream models generally deliver decent results. That is precisely the point: the benchmark is close to saturated. It works as a pass/fail screen, not as a way to separate strong models from each other.
At the highest reasoning tier, the output was a pelican in a red hat cycling along the coast: the wheels keep turning and the scarf trails behind. The page also included a play/pause button, a pedaling-speed slider, and a day/night toggle below the canvas. Setting the speed to 1.75x and exporting, then opening the SVG on its own with the network disabled, the animation still ran smoothly.
A strong SVG animation output usually has five characteristics:
- Clear layered structure. Wheels, frame, bird body, and scarf are grouped by layer, so each can be located and modified individually.
- Parameterized animation. Rotation speed, cycle length, and colors are managed through variables rather than hard-coded numbers scattered through the markup.
- Correct transform origins. Rotation is where animations most often go wrong; here is a reliable pattern.
<svg viewBox="0 0 320 200" xmlns="http://www.w3.org/2000/svg" role="img"aria-label="A pelican in a red hat riding a bicycle along the coast"><defs><!-- A single variable controls tempo and palette, so values can be tuned from outside --><style>:root { --speed: 1; --sea: #3c8dbc; --sun: #ffd76a; }.wheel-spin { animation: spin calc(1s / var(--speed)) linear infinite; }@keyframes spin { to { transform: rotate(360deg); } }</style></defs><!-- Background: the sky color is driven by the day/night variable --><rect class="sky" width="320" height="200" fill="var(--sky-color, #bfe5f5)"/><rect x="0" y="140" width="320" height="60" fill="var(--sea)"/><!-- Wheel: rotate(angle cx cy) spins around a given center directly, with no transform-origin needed --><g class="wheel"><circle cx="110" cy="160" r="18" fill="none" stroke="#333" stroke-width="3"/><circle cx="110" cy="160" r="18" fill="none" stroke="#333" stroke-width="1"stroke-dasharray="4 6"><animateTransform attributeName="transform" type="rotate"from="0 110 160" to="360 110 160" begin="0s"dur="1s" repeatCount="indefinite"/></circle></g><!-- Scarf: two keyframes fake the sine-like flutter of fabric trailing backward --><path class="scarf" d="M140 90 q20 -6 34 2 q14 8 28 2" fill="none" stroke="#e05b4b" stroke-width="6"><animate attributeName="d" values="M140 90 q20 -6 34 2 q14 8 28 2;M140 90 q20 -10 34 -2 q14 8 28 0;M140 90 q20 -6 34 2 q14 8 28 2"dur="1.2s" repeatCount="indefinite"/></path></svg>
- A closed interaction loop. The pause button actually calls a native API such as
svg.pauseAnimations()instead of only being drawn. - Offline and self-contained. Styles, scripts, and fonts are all inlined, with no reference to any external resource.
4. Case 2: “Drawing” a photograph with code
4.1 The task
“Using any code technique you know, draw a picture of interlocking light and shadow that captures the quiet atmosphere of an afternoon. Push it as far as you can.” This is a classic open-ended rendering task. The prompt contains no technical constraints, so the model has to make the technical choices itself: real-time or offline rendering? Rasterization or ray tracing?
4.2 Three routes under the same prompt
Public work on this task shows a wide spread. One example uses a pure-C++ path tracer for offline rendering: an afternoon room where window-pane shadows sweep across the wall, a chair, and parquet flooring, taking roughly three hours to render. Another uses pure JavaScript and WebGL for real-time rendering, with a CPU offline render alongside it — the patch of light slants in through the window panes, crosses a small stool, and lands near the bottom edge of the frame, all interactive and smooth. Yet another, produced in a specialized coding environment, yields a static image that is clearly better than the WebGL version above.
Given one identical prompt, the answers from different models can look like two entirely different things. That is the most direct demonstration of the gap in code-generation ability.
4.3 Trade-offs among the three routes
| Route | Strengths | Cost |
|---|---|---|
| Rasterization (GPU, real-time) | Frames in milliseconds, parameters adjustable interactively | Ray-traced effects and global illumination have to be approximated, which often makes the scene look like a stage set |
| CPU path tracing | Softer shadows, color bleed, and caustics look more believable | Tens of minutes to hours per render; sampling is mandatory just to preview |
| Offline rendering | Highest achievable image quality | Complex setup, high debugging cost |
For an open-ended prompt like this one, the route a model chooses is itself a signal of its planning ability. There is no single correct answer; what matters is whether the choice fits the goal of the task.
4.4 A minimal ray-tracing skeleton
// Cosine-weighted hemisphere sampling: decides the random direction a diffuse surface reflects towardfunction cosineSample(n, rng) {const r = Math.sqrt(rng());const phi = 2 * Math.PI * rng();const t = new Vec3(r * Math.cos(phi), r * Math.sin(phi), Math.sqrt(1 - r * r));// Transform the local coordinates (t.x, t.y, t.z) into the tangent space defined by normal nreturn toWorld(t, n);}// Diffuse path tracing with importance samplingfunction trace(ray, scene, depth) {if (depth <= 0) return new Vec3(0, 0, 0);const hit = scene.intersect(ray); // returns the nearest hit, its normal, and its materialif (!hit) return scene.background(ray); // sky light, instead of a flat "no ambient light" backgroundconst color = hit.material.emissive || new Vec3(0, 0, 0);const albedo = hit.material.albedo; // surface reflectanceconst nextDir = cosineSample(hit.normal, rng);const pdf = Math.max(nextDir.dot(hit.normal), 0) / Math.PI; // sampling probability densityconst throughput = albedo.multiplyScalar(Math.PI); // the cosine term cancels the pdf, leaving only albedo// Offset escaping rays so self-intersection does not produce black specklesconst nextRay = new Ray(hit.point.add(hit.normal.multiplyScalar(1e-3)), nextDir);return color.add(trace(nextRay, scene, depth - 1).multiply(throughput));}
The code is short, but it covers several details that decide whether the image works:
- Random sampling. Averaging multiple samples per pixel is the only way to remove the noise inherent to path tracing.
- Energy conservation. Pairing the diffuse BRDF with cosine sampling keeps brightness from blowing out or washing to gray.
- Environment light and escaping rays. Sky light supplies soft ambient illumination so shadows do not go dead black, while the ray offset prevents self-intersection artifacts.
- Precision versus time. Sample count, bounce depth, and resolution together determine render time. For offline renders that take hours, the common practice is layered rendering: render a low-sample preview first, confirm the composition, then raise the precision.
5. Explaining the gaps: three capability shortfalls visible in testing
Comparing your own results against high-quality work on the same task gives a reasonably objective view of where the gaps are, with one caveat: models iterate quickly, so any single test is only a snapshot in time. Even so, a few directions worth watching emerge.
5.1 Long-range code coherence
The SVG case is decided in a single turn; path tracing is a long-range planning problem. The latter requires holding a coordinate system, a lighting model, a material system, and a render loop consistent across thousands of lines. One mistake anywhere degrades the whole result. Whether a model can keep the output internally consistent largely decides the outcome.
5.2 Physics and optics fundamentals
“Hard black lines in the room corners” and “the potted plant looks like it was just placed on a set” are both symptoms of missing global illumination. In reality, light bounces between objects and color bleeds across surfaces, and shadow edges soften into a penumbra that scales with the size of the light source. Whether a model can express correct modeling of soft shadows, ambient occlusion, and color temperature shifts in code is a dividing line in graphics skill.
5.3 Aesthetics and composition
Subjective goals like “light and shadow atmosphere” cannot be settled by “it runs / it errors,” yet this is where flagship models separate most clearly. Whether the transition from the bright window area into the dark interior feels natural, and whether the tonal layering is rich, is a real gap between ordinary models and the best ones.
6. Practical advice for developers
6.1 Match the model tier to the task
The cost difference between reasoning tiers gets multiplied across batch calls. One option is to route by complexity: single-file tasks like SVG/HTML go to a standard tier, while tasks involving complex geometry, physical simulation, or large-scale refactoring move to the highest tier. This is the same idea as the general engineering practice of routing requests by task complexity.
6.2 Build your own small benchmark set
Examples circulating online are easy for training data to have “memorized.” Teams are better off constructing a benchmark from their own business cases, where each case carries a prompt, acceptance criteria, and a reference implementation, and is re-run on a fixed schedule in a fixed environment. That keeps judgment from being pulled around by whichever single example is trending.
6.3 Run the evaluation pipeline in the cloud
Enterprise-grade evaluation does not need to depend on a personal laptop. General-purpose cloud capabilities are enough to productionize it:
- Object storage: Keep generated artifacts, renders, and evaluation reports for each version in one place, with the full change history.
- Container platform: Start an isolated, reproducible runtime per case, which contains dependency conflicts.
- Logging and monitoring with alerts: Collect generation time, error rate, and pass rate per model, and set threshold alerts to catch shifts in capability or cost early.
- A unified LLM gateway: If an organization connects to different models through a single LLM service, such as the offering from Baidu AI Cloud, authentication, quotas, and call statistics can be managed centrally, which makes head-to-head comparison and cost control easier.
7. Closing thoughts
Progress in LLM code and 3D generation is hard to miss. “A pelican riding a bicycle,” once unsolvable for most models, is now a pass/fail bar, and work that is fully interactive and works offline is no longer rare. On harder tasks like “draw a photograph with code,” though, the gap between the top flagship models and the rest is still clear, and closing it will not happen overnight.
For developers, the better investment is not agonizing over one anonymous release or one leaderboard. Build your own evaluation method and benchmark set, and turn “is this model good?” into “which cases pass in my scenario, and at what cost?” Measuring models against engineering standards is far more valuable than chasing hype.
