How to Scientifically Evaluate LLM Code Generation and Multimodal Capabilities

    As general-purpose LLMs ship at an accelerating pace, developers face a harder question than "which model is best": whether a given model can reliably and reproducibly complete a task in their own scenario. This article presents a structured evaluation methodology built on standardized task sets, multi-dimensional scoring criteria, and reproducible run conditions. Using two representative code tasks—an SVG animation that tests structured output and interactive logic, and a pure-code photorealistic render that tests math and graphics skills—it shows how to turn vague "feels usable" impressions into verifiable conclusions, and closes with a practical checklist developers can apply directly.

    Introduction: Models Iterate Fast, but Evaluation Often Runs on “Gut Feeling”

    A new general-purpose LLM seems to ship every few weeks, and each release arrives with dense marketing, benchmark scores, and hands-on impressions from the community. But for developers, the genuinely useful question is not “which model is strongest” but rather whether a model can reliably and reproducibly complete a defined task in your own scenario. Scattered screenshot comparisons and subjective impressions are heavily influenced by prompt wording, inference settings, and the runtime environment, which makes them hard to distill into comparable conclusions.

    So instead of chasing single-point benchmarks, it pays to build a structured evaluation method: a standardized task set, clear multi-dimensional scoring criteria, and reproducible run conditions. Together these turn “does it feel good to use” into “which capabilities meet the bar and which fall short.” Below, two highly representative code tasks illustrate how this method works in practice.

    1. The First Principle: Standardization and Reproducibility

    Before running any test, fix three variables, or your results cannot be compared side by side:

    • Fixed prompts: Use exactly the same input for a given task across all candidate models, avoiding the noise introduced by “prompt tuning.”
    • Fixed inference tier: If the platform offers multiple reasoning-effort levels (for example, standard / high / very high), record and keep them consistent. Never compare results produced at different tiers.
    • Independently verifiable output: Prefer artifacts that can be exported as standalone files and opened or run offline (such as a single SVG file or a self-contained script), so that “it still runs with the network off and on a different machine” holds true.

    Once these three are locked down, the evaluation becomes reproducible, and any subsequent comparison actually means something.

    2. Task One: SVG Animation — Testing Structured Output and Interactive Logic

    “Ask the model to draw a pelican riding a bicycle” has become a classic entry-level benchmark in the community. Its value is not whether the picture looks nice, but that in a single pass it exercises several capabilities at once: can the model produce syntactically correct SVG, can it use tags like animateTransform to create sensible motion, can it add interactions such as pause, speed control, and day/night switching, and is the final export self-contained and able to run offline?

    A solid result usually looks something like this—the wheels rotate continuously around their own centers, and the whole file depends on no external resources:

    1. <svg viewBox="0 0 200 120" xmlns="http://www.w3.org/2000/svg">
    2. <!-- Wheel: spoke group rotates around center (100,80) -->
    3. <g>
    4. <circle cx="100" cy="80" r="30" fill="none" stroke="#333" stroke-width="4"/>
    5. <line x1="100" y1="50" x2="100" y2="110" stroke="#333" stroke-width="2"/>
    6. <animateTransform attributeName="transform" attributeType="XML"
    7. type="rotate" from="0 100 80" to="360 100 80"
    8. dur="1s" repeatCount="indefinite"/>
    9. </g>
    10. </svg>

    Score each item rather than judging the overall look:

    1. Structural correctness: Are the SVG tags properly closed, are the path coordinates reasonable, and is there no syntax error?
    2. Animation plausibility: Does the motion match physical intuition (wheel rotation direction, whether the parts move together naturally)?
    3. Interaction completeness: Are controls such as pause/play, speed adjustment, and day/night switching actually implemented and functional?
    4. Self-containment: When exported and opened on its own, does it run normally with the network disconnected? This is a point where many models lose marks.

    Today’s mainstream models mostly produce decent results on this task, so it is better suited as a baseline threshold test: if a model already produces syntax errors or stiff animation at this step, its code generation ability is not yet up to standard.

    3. Task Two: Pure-Code Photorealistic Rendering — Testing Math and Graphics Skills

    If SVG animation tests “structured output,” then rendering an image so realistic it is hard to distinguish from a photograph using code alone is a markedly harder step up. The prompt for this kind of task is usually something like: “Using any coding technique you know, draw a scene with dramatic light and shadow, aiming for maximum realism.”

    The difficulty lies in this: the model must combine graphics knowledge—lighting models, geometric modeling, sampling, and denoising—and translate physical intuition into runnable rendering logic through code. Different technical approaches lead to obvious differences:

    • Path tracing: High physical accuracy, with natural global illumination, soft shadows, and smooth light-to-dark transitions—but computationally expensive, often requiring long render times.
    • Real-time rasterization (such as WebGL): Fast to run and convenient for interactive preview, but typically less refined than path tracing for diffuse reflection, indirect lighting, and soft shadows.

    When evaluating, watch a few details that “expose true skill level”:

    • Tone transitions: From a bright window to a dim interior, are there obvious banding steps or posterization, or is the falloff continuous and natural?
    • Hard edges and artifacts: Do walls and object edges show unnatural hard black lines? Are floor textures, plants, and so on just crude “texture-mapped” scenery?
    • Light-shadow consistency: Do the direction and length of shadows agree with the light source position, and do reflections and shadows contradict each other?

    This task clearly separates models: whether sampling, convergence, and denoising are implemented correctly directly determines whether the output looks “photographic” or like “a scene built from blocks.” It is also a touchstone for whether a model’s coding ability reaches the state of the art.

    4. Extended Dimension: Agentic Ability and Tool Use

    Beyond single-pass code generation, modern evaluation should also cover a model’s agentic ability: whether it can autonomously break down steps across multiple turns, call available tools, and self-correct based on intermediate results. One way to test this is a composite task that involves several sub-steps—retrieval, computation, file read/write, and so on—to see whether the model can plan correctly and, on failure, roll back and retry rather than aborting at the first sign of an error.

    5. How to Interpret Results: Score by Category, Not by a Single Ranking

    A single “total score” or a lone screenshot easily misleads decisions. A more robust approach is to build a scoring matrix: score separately by task category (structured output, graphics rendering, tool use, and so on) and by evaluation dimension, then apply weights based on your own business priorities. This reveals a model’s relative strengths and weaknesses in programming, interaction, and visual expression, while avoiding the bias of “drawing a conclusion from one good showing.”

    6. A Practical Checklist for Developers

    1. Use 5–10 standard tasks covering different capability dimensions instead of scattered spot checks.
    2. Fix the prompt and inference tier, and record the version and runtime environment.
    3. Prefer artifacts that can be exported and reproduced offline as the deliverable.
    4. Build a per-category scoring sheet and avoid a single overall ranking.
    5. Sample the same task multiple times, focusing on the stability and variance of the output, not just the single best run.

    Conclusion

    Ultimately, a model’s capability comes down to a verifiable fact: whether it reliably meets the bar on a specific task. By establishing a standardized, reproducible evaluation process that scores by category, developers can see through marketing and benchmark scores to make technical choices that fit their own scenarios—which is far more valuable in the long run than chasing “who just released a new version.”