Evaluating LLM Code Generation: From Task Design to Multidimensional Comparison
A practical evaluation of LLM coding capabilities requires more than benchmark scores or working demos. This article uses a claw-machine game to explain how to design a reproducible engineering task, define objective acceptance criteria, and compare models across completion time, token usage, repair efficiency, and code quality. It also explores a code-only visual art challenge as a separate test of cross-domain capabilities.
1. Why Coding Models Need a Rigorous Evaluation Method
As AI coding tools become more widely adopted, developers have more models to choose from. Yet public claims often emphasize benchmark scores or carefully selected demonstrations, neither of which necessarily reflects performance in real software projects.
A model’s usefulness in development depends on several capabilities working together: decomposing long tasks, maintaining state, coordinating changes across files, and correcting its own errors. A single benchmark rarely captures all of them.
A more reliable approach is to define a task with clear boundaries, objectively verifiable acceptance criteria, and sufficient engineering complexity, then have each candidate model complete it under the same conditions. Consistent metrics make the results easier to compare.
This article uses a claw-machine game to illustrate how to design such an evaluation and interpret its results.
2. Designing the Evaluation Task
Test Multiple Engineering Capabilities Together
A claw-machine game may look simple, but implementing one involves several classic frontend engineering challenges:
- Physics-based interaction: The claw’s descent, forces during gripping, and the rope’s swinging motion require simplified physical models, such as spring–damper systems or interpolated curves that approximate physical behavior. This tests whether a model can implement numerically reasonable simulation logic without external dependencies.
- State-machine design: The sequence—idle → move → descend → evaluate grip → lift → release → settle the result—requires complete entry and exit conditions. Exceptional paths, such as an empty grab or a dropped toy, must also be handled. A missing transition can leave the interface stuck.
- Animation and rendering: Toy collisions, claw-opening and closing animations, and scoreboard updates require code that balances frame-loop performance with coordinated animation timing.
A single task that exercises several capabilities at once can reveal weaknesses more effectively than ten isolated code snippets.
Make Acceptance Criteria Objectively Verifiable
Define the functional checklist before running the evaluation. For example:
[ ] The user can move the claw left and right with smooth motion.[ ] Claw-opening and closing animations are synchronized with the descent.[ ] Grip detection uses the intersection between the closing claw and toy bounding boxes.[ ] Empty grabs, successful grabs, and mid-lift drops all return to the correct state.[ ] Scoring and remaining-attempt logic form a complete loop with no deadlock paths.
A checklist replaces vague judgments such as “it seems fine” with observable results.
3. Evaluation Metrics: Look Beyond Whether the Code Runs
Completion Time and Generation Speed
Total task time depends on both model output speed, measured in tokens per second, and the amount of human intervention required.
A preview-stage model may generate output slowly, extending its completion time. However, generation speed and first-pass correctness measure different things. Serving and generation performance can improve through engineering iterations, while first-pass correctness is a more direct indicator of coding capability.
Record these separately rather than treating elapsed time as a single measure of model quality.
Token Usage and Cache Hit Rate
Total token processing volume provides insight into how a model uses context during a task.
One pattern worth examining is the high input-cache hit rate observed among strong performers: in the referenced evaluation, some exceeded 95%. Such results can help assess context reuse and efficiency, but cache hit rate should be interpreted alongside overall token volume.
Also track output tokens. More output does not necessarily mean better results. Verbose rewrites and repeated trial and error can increase costs without improving the final implementation.
Effective Code and Revision Cycles
Some of the most useful differentiators are easy to overlook:
- Action granularity: Strong models tend to advance the task through larger, coherent changes, sometimes updating several related files together. Less efficient models may make frequent small edits and repeatedly revisit the same issues.
- Error-driven rewrite rate: After a runtime error, does the model identify and repair the specific cause, or rewrite large sections and introduce new problems? The average number of lines edited per error repair can help quantify this behavior.
- Effective engineering code ratio: What proportion of the final submission directly implements the required application logic? A higher proportion can suggest that the model produced a more complete solution with fewer unnecessary additions.
The following anonymized figures illustrate the data collected in one evaluation:
| Model | Completion Time (Minutes) | Total Tokens Processed | Output Tokens |
|---|---|---|---|
| Model A | 17 | 150,000 (96% cached) | 130,000 |
| Model B | 46 | 440,000 (99% cached) | 281,000 |
| Model C | 68 | 4,000,000 (98% cached) | 198,000 |
| Model D | 75 | 2,690,000 (95% cached) | 55,000 |
Model D produced the fewest output tokens—55,000—and was described as producing concise code. However, it also had the longest completion time in this table. The figures therefore should not be read as evidence that shorter output alone guarantees faster delivery.
The broader point is to examine whether concise output reflects effective, first-pass implementation rather than assuming that output volume measures progress. Total token processing volume and elapsed time also do not have a simple linear relationship; cache hit rates must be considered when interpreting them.
4. A Cross-Domain Challenge: Creating Visual Art Entirely Through Code
An additional task can test capabilities beyond conventional software engineering. One example is:
Create an ink-wash-style painting using only Python, NumPy, and PIL, without an image-generation model or a render-first, trace-or-sample-later shortcut.
This challenge examines three capabilities:
- Algorithmic abstraction: Can the model translate ink diffusion into a computational process? Possible approaches include layering Gaussian kernels to represent ink density and using random walks or noise fields to produce bleeding along brushstroke edges.
- Aesthetic parameterization: Can concepts such as negative space, dry-brush texture, ink density, and wetness become adjustable parameters rather than hard-coded pixels?
- Composition: Does the code create a deliberate balance between painted forms and empty space, rather than mechanically filling the canvas?
Score technical correctness and visual quality separately. This prevents aesthetic preferences from concealing algorithmic defects.
5. A Reusable Evaluation Workflow
1. Keep the Task Fixed
Use exactly the same prompt and execution environment for every model, including the same coding tool and version.
2. Record the Process
Enable session logging. Capture input and output token counts for each interaction, error stack traces, and the methods used to repair failures.
3. Use Blinded Acceptance Review
Have reviewers who did not participate in the evaluation assess the finished implementation against the functional checklist. Keep model identities hidden during review to reduce subjective bias.
4. Repeat Each Run
Run the same task at least two or three times to assess consistency. Variation between runs can sometimes reveal more than the average result.
5. Normalize Costs
Convert elapsed time, token usage, and platform credit consumption into a consistent cost basis. Compare that cost with quality scores to assess the relationship between quality and expense.
6. Conclusion
The purpose of evaluating coding models is not simply to declare a winner. It is to establish a reproducible, measurable framework for observing their behavior.
A compound engineering task exposes strengths and weaknesses across multiple capabilities. Cache hit rates, output volume, repair behavior, and the proportion of useful implementation code help explain efficiency. A separate cross-domain challenge can provide another perspective on generalization.
Results will change as models evolve. Keeping the evaluation process standardized is what makes those results useful—and comparable—over time.
