Evaluating Preview LLM Coding Capabilities: Engineering Insights from a Claw Machine Challenge
When evaluating large language models (LLMs) for programming, simply checking if they can "write code" is no longer sufficient. True development experience depends on a model's ability to understand complex requirements, advance tasks continuously, minimize unnecessary rewrites, and deliver a runnable, maintainable project. This article uses a claw machine game as a representative test case to evaluate the engineering capabilities of a preview LLM, analyzing its task progression stability, delivery quality, token efficiency, and applicable development scenarios.
When evaluating a large language model (LLM) designed for programming scenarios, simply checking whether it can “write code” is far from enough. What truly impacts the development experience is the model’s ability to understand complex requirements, continuously advance tasks, reduce unnecessary rewrites, and ultimately deliver a complete, runnable, and maintainable project.
A claw machine game serves as a highly representative test case. On the surface, it is just a simple web game; in reality, it simultaneously involves physics interactions, animation control, state machine design, collision detection, user input handling, resource layout, scoring logic, and front-end engineering organization. A model capable only of generating localized code will struggle to complete a genuinely usable product; only a model with strong task decomposition and continuous correction capabilities can successfully integrate multiple subsystems.
1. Why This Test Case is Representative
A complete claw machine game typically includes at least the following modules:
Lateral Movement Control
Players need to control the claw moving left and right along the track. The system must handle keyboard, mouse, or touch inputs and restrict the claw from exceeding the device boundaries.Descent and Grasping Actions
When the claw descends, it must not only change coordinates but also present visual changes such as rope lengthening and claw opening/closing. The descent process is usually driven by timers or animation frames rather than a one-time position modification.Rope and Swinging Effects
After grabbing an object, the rope should have a sense of sagging or swinging. Even without a full physics engine, natural movement must be simulated through interpolation, damping, and angle changes.Collision and Grasping Determination
Clear conditions are needed to determine whether the claw hits the target, whether the target is successfully grasped, and whether the object drops during movement. Overly lenient determination reduces the game’s credibility, while overly strict rules make users feel like they can never catch anything.State Machine Management
The game usually goes through states such as idle, moving, descending, grasping, ascending, retracting, and settling. Without clear state transitions, the code can easily suffer from repeated triggers, unresponsive buttons, or overlapping animations.Scoring and Feedback
After a successful grab, the score, attempts, or rewards need updating, accompanied by clear visual feedback. Scoring logic should be decoupled from the animation process to prevent score anomalies caused by repeated triggers.
Therefore, this type of task tests not only code generation capabilities but also the model’s engineering planning skills.
2. Actual Experience: Slower Speed, but Stable Task Progression
In an actual test, a preview LLM took over an hour to complete this project via an intelligent programming tool. This result does not imply low model efficiency, as the task itself contains many interactive details, and the generation process often includes requirement understanding, file creation, runtime verification, issue fixing, and visual adjustments.
One of the main factors affecting overall time consumption is the relatively limited output speed of the preview version. The observed generation speed in testing was about 33 tokens per second. For tasks requiring the continuous generation of multiple files, lengthy component logic, and style code, the output speed directly impacts waiting time.
However, a slow speed does not equate to low production efficiency. More importantly, it is essential to observe whether the model can:
- Complete large-granularity functional modifications in one go;
- Reduce meaningless back-and-forth confirmations;
- Locate the root cause when encountering errors rather than simply overwriting the original code;
- Preserve existing features from being broken by subsequent modifications;
- Continue handling interactive details and visual representations after completing the core logic.
From this task, although the model’s generation process was relatively slow, its overall progression was continuous. It did not frequently fall into the “modify-error-rewrite” cycle, and the proportion of valid engineering code generated was relatively high. For complex front-end tasks, this stability is often more important than single-response speed.
3. Delivery Quality: Complete Core Functions, Details Require Manual Polish
The final implementation basically covered the expected functions, including mechanical movement, rope changes, target collision, grasping process, and scoring logic. From an engineering perspective, the core workflow is complete, allowing users to perform a closed-loop operation from controlling the claw to obtaining results.
Several aspects are particularly commendable:
3.1 Natural Integration of Functions
The model did not just implement a static page; it combined control, animation, and result feedback into a complete process. After player input, the claw moves according to a preset sequence, the grasping action features staged changes, and success or failure provides corresponding feedback.
3.2 Animations Beyond Position Jumps
If only the horizontal and vertical coordinates of elements are modified, the claw machine can easily look like a demonstration prototype. A better implementation separately handles rope length, claw angle, target object position, and movement rhythm, ensuring the entire process maintains continuity.
3.3 Large Granularity in Code Modifications
During task progression, the model could often adjust component logic, styles, and interactions simultaneously rather than modifying just one or two lines at a time. This large-granularity operation helps quickly form a runnable version and reduces the cognitive bias caused by context switching.
Of course, the project still had some minor flaws. For example, some collision boundaries might not be precise enough, object behavior at extreme positions was not entirely natural, and certain animation rhythms still fell short of real mechanical movements. These issues do not affect the main workflow, but they indicate that the first version generated by the model still requires manual acceptance testing.
4. Understanding Token Consumption and Engineering Efficiency
In intelligent programming tasks, token consumption cannot simply be equated to cost, nor should model quality be judged solely by “the fewer, the better.”
A model using fewer tokens might mean it outputs concisely and plans accurately, but it might also mean it omitted necessary functions. Another model consuming more tokens might be repeatedly retrying, or it might be completing a more complex engineering implementation. Therefore, the following metrics should be observed simultaneously:
- Total processing volume;
- Output volume;
- Cache hit or reuse ratio;
- Actual proportion of valid code;
- Number of error rewrites;
- Total time from start to a runnable version;
- Time required for final manual fixes.
In a comparative test, the processing volumes of different intelligent programming solutions varied significantly: some had a total processing volume of tens of millions of tokens, while others were closer to a few million tokens; output volumes ranged from tens of thousands to over 200,000 tokens. One preview solution had a total processing volume of about 2.69 million tokens, an output of about 55,000 tokens, and a cache ratio of about 95%.
These numbers can only serve as process references and cannot be interpreted in isolation from task difficulty. A more valuable conclusion is that this solution did not obviously rely on massive repetitive generation during this task and could accomplish the main engineering goals with relatively restrained token consumption. In other words, its advantage is not necessarily having the absolute fastest speed, but having a higher “valid output ratio.”
5. What to Look for When Comparing with Other Solutions
If a horizontal evaluation of multiple models is required, it is recommended to establish a unified testing standard rather than just comparing final page screenshots.
5.1 Unified Requirement Description
All models should receive the same functional requirements, resource conditions, and operating environments. The requirement description should clearly specify which functions must be implemented and which visual details are bonus items.
5.2 Unified Acceptance Checklist
Scoring can be conducted from the following dimensions:
- Whether it can start normally;
- Whether core operations are complete;
- Whether state transitions are correct;
- Whether there are obvious control conflicts;
- Whether collision determination is stable;
- Whether both failure and success scenarios are handled;
- Whether the page is usable across different screen sizes;
- Whether the code has basic maintainability;
- Whether it can continue to fix issues based on runtime results.
5.3 Recording Process Metrics
In addition to the final result, the time to first runnable version, final delivery time, generation speed, number of errors, and number of manual interventions should be recorded. For enterprise R&D, subsequent maintenance costs are often more critical than initial generation speed.
5.4 Distinguishing “Model Capability” from “Tool Capability”
Intelligent programming tools typically also handle file management, code execution, error collection, and context compression. The final experience is the result of the combined action of the model and the tool. Therefore, test reports should ideally separately describe the performance of the model itself and the impact of the toolchain on task progression.
6. Suitable Development Scenarios for Preview Models
Based on its performance in these types of tasks, the preview model is more suitable for the following work:
- Rapidly building interactive prototypes;
- Generating small to medium-sized front-end projects;
- Completing component refactoring and style adjustments;
- Writing basic state management logic;
- Performing targeted fixes based on runtime errors;
- Translating natural language requirements into a runnable initial project.
However, for high-reliability systems, complex back-end services, financial transaction logic, or security-sensitive modules, you still cannot directly rely on model output. Developers must conduct code reviews, boundary testing, fault injection, and performance verification.
It is especially important to note that model-generated code often performs well on the “happy path,” but may handle the following situations inadequately:
- Rapid continuous clicks by users;
- Re-triggering operations before an animation finishes;
- Target objects located at boundary positions;
- Repeated page initialization;
- Network or resource loading failures;
- Simultaneous triggering of mobile touch events and mouse events.
These scenarios need to be actively covered through test cases.
7. Comprehensive Evaluation: Slower, but Completion and Stability are Noteworthy
Synthesizing this practice, we can make a relatively objective judgment about this preview model:
First, its generation speed still has room for improvement, and waiting times during long tasks are quite noticeable.
Second, its overall planning ability for complex front-end tasks is good, capable of continuously advancing toward the goal.
Third, its token usage is relatively restrained, with fewer repetitive rewrites and ineffective struggles.
Fourth, it can deliver a relatively complete engineering result rather than just providing scattered code snippets.
Fifth, the final product still requires manual visual optimization, boundary fixes, and quality acceptance.
Therefore, evaluating such models should not only ask “is it fast?” but should further ask “can it stably complete tasks?”, “does it reduce developer intervention?”, and “is the code easy to maintain?” If developers focus more on the one-time completion rate and process stability of complex tasks rather than simply pursuing the fastest output, then this type of preview model already possesses strong practical value.
For enterprise R&D teams, it is recommended to position it as an efficient engineering collaboration assistant: let the model handle requirement decomposition, initial implementation, and routine fixes, while developers handle architectural decisions, risk control, and final acceptance. Only by combining the model’s generation capabilities with a standardized testing process can token efficiency truly be transformed into R&D efficiency.
