Testing LLM Code Generation with a Basketball Game: An Exploratory Observation
This article presents an exploratory test assessing the code generation capabilities of Large Language Models (LLMs) using an interactive basketball game development task. It details a three-part test design, shares observations from testing two unnamed models, and analyzes hypotheses regarding one model's failure. Finally, it outlines the test's limitations and provides practical guidelines for conducting more reliable and standardized LLM comparisons in the future.
Introduction: What This Test Reveals
There are many online discussions regarding the code generation capabilities of Large Language Models (LLMs), and conclusions from different sources often contradict each other. Some claim a new model has surpassed mainstream ones, while others experience the opposite in daily use. Both perspectives lack verifiable data.
This article introduces an exploratory test based on developing an interactive basketball game. It is not a strict benchmark: each model was run only once, and their names remain undisclosed for now. The value of this test lies in demonstrating the test design approach, recording observed phenomena, and explaining how to make such evaluations more reliable. Readers should treat these observations as preliminary clues rather than direct criteria for model selection.
Test Design
The task was divided into three main parts.
Part 1: Basic Scene and Rendering Layer. This included drawing the basketball court, player entrance, warm-ups, and the referee’s whistle for the jump ball. This part tested the model’s ability to implement frontend rendering and basic UI interactions.
Part 2: Core Algorithm and Logic Control Layer. This involved dribbling, passing, shooting, dunking, stealing, blocking, and rebounding, along with simulating stamina consumption and foul calling. This part placed the highest demands on state machine management.
Part 3: Data Recording and Visual Optimization Layer. This required the presentation of an art style and the recording of various metrics during the test.
The interaction rules were as follows. The first-round prompt was identical across all models. In subsequent multi-turn conversations, the tester acted solely as a regular user pointing out observed bugs or logic flaws, without providing repair code or hinting at a solution. This design was intended to observe the model’s ability to locate issues independently. However, the way the tester described the flaws could influence the model’s repair path, a point discussed later in this article.
Reproduction Checklist
The original records of this test were incomplete. The table below lists the information required for reproduction and its current status. Unrecorded items are marked as “To be supplemented,” and no estimations are made in this article.
| Item | Current Status |
|---|---|
| Model names and versions | To be supplemented (referred to as Model A and Model B in this article) |
| Full text of the first-round prompt | To be attached |
| Runtime environment (APIs, temperature, and other parameters) | To be supplemented |
| Dialogue rounds and time consumption per round | To be supplemented |
| Bug list (ID, phenomenon, associated Part) | Partially shown in the results table below; full list to be supplemented |
| Scoring metrics (pass rate, fix success rate, lines of code, error types) | To be supplemented |
| Generated code and test screen recordings | To be released |
| Test time and author conflict of interest statement | To be supplemented |
Until verification is complete, this article will not disclose the specific model names.
Result Observations
The table below lists the observations from the original records by phase. “Not recorded” means there was no related description in the original material; it does not indicate a pass or fail.
| Phase | Model A | Model B |
|---|---|---|
| Part 1: Scene Rendering | Drew a net at the rim, but court proportions and boundary lines were chaotic with severe structural issues | Obvious deviations in rim position and court line drawing |
| Part 2: Player Movement | Game failed to start; players could not move | Character movement was basically usable, but speed was too fast |
| Part 2: Interactive Buttons | Shooting, passing, stealing, and other buttons were unclickable | Not recorded |
| Broadcast System | Not recorded | Completed |
| Part 3: Visuals and Data Recording | Could not proceed to this phase due to flow blockage | Not recorded |
| Multi-turn Repairs | Repairs failed; some character models were deleted after modifications, resulting in test failure | Repair process not recorded |
| Extra Features | None | The model independently designed an Auto mode to simulate the process of watching a match |
The Auto mode was a feature spontaneously implemented by the model. The original record only described its effect without analyzing the code structure. Therefore, this article does not interpret it as evidence that the model understands design patterns like the Observer pattern.
Interpreting the Results
This test has several obvious limitations.
First, the sample size is 1. The same model might yield different results at different times or with different random seeds.
Second, each model was run only once. It is impossible to determine if Model A’s failure was a stable phenomenon or an occasional glitch.
Third, the tester was simultaneously the task designer and the result evaluator, lacking a blind review.
Fourth, the original records lacked complete scoring metrics. Terms like “failure” or “high completion rate” in the text are only qualitative descriptions and cannot be treated as quantitative comparisons.
Therefore, the only conclusion this test supports is: under the conditions of this specific task, Model A encountered blocking issues at the basic rendering and interaction levels, while Model B had a higher completion rate. This does not indicate the overall capabilities of the two models, nor does it suggest which one is better suited for a particular team.
Hypotheses on the Causes of Failure
The original article attributed Model A’s failure to insufficient global state management, context degradation during multi-turn conversations, and a disconnect between the UI and underlying logic. These explanations are plausible but remain hypotheses. Verification methods are as follows:
- Global state management: Before and after each repair, record the list of character objects and their lifecycles to confirm whether objects were accidentally deleted or duplicated during the repair process.
- Context degradation: In subsequent repair rounds, re-provide the initial architecture description and compare whether the repair success rate changes.
- UI and logic disconnect: Check whether the buttons’ event listeners are correctly bound, and log in the console whether click events trigger the corresponding game logic.
How to Conduct More Reliable Comparisons
- Run each model at least 5 times, reporting the pass rate and result distribution rather than just a single outcome.
- Keep models anonymous and have a third party conduct a blind review using a scoring rubric fixed before testing.
- Fix adjustable parameters and record model versions, invocation times, and settings like temperature.
- Fully save conversations, generated code, and repair records for post-hoc review.
- Adopt a uniform rule for defect descriptions in each round, such as describing only the phenomenon without any repair clues.
- Define clear criteria for determining whether a task is “completed,” “partially completed,” or “not completed.”
Frequently Asked Questions
How to choose a model suitable for your team? First, extract task samples from your actual requirements, use these tasks to test candidate models, and then compare pass rates, repair success rates, and rework costs. Public leaderboards and online reviews can serve as references but cannot replace this step.
What are the most common mistakes during testing? First, using a single result to represent overall performance. Second, modifying prompts midway through testing. Third, only recording successful cases while ignoring the failure process.
What should a minimal reproduction include? A fixed prompt, a set of clear defect descriptions, model versions and parameter settings, as well as the complete dialogue and code output. Only when others can rerun the test using the same steps can the conclusions be validated.
