How to Evaluate LLM Coding Ability with Progressive Game Development Tasks
A single passing unit test is a weak proxy for real engineering skill. This article argues for shifting coding evaluation from single-function accuracy to process reliability, using a browser-based turn-based strategy game as the task vehicle. It walks through a seven-level chain of progressively accumulating requirements that simulate iterative development, a four-dimensional orthogonal scoring scheme, practical scoring weights, a FAQ on common pitfalls, and guidance on plugging the evaluation into a real R&D workflow.
Lead with the conclusion: from “single function correct” to “engineering process measurable”
Evaluating an LLM’s coding ability cannot stop at whether the model can produce a function that passes a unit test. Real engineering delivery is closer to a continuously evolving pipeline: start with a minimum viable version, then progressively add state, interaction, edge-case handling, algorithm optimization, and thematic refactoring. At every step the model must keep moving forward without breaking existing functionality. So the focus of evaluation should shift from point accuracy to process reliability. The approach proposed here uses a browser-based turn-based strategy game as the task carrier, simulates iterative development through seven levels of progressive requirements, and scores results across four orthogonal dimensions.
Why traditional coding evaluations fall short
Common approaches such as “code fill-in-the-blank” and “complete a single function and pass its unit tests” are easy to auto-grade and produce deterministic results. But their limitation is obvious: they narrow “writing code” down to “writing a snippet that passes a few cases.” Real projects demand long-context memory, cross-turn consistency, understanding of implicit intent, decomposition of complex mechanics, maintenance of code structure, and the ability to layer new requirements on top of old ones without breaking them. Single-function problems can hardly cover any of this.
What we need, then, is a task carrier that satisfies all of the following: controllable initial difficulty, smoothly increasing complexity, verifiable results, and natural state and interaction. Game development offers exactly such a stress-testing ground.
Why complex game development works as a yardstick
A small browser-based strategy game is well suited as a coding-evaluation carrier for three reasons.
First, it is naturally a state-machine system. Turn logic, unit health, resource values, tech progress, and win/loss conditions all require maintaining a continuously changing state, which directly tests the model’s control over global state consistency.
Second, its complexity can be sliced into non-overlapping layers. From rendering and interaction control to pathfinding, unit mechanics, and thematic refactoring, each layer can have its own independently defined acceptance points.
Third, its output is fully visual and interactively verifiable. An evaluator can open the page, perform a few operations, and judge whether a feature is genuinely implemented, rather than relying solely on a black-box accuracy number.
A game scenario therefore forms a high-fidelity miniature of engineering work. Compared with single-function problems, it can expose larger gaps in architecture and iterative maintenance, though this conclusion still needs validation through multi-model testing.
How to design the progressive task ladder
The core idea is to use a chain of requirements that grows in difficulty and accumulates context turn by turn, simulating real iterative development. When evaluating, do not clear the conversation each turn; instead, append each round’s requirement to the context and observe whether the model can preserve earlier work as the history grows longer.
The task chain has seven levels of requirements. It mainly assesses four capability types: instruction following, no functional regression, code maintainability, and creative output, and it is recommended to accumulate context every round.
The initial artifact is a simplified browser-based turn-based strategy game: a grid map or board, a turn button, units for both sides, resource values, and basic win/loss conditions. Before starting Requirement 1, first get the initial version running and verified, for example that the page loads, the turn button toggles, both sides’ units display, and resource values change. Only after the initial version passes acceptance should you proceed to later requirements.
Here is a neutral requirement specification:
Task theme: browser-based turn-based strategy game -> progressive evolutionRequirement 1: Convert the turn-based system into real-time strategy (RTS),with both sides advancing on a synchronized timelineRequirement 2: Add an "AI assistant" toggle that automatically advances techresearch when resources are sufficientRequirement 3: Give all units auto-targeting and pathfinding; actively searchfor enemy units and strongholdsRequirement 4: Interaction enhancements4.1 Provide a 1x-10x speed slider4.2 Change map tiles to regular hexagons4.3 Use arrow keys to pan the viewport4.4 Add icons for cities/units (emoji or system-font icons are both fine)Requirement 5: Add a debug panel that outputs runtime diagnostic info andcaptures exceptionsRequirement 6: Add new creative units6.1 Aerial heavy unit (high HP, area bombing)6.2 Energy-focus unit: units within a certain distance can link to each other;when multiple units stack, attack power multiplies, rendered as a chainedbeamRequirement 7: Refactor the overall theme into a "cyberpunk" art style whilepreserving all existing gameplay
Each level targets different capabilities:
- Requirement 1 tests understanding of abstract concepts. The model must grasp the mechanical difference between real-time strategy and turn-based play, such as both sides advancing on a synchronized timeline, rather than simply deleting the word “turn.”
- Requirement 2 tests inference of implicit intent. What the user actually wants may not be a checkbox but a full set of automatic decision-making logic.
- Requirement 3 tests algorithm implementation, involving targeting, path planning, and objective search.
- Requirement 4 tests completeness of interaction engineering, and the hexagonal map introduces non-Cartesian coordinate handling.
- Requirement 6.2 is where differences tend to widen; it involves dynamic linking of multiple units, distance checks, stacking of attack power, and spatial-relationship computation.
- Requirement 7 tests creative expression and style transfer, while also requiring all existing gameplay to be preserved.
How to quantify four-dimensional scoring
A single “pass/fail” is too coarse. It is better to break the evaluation into several orthogonal dimensions, score each separately, and then aggregate with weights. Here is a schematic main loop:
def evaluate(model, tasks):context = []verified_history = [] # only records artifacts and assertions that passed verificationreport = []for task in tasks:context.append(task.spec) # accumulate context; do not clear it each roundoutput = model.generate(context) # output can be a full file bundle or a patch; see conventions belowok, detail = run_and_verify(output, task.assertions)dim = {"instruction_following": score_following(task, output, context),"no_regression": score_no_regression(output, verified_history),"code_maintainability": score_structure(output),"creativity": score_style(task, output),}if ok:verified_history.append({"artifact": output,"passed_assertions": task.assertions,})report.append({**dim, "passed": ok, "detail": detail})return aggregate(report) # aggregate is a schematic function used for weighted summation
Prerequisites and output conventions: before an official run, the output form of each round must be defined. You can require the model to generate a “complete runnable file” each round, such as a single HTML file or a full directory; or you can require the model to output patches keyed by file path, applied to the previously verified artifact. The two forms cannot be mixed, otherwise “preserving earlier work” cannot be reproduced. run_and_verify should include at least three kinds of steps: page load, core operations, and assertion checks. Only artifacts that pass verification are written to verified_history; the subsequent no-regression score re-runs regression checks against the accumulated set of assertions in verified_history, rather than simply comparing new and old outputs.
Dimensions, focus points, and scoring methods
| Dimension | Focus points | Scoring method |
|---|---|---|
| Instruction following | Whether each requirement clause is hit point by point, and whether implicit intent is correctly understood | Check against requirement points, distinguishing explicit from inferred requirements |
| No functional regression | Whether historically passing features still work after new features are added | Re-run regression against assertions that passed in verified_history |
| Code maintainability | Whether naming, module division, and state management versus rendering are decoupled | Inspect structural clarity, extension cost, and ability to keep layering on requirements |
| Creativity | Completeness and imagination of visual, interaction, and theme design | On the basis of preserved gameplay and values, add points for visual creativity |
The four dimensions break down as follows:
- Instruction-following accuracy: whether each constraint in the requirements is hit point by point, with particular attention to whether implicit intent is correctly inferred.
- No functional regression: whether adding features accidentally breaks previously passing functionality. This is the key metric distinguishing “can write code” from “can maintain code,” and it should usually carry a relatively high weight.
- Code maintainability: whether naming, module division, and the separation of state management from rendering can support continuing to layer on requirements.
- Creativity: the completeness and imagination of visual, interaction, and theme design. This dimension is scored primarily on “whether all existing gameplay and values are preserved after the Requirement 7 theme refactor,” with visual imagination serving only as a bonus, to avoid overly subjective scoring.
The specific weight ratios can be adjusted according to team goals. A fairly safe range is: no functional regression 35%-50%, instruction following 20%-30%, code maintainability 15%-25%, creativity 10%-20%. When running multiple models over the same task chain, this setup offers relatively high discrimination in cross-model comparison (this needs validation through multi-model testing).
FAQ: common issues in putting the evaluation into practice
How do you set scoring weights for game-task evaluation?
Start with what the team cares about most. If the model will be used for long-term iterative projects, raise the weight on no functional regression; if it will be used in requirement-understanding-intensive scenarios, raise the weight on instruction following. Weights are not arbitrary; they should be repeatedly calibrated across multiple models and rounds.Should the conversation be cleared each round?
Clearing is not recommended. In real engineering, later requirements usually depend on earlier context. During evaluation, keep the history of requirements and verified artifacts, and observe whether the model maintains consistency within a long context.What are common mistakes?
Common mistakes include comparing only the final artifact while ignoring the no-regression dimension; writing failed rounds into the regression baseline; adding only UI without wiring up logic; deleting old features during style refactoring; and using too few assertions, so a minimal implementation fools the automated checks.How do you handle overly long context?
You can do structured compression, for example keeping the original requirement text, a list of verified files, key state descriptions, and a summary of failure reasons, while retaining raw logs for auditing. After compression, ensure the acceptance assertions remain traceable.How do you prevent the model from fooling automated assertions with minimal code?
Beyond page load and surface-level operations, add hidden assertions, boundary assertions, and manual spot checks. For example, Requirement 2 should check not only whether the toggle exists but also whether tech automatically advances when resources are sufficient; Requirement 3 should check not only whether units move but also whether they go around obstacles.
How to integrate evaluation into the R&D workflow
To adopt this on a routine basis at a team or enterprise level, start from three areas.
Build a layered case library. Classify and accumulate the task chain by layer: rendering, logic, algorithm, and creativity. After each iteration of the model, prompt, or code architecture, run regression tests to form comparable historical curves.
Stick to context-accumulating evaluation. Do not clear the conversation each turn just to save effort. Accumulating context is closer to real engineering and more likely to expose long-range consistency weaknesses.
Use platform capabilities for support. Enterprises can connect to and schedule different models through mainstream cloud vendors’ LLM service platforms, write evaluation artifacts to object storage for version archiving, use logging and monitoring/alerting services to track pass-rate trends, and hook into a CI pipeline to trigger regressions automatically. This turns a one-off evaluation into a sustainable, operable quality metric.
Typical failure cases and how they are penalized
- The AI assistant adds only a toggle without wiring up decision logic: Requirement 2 looks like it has a control, but there is no automatic decision-making. Instruction following should be heavily penalized, and this item may even score zero.
- The Requirement 7 style refactor loses the Requirement 3 pathfinding: this is a typical no-regression case. No functional regression should be penalized heavily, and creativity points cannot offset lost functionality.
- Long context causes old logic to be overwritten: the model remembers the new requirement but deletes an old mechanism. No functional regression scores low or zero, and code maintainability should also be penalized.
- Minimal code fools the assertions: the page opens but the logic is incomplete. Hidden assertions should be added during acceptance, and this item should be treated as instruction following not meeting the bar.
Conclusion
Evaluating LLM coding ability through progressive complex game development is, in essence, replacing an isolated problem with a complete engineering effort. It requires the model to understand abstract concepts, infer implicit intent, implement complex algorithms, maintain long-range consistency, and ultimately complete a global thematic refactor. This method is useful not only for model selection but also, in reverse, for guiding prompt engineering, code architecture, and R&D evaluation workflows. When an evaluation can stably observe a model’s true level across multiple rounds of iteration, it is no longer merely a point test but a reusable, quantifiable capability-assessment framework that plugs into the R&D process.
