Assessing AI in Game Development: From Impressive Demos to Engineering Usability

Over the past year, videos showcasing Large Language Models (LLMs) generating mini-games and prototypes have frequently gone viral in developer communities. Inputting a natural language prompt and getting a playable mini-level running in a browser within seconds is undeniably intuitive. However, once the initial hype fades, developers who actually integrate AI into game development pipelines encounter a host of issues masked during the demo phase: generated code fails to run, physics feel unnatural, asset loading throws errors, and the codebase collapses by the third iteration.

This article aims to abstractly discuss a technical question: When treating “making games with AI” as an engineering task, what are the capability boundaries of current mainstream LLMs? Which stages are already usable, and which are still stuck at the prototype phase?

1. Deconstructing “AI Game Development”: Core Task Layers

Much of the confusion in current discussions stems from lumping vastly different tasks together. By engineering complexity, AI game development can be roughly divided into four layers:

Layer 1: Pure Code Generation. For example, writing a Snake or Breakout game using Canvas, WebGL, or a specific game framework. These tasks have simple logic, minimal state, no external asset dependencies, and are well-covered in model training data. Models generally perform well here.

Layer 2: Game Logic + State Management. This involves small platformers or similar games with level progression, save states, scoring, and collision detection. This introduces cross-frame states, event sequencing, and boundary conditions. Here, models begin to exhibit the “locally correct, globally non-convergent” problem.

Layer 3: Asset and Pipeline Integration. The model needs to load images, audio, and sprite sheets, dealing with paths, formats, asynchronous loading, and memory release. Failures at this layer are rarely logic errors but rather incorrect environment assumptions—the model assumes certain files exist or certain APIs are available, which is not true in practice.

Layer 4: Multi-turn Iteration and Maintainability. When a user requests to “make the jump feel lighter” or “add a double jump,” the model must make incremental modifications to existing code without breaking the original structure. This is the layer closest to real-world R&D scenarios and the most likely to expose a model’s shortcomings.

Distinguishing these four layers is crucial: The same model might score near-perfectly on Layer 1 but barely pass on Layer 4. Discussing whether “AI can make games” without context of the task layer is meaningless.

2. Evaluation Dimensions for AI Game Development

If you are conducting a serious horizontal evaluation, you should cover at least the following dimensions rather than just checking if “it can run.”

Runnability. Can the generated code run directly with zero human intervention? What is the first-run success rate? Can it self-repair and run successfully after an error? This is the most fundamental hard metric.

Correctness. Do the game rules match the description? Are win/loss conditions, scoring, and collisions accurate? It is recommended to verify this using automated test cases rather than manually playing for two minutes.

Game Feel. Jump height, acceleration, damping, and input response latency. This is subjective but can be compared via parameterization: does the model extract physics parameters into adjustable constants, or hardcode them into the logic?

Extensibility. When users add requirements, does the model make local modifications or rewrite everything? The former indicates an understanding of the code structure; the latter indicates it is merely “regenerating.”

Dependency Handling. Does it use placeholder assets? Does it explicitly declare dependencies? Does it provide a fallback plan when assets are missing?

Consistency. After multiple turns of dialogue, are the early settings preserved? This is a direct reflection of long-context capabilities.

A simple scoring table can be structured as follows:

Dimension	Suggested Weight	Verification Method
Runnability	High	Zero-intervention run + auto-repair turns
Correctness	High	Automated test case coverage
Game Feel	Medium	Parameterization degree + subjective comparison
Extensibility	High	Diff scale after adding requirements
Dependency Handling	Medium	Behavior on missing assets
Consistency	Medium	Setting fidelity after multiple turns

3. Real-World Model Performance: Strengths and Clear Weaknesses

Based on various public tests, a relatively stable profile can be summarized.

The strengths are concentrated in niche capabilities. In Chinese language understanding, comment and documentation generation, structured output (such as generating level configuration JSONs according to a specified schema), and familiarity with domestic tech stacks, models like ERNIE, Qwen, and GLM have reached a level where they can compete head-to-head with leading overseas models. Particularly when assigning tasks like “Game Design Document -> Structured Level Data” to these models, the stability of the output format and field completeness often exceed expectations.

The weaknesses are concentrated in long-chain engineering tasks. Typical issues include:

• One-shot generation, multi-turn collapse. The first version runs, but modifying requirements in the second round introduces regression bugs. By the third round, “patch-style fixes” begin, and the code structure degrades rapidly.
• Incorrect environment assumptions. Assuming certain global objects or build tool configurations exist, which are not present in actual projects.
• Overconfidence. Failing to state assumptions about uncertain APIs, directly writing seemingly reasonable but practically non-existent calls.
• Rough physics and numerical tuning. Parameters like gravity, speed, and collision boxes tend to be “reasonable-looking” integers, lacking design for adjustability.

The common thread among these issues is: they do not surface during the “generation” phase, but during the “iteration” phase. This explains why demo videos always look great—demos usually only reach Layer 1 or Layer 2.

4. Engineering Solutions: Practical Recommendations

If you plan to introduce LLMs to assist game development in real projects, the following experiences are worth referencing.

1. Isolate the Model and Engine with a Structured Intermediate Layer.

Do not let the model directly output final runnable code. Instead, have it output a structured description, which is then processed by deterministic code to generate the runtime. For example:

{
  "level": 1,
  "gravity": 980,
  "player": { "speed": 220, "jumpVelocity": -420 },
  "platforms": [
    { "x": 0, "y": 400, "w": 320, "h": 24 },
    { "x": 400, "y": 320, "w": 160, "h": 24 }
  ],
  "goal": { "x": 520, "y": 280 }
}

The model is only responsible for filling this structure; rendering and physics are handled by fixed code. This way, even if the model output deviates, it will not directly break the runtime logic. Concurrently, parameters are inherently adjustable, testable, and regression-friendly.

2. Treat “Runnability” as the Primary Acceptance Criterion, Not “Looks Correct.”

Add a pipeline in CI: Model output -> Static analysis -> Headless run -> Automated test cases -> Screenshot comparison. If any step fails, roll back and retry, rather than relying on human judgment to decide if “it should run.”

3. Explicitly Declare Environment Assumptions.

Pin the tech stack version, available API whitelist, and asset directory structure in the prompt. Force the model to proactively throw errors when assumptions are not met, rather than hallucinating solutions.

4. Control Single-Turn Modification Scale.

Break down “add a double jump” into three steps: “confirm jump state machine structure -> insert double jump branch -> tune parameters.” Verify each step independently. Delivering large modifications all at once almost inevitably introduces regressions.

5. Use Logs and Monitoring as a Safety Net.

Game runtime metrics like frame rate, memory, and exception stack traces should be uniformly collected to a logging service, paired with monitoring alerts. This allows quick localization of issues when unexpected model-generated code goes live. This part is essentially no different from observability construction in general backend systems.

5. Conclusion: The Gap is Narrowing, but “Usable” and “Impressive” are Different

Overall, the current positioning of LLMs in game development can be summarized as: a reliable accelerator for bounded, small-to-medium tasks, but still requiring human safety nets for open-ended, long-chain tasks.

Models like ERNIE, Qwen, and GLM have become competitive in Chinese understanding, structured output, and niche tasks, capable of competing with leading overseas models in certain scenarios. However, gaps remain in multi-turn iteration stability, environment assumption rigor, and long-code structure maintenance. This gap cannot be simply summarized as “a few months”; it depends on the task layer: the deeper you go into engineering, the more obvious the gap becomes.

For developers, the pragmatic approach is not to wait for models to become stronger, but to break tasks down, lock down structures, and automate verification. Achieve these three points, and today’s models can already significantly improve efficiency during the prototyping and configuration generation phases. Fail to do so, and even the strongest model will only produce more “looks like it should run” demos.

The ultimate criterion for tech selection is not whose demo is flashier, but whose output can enter your CI pipeline.