Developing a UEFI Graphics Game with LLMs: A Complete Test and Lessons Learned
This article reports a hands-on evaluation of multiple domestic Chinese LLMs on an unusual but verifiable coding task: building a complete UEFI graphics tank game from scratch using the GOP protocol under QEMU and OVMF. It covers the rationale for the benchmark, prompt and acceptance design, an EmulatorPkg build-system failure mode, token-cost controls, and an overall assessment of code generation versus engineering-environment adaptation.
Developing a UEFI Graphics Game with LLMs: A Complete Test and Lessons Learned
1. Why This Benchmark Task
When evaluating an LLM’s coding ability, the biggest risk is benchmark leakage. If the model has already seen similar problems in its training corpus, the output may look impressive but fail to reflect real engineering capability. A qualified evaluation benchmark should therefore have three characteristics: it should be obscure, self-contained, and verifiable.
Developing a NES-style tank battle game under the UEFI Shell satisfies all three:
- Obscure: UEFI application development is a niche area, and publicly available GUI game code is extremely rare, so a model is unlikely to simply “recall” the answer.
- Self-contained: The task does not depend on an operating system or third-party libraries. It relies only on the GOP (Graphics Output Protocol) defined by the UEFI specification, so the model must genuinely understand the low-level interface.
- Verifiable: Whether the game runs, whether the graphics are correct, and whether the frame rate is stable are immediately visible. There is little room for ambiguity.
The specific task requirements were: a top-down view with a fixed map; the player controls a tank, moves, fires shells, destroys enemy tanks that spawn in waves, and protects the player base; brick terrain can be destroyed; power-ups such as invincibility, armor-piercing shells, and extra lives can be picked up; graphics output uses the GOP protocol; no sound is required. The supporting environment was QEMU with OVMF firmware, targeting stable output at 640×480 resolution, 32-bit BGRA pixel format, and approximately 30 frames per second.
The final implementation contained eight maps, four enemy tank types, five power-ups, and a particle-effects system. It comprised thousands of lines of code and was compilable, runnable, and completable. This shows that current leading LLMs have crossed the usability threshold for “building a complete application from scratch in an obscure domain.”
2. Evaluation Method: Fairness and Reproducibility
Multiple domestic Chinese LLMs were tested, all using exactly the same prompt and acceptance criteria. The core structure of the prompt deserves separate explanation because it directly determines the lower bound of output quality:
- Specify the runtime environment: a pure software UEFI application running in an emulated environment, with no real-hardware flashing involved.
- Specify technical constraints: graphics output may use only the GOP protocol, and the model must not assume the existence of operating-system APIs.
- Specify the feature list: core gameplay, terrain destruction, the power-up system, and wave logic were listed item by item as an acceptance checklist.
- Require a project design document first: the model was asked to output development conventions and an architecture description before writing code, effectively making it “write the design before starting implementation.”
The essence of this approach is to structure a natural-language requirement into a miniature technical specification. In practice, models that skipped the design document and were asked directly for code had a noticeably higher rework rate later. In contrast, the workflow that produced a conventions document first consumed only about RMB 0.3 in token cost for the first generation step, which was highly cost-effective.
As for why only domestic Chinese models were tested and no China-versus-foreign comparison was made, the principle is simple: more usage and more feedback allow domestic models to form a scale effect and improve. This is not exclusion; it is a choice.
3. Key Lesson: The RMB 50 Tuition on the EmulatorPkg Route
The biggest pitfall during testing appeared in the choice of compilation and debugging route.
The standard process was originally to place the code under the EmulatorPkg directory included in an open-source firmware development kit, compile it with Visual Studio on Windows, and then run it in the bundled emulator. Among previously tested models, only two completed this step successfully; all the others failed. The new model tested this time also failed here.
The model’s failure mode was very typical: it repeatedly tried to modify the DSC project description file, adjust the FDF firmware volume layout, and wrestle with module dependencies. Each attempt produced new compilation errors, and then it patched those new errors, falling into a loop of “fix one place, break three others.” Tokens kept burning, RMB 50 was spent, and the problem did not move.
There are three layers to this lesson.
First, the model has a blind spot in its understanding of build systems. The build system of the open-source UEFI firmware kit is interwoven from four types of files: DSC, DEC, FDF, and INF. Dependencies are implicit, and error messages are unfriendly. There are few public examples of this kind of troubleshooting in the model’s available corpus, so it can only try blindly. This suggests that evaluating a model’s engineering ability cannot focus only on “writing code”; it must also examine “handling the build system.”
Second, timely stop-loss is more important than persistence. When the model enters an ineffective loop, the most economical action is to switch routes immediately rather than continue feeding tokens in the hope of a sudden insight. In the actual test, after switching to the instruction “Stop compiling the emulator package; use QEMU for debugging and running,” the problem was solved immediately.
Third, route selection is itself an engineering decision. EmulatorPkg is suitable for rapid iteration, but environment setup has a high barrier. The combination of QEMU and OVMF is closer to real firmware behavior, has a higher degree of standardized configuration, and has abundant public documentation. The model’s success rate on this route was noticeably higher. For AI-assisted development, “choose a toolchain the model is familiar with” is often more practical than “choose the theoretically optimal toolchain.”
4. Cost Control in AI-Assisted Low-Level Development
This test cost RMB 150 in total, about one-third of which was consumed by ineffective compilation trial and error. Here are several practical lessons for controlling cost:
- Set a per-issue budget cap: if the model makes three or four consecutive attempts without progress on the same issue, intervene manually and change the approach immediately. Do not let it keep burning tokens.
- Use staged acceptance: first verify a minimal program, such as drawing only a filled rectangle through GOP output, and then add features incrementally. Avoid generating thousands of lines at once and having to rework the whole thing.
- Keep rollback-capable intermediate states: save a compilable version at every milestone so that when the model “makes things worse with each change,” you can quickly revert.
- Use explicit correction instructions: the model’s default path is not necessarily the path you want. A direct instruction such as “switch to QEMU debugging” is worth more than ten rounds of hints.
5. Overall Assessment and Outlook
Returning to the original question: how should this new-generation model’s performance be assessed?
The conclusion is: code generation has improved significantly, but engineering-environment adaptation remains a weak point. At the pure code level, it can produce a structurally complete, fully playable, and stably running UEFI graphics game, which would have been difficult to imagine a year ago. But when facing a real build system, it still easily falls into blind trial and error and needs a human to steer the direction.
The implication for developers is that, at the current stage, an LLM is more like “a highly capable engineer without on-site experience.” Give it clear specifications, a standardized toolchain, and explicit correction instructions, and it can deliver results beyond expectations. Give it vague engineering decisions that depend on tacit knowledge, and the bill will teach you a lesson.
The implication for evaluation methods is that using the same obscure, verifiable benchmark task to compare multiple models horizontally is a more reliable way to select models than running public leaderboards. Before a model enters your real workflow, having it run through your own “UEFI tank battle” test says more than anything else.
