Writing a Game Under UEFI Shell: A Hardcore Benchmark for AI Code Generation
This article presents UEFI Shell game development as a demanding benchmark for evaluating the code-generation ability of large language models. It explains why the task leaves models almost no fallback, how to make the benchmark reproducible, the six most common failure modes, practical prompt-engineering techniques, and a scoring rubric. It also shows how to turn the benchmark into an automated enterprise evaluation pipeline. Rather than ranking specific models, it delivers a reusable method and a set of failure patterns that teams can apply in their own environments.
1. Why “Writing a Game Under UEFI Shell” Is a Touchstone for Code Ability
Before this exercise, I had used the same benchmark task—developing a classic-style mini-game under UEFI Shell—to test several mainstream large language model versions in succession. Many readers kept asking me to add a particular vendor’s next-generation model, but I had doubts about its predecessor’s code ability, so I hesitated. That hesitation is itself proof that the benchmark is worth something.
The reason is that this is a task that leaves the model almost no way out.
When you write a game on Linux, Windows, or some high-level language runtime, the model can lean on standard libraries, graphics frameworks, and ready-made input abstractions. If something is wrong, the interpreter hands you a stack trace. If something is missing, the package manager fills the gap. In the UEFI Shell environment, all three of those safety nets are gone:
- There is no operating system—only a layer of Boot Services and Runtime Services provided by the firmware.
- There is no C standard library;
stdio.h,malloc, and even floating-point arithmetic all have to be reconsidered. - Graphics output, keyboard input, memory allocation, and timed waits are all reachable only through the protocol interfaces defined by the specification.
- Every function-signature parameter and every
EFI_STATUSreturn value has to be exact. Get a singleUINTNversusUINT32wrong and the stack can misalign.
So what this task really tests is not “can you write loops and collision detection,” but whether the model can keep itself self-consistent at the interface level under the extreme constraints of long context, strong typing, missing libraries, and no feedback loop. This mirrors the most painful part of putting AI-assisted development into production: the problem is rarely that the algorithm cannot be written—it is that the API being called does not line up.
2. Making the Benchmark Rigorous: Keeping It Reproducible
An evaluation without a reproducible experimental design produces conclusions that amount to nothing more than personal impressions. In practice, I fixed the following elements:
1. A fixed task definition
The same requirement statement and the same acceptance checklist apply to every model version tested, with no “tilting” in the prompt.
2. A fixed environment matrix
- Build side: a C compiler and linker script that support PE/COFF and the EFI application subsystem.
- Runtime side: a general firmware emulation environment plus a reference firmware image provided by the official specification, which shields against machine-specific differences.
- Interaction side: driven by the emulator’s graphics window and keyboard events, not by eyeballing results manually.
3. Fixed deliverables
The model must produce a self-contained, single-file project (with no third-party dependencies) in one shot, and the file header must explicitly list: the entry-point signature, every protocol used, whether Boot Services are touched, and the stack and heap budget.
4. A fixed sampling method
The same prompt is sampled independently multiple times, and I take the pass rate and median performance rather than reporting “the best single run.” This point is critical—early model versions often show high variance, and a single success does not prove stable usability.
Note that this article offers no specific model scores or rankings. Such data depends heavily on the firmware version, the toolchain version, and the exact wording of the prompt; only numbers produced in your own environment carry real decision-making value. What follows is the method and the failure modes, which can be reused directly.
3. High-Frequency Failure Modes: Where Models Break Down
After running many rounds, the failures clustered tightly and can be grouped into six categories.
1. Getting the parameter form for protocol acquisition wrong
The graphical output protocol and the simple text input protocol are acquired differently—one by locating the protocol interface, the other taken directly from the system table. Models frequently mix the two up, or dereference a pointer without initializing it first.
/* Illustrative code: shows the correct form for protocol acquisition; not a complete project */EFI_STATUS InitGraphics(EFI_SYSTEM_TABLE *ST, EFI_GRAPHICS_OUTPUT_PROTOCOL **Out){if (ST == NULL || ST->BootServices == NULL || Out == NULL) {return EFI_INVALID_PARAMETER;}return ST->BootServices->LocateProtocol(&gEfiGraphicsOutputProtocolGuid, /* Use the GUID defined in the official specification */NULL,(VOID **)Out);}
2. Mixing CHAR8 and CHAR16
Shell text output, file names, and protocol names rely heavily on wide characters. Handing an ASCII string constant straight to OutputString is the single most common compile-time or runtime error.
/* Illustrative code: explicit wide-character conversion */STATIC CHAR16 mTitle[64];VOID SetTitle(CHAR8 *Ascii){UINTN i;for (i = 0; Ascii[i] != '\0' && i < (sizeof(mTitle) / sizeof(mTitle[0]) - 1); ++i) {mTitle[i] = (CHAR16)Ascii[i];}mTitle[i] = L'\0';}
3. The time base and frame-rate independence
Models tend to write a while (1) busy loop with a fixed delay, so the game runs wildly differently depending on host performance. A subtler error is treating timer ticks as equivalent to milliseconds, which throws the game speed off by several orders of magnitude. The correct approach is a fixed time step: measure real elapsed time, accumulate it into whole logic frames, then advance, keeping rendering decoupled from logic updates.
/* Illustrative code: fixed-timestep main loop skeleton */while (Running) {Accumulator += ElapsedMs(); /* Current time from runtime services */while (Accumulator >= kStepMs) {UpdateGame(kStepMs); /* Logic update, independent of real frame rate */Accumulator -= kStepMs;}RenderFrame(); /* Rendering should depend on state only, not modify it */ST->BootServices->Stall(1000); /* Yield the time slice to avoid monopolizing the CPU */}
4. Input handling that is too naive
Reading a keypress once and responding immediately—ignoring “repeat while held” and “only allow the next trigger after release”—makes the controls feel disastrous. A robust implementation needs explicit key-state bits and edge detection.
5. Resource and stack constraints
Declaring large arrays inside a function is a classic hidden landmine: the default stack for an EFI application is limited, and a local buffer of tens of kilobytes can blow straight through it. Models are often completely unaware of this. Likewise, memory obtained from AllocatePool that is not freed on the exit path will leak across repeated launches.
6. Missing boundary and exit paths
With no handling for branches like “launch failed,” “resolution is not as expected,” or “user pressed the exit key,” the program hangs silently and can only be judged unusable during evaluation.
4. Prompt Engineering: Turning the Uncontrollable into the Controllable
For the same model, a different prompt can make the one-shot pass rate differ dramatically. The techniques I found effective are below.
Contract first, then implementation. Work in two steps: first let the model output only the interface contract (data structures, module breakdown, state machine), have a human confirm it, and only then ask it to write the implementation. Skip this step and rework on protocol usage is almost guaranteed.
Provide a minimal vertical slice. Do not ask for a whole game at once; break it into four independently verifiable steps: clear the screen and print a title → draw a block and support arrow-key movement → add a grid and collision detection → add scoring and an end condition. Each step must run before moving on to the next.
Enforce a self-check list. Require the model to verify item by item before delivery: Did it mix character widths? Does it call Boot Services after exiting Boot Services? Are there unfreed memory pools? Is there any floating-point arithmetic? Do the local buffers exceed the stack budget? Is there a CPU-yielding call inside the loop?
Feed compiler errors back. Keep a loop of at most two to three rounds of “compile → paste back the error → fix.” In practice, a single round of compiler feedback clears most of the low-level interface mistakes. But if it still fails after three rounds, the model’s context-level self-consistency has collapsed, and pressing on only wastes time—switch approaches or step in manually.
A prompt skeleton you can apply directly:
Role: A C developer targeting a no-OS firmware environment.Constraints:1) Do not use any standard library or third-party library;2) Do not use floating-point arithmetic;3) Every memory-pool allocation must be freed on the exit path;4) Total local buffers must not exceed 8KB;5) Decouple logic updates from the real frame rate, using a fixed time step.Deliverable: A single .c file whose header lists the entry-point signature, the dependency protocol list, and the stack budget.Step one: Output only the data structures and module breakdown—no implementation yet. Wait for my confirmation.
5. Making the Evaluation Table Permanent
Subjective impressions cannot support a selection decision. I recommend adopting the scoring dimensions below (the weights are examples and should be adjusted per business scenario):
| Dimension | How It Is Judged | Suggested Weight |
|---|---|---|
| First-compile success rate | Zero errors on the first compile against the target toolchain | 20% |
| Boot success rate | Loads in the firmware environment without crashing or exiting immediately | 20% |
| Gameplay completeness | Checked off item by item against the acceptance list | 25% |
| Timing and feel | Speed consistency across different host loads, and input latency | 15% |
| Resource robustness | Memory release, stack usage, exception-branch handling | 10% |
| Maintainability | Module breakdown, naming, and layering of protocol wrappers | 10% |
A more advanced approach is to productize this benchmark into an internal enterprise evaluation pipeline: containerize the fixed prompts, firmware images, compiler, and acceptance scripts, then automatically trigger a full regression run whenever a model or tool version changes. Results are pushed to a message queue, then archived by log and monitoring/alerting services and shown on a dashboard. The code-hosting and CI platform handles triggering and artifact retention, while object storage keeps images and reports over the long term. Baidu AI Cloud and similar cloud platforms provide general-purpose capabilities—container platforms, object storage, message queues, log services, and monitoring/alerting—that are by themselves sufficient to carry a pipeline like this without any extra customization. The benefit is that evaluation shifts from “one colleague’s experiment” to “a traceable, comparable, continuously evolving quality baseline.”
6. Conclusion: The Variable Is the Model, the Method Is Yours
Back to the original question: how do you judge a new generation of models?
I have three conclusions, none of them tied to a specific product.
First, generational differences are real, but they mainly show up in the structural self-consistency of long contexts. Earlier versions had a high error rate and high variance on hard constraints such as protocol parameters, character widths, and memory management; newer versions are more likely to produce a clear, boundary-complete solution in one pass, which noticeably lowers the cost of manual cleanup. This is, however, a qualitative observation—the specific numbers must come from your own benchmark.
Second, the ceiling of a model’s ability is largely determined by your prompts and process. Contract-first, vertical slices, error feedback, and a self-check list—these four practices tend to affect the one-shot pass rate more than switching to a newer model generation does.
Third, “no-library, no-runtime, strong-typing” benchmark tasks like this one deserve to become a regular part of a team’s evaluation routine. They sit closer to the real pain points of AI-assisted development than grinding through algorithm problems does: whether the model can stay faithful to the interface contract when information is incomplete.
Tools iterate quickly; the method stays relatively stable. Make the evaluation design solid, and you won’t have to judge from scratch every time a new model ships—run your own pipeline once, and the answer is right there.
