Benchmarking AI Coding Tools for UEFI Low-Level Development: A Practical Evaluation

    This article presents a hands-on evaluation of AI coding tools in UEFI Shell environments, where no operating system runtime, threading, graphics libraries, or debuggers are available. By building a simplified Pac-Man-style game on x86_64 UEFI using OVMF and QEMU, the author examines how AI tools handle graphics output, keyboard input, timing, and game state machines. The article covers the test task design and constraints, the first-round results, a breakdown of AI capabilities across four common low-level problem types, a detailed defect case in the power-pellet logic, a checklist of frequent UEFI mistakes, and a framework for defining verifiable low-level development tasks. It concludes that AI coding tools are best positioned as efficient coding and exploration partners rather than fully autonomous developers, and that combining clear constraints, simulated verification, modular progression, and manual review is essential for reliable results.

    1. Scope: Why Use UEFI Shell for AI Coding Tests

    UEFI (Unified Extensible Firmware Interface) runs before the operating system loads and serves as a firmware-level development environment. Unlike web, desktop, or ordinary system applications, the UEFI Shell typically has no mature operating system runtime, thread scheduler, graphics library, input framework, or debugger support. Asking an AI coding tool to complete tasks in this environment is therefore well suited to testing its understanding of system interfaces, memory layout, protocol calls, and low-level debugging.

    It should be noted that this article focuses on developing UEFI Shell applications. It does not apply to graphics programs, desktop games, or ordinary user-space applications that run after the operating system has loaded. Tasks of this kind share the following characteristics:

    • No ready-made runtime libraries; many conventional capabilities must be achieved by calling firmware protocols directly.
    • Graphics output, keyboard input, and timing control must all be implemented through UEFI interfaces.
    • Debugging options are limited; illegal memory access or abnormal protocol states can cause a direct crash.
    • High demands on determinism; generated code must not only “look like it runs” but also reliably pass boundary-condition validation.

    For this reason, the UEFI environment is not simply “low-resource development.” It is a stress test of AI-generated code’s protocol understanding, boundary handling, verifiability, and risk control.

    2. Test Task Design and Constraints: Simulating Realistic Development of an Unfamiliar Low-Level Module

    The task for this evaluation was to develop a simplified Pac-Man-style game in the UEFI Shell environment. To make the test closer to real low-level development, the following constraints were deliberately set:

    1. Target platform and runtime environment
      The target architecture was x86_64 UEFI, and the runtime environment was a UEFI Shell application. The validation environment used OVMF firmware with the QEMU emulator, in order to avoid the uncontrolled risk of repeatedly writing display-related resources directly on real hardware.

    2. Toolchain restrictions
      Only a basic integrated development environment, standard C/C++ compiler, and linker tools were used. EDK II, or GCC/Clang with the LLD linker, was permitted. No mature graphics library, game engine, or third-party UI framework was allowed.

    3. Zero starting point
      The local codebase was empty. Starting from a blank project, the AI was required to design the directory structure, build scripts, entry function, and module division on its own.

    4. Limited human guidance
      Only the task goal and runtime constraints were provided. No complete scaffold was given, and protocol-call details were not offered step by step. The AI had to carry out requirements analysis, interface selection, implementation, testing, and fixes independently.

    5. Clear acceptance criteria
      Acceptance was not based solely on “code was generated successfully.” Instead, the program had to be able to, in the UEFI Shell:

      • Correctly display the maze, player character, ghosts, pellets, and power pellets;
      • Respond to keyboard arrow input;
      • Support character movement, ghost chasing, and basic collision detection;
      • Provide a verifiable power-pellet mechanism, including ghost state switching, a capturable state, and respawn logic;
      • Run continuously without crashing and support repeatable startup testing.

    These constraints resemble a developer taking over an unfamiliar low-level module: incomplete information, little reference code, high debugging cost, and strict correctness requirements. This is closer to a real engineering scenario than merely testing the ability to “generate nice-looking code.”

    3. Results: Runnable on the First Round, but a Verification Loop Is Still Needed

    After the task, environment, and acceptance criteria were defined, the first-round AI-generated result could run in the UEFI Shell. The program displayed the maze scene, rendered elements such as the character, ghosts, pellets, and power pellets, and keyboard input drove basic movement.

    Here, it is important to avoid subjective exaggeration. A more accurate statement is that the first-round result reached a “runnable prototype” state, not a “completed and fully correct in one pass” state. Based on the test conditions:

    • The runtime environment was OVMF + QEMU emulating UEFI firmware.
    • The first-round generated code already included the power-pellet logic framework.
    • Display, input, movement, and chasing logic were basically functional.
    • The power-pellet-related state switching had defects that needed to be fixed through debugging.
    • A local simulation test harness was also built to enable rapid regression, allowing some logic to be verified without repeatedly booting the firmware.

    For low-level development, “boots, displays, accepts input” is only the first step. What really matters is whether a verification loop can be established: after each change, whether one can determine if the behavior matches expectations. The fact that the AI tool proactively built a local test framework shows that it was not merely generating code but was also attempting to solve the problem of “how to know whether it is correct after a change.”

    4. Breaking Down AI Capabilities in UEFI Low-Level Tasks

    To more clearly judge the boundaries of AI capability, the task can be broken into four typical categories of low-level problems. These four categories are also common pitfalls for AI in UEFI development.

    Task type Key interface or concept Key point Common failure cause
    Graphics output EFI_GRAPHICS_OUTPUT_PROTOCOL Acquire the protocol, confirm the mode, handle PixelFormat, stride, and boundaries Failure to initialize graphics output, wrong pixel format, out-of-bounds memory writes
    Keyboard input EFI_SIMPLE_TEXT_INPUT_PROTOCOL Poll for key presses, distinguish scan codes from characters, handle repeated triggering Misusing scan codes, not handling key release, logic deadlocks
    Timing and refresh Boot Services delay or event mechanism Establish frame pacing, avoid busy-waiting, control movement speed Relying on busy-waiting, causing sluggish response and jumpy animation
    Game state machine State switching, timers, collision detection Isolate states such as normal, frightened, eaten, and respawn Confused state conditions, wrong respawn location or timing

    4.1 Graphics Output in an Environment Without a Standard Library

    In UEFI, you cannot call a high-level graphics library directly as you would in a desktop program. Typically, you must acquire screen information through the graphics output protocol and then write pixels based on coordinates. The example below is pseudocode used to illustrate the idea; the actual implementation must be adjusted according to the firmware environment and pixel format.

    1. // Pseudocode: draw a single pixel via the UEFI graphics output protocol
    2. EFI_STATUS DrawPixel(
    3. EFI_GRAPHICS_OUTPUT_PROTOCOL *Gop,
    4. UINTN X,
    5. UINTN Y,
    6. UINT8 R,
    7. UINT8 G,
    8. UINT8 B
    9. ) {
    10. if (Gop == NULL || Gop->Mode == NULL || Gop->Mode->Info == NULL) {
    11. return EFI_INVALID_PARAMETER;
    12. }
    13. if (X >= Gop->Mode->Info->HorizontalResolution ||
    14. Y >= Gop->Mode->Info->VerticalResolution) {
    15. return EFI_INVALID_PARAMETER;
    16. }
    17. // When writing directly to the frame buffer, you must confirm PixelFormat and stride.
    18. // This is only an illustration; production code should prefer Blt or a full wrapper.
    19. UINT32 *Fb = (UINT32 *)Gop->Mode->FrameBufferBase;
    20. if (Fb == NULL) {
    21. return EFI_NOT_READY;
    22. }
    23. UINTN Stride = Gop->Mode->Info->PixelsPerScanLine;
    24. UINT32 Packed = 0;
    25. if (Gop->Mode->Info->PixelFormat ==
    26. PixelRedGreenBlueReserved8BitPerColor) {
    27. Packed = (R << 16) | (G << 8) | B;
    28. } else if (Gop->Mode->Info->PixelFormat ==
    29. PixelBlueGreenRedReserved8BitPerColor) {
    30. Packed = (B << 16) | (G << 8) | R;
    31. } else {
    32. return EFI_UNSUPPORTED;
    33. }
    34. // Pseudocode: the actual write requires confirming the memory mapping and firmware support.
    35. Fb[Y * Stride + X] = Packed;
    36. return EFI_SUCCESS;
    37. }

    The key is not this small snippet itself, but whether the AI can proactively realize that it must first confirm whether the graphics output protocol is available, whether a display mode has been set, what the pixel format is, what the unit of stride is, and whether the coordinates are out of bounds. Once a low-level write crosses the boundary, the problem is often not an “error message” but rather a “black screen, garbled display, or an outright reboot.”

    A safer implementation should give priority to the Gop->Blt interface, because the firmware has already encapsulated part of the format handling. If you operate on the frame buffer directly, you must first verify the pixel format, scan-line stride, and color ordering in an isolated environment or simulator.

    4.2 Keyboard Input: Scan Codes, Characters, and Repeated Triggering

    UEFI input is typically obtained through the simple text input protocol. Developers must distinguish Unicode characters from scan codes. For arrow keys, scan codes usually need to be handled; for ordinary characters, UnicodeChar may need to be read. If key release or repeated triggering is ignored, character movement will exhibit rapid consecutive moves, stuck keys, or wrong directions.

    A verifiable input test case should include:

    • After pressing an arrow key, the character moves one grid cell or continues moving;
    • After releasing the arrow key, movement stops or returns to the default state;
    • Rapid consecutive key presses do not cause lost state;
    • Non-game keys do not affect the game logic.

    4.3 Timing Baseline: No Scheduler, but Still a Stable Rhythm

    Without an operating system thread scheduler, the AI must establish its own timing baseline. Common approaches include Boot Services delays, event timers, or frame counters. Note that a simple busy-wait, while feasible, wastes CPU and degrades response quality. A more reasonable approach is to update the logic at a fixed frame interval and, within each frame, handle input, movement, collision, and rendering.

    4.4 Game Logic and Character AI: The Finite State Machine Is Central

    In Pac-Man-style games, ghost behavior is essentially a state machine. At minimum, it must distinguish:

    • Normal chasing state;
    • Frightened state after a power pellet takes effect;
    • Disappearance or return state after being eaten;
    • Respawn delay state;
    • Return to normal state.

    If the AI generates only ghosts that “look like they are moving” without complete state switching, timing, and position restoration logic, problems such as “the ghost does not flee after a power pellet is eaten,” “the ghost cannot be eaten,” and “the respawn position is wrong” will easily appear.

    5. Defects and Debugging: The Power-Pellet Mechanism Exposed the Gap Between “Generating Correctly” and “Running Correctly”

    A typical problem found in the first-round code was: after the player ate a power pellet, the ghost did not enter a capturable state, neither fleeing nor being eatable.

    This problem should not be simply dismissed as “code generation failure.” It should be reviewed as a typical case in low-level AI development. The reproducible behavior is as follows:

    1. Launch the game program in the UEFI Shell;
    2. Move the character to a power pellet’s position and eat it;
    3. Observe the ghost’s behavior: the ghost did not change color, did not change movement direction, and did not enter an eatable state;
    4. Continue colliding with the ghost; the expected capture logic did not trigger.

    The diagnosis can be divided into three layers:

    • Symptom layer: the power pellet triggered successfully, but the ghost state did not change;
    • Logic layer: the state variable or boolean flag was not updated when the power pellet was triggered;
    • Implementation layer: collision detection may have handled only normal pellets and did not call the power-pellet-specific state-switching function.

    Before the fix, test cases should be designed:

    • The character eats a normal pellet: the score increases, and the ghost state remains unchanged;
    • The character eats a power pellet: the ghost state switches to frightened;
    • A frightened ghost meets the character: it should be eatable;
    • An eaten ghost should enter a disappearance or respawn flow;
    • After the respawn delay ends, the ghost returns to normal chasing.

    After the fix, the same test cases must be run again. Only when “power-pellet triggering, state switching, collision detection, and respawn recovery” all pass can the module be considered basically correct.

    This also shows that generating a runnable program in one pass does not mean the logic is fully correct. Code involving state switching must rely on reproducible scenarios, clear diagnostic evidence, and before-and-after comparison, rather than being judged as successful simply because “something moves on the screen.”

    6. Common Mistakes: Pitfalls AI Development Often Hits Under UEFI

    In the UEFI environment, common errors in AI-generated code are often not syntax issues but rather an insufficient understanding of firmware constraints. The following checklist can serve as a review focus:

    1. Graphics output protocol not initialized or not enabled
      Writing directly to the frame buffer without first confirming that the graphics output protocol is available.

    2. Wrong pixel format judgment
      Different firmware may use different PixelFormats, and the color ordering and bit layout may differ.

    3. Misunderstanding stride
      Mistaking PixelsPerScanLine for the screen width or a byte count, causing a misaligned display.

    4. Mixing up scan codes and characters
      Arrow keys usually require scan codes, while ordinary input may require UnicodeChar.

    5. Relying on busy-waiting to control frame rate
      This easily consumes CPU and leads to poor input responsiveness or system instability.

    6. Missing boundary checks
      Writing out-of-bounds coordinates to the frame buffer can cause display anomalies or memory corruption.

    7. Omitted state machine conditions
      States such as power pellet, respawn, and frightened are not fully covered, causing logic that “appears to exist but cannot be triggered.”

    8. No isolated validation
      Repeatedly writing directly to the target device or a critical firmware environment is high-risk.

    7. How to Define Verifiable Low-Level Development Tasks for AI

    Drawing on this practice, the acceptance methods for AI low-level programming tasks can be summarized into several questions:

    7.1 Are the Constraints Clearly Written?

    At minimum, the following should be specified:

    • Target architecture: x86_64, ARM, etc.;
    • Runtime environment: UEFI Shell, bare metal, or emulator;
    • Toolchain: EDK II, GCC, Clang, LLD;
    • Whether third-party libraries are allowed;
    • What the acceptance criteria are.

    The clearer the constraints, the less likely the AI is to make wrong assumptions under ambiguous premises.

    7.2 Is the Verification Environment a Closed Loop?

    Low-level tasks cannot focus only on “code was generated.” The AI should be made to first establish a runnable environment, such as:

    • QEMU + OVMF booting the UEFI Shell;
    • A local host simulation test harness;
    • Unit tests for key input and rendering logic;
    • Automated screenshots or state-output logs.

    Without a verification loop, the AI easily falls into a repetitive “generate—fail—regenerate” cycle.

    7.3 Is Validation Done Module by Module?

    It is recommended to proceed module by module:

    1. First confirm that graphics output can display a solid-color background;
    2. Then draw a single pixel, rectangle, or character;
    3. Then handle keyboard input;
    4. Then implement movement and boundaries;
    5. Finally, connect the ghost state machine and power-pellet logic.

    Layered validation reduces the cost of locating problems when they would otherwise erupt all at once.

    7.4 Is the State Machine Tested Separately?

    For logic such as power pellets, respawn, and frightened states, test cases should be designed separately. The fact that the character can move does not mean the game logic is correct.

    7.5 Is Hardware Interaction Reviewed Manually?

    When code involves the frame buffer, registers, memory mapping, or firmware interfaces, manual review should be retained. It is advisable to validate in an isolated environment or simulator first, avoiding direct writes to the frame buffer or registers that could cause display anomalies, memory corruption, or device malfunction. When real hardware is involved, be sure to confirm the firmware version, display mode, pixel format, and the protocol return status.

    8. Summary and Developer Checklist

    In environments like the UEFI Shell that have no operating system support, intelligent coding tools can participate in tasks such as graphics output, input response, game logic, and test environment setup, demonstrating strong code-generation and problem-exploration capabilities. However, low-level development demands extremely high correctness, and a clear gap still exists between one-time generation and a finally usable result.

    A practical developer checklist could include:

    • Have the target architecture, firmware environment, and toolchain been clearly defined?
    • Have executable acceptance criteria been defined?
    • Has validation first been performed in a simulator or isolated environment?
    • Have the graphics output protocol, pixel format, stride, and boundaries been confirmed?
    • Have input scan codes, characters, and repeated triggering been tested?
    • Has the state machine been validated separately?
    • Is manual review retained for hardware-related code?
    • Has a closed loop of “detect a problem—describe the symptom—locate and fix—regression-test” been formed?

    Overall, AI coding tools are better positioned as efficient coding and exploration partners in low-level development, rather than as fully autonomous development tools that can completely replace human judgment. Only by combining clear constraints, simulation-based validation, modular progression, and manual review can AI be used more reliably in low-level scenarios such as UEFI.