Deep Dive: Testing Domestic LLMs in UEFI Low-Level Development

Deep Dive: Testing Domestic LLMs in UEFI Low-Level Development

When evaluating the code generation capabilities of Large Language Models (LLMs), standard web development or simple scripting tasks are no longer sufficient to differentiate their performance. To truly test a model’s grasp of underlying hardware architectures, memory management, and programming in OS-less environments, we designed a series of extreme tests: tasking AI to autonomously develop five classic games within the UEFI (Unified Extensible Firmware Interface) Shell environment.

I. Test Background and Strict Benchmark Environment

UEFI development serves as a true “touchstone” for LLM capabilities because it strips away the protections provided by modern operating systems. There is no standard C library, no mature graphical UI framework, and not even a basic file system abstraction layer. Developers must interact directly with the system table provided by the firmware, utilizing Boot Services and Runtime Services to allocate memory and handle input/output operations.

The rules for this series of tests were exceptionally strict:

1. Environment Constraints: The physical Windows machine was installed only with a basic compiler toolchain (such as Visual Studio 2019), without pre-installing any specific UEFI development packages.
2. Zero Codebase: No historical code or templates were available locally. The AI had to generate all code from scratch, including boot protocols and graphics driver calls.
3. Autonomous Decision-Making: Only high-level requirement prompts (e.g., “Write a Pac-Man game”) were provided. No manual technical guidance was given. The AI autonomously planned the architecture, resolved compilation errors, and completed debugging throughout the entire process.

In the first four rounds of testing, different LLMs exhibited vastly different levels of technical maturity. Some models successfully compiled a Breakout game on the first attempt and ran it in the BIOS within two and a half hours. Other models spent hours trapped in UEFI-specific compilation and linking errors. Furthermore, while some models managed to complete the core logic of a Snake game, they stumbled over underlying pitfalls in graphics rendering and keyboard interrupt responses. These differences fully exposed the gap between models in terms of system-level programming knowledge reserves and logical error-correction capabilities.

II. The Fifth Round Challenge: Autonomous Implementation of UEFI Pac-Man

In the fifth test, the task objective was set to develop a Pac-Man game under the UEFI Shell. Compared to Breakout or Snake, the complexity of Pac-Man increases exponentially. It requires not only basic graphics drawing but also complex maze pathfinding algorithms, independent AI state machines for four ghosts, collision detection for dots and power pellets, as well as the special mechanism where power pellets allow the protagonist to eat ghosts.

III. Core Technical Highlights and Capability Breakdown

During this task, the LLM demonstrated remarkable system-level engineering capabilities, specifically breaking through the following core technical dimensions:

1. Proactive Construction of a Win32 Simulation Test Environment
This was the most prominent engineering practice in the test. Typically, UEFI programs must be verified by rebooting on real hardware or virtual machines like QEMU, resulting in a very long debugging cycle. Without receiving explicit instructions, the LLM proactively determined that the verification cost in a pure UEFI environment was too high. Consequently, it independently designed and generated a Win32 simulation environment.
In this simulated environment, the model bridged the UEFI EFI_GRAPHICS_OUTPUT_PROTOCOL (Graphics Output Protocol) to Windows GDI or Direct2D APIs, and mapped the UEFI Simple Text Input Ex Protocol to the Win32 message loop mechanism. This dimensional reduction simulation approach allowed the game logic to undergo rapid trial-and-error verification in Windows at millisecond speeds, significantly boosting R&D efficiency.

2. Complete Graphics Rendering and State Machine Implementation
After the first prompt was issued, the main game framework surprisingly ran successfully on the first try. The screen successfully rendered the classic maze layout, the yellow protagonist, four ghosts with eye details, and scattered dots. This proved that the model not only retrieved the classic visual layout memory of Pac-Man but also accurately translated it into underlying pixel drawing or character rendering instructions.
Regarding AI logic, the model implemented a basic ghost chasing algorithm (such as greedy pathfinding based on Manhattan distance) and constructed a complete game frame loop with extremely low operation response latency. This demonstrated the model’s profound understanding of real-time system event loops.

3. Agile Logic Repair and Closed-Loop Capability
Despite the stunning start, the model exhibited a logical flaw in the power pellet mechanism: after consuming a power pellet, the ghosts’ state did not switch to “fleeing,” and they could not be eaten by the protagonist. This was a classic case of missing state machine transition conditions.
When this issue was pointed out, the model quickly conducted a code review. It accurately located the interaction defect between the collision detection module and the ghost state management module, added a global timer variable that activates when a power pellet takes effect, and rewrote the decision weights for the ghost AI. The repaired code was recompiled and executed, and the power pellet mechanism worked perfectly, demonstrating a strong self-correcting closed-loop capability.

IV. Underlying Logic Perspective: From Prompts to System Architecture

The reason the LLM achieved such breakthroughs in UEFI development is not simple “probabilistic stitching,” but rather based on a deep understanding of system architecture:

• Protocol Abstraction Capability: The model can accurately understand and apply core concepts in the UEFI specification. For example, during the initialization phase, it generates standard EFI_STATUS return value checks and correctly uses gBS->LocateProtocol to obtain the graphics output protocol interface. A representative code snippet is as follows:

  EFI_STATUS Status;
  EFI_GRAPHICS_OUTPUT_PROTOCOL *GraphicsOutput;
  Status = gBS->LocateProtocol(&gEfiGraphicsOutputProtocolGuid, NULL, (VOID**)&GraphicsOutput);
  if (EFI_ERROR(Status)) {
      return Status; // Error handling
  }
  GraphicsOutput->Blt(GraphicsOutput, ...); // Execute Blt operation

• Memory Management Adaptation: In an OS-less environment, memory allocation must rely on AllocatePool or AllocatePages. When generating dynamic data structures (such as maze node lists and ghost entity queues), the model consciously avoids using the standard malloc, opting instead for underlying UEFI memory allocation interfaces. This ensures the firmware-level security of the program.

V. Conclusion and R&D Implications

Through these five rounds of extreme testing, we can clearly see that leading domestic LLMs have acquired the critical capability to transition from “usable” to “practical” in the field of low-level programming. They can not only understand complex hardware specifications but also proactively construct engineered simulation environments, achieving end-to-end autonomous development from requirement comprehension to compilation and execution.

For enterprise-level R&D, this capability evolution signifies massive efficiency dividends. In high-barrier fields such as driver development, firmware customization, and embedded system porting, intelligent programming tools can already serve as capable pair programming partners. Developers can focus their energy on architecture design and protocol definition, leaving the tedious underlying code implementation, cross-platform simulation bridging, and basic logic debugging to the LLMs. As models further deepen their grasp of system-level details, a fully automated low-level R&D process is no longer out of reach.