Testing LLMs on Bare-Metal Programming: Why Passing in Simulation Isn't the Same as Running on Real Hardware

    This article examines how coding-oriented large language models perform on a demanding class of task: building a runnable program from scratch in a bare-metal environment. Using a UEFI Shell Pac-Man game as the case study, it walks through what the model did well (environment self-bootstrapping, architectural intuition, fast debugging, and interaction polish) and where it fell short (UEFI-specific type, memory, input, entry-point, and timing conventions that a self-built Win32 simulation quietly hid). It then lays out an engineering workflow to turn code that "looks like it runs" into code that "actually runs on real hardware": interface-first design with platform-agnostic contracts, build-time constraints and static assertions, a three-layer build/simulation/regression verification loop, and clear boundaries for when the model is a safe accelerator versus when human review is mandatory.

    Introduction: The Bare-Metal Environment Is a Stress Test for Coding Models

    Evaluating coding models with algorithm problems mainly measures reasoning ability. Evaluating them with business systems measures how familiar they are with framework ecosystems. But there is a third category of task that concentrates and exposes a model’s weak spots: developing a working small program from zero in a bare-metal environment.

    Take a UEFI Shell mini-game as an example. This kind of task has four defining characteristics at once:

    • A barren runtime: no OS scheduling, no standard C library, and no mature windowing or graphics framework.
    • Strict entry-point conventions: the program is loaded directly by firmware, with a well-defined entry signature and status-code convention.
    • A long feedback loop: a single struct definition error may compile cleanly yet only crash at load time.
    • Few debugging tools: failures usually show up as an unresponsive screen or a machine reboot, not as a clear error log with a readable stack trace.

    For exactly these reasons, this kind of task has become a representative sample for observing a coding model’s engineering ability. Using the case of “a model building a UEFI Shell Pac-Man game from scratch,” this article maps out its performance across two dimensions—highlights and traps—and offers an engineering approach to turn “looks like it runs” into “actually runs on real hardware.”

    Part 1: Experiment Design—Clear Rules Make Results Reproducible

    Task definition: Implement a Pac-Man game under the UEFI Shell, including the following features:

    • Maze rendering, pellets, and power pellets;
    • Player movement and pellet-based scoring;
    • Ghost tracking and collision detection;
    • State transitions during the power-pellet effect window;
    • Restarting a new round;
    • Game state display and level progression.

    Environment constraints:

    • The host machine runs Windows with only a standard C/C++ compiler installed;
    • No cross-compilation toolchain is provided, and there is no existing local codebase;
    • Apart from the launch instruction, essentially no expert guidance is given.

    Evaluation criteria: The assessment breaks down into observable metrics, including whether the code can build, whether it can run in a UEFI environment, feature completeness, architectural portability, and how efficiently problems get fixed. In practice there is one additional dimension: whether the model honestly states the boundaries of its verification.

    It’s worth stressing that a free-form, single case study is good for surfacing strengths and risks but is hard to use for head-to-head rankings. The same model can show noticeable variation depending on prompting style, environment configuration, and version updates. This article only discusses the capability characteristics reflected by this one case and the engineering lessons it offers.

    Part 2: Highlights—Four Standout Performances in a Bare-Metal Task

    2.1 Environment Self-Bootstrapping: Building Its Own Verification Tools Instead of Waiting to Be Handed One

    The constraints clearly stated that the host had only a standard compiler, but they didn’t say the model couldn’t build its own test environment. After the first instruction, the model not only produced the game code but also proactively stood up a Win32 simulation framework that runs directly on the host, and the game basically worked on the first run: maze, characters, ghosts, pellets, and power pellets all present, smooth movement, and correct score display.

    This is worth praising. Windows desktop development is richly represented in a coding model’s training corpus, so having the model bypass a complex toolchain and use familiar APIs to quickly validate game logic is a pragmatic choice.

    But we must stay clear-eyed: a self-built simulation environment is a double-edged sword.

    • Upside: It gives the model immediate feedback across a long development cycle and reduces “writing code blind.”
    • Risk: The model plays both “developer” and “referee” at once, and when the test environment and the implementation come from the same line of thinking, blind spots are easy to form.

    2.2 Architectural Intuition: Abstraction, a Main Loop, and a State Machine

    Looking further at the source code, the model applied reasonable layering to the project, roughly as follows:

    • Platform layer: abstracts away the differences between the host and UEFI in rendering and input;
    • Game layer: pure logic such as maps, characters, collisions, and scoring;
    • Presentation layer: rendering and text output.

    This “decoupling logic from platform” structure itself reflects an “interface-first” architectural awareness. The runtime behavior further shows that the model understood Pac-Man’s mechanics well:

    • Map encoding, character movement and wall collisions, ghost target-point selection;
    • Scoring for pellets and power pellets, state switching after a power pellet takes effect, game-over-on-collision rules.

    All of this is essentially consistent with the classic gameplay.

    2.3 Problem Localization: A Fast Fix for a Power-Pellet Flaw

    During playtesting, a functional flaw surfaced: after eating a power pellet, the ghosts did not switch into a fleeing state and could not be eaten in reverse, so the power pellet’s core mechanic didn’t actually take effect. After the problem was fed back to the model, it completed the fix in a short time.

    The key to the fix was the state model. A ghost’s state should be a continuously timed interval, not an instantaneous event:

    • On collision, if the power pellet has already expired or the ghost is fleeing, the player loses;
    • If the ghost is fleeing and hasn’t been eaten yet, the ghost is eaten and returns to its spawn point.

    Measured against this model, the earlier flaw most likely stemmed from missing state flags and timing logic—a typical case of an omitted rule implementation. This also shows that for this kind of functional problem, coding models often have strong localization and correction ability; paired with concrete symptom feedback, the iteration efficiency is high.

    2.4 Interaction Experience: Naturally Friendly Handling of Edge Cases

    The author noted that in the model-generated visuals the ghosts had eyes with a clear sense of direction, and the interaction feel was close to the classic Pac-Man in memory. For this kind of graphical interactive program, detail expression often depends on designs like “reusing rendering subroutines” and “managing character-direction state.” That the model proactively achieved this shows it has good expressive ability when it comes to reproducing the classic gameplay experience.

    Part 3: Traps—UEFI Real-Hardware Differences Masked by the Self-Built Simulation

    However, a closer review of the source code revealed a problem: despite abstracting UEFI to the Win32 API, the abstraction layer was still thin, and some definitions clearly did not conform to the UEFI spec and would likely fail to compile in a pure UEFI environment. This is exactly the typical side effect of a “self-built simulation.”

    3.1 Type System: Representations Disconnected from the UEFI Spec

    UEFI has strict conventions for data types:

    • Characters use the 16-bit CHAR16, i.e., UTF-16 encoding;
    • EFI_STATUS is a UINTN type, which the compiler pads to 64 bits on x86_64;
    • A status code is not a plain integer; errors are represented in the form “status code | (1 << 63)”.

    Comparing against this, the model’s code had several imprecise spots:

    • Character positions and map rows/columns were represented with int or even float, whereas in the UEFI ecosystem such values should use UINT32, UINT16, UINT8;
    • Error codes were defined directly as ordinary integers like 0, 1, 2, or structs were declared arbitrarily without including the official headers.

    In the Windows simulation environment, all of this runs perfectly, because the simulation layer silently performs implicit conversions and the compiler handles types by the host’s rules. But in a UEFI environment, struct layout, status-code checks, and character encoding all get misaligned—or linking fails outright.

    A typical example: the model defined a slimmed-down “input event struct” that didn’t match EFI_INPUT_KEY in the UEFI spec. In the simulation layer, reading keyboard input worked fine; on real hardware, the firmware fills in a spec-conforming struct, the field meanings on both sides don’t line up, and the result is either that key presses can’t be read or the wrong content gets read.

    3.2 Memory Layout and Bitwise-Operation Assumptions

    UEFI’s framebuffer descriptor includes fields such as the starting address, width, height, and pixels per line. If the model defined its own slimmed-down struct without accounting for field order or type sizes, the pointer read in the simulation environment would be wrong. Similarly, color packing, pixel-format detection, and RGB-to-BGR conversion, if they rely on the host’s little-endian byte order, will produce garbled colors on real hardware.

    3.3 Input Event Model Differences

    A Windows simulation layer usually captures key presses via GetAsyncKeyState or a message queue—an “instantaneous state” model. UEFI keyboard input, by contrast, is implemented through ConIn‘s ReadKeyStroke, which returns an EFI_INPUT_KEY struct containing two fields, UnicodeChar and ScanCode, and returns EFI_NOT_READY when no key is available. UEFI provides no instantaneous-query interface like GetAsyncKeyState, so:

    • Input must be polled actively in the main loop, and the “no input” state must be handled;
    • Arrow keys, the space bar, and the like require reading scan codes, not character codes;
    • The timing of event reads must be coordinated with the render loop, or key presses get dropped or the system hangs.

    The model’s simulation layer very likely didn’t handle these differences, so the code ended up using a “key takes effect immediately” calling style that simply doesn’t exist in UEFI.

    3.4 Entry Point and Lifecycle

    A UEFI application must implement an entry signature like EFI_STATUS EFIAPI UefiMain(IN EFI_HANDLE ImageHandle, IN EFI_SYSTEM_TABLE *SystemTable) and be compiled with the EFIAPI calling convention. If the code uses a main() entry, or calls standard-library functions without linking the UEFI libraries, it will simply fail to load or run on real hardware.

    3.5 Time Handling

    A Windows simulation layer can use time mechanisms such as Sleep, timer messages, or QueryPerformanceCounter. A UEFI environment, by contrast, has no OS clock scheduling; time handling must rely on timer events provided by Boot Services, or approximate the frame rate with a loop counter. The timing approach the model relied on in the simulation layer either doesn’t exist in UEFI or behaves completely differently.

    3.6 Summary: The Nature of the Problem

    The model did do “abstraction,” but this abstraction only maps UEFI APIs onto Win32 APIs at a shallow, surface level. It didn’t form a genuinely “platform-agnostic interface”; instead, it wrapped UEFI concepts in Windows idioms. When the runtime switches to UEFI, this adaptation layer simply can’t compile.

    Part 4: An Engineering Approach—Turning “Looks Like It Runs” into “Actually Runs on Real Hardware”

    4.1 Core Idea: Interface-First, Decoupling Logic from Platform

    The key to breaking this deadlock isn’t having the model “write more code”; it’s defining clear interface contracts first:

    • Interfaces use only native UEFI types: interface headers depend only on UEFI-defined base types (UINT8, UINT16, UINT32, CHAR16, and so on), with no Windows types;
    • Design interfaces as “platform-agnostic”: game logic calls platform capabilities only through interfaces, without caring whether the underlying layer is Win32 or UEFI;
    • Provide a separate adapter layer for each target platform: the UEFI adapter layer and the simulation adapter layer are implemented independently; the interface layer only defines the contract and holds no logic.

    Example of a Platform-Agnostic Interface

    1. /* Platform-agnostic interface: uses only native UEFI types, no dependency on any specific platform */
    2. typedef unsigned char UINT8;
    3. typedef unsigned short UINT16;
    4. typedef unsigned int UINT32;
    5. typedef unsigned short CHAR16;
    6. /* Keyboard event: ScanCode == 0 means a normal character key */
    7. typedef struct {
    8. UINT16 Unicode;
    9. UINT16 ScanCode;
    10. } KeyEvent;
    11. typedef enum {
    12. KEY_NONE = 0,
    13. KEY_UP, KEY_DOWN, KEY_LEFT, KEY_RIGHT,
    14. KEY_FIRE, KEY_RESTART, KEY_QUIT
    15. } KeyAction;
    16. typedef struct {
    17. int width, height; /* logical resolution */
    18. unsigned char *pixels; /* RGB888 framebuffer */
    19. } FrameBuffer;
    20. typedef struct {
    21. void (*init)(void);
    22. int (*poll_key)(KeyEvent *out); /* 1=key read, 0=no input, non-blocking */
    23. KeyAction (*map_key)(const KeyEvent *key);
    24. void (*sleep_ms)(UINT32 ms);
    25. void (*present)(const FrameBuffer *fb);
    26. void (*abort)(void); /* abnormal exit: print diagnostics then end the program */
    27. } GamePlatform;
    28. /* UEFI adapter layer declaration */
    29. extern GamePlatform uefi_platform;
    30. /* UEFI adapter layer implementation (key logic only, illustrative) */
    31. static int uefi_poll_key(KeyEvent *out)
    32. {
    33. EFI_STATUS st;
    34. st = gBS->CheckEvent(gST->ConIn->WaitForKey); /* non-blocking check */
    35. if (EFI_ERROR(st)) {
    36. return 0; /* EFI_NOT_READY: no key pressed */
    37. }
    38. st = gST->ConIn->ReadKeyStroke(gST->ConIn, &key);
    39. if (EFI_ERROR(st)) {
    40. return 0;
    41. }
    42. out->Unicode = key.UnicodeChar;
    43. out->ScanCode = key.ScanCode;
    44. return 1;
    45. }

    Game Logic: Depends Only on the Interface

    1. /* Game logic: depends only on the GamePlatform interface, agnostic to whether the
    2. underlying layer is Win32 or UEFI */
    3. void game_run(GamePlatform *pf)
    4. {
    5. pf->init();
    6. for (;;) {
    7. KeyEvent ev;
    8. KeyAction act;
    9. if (pf->poll_key(&ev)) {
    10. act = pf->map_key(&ev);
    11. /* handle movement, fire, restart, quit, and other actions */
    12. }
    13. game_update(); /* update logic, collisions, scoring, etc. */
    14. pf->present(&fb); /* render and display the current frame */
    15. pf->sleep_ms(16); /* control frame rate, ~60 FPS */
    16. }
    17. }

    The UEFI adapter layer needs to handle a few more things:

    • No input: when ReadKeyStroke returns EFI_NOT_READY, return 0 to the caller, and the game loop continues accordingly;
    • Scan-code mapping: scan codes for arrow keys and the like must be mapped to a unified KeyAction, distinct from character input;
    • Exit: the ESC-key exit logic might just close a window in a host simulation environment, whereas in UEFI it should print a line of diagnostics and return directly, handing control back to the UEFI Shell;
    • Fallback on exceptions: on a rendering or logic error, call abort to print the error code and return a failure status.

    The host simulation adapter layer fills the same KeyEvent struct from window messages or input events and calls the same game logic. This way, the code is written once and can be validated in both environments.

    4.2 Build-Time Protection: Blocking “Out-of-Bounds” Code with Constraints and Compile-Time Assertions

    During free generation, the model tends to “improvise,” so it’s important to state constraints clearly at the prompt level:

    • Entry point: must use the UEFI application entry signature; main() entry is forbidden;
    • Standard library: standard-library functions are forbidden, and so are host-platform-specific types;
    • Types: using int/float to represent character positions, color packing, and the like is forbidden; must use UEFI base types;
    • Input: must poll via ConIn‘s ReadKeyStroke; relying on the “key takes effect immediately” assumption is forbidden;
    • Time: must specify the frame-rate control method; relying on the system clock or Sleep-style functions is forbidden;
    • Structs: if you define your own structs, you must match the UEFI spec’s field order and types, and use static assertions to verify sizes where necessary;
    • Verification boundaries: must state that “passing in simulation doesn’t mean passing on real hardware,” and proactively require or recommend real-hardware verification.

    You can also add compile-time assertions to the interface header so that non-compliant code fails at compile time:

    1. /* On compilers that support C11, use _Static_assert to verify struct size */
    2. _Static_assert(sizeof(KeyEvent) == 4, "KeyEvent layout mismatch");
    3. /* If the compiler doesn't support _Static_assert, use the typedef array trick */
    4. typedef char keyevent_size_check[
    5. (sizeof(KeyEvent) == 4) ? 1 : -1
    6. ];

    4.3 Validation-Phase Approach: A Three-Layer Closed Loop of Build, Simulation, and Regression

    Build Verification

    • Use the official UEFI build toolchain to ensure the code compiles and links on the target architecture;
    • Load the build artifact as a UEFI application to confirm the entry point is correct and there are no undefined symbols;
    • At the link stage, check for any accidental dependency on host-platform libraries.

    Simulation Verification

    • Boot with a UEFI-compatible firmware environment (such as VM firmware) and load the compiled program;
    • Output key state over the serial log (initialization, per-frame score, game over, and so on) for automated assertions;
    • Use a screenshot tool to capture the screen, combined with scripted key sequences, for manual acceptance;
    • Cover edge cases: no input, arrow keys and character keys arriving simultaneously, ghost state transitions during the power-pellet window, pause and resume, and so on.

    Regression Verification

    • Create a test case for every fixed defect (such as “ghosts don’t flee”) to ensure later changes don’t cause regressions;
    • Fold these cases into an automated pipeline as must-pass gates before merge;
    • Archive test evidence such as simulation logs, screenshots, and serial output in a unified store for traceability;
    • In the cloud, leverage common capabilities like object storage, container platforms, log services, and monitoring/alerting to chain these steps into a pipeline: build trigger, simulation execution, log archival, and result alerting, forming a continuous feedback loop.

    4.4 Usage Boundaries: When to “Let Go” and When to “Tighten Up”

    Scenario Model’s Role Human Responsibility Risk Level
    Learning UEFI development Teaching assistant Validate examples against the spec Low
    Prototype validation Quickly generate a playable demo Accept the experience and rules Medium
    Product-grade firmware development Generate module code Build verification, real-hardware testing, code review High
    Safety-related firmware Reference only Line-by-line human review, following secure coding standards Very high

    As you can see, a self-built simulation environment is a plus for learning and prototyping, but for product-grade delivery it only “quickly validates logic”—it can’t replace “validating platform adaptation.”

    Part 5: Conclusion—Credit the Highlights to the Model, Attribute the Risks to the Process

    Looking back at this experiment, the coding model’s performance can be summarized in three points:

    • Highlights: architectural intuition, fast fixing, and environment self-bootstrapping turned bare-metal programming from a “high barrier to entry” into a “task you can pick up.”
    • Trap: the self-built simulation had the model repeatedly proving itself in an “unreal” environment, masking the differences from real UEFI hardware.
    • Engineering lesson: “interface contracts, build verification, and simulation regression” must be set as hard constraints up front, not patched on after the fact.

    For developers, the following checklist can help judge whether a coding model is ready for product-grade firmware development:

    • Does the generated code use native UEFI types and entry points?
    • Does it include a note that “passing in simulation doesn’t mean passing on real hardware”?
    • Can it integrate with build, simulation, and regression workflows?
    • Are boundary conditions spelled out at the prompt level?

    If the answer to any of these is no, then it’s better suited as a “prototype accelerator” or “learning assistant” rather than a “product-grade code generator.” The value of bare-metal programming lies in rigor—and rigor isn’t something a model is born with; it’s something a process gives it.