Evaluating LLMs in Low-Level Graphics Development: Practical Performance and Methodology

    This article explores the practical capabilities of Large Language Models (LLMs) in low-level graphics development, such as firmware and bare-metal environments. Drawing from a real-world project of developing a Breakout game in a boot firmware environment, we examine how LLMs contribute to code review, debugging, and iteration. We also cover typical debugging challenges like pointer lifetimes and protocol ID misconfigurations, propose fair evaluation methods for comparing LLMs, and offer a two-iteration approach to transition from a runnable prototype to deliverable engineering code.

    Many developers assume that LLMs are primarily suited for generating business scripts, UI components, or common API calls. However, when tasks move into firmware, boot environments, graphics protocols, and bare-metal debugging, complexity rapidly increases: there is no full OS runtime support, debug information is limited, memory and pointer errors are hard to trace, and protocol interfaces can be affected by hardware and firmware discrepancies.

    A recent low-level graphics game development practice clearly demonstrated the actual value of LLMs in such tasks. The goal was to implement a Chinese UI Breakout game in a boot firmware environment and have the model take over a failing Tetris project. The results show that LLMs can already handle a significant amount of engineering work, but their capability is not about “inputting requirements and getting a finished product automatically.” Instead, their value lies in code review, root cause analysis, debugging strategy design, and repeated iteration.

    1. Why Low-Level Development is a Better Test for LLMs

    Standard application development usually operates in a mature environment where developers have access to debuggers, logging systems, dependency managers, and exception stacks. Firmware environments are different; programs often run directly in the boot phase, lacking a file system, process management, standard I/O, and comprehensive dynamic debugging capabilities.

    In this environment, a seemingly simple game actually involves at least the following technical steps:

    1. Initialize the firmware application entry point;
    2. Obtain the graphics output protocol;
    3. Query the display mode and frame buffer;
    4. Draw the background, paddle, bricks, and ball;
    5. Read keyboard input;
    6. Implement a timed loop and collision detection;
    7. Handle memory layout, pointer validity, and protocol object lifecycle;
    8. Configure the build environment and generate a bootable file;
    9. Verify the results in a virtual machine or on real hardware.

    Any error in these steps can cause the program to fail compilation, exit immediately upon boot, fail to display graphics, or hang without any obvious warning. Therefore, such tasks test not just whether the model can generate code, but whether it possesses systematic analytical capabilities.

    2. Key Steps for Successful Development

    In practice, an LLM can quickly complete a basic version not because it generates perfect code in one shot, but because the task is broken down into multiple verifiable phases.

    1. Build a Minimum Viable Framework First

    A reasonable approach is not to ask the model for the complete game right away, but to generate a minimal program that can:

    • Enter the firmware application entry point;
    • Find the graphics output protocol;
    • Get the current display resolution;
    • Clear the screen and draw a simple rectangle;
    • Output basic debug information;
    • Return normally or enter the main loop.

    Only after the minimal framework is confirmed to work should game logic be added. This separates “environmental issues” from “business logic issues,” preventing the model from handling too many variables at once.

    2. Add Features Modularly

    The basic graphics program can be split into several modules with clear responsibilities:

    • Graphics initialization module: Obtains display capabilities and the frame buffer;
    • Drawing module: Renders rectangles, borders, and simple text;
    • Input module: Reads key states;
    • Game state module: Records positions of the ball, paddle, and bricks;
    • Collision detection module: Handles boundary and object collisions;
    • Main loop module: Controls frame rate, updates state, and triggers redraws.

    Modular design is particularly important for LLMs. Compared to stuffing all code into a single entry function, clear responsibility boundaries make it easier for the model to review, locate issues, and make local modifications, while also helping developers take over the code later.

    3. Translate Runtime Behavior into Debugging Clues

    Low-level programs rarely tell developers directly “which line is wrong.” Therefore, feedback must be as specific as possible. For example:

    • Did compilation succeed?
    • Did the program enter the entry function?
    • Was the graphics protocol successfully obtained?
    • Did querying the display mode return valid results?
    • Can original information still be accessed after setting the display mode?
    • Is the main loop executing?
    • Is input reading blocking?
    • Was the first draw completed?
    • What was the last log before the program exited?

    Instead of a vague description like “the program doesn’t run,” a more valuable clue is “it exits immediately after entering the main loop, and the last log appears right after the graphics protocol call.” The model can narrow down the troubleshooting scope based on the execution path rather than regenerating an entirely different set of code.

    3. Typical Issue: Boot Crashes Caused by Pointer Lifetimes

    During one debugging session, the program appeared to compile successfully and printed a message indicating it entered the game loop upon booting, but then exited immediately. After adding serial logs, we confirmed the exception occurred near a graphics output protocol call.

    This type of issue is likely related to pointer lifetimes. Some query interfaces return structure pointers describing the display device state, and developers might continue using old pointers after setting the display mode. However, a mode switch might reallocate, refresh, or invalidate the relevant structures, making the original pointers unreliable.

    This error has several typical characteristics:

    • The compiler usually cannot catch it;
    • It sometimes runs, sometimes crashes;
    • It behaves inconsistently across different firmware or virtual environments;
    • The error location might be far from the actual cause;
    • It might even temporarily disappear under single-step debugging.

    A safer handling approach is:

    1. Use the old state only for necessary checks before switching the display mode;
    2. Execute the mode switch;
    3. Re-query the graphics output information after a successful switch;
    4. Re-confirm the frame buffer address, resolution, and pixel format;
    5. Proceed to the drawing phase.

    The value of the LLM here is not just pointing out “there might be a wild pointer,” but rather connecting the error to the interface call sequence and providing a fix that aligns with lifecycle logic.

    4. Typical Issue: Protocol ID Configuration Errors

    Another common issue is that the graphics interface doesn’t display, but the program itself seems to keep running. Developers might suspect the drawing algorithm, color format, or coordinate calculation. However, in firmware environments, protocol ID configuration is also a high-frequency failure point.

    Low-level environments typically define multiple protocol IDs with similar purposes. They might correspond to:

    • The protocol interface itself;
    • Protocol-related data structures;
    • Different versions or services;
    • Declaration objects in the build environment.

    If the wrong ID is used, the program might fail to find the correct graphics output interface. More troublingly, this error might not cause an obvious exception but instead manifest as “cannot obtain protocol,” “drawing function has no effect,” or “black screen after running.”

    When handling this, the model should be tasked with checking:

    • Whether the protocol ID matches the interface type;
    • Whether the variable type passed when getting the protocol is correct;
    • Whether the return status is checked;
    • Whether the protocol pointer is null;
    • Whether the graphics mode has been successfully set;
    • Whether the frame buffer address is valid;
    • Whether the pixel span per row is correctly used;
    • Whether the pixel format matches the color arrangement of the drawing function.

    This shows that low-level debugging cannot just look at business code. Entry points, protocols, data structures, memory layout, and drawing logic must be analyzed together.

    5. How to View “Comparative Testing” Between Models

    Developers often have two models perform different tasks and judge which is stronger based on the final results. For instance, one model completes a running Breakout game, while another tries to implement Tetris but fails to produce an executable program.

    These results can serve as personal user experiences but cannot be directly viewed as strict benchmarks. There are three main reasons. First, task difficulty differs. Tetris usually requires handling more states, including piece generation, rotation, boundary checks, landing, line clearing, and game over logic. Second, initial environments differ. One task might start from a blank project, while another is based on structurally complex but flawed legacy code. Third, debugging costs differ. A brand-new project easily forms a unified design, while taking over legacy code requires understanding the existing structure first and then fixing hidden issues.

    However, this type of testing is not entirely meaningless. In real development, developers often need models to take over half-finished products, fix legacy code, or solve unreproducible faults. Therefore, “whether a model can clean up structurally chaotic but not completely worthless code” is an important capability in itself.

    A fairer evaluation method should:

    • Use the same hardware and build environment;
    • Provide the same requirements;
    • Specify the same time and interaction limits;
    • Use the same initial code quality;
    • Unify the criteria for “completion”;
    • Record compilation success, first run, key feature pass, and defect count;
    • Distinguish between autonomous model completion and human intervention.

    6. Two Iterations from “Runnable” to “Deliverable”

    Low-level projects should not just use “whether it boots successfully” as the sole metric. A more reasonable approach is to divide it into two rounds.

    Round 1: Restore Basic Runnable State

    The goals for the first round include:

    • The project can compile;
    • The application can boot normally;
    • The graphics protocol can be correctly obtained;
    • The screen can display basic content;
    • The main loop can run continuously;
    • Input events can be read;
    • The program does not exit due to obvious pointer or state errors.

    The focus of this stage is to restore the system’s vital signs, without prematurely pursuing UI aesthetics or complex features.

    Round 2: Engineering Optimization

    After confirming the program runs, address the following issues:

    • Refactor duplicate logic into independent functions;
    • Reduce unnecessary full-screen redraws;
    • Unify coordinate and color representations;
    • Add boundary condition checks;
    • Prevent input handling from blocking the main loop;
    • Control the loop pace to avoid speeds dependent on device performance;
    • Clean up invalid variables and unused modules;
    • Add error states and debug logs;
    • Check return values for key interface calls;
    • Improve code comments and module documentation.

    This distinction is crucial. Many AI-generated programs can “demo successfully,” but they are still far from maintainable, scalable engineering code.

    7. Practical Tips for Developers Using LLMs

    To improve efficiency, it is recommended to follow these principles:

    1. Let the model explain before modifying
      First ask it to describe the call chain, data flow, and potential risks before deciding whether to accept the modification.

    2. Solve only one type of problem at a time
      Do not simultaneously ask to fix protocols, input, drawing, and collision logic, otherwise it will be difficult to judge the effect of the changes.

    3. Provide complete error context
      Include compile output, runtime logs, the last successful step, and reproduction steps.

    4. Require change boundaries
      Have the model specify which files and functions were modified, and why it made those changes.

    5. Verify immediately after each modification
      Do not accumulate multiple unverified patches, or it will be very difficult to backtrack if they fail.

    6. Manually review critical low-level code
      Especially for pointers, buffers, protocol handles, memory release, and concurrent access.

    Conclusion

    The takeaway from this practice is not that a specific model can replace developers, but that LLMs can already undertake relatively complete engineering collaboration tasks under specific conditions. They are particularly good at reading modular code, organizing issue lists, making hypotheses based on logs, and gradually restoring an unrunnable project to a verifiable state through multiple rounds of modifications.

    However, the model’s output still requires testing, observation, and human judgment. For firmware, drivers, embedded systems, and other high-risk low-level scenarios, “generating code” is just the starting point; “verifying hypotheses, controlling changes, and confirming results” is the core that determines delivery quality. Positioning the LLM as a development assistant capable of fast analysis and continuous iteration, rather than an unsupervised auto-programmer, is a more reliable and realistic way to use it.