Evaluating LLM Coding Ability Through Firmware Game Development: Code Repair in Practice

    This article examines how large language models handle real engineering work through a practical test. An AI coding assistant built a simple graphical game in a firmware boot environment. The article explains why firmware development suits this kind of evaluation, how to design a fair test covering environment setup, acceptance criteria, and repair ability, and how the model debugged startup exits and blank screens. It also outlines where models help most and where human review remains necessary.

    As large language models become part of software development workflows, their ability should not be judged only by whether they can generate a piece of code. What matters more is whether they can understand a real engineering project, set up a working environment, locate low-level errors, and keep fixing problems after failures.

    One representative practice was to have an AI coding assistant develop a simple graphical game in a firmware boot environment. On the surface, the task only involves drawing blocks, handling keyboard input, and implementing collision detection. In practice, the developer must handle several layers at once: the boot flow, graphics protocols, memory lifetimes, build configuration, input devices, and debug output.

    The value of this task lies in what it reveals about a model’s ability to handle complex engineering problems, not in the game itself.

    1. Why Firmware Environments Suit LLM Evaluation

    In typical application development, failures can usually be traced quickly through logs, debuggers, or exception stacks. Firmware environments are more constrained:

    • There are no complete operating system services.
    • Standard input and output are limited.
    • Graphics output depends on low-level firmware protocols.
    • Memory and pointer errors can cause the program to exit immediately.
    • The build environment, boot media, and runtime environment must match each other.
    • Some errors only appear on real hardware or in a virtual firmware environment.

    Firmware development is therefore closer to a complete engineering task than to a code generation exercise. The model must do more than write code. It also needs to understand the project directory, module responsibilities, the boot entry point, low-level interfaces, and runtime constraints.

    In practice, the most visible differences between models often lie not in the amount of code they produce, but in the following areas:

    1. Whether the model reads and understands the existing project first.
    2. Whether it establishes an executable troubleshooting sequence.
    3. Whether it narrows the problem scope based on logs.
    4. Whether it re-verifies after modifying code.
    5. Whether it handles lifetime and compatibility issues in low-level interfaces.

    2. A Complete Test Should Not Compare Only Final Code

    If the comparison only looks at how much code a model generated, the conclusion can easily be misleading. Task complexity, initial code quality, and environment setup can all affect the results.

    A sounder test method should include at least four stages.

    1. Unified Development Environment

    First, fix the build toolchain, project template, run method, and test environment. Otherwise, one model may start from a project that is already configured, while another has to solve compiler, boot media, or virtual machine problems from scratch. The results would then be hard to compare.

    Where possible, the environment should make the following explicit:

    • The build target and architecture.
    • The project directory structure.
    • The boot entry point.
    • How the firmware runs.
    • The conditions for graphics output.
    • How input devices are simulated.
    • Where logs and serial output go.

    2. Clear Acceptance Criteria

    “It runs” is not enough as the only standard. Criteria should be split into:

    • Whether it compiles successfully.
    • Whether it starts normally.
    • Whether it enters the main loop.
    • Whether it displays the graphical interface.
    • Whether it responds to input.
    • Whether the game logic is correct.
    • Whether there are obvious resource leaks or out-of-bounds risks.
    • Whether the code remains maintainable after modification.

    3. Separate Generation Ability from Repair Ability

    Building a project from scratch mainly tests a model’s architecture design and code generation. Taking over flawed existing code better tests its code comprehension, debugging, and refactoring ability.

    In real development, the latter is more common. Enterprise projects rarely start from blank files. More often, teams inherit legacy code, fix outstanding issues, add features, or migrate runtime environments.

    4. Keep the Full Process Record

    A test should record more than the final result. It should also keep:

    • The model’s analysis process.
    • The build commands it ran.
    • Compiler errors.
    • Runtime logs.
    • Diffs before and after each change.
    • Whether each fix was re-verified.

    Only with this record can one judge whether a model solved the problem through sound reasoning or changed things until something happened to work.

    3. Troubleshooting from “Won’t Run” to “Basically Usable”

    In one repair session on a firmware graphics program, the initial project already had some structure. It included an entry module, a game loop, graphics drawing, input handling, and game logic. However, the program exited soon after startup and never reached a normal interface.

    In this situation, asking the model to simply “fix it” is usually inefficient. A better approach is to have the model perform a structured review first.

    1. Start with a Code-Level Review

    The model first needs to confirm:

    • Whether the entry function is correct.
    • Whether the initialization order makes sense.
    • Whether the main loop can keep running.
    • Whether the graphics output object was obtained successfully.
    • Whether the input device exists.
    • Whether key pointers are checked for null values.
    • Whether the drawing area uses the correct width and height.
    • Whether game objects are accessed before initialization.
    • Whether there are early returns or abnormal exit paths.

    The focus at this stage is not to change code right away but to build a list of problems. For low-level projects, identifying whether a fault belongs to the build stage, startup stage, protocol acquisition stage, drawing stage, or logic stage can greatly reduce wasted attempts.

    2. Increase Observability

    When a program exits directly, the developer usually cannot tell how far it got. The model should therefore add debug output at key points, such as:

    • Entering the entry function.
    • Completing initialization.
    • Obtaining the graphics output interface.
    • Setting the display mode.
    • Initializing the input device.
    • Entering the game main loop.
    • Completing the first frame.
    • Reading user input.

    These logs should follow a clear pattern of stage markers rather than being added at random. The last successful message shows which initialization step failed.

    In resource-constrained environments, serial logging is especially important. It cannot replace a debugger, but it gives developers a basic execution trace.

    3. Watch Pointer Lifetimes

    A typical problem occurs when the program gets a pointer to a structure while querying display mode information, then calls an interface that switches the display mode. After the switch, the memory behind the earlier information pointer may no longer be valid, yet the code keeps using it.

    In a general application, this might appear as an occasional crash. In a firmware environment, it can cause the program to exit directly.

    A safer approach is:

    1. Obtain the necessary information before switching the display mode.
    2. Perform the mode switch.
    3. After the switch, query the current display information again.
    4. Stop using the old pointer, which may now be invalid.
    5. Check the result of every interface call.

    This reflects an important skill when an LLM debugs low-level code. It must not only check syntax and function calls. It also needs to understand object lifetimes and the state changes that interface calls can cause.

    4. When the Screen Stays Blank, the Problem May Not Be in the Drawing Logic

    A program that no longer crashes is not necessarily finished. A second common problem is that the program runs normally but nothing appears on screen.

    A developer may immediately check rectangle coordinates, color values, or the drawing loop. In firmware graphics development, however, a more basic issue may be that the graphics output protocol was never acquired correctly.

    1. Protocol Identifiers Must Be Exact

    Firmware usually locates graphics output capabilities through protocol identifiers. A project may contain several similar identifiers with similar names but different purposes. If the wrong identifier is used, the program may compile without obvious errors yet fail to obtain a valid graphics output object.

    The developer should confirm that:

    • The identifier used is the one for the Graphics Output Protocol.
    • The arguments to the protocol query function are in the correct order.
    • The return status is checked.
    • The returned object is not a null pointer.
    • The current firmware actually supports graphics output.
    • The query result matches the drawing interfaces used afterward.

    2. Do Not Assume the Drawing Function Is at Fault

    Graphics problems can be divided into at least four layers:

    1. Protocol acquisition failure.
    2. Display mode setting failure.
    3. An invalid frame buffer or drawing target.
    4. Errors in coordinates, colors, or drawing algorithms.

    The correct approach is to verify from the b