Large Language Model Coding in Practice: What Firmware Game Development Reveals About Code Repair

    As large language models enter software development workflows, their value is increasingly measured by more than code generation. This article examines a firmware-based graphics game development exercise to evaluate how well models understand existing projects, configure environments, diagnose build and runtime failures, and iteratively move imperfect code toward a runnable and maintainable state. It also outlines a practical framework for designing fairer coding evaluations and improving developer–model collaboration.

    As large language models become part of software development workflows, developers are paying attention to more than whether a model can write code. The more important questions are whether it can understand a real project, configure the execution environment, locate build or runtime failures, and repair code that is already incomplete or poorly structured.

    A firmware graphics game development exercise provides a useful way to examine these capabilities. One group of models was asked to build a simple graphics game from scratch, while another attempted to implement a different classic game. The first group completed code generation, compilation, and an initial run in a relatively short time. The second produced multiple source files with some degree of modular structure, but repeatedly encountered problems during environment setup and startup. It ultimately failed to produce a working application.

    These results should not be interpreted as proof that one model is universally stronger than another. The tasks were not identical: their complexity, input methods, game logic, and debugging difficulty differed. However, the exercise still highlights an important point. When evaluating an LLM’s programming ability, the most meaningful metric is not how much code it generates in one attempt, but whether it can move that code toward a verifiable, runnable, and maintainable state.

    1. Testing Should Go Beyond Code Generation

    Traditional code-generation tests usually provide a model with a requirement, ask it to output a function or complete program, and then evaluate whether the result behaves as expected. In real development, however, code is only the starting point. Follow-up work often includes:

    • Establishing the correct build environment;
    • Understanding the project directory and module dependencies;
    • Identifying how interfaces should be used;
    • Handling platform differences;
    • Locating failures from runtime logs;
    • Modifying and validating the result repeatedly;
    • Ensuring that a fix does not break other functionality.

    Firmware development makes these requirements especially visible because the program cannot assume that the conveniences of a conventional operating system are available. Developers may not have a filesystem, standard input and output, a mature graphics library, or a fully featured debugger. The program may interact directly with boot services, graphics output protocols, keyboard input protocols, and memory layouts.

    An LLM’s engineering ability should therefore be assessed from at least four perspectives:

    1. Requirement understanding: Can it break down the game logic while respecting platform limitations?
    2. Code organization: Can it separate the entry point, rendering, input, collision detection, and state management?
    3. Debugging reasoning: Can it propose testable failure hypotheses based on observed behavior?
    4. Execution of the feedback loop: Can it modify the code, rebuild the project, and confirm whether the issue has been resolved?

    2. Giving the Model an Existing Codebase Is Closer to Real Development

    To examine code-repair ability more directly, an existing project can be copied into a new development environment without modifying the files or reorganizing the structure in advance. The model is then asked to review the code before attempting to build and run it.

    This approach is closer to real engineering than asking a model to create a program from an empty directory. Real projects rarely start from scratch. Developers often inherit legacy code, experimental code, or code generated by another system. Such projects may appear well organized while containing lifecycle errors, incorrect interface calls, platform-configuration problems, and missing error handling.

    During the review stage, a model will typically inspect questions such as:

    • Does the program entry point initialize the runtime environment correctly?
    • Does the main loop have a stable termination condition?
    • Are game state and rendering state synchronized?
    • Can input events be lost?
    • Is the graphics buffer acquired correctly?
    • Do resource pointers remain valid after an interface call?
    • Are data structures consistent across modules?
    • Do error paths provide logs or handle return values?

    One important lesson is not to ask the model to “fix everything” immediately. A more reliable process is to have it produce an audit first, then verify issues one by one according to priority. This reduces the risk of large-scale rewrites before the root cause has been confirmed.

    3. A Typical Failure: Immediate Exit After Startup

    In one practical scenario, the program completed initialization and printed a message indicating that the main loop had started, but then exited immediately. It would be easy to conclude that the main loop never ran. In reality, the cause could also be a failed graphics-interface call, an unexpected return path, an invalid pointer, or an initialization function that ended prematurely.

    The key to resolving this type of problem is observability. The model can be asked to insert serial logs or debug output at critical stages, for example:

    1. Entering initialization
    2. Starting graphics output protocol lookup
    3. Display mode configured
    4. Frame buffer information acquired
    5. Entering the game loop
    6. Reading input events
    7. Drawing the first frame

    The last successful message helps define the failure boundary. If logging stops after a graphics-output protocol call, the investigation can be narrowed from “the entire program” to protocol acquisition, mode configuration, or the related data structures.

    A common error in this area occurs when a program queries display information and saves an information structure or buffer pointer, then calls a mode-setting interface that invalidates the previously saved address. If the old pointer is used afterward, the program may encounter an exception or exit immediately.

    A more reliable sequence is:

    1. Query the current display mode;
    2. Set the display mode;
    3. Retrieve the display information again after mode configuration;
    4. Calculate the frame-buffer address, row stride, and pixel format from the latest information;
    5. Validate the result of every relevant call.

    This issue is valuable because it tests more than syntax generation. It tests whether the model understands interface lifecycles and low-level memory semantics. A model that relies only on surface-level error messages may repeatedly try unrelated changes. A model that can reason that a resource’s state may change across an interface call has a better chance of identifying the root cause quickly.

    4. Another Typical Failure: Incorrect Protocol Identifier Configuration

    A program no longer crashing does not mean that its graphics interface will appear. Another common symptom is that startup proceeds normally and logs continue, but the screen remains blank.

    In a firmware environment, graphics output is often exposed through a protocol mechanism. A protocol includes more than a function table; it must also be located through the correct global identifier. A platform may contain several similar protocol identifiers. If the wrong identifier is used, the program may fail to obtain a valid graphics interface.

    This type of problem should not be investigated by looking only at the drawing function. The checks should proceed in the following order:

    • Is the protocol identifier exactly the one required by the target interface?
    • Did protocol acquisition succeed?
    • Is the returned interface pointer null?
    • Does the current device actually support graphics output?
    • Did display mode configuration succeed?
    • Is the frame-buffer address valid?
    • Are pixel format and row stride handled correctly?
    • Does the drawing region remain within the screen boundaries?

    The model’s role here is not simply to add a few lines of code. It should help establish the complete chain from interface discovery to pixel writes. Developers should also ask the model to explain the basis for each check rather than accepting unverified modifications.

    5. Why Modular Code Does Not Automatically Mean Runnable Code

    Automatically generated projects often have a reasonable surface structure. For example, the entry point, game logic, input handling, and rendering may be placed in separate modules, making the project appear more organized than a single-file program. However, modularization only shows that the code has been divided into files. It does not prove that the contracts between those modules are correct.

    Common problems include:

    • One module uses a data structure that differs from the one expected by another module;
    • The coordinate range returned by the input module does not match the assumptions of the rendering module;
    • Game logic uses screen coordinates while collision detection uses grid coordinates;
    • The rendering module assumes a contiguous buffer even though the actual row stride does not equal the screen width;
    • Initialization order does not satisfy dependency requirements;
    • Error codes are ignored, causing later modules to use invalid resources.

    Code review should therefore go beyond asking whether the files are divided sensibly. It should examine whether each module’s interface satisfies its contract. A useful approach is to ask the model to list each module’s inputs, outputs, lifecycle, and failure conditions, then verify that callers actually handle those cases.

    6. How to Design Fairer LLM Programming Evaluations

    When the goal is to compare different models, the evaluation should control as many variables as possible. The following practices are useful.

    1. Use the Same Task

    Every model should receive exactly the same requirements, including the target platform, input method, display requirements, build process, and acceptance criteria. It is not meaningful to compare a model building a simple animation with another building a complex game and then compare their completion times directly.

    2. Standardize the Environment

    Compiler versions, directory structures, startup procedures, and available dependencies should remain consistent. Environmental differences may have a greater effect on the outcome than differences between models.

    3. Record the Full Process

    Do not record only the final result. Also track:

    • Time required for the initial generation;
    • Number of build errors;
    • Number of runtime errors;
    • Number of model revisions;
    • Whether the model can read and use logs proactively;
    • Whether it repeats the same class of error;
    • Whether the final program is maintainable.

    4. Separate Generation from Repair

    Two categories of tasks can be evaluated separately: building a project from scratch and taking over a broken project. The first measures requirement implementation. The second measures code comprehension, debugging, and engineering repair.

    5. Define Clear Acceptance Criteria

    For example:

    • Can the project build successfully?
    • Can it enter the main loop?
    • Can it display graphics?
    • Can it respond to input?
    • Are the core game rules correct?
    • Does it remain stable during extended execution?
    • Are there obvious risks of resource leaks or out-of-bounds access?

    7. How Developers Can Improve Collaboration with Models

    For tasks of this kind, prompt quality directly affects the outcome. A staged collaboration process is often more effective.

    In the first stage, ask the model only to review the code and produce a list of issues with supporting evidence.

    In the second stage, ask it to fix only the highest-priority problems that prevent the program from starting, and to identify the files it changes.

    In the third stage, rebuild and run the project, then provide the complete logs to the model.

    In the fourth stage, address display, input, and game-logic problems.

    In the fifth stage, clean up the code, check boundary conditions, and improve the structure.

    Developers should also avoid giving feedback such as “it still does not work.” More useful information includes the last log message, the location of an exception, return values, visible screen behavior, and the differences between the code before and after a change. The more structured the feedback, the easier it is for the model to establish an effective path for fault isolation.

    Conclusion

    This exercise shows that the value of LLM-assisted programming is not limited to generating code quickly. A more important question is whether a model can complete the full process of understanding code, locating problems, making changes, and validating the result in a complex, low-level environment with limited runtime support.

    For developers, the most practical approach is not to treat an LLM as a one-time code generator, but as a collaboration tool that can participate in review, diagnosis, modification, and verification. Evaluations of model capability should likewise look beyond code volume or surface-level structure. The critical questions are whether the model can form technically sound hypotheses, use logs to validate its reasoning, and move a project through multiple iterations toward a genuinely runnable and maintainable state.