LLM Programming in Action: From Firmware Game Development to Complex Code Repair
While Large Language Models (LLMs) can easily generate simple application code, their true engineering capabilities are tested in complex, low-level environments like firmware development. This article explores how to evaluate LLMs on firmware-based game development tasks, highlighting the challenges of graphics protocols, memory lifecycles, and limited debugging tools. It outlines a structured, closed-loop workflow for using LLMs to diagnose and fix code, shifting the focus from raw code generation to managing an effective debugging partnership between developers and AI.
In standard application development, generating a simple program with a Large Language Model (LLM) is not difficult. What truly reflects an LLM’s engineering capability, however, is its performance on low-level projects characterized by complex runtime environments, limited debugging tools, and strict interface specifications. For instance, implementing an interactive graphical mini-game in a firmware boot environment simultaneously involves engineering configuration, the compilation toolchain, boot processes, graphics protocols, input devices, memory lifecycles, and debug output.
Practical testing shows that when assigning similar game development projects to LLMs, results can vary significantly. One model successfully generated a firmware-based Breakout game in a short time, compiled smoothly, and booted successfully on the first try. Another model generated well-structured Tetris code but failed at the environment configuration stage, ultimately yielding no runnable result.
Such outcomes cannot simply be interpreted as “one model is inherently stronger.” Task complexity, context completeness, environment preparation, debug feedback quality, and the level of developer intervention all significantly impact the final performance. A more valuable approach is to break down the code generation process into an observable, reproducible engineering workflow to analyze exactly where the LLM contributed.
1. Look for Closed-Loop Engineering, Not Just Code Volume
For low-level development tasks, having many code files and clear module divisions does not guarantee that the project is actually usable. A seemingly standardized project might still suffer from the following issues:
- Mismatched build files and directory structures;
- Entry function declarations that do not meet runtime environment requirements;
- Incorrect use of protocol identifiers or data structures;
- Pointers referencing invalid memory regions;
- Uninitialized input events;
- Inconsistencies between graphics buffer resolution and the actual mode;
- Main loops missing exit conditions or exception handling;
- Generated code that has not been verified through actual compilation and booting.
Therefore, when evaluating LLM coding capabilities, you should focus on at least four levels:
- Can it generate a complete project? This includes source code, build configurations, and necessary resource files.
- Can it pass compilation? It must not only have correct syntax but also meet the linking and interface requirements of the target environment.
- Can it actually run? The program needs to complete booting, initialization, and core flows, not just stop at successful compilation.
- Can it fix issues based on feedback? When faced with real error messages, can the model narrow down the problem scope and provide effective fixes?
The last level is particularly important. Engineering development is rarely completed in one go. What truly impacts efficiency is whether the model can continuously work within the “run—observe—analyze—modify—rerun” loop.
2. Why Firmware Environments Severely Test LLMs
Unlike desktop applications, firmware programs typically lack mature graphics libraries, file systems, and debugging tools. Developers must directly confront low-level interfaces, making issues that are insignificant in standard apps highly critical.
Take graphics output as an example. A program generally needs to complete the following steps:
- Obtain the graphics output protocol provided by the current firmware;
- Query available display modes;
- Select the appropriate resolution and pixel format;
- Switch to the target mode;
- Re-confirm the framebuffer address, width, and scanline stride;
- Write image data according to the actual pixel format;
- Continuously refresh the screen in the main loop.
An error in any of these steps can lead to a black screen, garbled display, direct crash, or entirely different behaviors across different machines.
LLMs can usually infer the general workflow based on interface names, but the key challenge of low-level interfaces is not “knowing what the function is called.” It lies in understanding object lifecycles, return value semantics, and environmental differences. For example, a mode switch might invalidate previously queried buffer information. If the program continues to use the old pointer, an access exception may occur. The model only has a chance to accurately diagnose the problem after seeing serial logs, exception locations, or register information.
This indicates that the value of LLMs in low-level development is not to replace the debugging environment, but to help developers establish problem hypotheses faster and modify them based on feedback.
3. An Effective Engineering Repair Process
In practice, tasks can be divided into two rounds: “restoring to a runnable state” and “feature refinement,” rather than expecting the model to complete all features simultaneously from the start.
Round 1: Get the Project Booting
When dealing with non-runnable code, the first step should not be to immediately ask the model to rewrite it, but to have it perform a structural review. The review should cover:
- Whether the entry function complies with target environment specifications;
- Whether global variables and module dependencies are complete;
- Whether build files include all source files;
- Whether the main loop can be entered stably;
- Whether protocol retrieval checks return values;
- Whether memory allocations and frees are paired;
- Whether input devices are correctly registered;
- Whether the graphics buffer is initialized at the correct time;
- Whether there are uninitialized structure fields;
- Whether temporary variable addresses are saved for long-term use.
After the review, the model should be required to output findings in the format of “problem location, cause, impact scope, fix solution, and verification method,” rather than just a vague conclusion. This reduces omissions and allows developers to verify item by item.
Round 2: Increase Observability
When a program crashes immediately, it is usually impossible to visually determine where it stopped. At this point, debug information should be added, such as logging when:
- Entering the entry function;
- Completing graphics protocol retrieval;
- Querying a display mode;
- Completing the mode switch;
- Initializing input devices;
- Entering the main loop;
- Detecting a key press;
- Completing a frame render;
- Triggering an exception exit.
In environments lacking advanced debuggers, serial output is a highly practical tool. Printing a unique marker at each critical stage can quickly pinpoint whether the failure occurs during initialization, protocol calls, rendering, or input handling.
Note that logging should be kept moderate. Outputting massive amounts of information per frame can slow down the program and even affect timing. A more reasonable approach is to only log state changes, error branches, and periodic statistical information.
4. Typical Issue: Crash After Protocol Call
In one run, the program could display a boot prompt but immediately crashed upon entering graphics initialization. By adding stage logs, it was confirmed that the exception occurred right after a graphics protocol call.
Further inspection revealed that the program first saved a copy of the framebuffer information from the mode query results, then switched the display mode, and continued to use the previously saved address. The problem is that switching modes can change the underlying buffer layout, and previous descriptor information is not guaranteed to remain valid.
The correct approach is:
- Complete the display mode switch;
- Re-fetch the graphics information corresponding to the current mode;
- Re-read the buffer address, width, height, and scanline stride;
- Use the latest information for drawing;
- Check the return status after every critical call.
This type of issue is highly universal. In systems programming, whenever an operation might reallocate resources, switch contexts, or change device states, you cannot assume old objects remain valid. LLMs can help identify such risks, but developers still need to confirm them against interface specifications and runtime logs.
5. Typical Issue: Protocol ID or Interface Description Mismatch
When the program no longer crashes but still lacks a graphical interface, the problem might lie in protocol identifiers, structure definitions, or interface matching. In low-level environments, multiple interfaces may have similar purposes but different identifiers. If the wrong identifier is used, retrieving the interface might return an empty result, preventing the program from completing graphics initialization.
When handling such issues, the LLM can be asked to check three aspects:
- Whether the identifier currently used perfectly matches the target protocol;
- Whether the field order and types of the protocol structure are correct;
- Whether the caller checks the return status to determine if the interface is available.
However, you cannot rely solely on the model’s memory. A more reliable verification method is to compare against the header files, interface documentation, and actual compilation errors provided by the target environment to confirm names, types, and calling conventions. Especially in cross-environment development, identical concepts may have different implementations, and model-generated code must be validated by the target toolchain.
6. From “Writing Code” to “Managing the Debug Process”
These practices reflect that the most valuable capability of an LLM is not just generating code in one pass, but participating in the complete engineering reasoning process. Developers can adopt the following collaboration methods:
1. Provide Full Context
Do not just paste a single error message. Provide simultaneously:
- The target runtime environment;
- The project directory structure;
- Build commands or build results;
- The complete error log;
- Modifications already attempted;
- Expected behavior vs. actual behavior.
The more complete the context, the easier it is for the model to distinguish between configuration issues, syntax issues, lifecycle issues, and runtime issues.
2. Request Small-Step Modifications
Modifying multiple modules at once makes the source of the problem difficult to determine. A better approach is to handle only one clear goal per round, such as fixing the boot process first, then graphics initialization, and finally input and game logic.
3. Verify Every Modification
LLMs might provide fixes that look reasonable but are unverified. Developers should execute compilation and running at each stage and feed the real results back. Code review without a verification loop is of very limited value.
4. Ask the Model to Explain Trade-offs
For key modifications, require the model to explain why the change was made, what side effects it might introduce, and how to verify the fix is effective. This not only helps with the current issue but also helps developers judge whether the model truly understands the code.
7. How to Fairly Evaluate LLM Coding Capabilities
If comparing models, test designs must be as unified as possible. At least the following conditions should be kept consistent:
- Using the same hardware or virtual environment;
- Providing the same requirements description;
- Granting the same code access permissions;
- Using the same compilation and running process;
- Specifying whether internet access or external resources are allowed;
- Recording the time to first runnable state, the number of fixes, and final feature completion;
- Simultaneously evaluating code quality, maintainability, and exception handling capabilities.
Simply comparing “how many hours were spent” is insufficient. Some tasks seem functionally simple but have extremely complex underlying environment setups; others have large codebases but can be completed quickly using mature frameworks. Therefore, test results should be judged comprehensively based on task difficulty, environment preparation cost, and the number of debugging iterations.
8. Conclusion: LLMs as Engineering Debugging Partners
From the process of developing firmware graphics programs, it is evident that LLMs can already participate in code review, error location, interface checking, and local repairs. They are particularly suited for engineering tasks with many modules and high manual troubleshooting costs. However, they are not automated delivery systems; they still require developers to provide accurate context, build observability, and continuously correct them through real runtime results.
A more mature usage pattern is not to throw requirements at the model and wait for the final code, but to integrate the model into the engineering workflow: decompose goals first, then establish a minimal runnable version; collect logs first, then locate faults; fix infrastructure first, then gradually refine features. For low-level development, this collaborative model of “developers responsible for verification, LLMs responsible for accelerating analysis and implementation” is often of more practical value than simply comparing who can generate more code in one go.
