Evaluating LLM Capabilities in Complex Firmware Development: A Practical Case Study

    As Large Language Models (LLMs) evolve from generating simple scripts to managing multi-file projects, their role in complex firmware development remains challenging. This article explores the practical application of LLMs in firmware development through a graphical game project. We discuss the limitations of evaluating models solely by code volume, the unique difficulties of firmware environments, and effective strategies for integrating LLMs into the development lifecycle, including structured code review, systematic debugging, and iterative validation.

    Recent years have witnessed significant improvements in LLM code generation capabilities. While they were previously best suited for completing functions, explaining errors, or generating simple scripts, some models can now handle multi-file projects, understand inter-module dependencies, and participate in the complete workflow of compiling, running, debugging, and optimizing.

    However, complex software development—especially firmware development—is not truly tested by a model’s ability to write hundreds of lines of code. The real test is whether it can continuously locate problems in an uncertain environment with minimal runtime feedback and strict interface constraints, and push a “plausible” project into a “fully functional” state.

    This article discusses the practical value, common limitations, and more reliable usage methods of LLMs in underlying engineering, based on the development practice of a graphical firmware game.

    1. Evaluating LLMs: Beyond Code Volume

    In one test, developers asked two general-purpose programming models to complete different game projects: a graphical Breakout game and a Tetris game. The former compiled successfully and launched in a short time. The latter, despite generating multiple source files and a relatively complete directory structure, never resulted in a runnable program.

    This outcome should not be simply interpreted as “the first model is stronger, and the second is weaker,” because the two tasks are not entirely equivalent. Differences in input logic, rendering processes, state management, and firmware interface calls, as well as the smoothness of project initialization, can significantly impact the final result.

    Therefore, when evaluating a programming model, at least the following factors should be considered:

    1. Task Comparability
      The two models should ideally handle identical requirements, development environments, and acceptance criteria; otherwise, conclusions are easily skewed by task difficulty.

    2. Coverage of the Full Development Loop
      Simply looking at generated code is insufficient. It is necessary to observe whether the model can complete project configuration, compilation, startup, log analysis, troubleshooting, and feature verification.

    3. Documentation of Human Intervention
      If developers manually modify a large amount of code at critical stages, the final result cannot be entirely attributed to the model itself.

    4. Distinction Between “Structural Correctness” and “Execution Correctness”
      Modularity, naming conventions, file count, and comment quality only indicate that the code has an engineering form, not that the program is actually usable.

    For firmware projects, “passing the first compilation and successfully starting” is an important milestone, but it is only the first phase. A truly comprehensive evaluation should also include input response, screen refresh, exception handling, compatibility across different hardware environments, and long-term operational stability.

    2. Why Complex Firmware Projects Are More Challenging

    Compared to standard desktop applications, firmware development typically involves stronger environmental dependencies and a much lower tolerance for errors. The program may run directly during the boot phase, lacking mature operating system services, conventional file systems, graphics libraries, and debugging tools.

    Taking a graphical mini-game as an example, a project usually includes at least the following components:

    • Program entry and startup process;
    • Main loop and frame rate control;
    • Game state management;
    • Data structures for characters, bricks, or blocks;
    • Collision detection;
    • Keyboard input handling;
    • Graphics output protocol calls;
    • Framebuffer operations;
    • Compilation and target platform configuration;
    • Serial port or other low-level log outputs.

    Individually, these modules are not very complex, but they carry numerous implicit constraints between them. For instance, the rendering module must know the screen resolution and pixel format, the input module must correctly handle key states, the main loop must avoid blocking, and memory pointers must remain valid after protocol calls. A single mistake with a low-level interface can cause the program to exit immediately, often without a clear error message.

    This is the primary reason LLMs often “write well but fail to run” in firmware projects: models can generate reasonable structures based on common code patterns but cannot rely solely on static text to verify pointer lifecycles, protocol states, and hardware behaviors in the runtime environment.

    3. Letting LLMs Take Over Existing Code: Review Before Execution

    In practical development, the most effective approach is often not to have the model generate all code from scratch, but to hand over an existing project for review, repair, and iteration.

    The first step should be to have the model perform a comprehensive code review, but the prompt should be more specific than just “check the code.” Better instructions include:

    • Explain the responsibilities of each source file;
    • Map the primary function call relationships;
    • Inspect the entry function and main loop;
    • Verify memory allocation and deallocation;
    • Inspect protocol acquisition and handle lifecycles;
    • Examine the data flow between input, rendering, and game logic;
    • List high-risk issues and prioritize them;
    • Explain the reproduction conditions, impact scope, and suggested fixes for each issue.

    The value of this approach is that the model does not blindly rewrite code immediately, but first builds a comprehensive understanding of the project. For multi-file projects, structural understanding is far more critical than localized patches.

    Concurrently, developers should require the model to explicitly categorize its findings into three types:

    1. Issues determinable from the source code;
    2. Issues inferred from experience but requiring runtime verification;
    3. Issues with insufficient information, requiring additional logs or environment details.

    This prevents the model from presenting guesses as conclusions and helps streamline subsequent debugging steps.

    4. First Execution Failure is Usually Not “Completely Wrong Code”

    During the debugging of a firmware game, the program entered the startup phase before the main loop and output a log similar to “starting execution,” but then exited immediately. This behavior indicated that the compilation phase was generally fine, and the fault occurred during runtime initialization or a protocol call.

    After the developer asked the model to add serial port logs, the problem scope quickly narrowed down to a graphics output protocol call. Further inspection revealed that the program obtained an information pointer when querying the display mode and then called an operation to set the mode. Setting the mode could cause the underlying buffer to be reallocated or invalidated, rendering the previously saved pointer unreliable. If subsequent code continued to use that pointer, it could trigger an abnormal exit.

    This type of issue has typical low-level development characteristics:

    • Compilers may not detect it;
    • Static code reading cannot directly confirm it;
    • It only occurs under specific call sequences;
    • The error symptom might just be a silent program exit;
    • The fix relies on understanding the lifecycle of the interface’s return value.

    A robust fix approach is to re-fetch relevant information after calls that might alter the underlying state, rather than continuing to use pointers or struct addresses cached before the call.

    This case illustrates that the true value of LLMs lies not just in “writing the fix code,” but in correlating log phenomena, call sequences, and pointer lifecycles to propose a troubleshooting direction aligned with low-level execution rules.

    5. Protocol Identifier Errors: A High-Frequency Issue

    Once the program stops crashing, the next issue might be “it runs but displays nothing.” At this point, the problem often lies not in the game logic, but in the identification and acquisition process of the graphics output protocol.

    In firmware environments, protocols are typically retrieved via globally unique identifiers. Different versions of development headers, different protocol objects, and different calling methods can result in multiple similar identifiers. If the identifier is configured incorrectly, the program can enter the initialization flow but fail to obtain a valid graphics output interface.

    When troubleshooting such issues, follow this sequence:

    1. Confirm that the current target environment actually provides graphics output capabilities;
    2. Check if the protocol identifier used matches the interface definition;
    3. Inspect the return status of the protocol query function;
    4. Verify whether the returned object is null;
    5. Output the current mode count, resolution, and pixel format;
    6. Execute a minimal single-color fill test before actual drawing;
    7. Confirm that the calling sequence of the graphics interface complies with the firmware specification.

    Do not start by simultaneously checking collision algorithms, keyboard logic, and drawing coordinates. A more efficient principle is to establish a “minimal verifiable path”:

    Get graphics protocol → Query display info → Set display mode → Fill monochrome screen → Draw a rectangle → Draw complete game objects.

    Proceed to the next step only after the previous one succeeds. This breaks down complex problems into multiple observable stages.

    6. Why “Modular Code” Still Might Not Run

    Some models generate projects containing entry files, game logic files, rendering files, and input handling files, with well-organized directories. While this structure aids future maintenance, it does not automatically guarantee the project is executable.

    Common reasons include:

    • Inconsistent struct definitions across modules;
    • Mismatched function signatures between declarations and implementations;
    • Compilation configurations missing newly added files;
    • Use of library functions not present in the target environment;
    • Default assumptions about a specific pixel format;
    • Input device names or scan codes incompatible with the current environment;
    • Main loop lacking a reasonable delay, causing excessive resource consumption;
    • Error paths failing to release resources;
    • Continued execution of subsequent logic after initialization failure.

    Therefore, code review must look beyond neat directories and check cross-file contracts. It is recommended to have the model output an interface table containing at least the following:

    Module External Function Input Output Dependency Error Handling
    Initialization init_function System Handle Game Context Graphics & Input Interfaces Return Error
    Input read_function Input Device Key State Input Protocol Handle Null Input
    Update update_function Game State, Time New State Collision Rules Boundary Protection
    Rendering draw_function Game State, Canvas None Graphics Interface Check Canvas Validity

    Such an interface table helps the model identify issues where “every file looks reasonable individually, but the combination fails to work.”

    7. Two-Round Development Process vs. One-Shot Generation

    Complex tasks are better suited for a two-round or even multi-round process.

    Round 1: Get the Program Alive

    The goal of the first round should not be beautiful graphics or complete features, but achieving a minimal closed loop:

    • The project can compile;
    • The program can start;
    • The graphics protocol can be acquired;
    • The screen can display;
    • Input can be read;
    • The main loop can run continuously;
    • Logs can be output when errors occur.

    Functionality should be minimized at this stage. For example, start by drawing a fixed rectangle, then add movement, then collision, and finally scoring and end states.

    Round 2: Feature and Engineering Optimization

    Once the baseline path is stable, address:

    • Collision boundaries;
    • Input debouncing;
    • Refresh rate;
    • Screen flickering;
    • Memory usage;
    • Error recovery;
    • Code duplication;
    • Data structure optimization;
    • Multi-resolution adaptation;
    • Long-term operational stability.

    If the model is asked to implement all features simultaneously from the start, a single low-level error could render the entire project unrunnable, and the model would struggle to identify the source of the problem.

    8. Improving LLM Debugging Efficiency

    To help LLMs participate more effectively in engineering development, the following practices are recommended.

    1. Provide Complete Context

    Do not just paste the error line. Provide simultaneously:

    • Directory structure;
    • Compilation commands or build configuration;
    • Target runtime environment;
    • Complete error logs;
    • The most recent changes made;
    • Expected behavior vs. actual behavior;
    • Reproducibility and reproduction steps.
      The more complete the context, the easier it is for the model to analyze the problem from a systemic perspective.

    2. Request Limited Scope per Modification

    Each fix should ideally focus on a single issue, listing the modified files and the reasons. This prevents the model from rewriting large swathes of code, which could break previously working sections.

    3. Include “Verification Methods” in Task Requirements

    For example, ask the model to explain after making changes:

    • What should be recompiled;
    • What logs should be observed;
    • What phenomena should be expected;
    • What information to collect next if it still fails.
      This shifts the model from “giving an answer” to “designing a verification experiment.”

    4. Establish Clear Lifecycle Rules for Low-Level Resources

    For protocol objects, buffers, handles, and dynamic memory, explicitly require the model to annotate:

    • Creation location;
    • Usage scope;
    • Invalidation conditions;
    • Release location;
    • Rollback method upon failure.
      Clearer lifecycles lead to fewer runtime errors.

    9. An Objective View of LLM Programming Capabilities

    This practice reflects that LLMs are already capable of participating in fairly complex development work, particularly excelling in the following scenarios:

    • Rapidly establishing project scaffolding;
    • Reading multi-file codebases;
    • Explaining unfamiliar modules;
    • Summarizing potential issues;
    • Proposing troubleshooting paths based on logs;
    • Generating highly repetitive boilerplate code;
    • Assisting with refactoring and documentation.

    However, it still cannot replace validation in a real environment. When dealing with low-level interfaces, hardware variances, pointer lifecycles, and compilation configurations, models might generate highly persuasive yet inapplicable code.

    Therefore, a more accurate conclusion is not that “models can now independently complete complex projects,” but rather:

    LLMs are evolving from code generators into software collaborators capable of participating in analysis, implementation, debugging, and optimization; but the final quality still depends on the testing loop, runtime feedback, and the developer’s engineering judgment.

    For enterprises and individual developers, the most critical aspect to observe is not a model’s advertised coding ability, but whether it can integrate stably into a real development workflow: whether it understands project context, can handle multi-turn feedback, is willing to correct judgments based on logs, and can maintain a traceable modification history after failures.

    Only by combining model capabilities with phased validation, automated builds, log collection, and human review can we truly transform “a pile of generated code” into “a deliverable, runnable software.”