Evaluating Large Language Models in Low-Level Software Development: Practical Capabilities and Methods

    Large language models are increasingly capable of handling complete software development workflows, from project analysis and code generation to debugging and repair. However, firmware and other low-level environments expose limitations that are less visible in application development. This article examines an embedded graphics game development exercise, explains common failures involving interface lifecycles and protocol configuration, and proposes a practical framework for evaluating model-assisted programming.

    In recent years, large language models have made substantial progress in code generation. They are no longer limited to completing functions or explaining compiler errors. They can now attempt to decompose projects, design modules, review code, debug builds, and repair problems across an entire development workflow.

    However, there is still a significant gap between “writing code” and “making a program actually run.” This is especially true in low-level scenarios involving firmware, boot environments, graphics protocols, and driver interfaces. These programs run in constrained environments with limited debugging facilities and strict interface requirements. A task that looks simple at the code-generation stage may require multiple rounds of diagnosis and validation before it works reliably.

    Based on a firmware graphics game development exercise, this article analyzes how large language models perform on complex low-level tasks and presents a more practical method for evaluating their capabilities.

    Why Low-Level Development Is a Stronger Test of Model Capability

    In conventional application development, programs usually run on mature operating systems and standard runtimes. File systems, memory management, graphics output, input devices, and exception handling are largely provided by the platform. Developers mainly need to call relatively stable APIs.

    Firmware environments are different in several important ways.

    1. The Runtime Environment Is Incomplete

    After a firmware program starts, it may not have a complete process model, file system, or device-driver stack. Many capabilities must be accessed through firmware-provided protocols, and even a small misuse can cause the program to terminate immediately.

    2. Debugging Capabilities Are Limited

    Application developers can use debuggers, logging frameworks, and graphical tools to locate problems quickly. Firmware programs may provide little more than serial output, screen messages, or memory-state inspection. As a result, every critical execution path should be designed with observability in mind.

    3. Interface Lifecycles Are More Strict

    Pointers, handles, and buffers returned by low-level interfaces may only remain valid during a specific stage. Some operations change the display mode, memory layout, or device state. When that happens, previously obtained addresses and objects may no longer be valid.

    4. Failures Are Often Difficult to Interpret

    Application-level failures commonly produce explicit exceptions or error codes. Firmware failures may appear only as a blank screen, a reboot, or immediate termination. Developers must infer the failure location from limited evidence.

    For these reasons, low-level tasks test more than code generation. They also evaluate context comprehension, interface knowledge, debugging reasoning, and the ability to collaborate on an engineering workflow.

    How to Decompose a Complete Task

    To avoid reducing an evaluation to the final output alone, the task can be divided into several stages.

    1. Analyze the Project Structure

    First, ask the model to inspect the existing code and explain the relationships among the entry point, main loop, rendering module, input module, and game logic.

    A simple graphics game should generally include responsibilities such as:

    • Initializing the firmware environment;
    • Obtaining the graphics output protocol;
    • Selecting a screen resolution or display mode;
    • Creating or acquiring a frame buffer;
    • Reading keyboard input;
    • Updating game state;
    • Drawing the background, paddle, ball, and bricks;
    • Defining the main-loop exit condition;
    • Releasing or restoring relevant resources.

    At this stage, the model should not immediately modify the code. Instead, it should list potential problems and explain the evidence for each one. A useful code review should describe not only that “this may be a problem,” but also the conditions under which it occurs, the symptoms it may produce, and how it can be verified.

    2. Establish a Runnable Baseline

    The next step is to prioritize compilation and startup problems rather than visual polish.

    A practical sequence is:

    1. Confirm the build environment and target architecture;
    2. Confirm the entry-point signature;
    3. Confirm that the required protocol can be obtained;
    4. Confirm that the display mode can be selected successfully;
    5. Confirm that the main loop can continue running;
    6. Add the complete game logic only afterward.

    This order narrows the failure scope. If rendering, input, collision detection, and scoring are all introduced at once, an early exit could result from initialization, protocol access, or game logic, making diagnosis much more difficult.

    3. Add Runtime Observability

    In low-level environments, logging is not optional. It is the foundation of troubleshooting. A model can help add serial logs or on-screen status messages such as:

    1. [BOOT] entry reached
    2. [GRAPHICS] protocol located
    3. [GRAPHICS] mode selected
    4. [FRAMEBUFFER] address acquired
    5. [INPUT] keyboard initialized
    6. [LOOP] entering main loop

    Each stage should record whether it succeeded or failed. Where necessary, the program should also print error codes, important structure fields, and the current execution phase.

    If the program displays one startup message and then exits, the problem most likely occurs in one of the initialization calls that follows. Adding logs incrementally can reduce the problem from “the entire program does not run” to “the program fails after a specific protocol call.”

    A Typical Failure: Stale Pointers and Interface Lifecycles

    During one firmware graphics debugging exercise, the program entered the startup sequence but terminated immediately after obtaining display information. Further inspection showed that the code first read the current display-mode information, then called a mode-switching function, and finally continued using the previously saved structure pointer.

    This pattern is dangerous in low-level development.

    Some interfaces return a structure that is not a permanently valid, independent copy. Instead, the structure may point to memory managed internally by the protocol. Changing the display mode may cause that internal buffer to be reallocated, invalidating the original pointer.

    Continuing to access the old pointer can lead to:

    • Reading an incorrect width or height;
    • Using an invalid frame-buffer address;
    • Writing to memory that does not belong to the current mode;
    • Triggering an exception or causing immediate termination.

    A safer approach is to:

    1. Read only the information required before changing the display mode;
    2. Complete the mode switch;
    3. Obtain the current mode information again;
    4. Save the new frame-buffer address, size, and resolution;
    5. Use only the post-switch data for subsequent rendering.

    An illustrative implementation is:

    1. mode_info = query_current_mode();
    2. if (mode_info == NULL) {
    3. log_error("query mode failed");
    4. return ERROR;
    5. }
    6. status = set_graphics_mode(target_mode);
    7. if (status != SUCCESS) {
    8. log_error("set mode failed");
    9. return status;
    10. }
    11. mode_info = query_current_mode();
    12. if (mode_info == NULL) {
    13. log_error("refresh mode info failed");
    14. return ERROR;
    15. }
    16. frame_buffer = mode_info->frame_buffer;
    17. width = mode_info->width;
    18. height = mode_info->height;

    The key lesson is not the names of the functions. It is the principle of reacquiring dependent data after a state-changing operation. Whenever an operation changes device state, memory layout, or protocol context, the validity of related pointers and handles should be verified again.

    A Typical Failure: Incorrect Protocol Identifiers

    Once the program stops crashing but still produces no graphics, the problem may have shifted from memory lifetime management to protocol discovery or device configuration.

    Firmware environments commonly use protocol identifiers to distinguish different services. Graphics-related capabilities may have several similar-looking identifiers with different meanings. Using the wrong one can cause:

    • Protocol discovery to fail;
    • The query result to be empty;
    • The returned interface to lack the expected capability;
    • The program to appear to run while producing no screen output.

    This type of failure demands more than syntactic pattern matching. The model must understand the semantics of the firmware interface, the definitions of the identifiers, and the constraints of the target environment.

    A practical investigation should include the following steps:

    1. Check where the protocol identifier is declared;
    2. Confirm that the query function expects a protocol-type identifier rather than another related identifier;
    3. Check whether the interface pointer is null;
    4. Validate the query result explicitly;
    5. After successful discovery, log the interface address or important capability fields;
    6. Confirm that the target firmware actually provides graphics output support.

    This illustrates an important point: syntactic correctness does not guarantee semantic correctness. A model may generate code that looks consistent while still using the wrong interface definition. Documentation, header files, and the real execution environment remain essential for validation.

    From “Runs” to “Usable”

    The first successful run only proves that the basic execution path is working. A usable firmware game still requires additional engineering.

    1. Input Handling

    Keyboard input in firmware is often read through polling or a firmware protocol rather than through application-level event callbacks. Important questions include:

    • Does the input operation block when there is no input?
    • Can one key press trigger repeated actions?
    • Is the input protocol available in all target environments?
    • Can the exit action be handled reliably?
    • Can the main loop continue if input reading fails?

    2. Time Control

    If the main loop relies only on execution speed, the game will run at different speeds on different devices. A better approach is to calculate movement from elapsed time:

    1. position = position + velocity × elapsed_time

    This prevents the program from running too quickly on faster hardware and makes collision detection more stable.

    3. Collision Detection

    Collision detection between the ball, paddle, and bricks should not depend only on exact coordinate equality. It should consider rectangle intersection, boundary crossing, and movement direction. Otherwise, fast-moving objects may pass through other objects.

    4. Frame-Buffer Boundaries

    Pixel rendering must validate every coordinate. Negative coordinates, coordinates beyond the screen dimensions, or an incorrect row-stride calculation can all cause out-of-bounds memory access.

    5. Resource and Exit Handling

    Before exiting, the program should restore the display state where possible, stop input processing, and release temporary resources. Even if firmware resources disappear after a reboot, cleanup logic should not be ignored.

    How to Evaluate Large Language Model Programming Ability More Objectively

    Comparing models only by the number of hours they require is not rigorous. More meaningful evaluation dimensions include:

    • Whether the model can understand an existing project structure;
    • Whether it can identify real interface-lifecycle problems;
    • Whether it can narrow the fault scope using logs;
    • Whether it can modify the code and verify the modification;
    • Whether it can handle compilation, linking, startup, and runtime issues;
    • Whether it proactively adds tests and diagnostic information;
    • Whether it can explain why a fix works instead of only producing replacement code;
    • Whether it avoids introducing new risks after the repair.

    Task difficulty should also be made as comparable as possible. If two models perform different types of tasks, the results describe their performance on those specific tasks rather than establishing a general ranking.

    A better approach is to build a standardized task set covering:

    1. Conventional application development;
    2. Maintenance of multi-file projects;
    3. Low-level interface usage;
    4. Build-environment configuration;
    5. Runtime fault diagnosis;
    6. Performance and security optimization.

    Each task should preserve complete logs, modification history, build results, and final runtime status. Conclusions should not be based solely on subjective impressions.

    How Developers Should Collaborate with Large Language Models

    Large language models are best suited to frequent, complex, and verifiable engineering work. They should support final judgment, not replace it. The following practices are useful:

    • Ask the model to explain before asking it to modify code;
    • Address one primary failure at a time;
    • Require the model to provide its reasoning and validation method;
    • For critical interfaces, ask it to reference the definition or explain the lifecycle;
    • Compile and run the program immediately after each change;
    • Manually review low-level memory, protocol, concurrency, and security issues;
    • Save each successful version so that changes can be rolled back.

    The value of a model is not simply that it can produce more code. Its greater contribution is helping developers form hypotheses, build diagnostic paths, and complete verification more quickly.

    For firmware and low-level systems, a reliable result must satisfy three conditions at the same time: the code compiles, the program runs, and its behavior conforms to the interface specification.

    Those are the core standards that should guide evaluations of AI-assisted programming.