LLM-Assisted Firmware Graphics Development: A Practical Guide to Code Takeover and Tuning

    This article walks through a real-world engagement of taking over an untested firmware graphics codebase with the help of a large language model (LLM) and tuning it into something that actually runs correctly. It explains why bare-metal firmware is a fundamentally different runtime from desktop development, how to structure a code review before touching anything, and two representative failure modes: a mode-switch crash rooted in a lifecycle/dangling-pointer bug, and a "runs but shows nothing" problem caused by a mismatched protocol GUID. It also reflects on how to make cross-model comparisons fair, and closes with a reusable takeover checklist and supporting engineering infrastructure. Readers will gain a review-first workflow, an appreciation for surfacing runtime facts through logging, and a method for feeding those facts back to the model so its fixes converge.

    1. Why the firmware stage is the runtime platform for graphics programs

    Developing a graphics program in a pre-OS firmware stage uses a completely different mental model from writing an ordinary desktop application. There is no runtime, no system calls, and no ready-made window or font services. All you can rely on is the set of services defined by the firmware specification itself: a graphics output protocol that provides a linear frame buffer and display mode management, a handful of timer and input device protocols that supply the tick and keypresses, and a serial output channel that serves as the only “console.”

    This leads to three immediate consequences. First, memory lifetime is entirely your responsibility. Static buffers and explicit allocation are the norm, and a single dangling pointer can trigger a processor exception rather than a catchable exception stack. Second, hardware timing is exposed: frame rate is not something a driver papers over for you — you advance it yourself, tick by tick. Third, the debugging loop is extremely long: compile, flash, board up, observe — one iteration costs far more than in ordinary engineering.

    For exactly these reasons, this kind of task is a good test of an intelligent coding assistant’s real capability — there is no large body of near-identical answers on the internet to imitate, and correctness depends entirely on how deeply the model understands protocol semantics and state machines.

    2. The scenario: taking over “looks-fine” legacy code

    A typical scenario looks like this: an early firmware game project generated by another model is copied verbatim into a new environment. The first impression isn’t bad — the directory layout is clearly layered, with entry point, main loop, game logic, rendering, and input each isolated; naming is consistent, and interface boundaries are clear. In other words, the code is organized in a maintainable way. The problem is that it has never actually run.

    This kind of “well-structured but zero-validation” code is precisely the ideal candidate to hand to an LLM: the structure is understandable, defects are locatable, and the fix cost is low. Throwing it away and rewriting is a waste; reading it line by line yourself takes too long. The approach here treats it as a real engineering takeover task and proceeds in two rounds: the first aims to “get it running,” and the second aims to make it “run correctly and run stably.”

    3. Step one: have the model produce a structured code review, not direct edits

    The most common mistake for beginners is the instruction “just fix it for me.” The problem with this kind of instruction is that the model will change multiple modules at once without enough information, and if it introduces a new defect, you can’t tell which step caused it.

    A more effective approach is to first ask for a graded list of issues, and to explicitly constrain the output format:

    • Require issues to be graded into three tiers: “blocks execution / affects functional correctness / code quality”;
    • Each issue must include the evidence location (module and function name) and the reasoning behind the judgment;
    • Each issue should include a draft of the minimal-change proposal, but no direct implementation is allowed this round;
    • Require a separate section listing “risks that can only be confirmed at runtime,” explicitly marking the parts that cannot be determined statically.

    The fourth point is especially critical. In firmware graphics programs, a large number of issues are runtime-contract issues — for example, the call order of protocols, the validity window of returned structures, and state invalidation after a mode switch. Static reading can only raise suspicion, not reach a conclusion. Making the model clearly distinguish “definite errors” from “suspicions that need on-board verification” prevents wasting time later on pseudo-defects.

    A usable review output looks roughly like this:

    Module Risk level Typical issue Verification method
    Entry & init Blocking Protocol lookup result’s status code not checked Print status on board
    Rendering Blocking Old frame buffer info still used after mode switch Observe whether it crashes
    Rendering Severe Protocol GUID config inconsistent with firmware’s actual definition Print lookup count
    Game logic Severe Logic advancement tightly coupled to render frame rate Change resolution and observe speed
    Input Minor Key presses lack debounce and repeat-trigger protection Long-press / rapid-tap test
    Resources & memory Minor Temporary buffers repeatedly allocated in main loop Long-run stability

    4. Typical fault 1: switching modes causes an instant crash — a lifecycle error at heart

    On the first on-board run, the program printed its startup messages and exited immediately, with the exit point right after the graphics output protocol’s mode-set call.

    Here the protocol semantics need to be spelled out. A graphics output protocol typically provides two actions: query the list of available modes, and set the current mode. The query action returns mode information descriptions (resolution, pixel format, frame buffer base address and size, and per-row stride), but this description is often stored in a temporary structure or buffer inside the protocol; once you call the set action, the firmware rebuilds the current mode information, and the memory occupied by the old description may be reused or released.

    In other words, the code below looks natural on the surface but actually holds a dangling pointer:

    1. // Counter-example: keep using the old query result after switching modes
    2. const mode_info *m = gfx_query_mode(mode_index);
    3. gfx_set_mode(mode_index);
    4. frame_base = m->frame_base; // memory m points to is already invalid
    5. frame_size = m->frame_size;

    The correct approach follows one principle: any handle that describes the hardware’s current state must be re-acquired after the state changes, not cached and reused.

    1. // Illustrative pseudocode; function names are custom wrappers
    2. Status gfx_enter_graphics(u32 want_w, u32 want_h) {
    3. Status st;
    4. mode_info *table = NULL;
    5. UINTN count = 0;
    6. st = gfx_query_modes(&table, &count); // step 1: get the mode list
    7. log_info("query_modes status=%r count=%u", st, (unsigned)count);
    8. if (IsError(st) || table == NULL || count == 0) return kNotFound;
    9. UINTN idx = gfx_pick_mode(table, count, want_w, want_h);
    10. st = gfx_set_mode(idx); // step 2: switch mode
    11. gfx_release_modes(table); // release the list once done
    12. if (IsError(st)) return st;
    13. // step 3: mode has changed, must re-query current mode info
    14. mode_info *cur = NULL;
    15. st = gfx_query_current(&cur);
    16. log_info("query_current status=%r", st);
    17. if (IsError(st) || cur == NULL) return kNotFound;
    18. g_ctx.width = cur->width;
    19. g_ctx.height = cur->height;
    20. g_ctx.stride = cur->stride;
    21. g_ctx.fb_base = cur->frame_base; // only use the refreshed values from here
    22. g_ctx.fb_size = cur->frame_size;
    23. return kOk;
    24. }

    This defect is typical because during static review it can only be marked as a “suspicion” and must be resolved through runtime observation. The success criterion after the fix is clear: the program no longer exits right after the mode set, and it can print field values matching the expected resolution.

    A note on debugging: when there is no image at all and even the exit reason isn’t obvious, prefixing logs by module and sending them all to the serial port is the most time-saving move. Having the model add logs first and then reason backward from the logs is far more efficient than letting it guess blindly.

    5. Typical fault 2: protocol lookup “succeeds but shows nothing” — a misconfigured GUID

    The second obstacle was a program that could run stably — the main loop was running, the serial output was scrolling — yet the screen stayed blank. After tracing it to the root cause, the problem turned out to be in the identifier configuration of the graphics output protocol.

    This is a classic trap in firmware development: the same functionality has multiple similar identifiers across different spec versions, different vendor implementations, and different encapsulation layers. The constant used to locate a service must strictly match the instance actually exported by the firmware. Using the wrong one won’t cause a compile error — it will only fail to find the instance at runtime, or return an instance with a different subset of capabilities.

    The right way to investigate is not to keep reading code but to make the lookup process itself observable:

    1. // Illustrative pseudocode: diagnostic output at the protocol-binding stage
    2. Status gfx_bind_display(void) {
    3. Status st;
    4. VOID *handles = NULL;
    5. UINTN count = 0;
    6. gfx_proto *gop = NULL;
    7. st = gfx_locate_handles_by_protocol(&kDisplayProtocolGuid, NULL, NULL, handles, &count);
    8. log_info("locate status=%r count=%u", st, (unsigned)count); // is there an instance?
    9. if (IsError(st) || count == 0) return kNotFound;
    10. for (UINTN i = 0; i < count; i++) {
    11. st = gfx_open_protocol(handles[i], &kDisplayProtocolGuid, &gop);
    12. log_info("open[%u] status=%r", (unsigned)i, st);
    13. if (IsError(st) || gop == NULL) continue;
    14. log_info("rev=0x%x mode_count=%u current=%u",
    15. gop->revision, (unsigned)gop->mode_count, (unsigned)gop->current_mode);
    16. return kOk; // got a usable instance
    17. }
    18. return kNotFound;
    19. }

    Three log lines answer three questions: whether the firmware exports this service at all, whether the handle can be opened, and what capabilities and version the obtained instance has. Once the answers land in the logs, the model’s localization shifts from “guessing” to “inferring,” and the fix naturally converges.

    From an engineering standpoint, one convention should be established here: define all protocol identifiers centrally in one configuration header, and add a startup self-check — when the lookup fails, print a clear reason and fall back to a serial mode, rather than continuing to run a main loop that is bound to show nothing.

    6. A methodological reflection on “unfair comparison”

    This kind of cross-model comparison experiment is often criticized for lacking fairness, and that criticism is methodologically correct. Handing the same task to different subjects, with different prompts, different environments, and different starting and ending states, yields conclusions that aren’t comparable; a single sample doesn’t constitute any statistically meaningful conclusion either.

    If you really want to do a meaningful side-by-side observation, you should at least control the following variables:

    • Unified input: the same task description, the same set of constraints, and the same acceptance criteria;
    • Unified starting point: either both start from a blank project, or both take over the same existing codebase;
    • Unified environment: the same build and flash process, the same hardware, and the same resolution;
    • Unified judgment: break “compiles successfully,” “first run doesn’t exit,” “correct image,” “runs continuously without crashing,” and “logic decoupled from frame rate” into separate checkable items;
    • Multiple samples: repeat the same task multiple times, record the success rate and average time, rather than picking the single best-looking run.

    There is also a more practical angle: don’t treat the model as a black box being evaluated, but as a collaborator that needs context filled in. What it lacks is never the code, but runtime facts — real logs, real return codes, real resolutions. Feed these facts in and its fix accuracy improves markedly; don’t feed them in and it can only output patches that “look reasonable.”

    7. Second round: from running to running correctly

    Getting it running in the first round is only the baseline. The second round of refinement usually starts from these directions:

    • Decouple frame rate from the tick: change logic advancement to a fixed time step, and refresh rendering by dirty regions, so the logic speed doesn’t vary with resolution.
    1. const u64 kTickNs = 16666666ULL; // ~60Hz
    2. u64 last = timer_now_ns(), acc = 0;
    3. while (running) {
    4. input_poll(&keys);
    5. u64 now = timer_now_ns();
    6. acc += now - last;
    7. last = now;
    8. while (acc >= kTickNs) { // fixed step; game speed is unchanged regardless of frame rate
    9. game_update(&state);
    10. acc -= kTickNs;
    11. }
    12. renderer_present(&state); // only redraw changed regions
    13. }
    • Input handling: add a decoding layer from scancodes to actions, key debounce, and long-press repeat, to avoid multiple triggers per frame;
    • Memory strategy: change temporary buffers to static arrays, avoiding repeated allocate/free in the main loop;
    • Portability: derive all geometric quantities from current mode information, don’t hardcode resolution, so that changing resolution only affects appearance and not logic;
    • Robustness: check return values at every step of the init chain, and on failure output a locatable log and exit safely;
    • Playability: difficulty curve, collision boundary judgment, and consistency of scoring and state transitions — these belong to the logic layer and need dedicated verification.

    8. A reusable collaboration checklist and engineering infrastructure

    Abstracting the process above into a reusable takeover workflow:

    1. Review first, don’t edit: require a graded list of issues, and separately list the items that “need runtime confirmation”;
    2. Establish observability: add logs and self-checks first, then have the model fix bugs;
    3. Single-point fixes: handle only one blocking item at a time, and verify on board immediately after each change;
    4. Contracts first: for any code involving state changes, resource lifetimes, or protocol versions, always “re-acquire after the change”;
    5. Lock in the baseline: commit a runnable version as soon as it runs, as a rollback point for all subsequent refinement;
    6. Quantify acceptance: use checklist items instead of subjective judgment.

    The supporting engineering infrastructure is equally worth investing in. A cloud-based containerized environment can guarantee consistent build toolchain versions and reproducible results; object storage is used to archive each build artifact and firmware image for easy retrospective comparison; and a logging service aggregates serial output and build records so that “which change caused which behavior change” becomes traceable. These fall into the category of general cloud capabilities, unrelated to this task’s technical path, but they determine whether a debugging workflow can be reused over the long term.

    9. Conclusion

    The truly valuable part of this practice isn’t “how long a certain model took to finish a game,” but three transferable conclusions: for taking over legacy code, review first and act second gives the best return; the pitfalls of firmware graphics development almost all lie in lifetimes and protocol contracts, where static reading can only raise suspicion while runtime logs alone give answers; and the prerequisite for a model to output correct fixes is that you feed runtime facts to it in a structured way. Bake these three points into your workflow, and the tools can be swapped while the methodology still holds.