Firmware Graphics with AI Assist: Development and Debugging
Firmware graphics run without a general-purpose OS, window system, or mature debugger. This article explains how to use AI coding tools in three phases—review, get a first observable loop, then polish—so modular code that compiles actually lights the framebuffer and stays verifiable.
Firmware has no general-purpose scheduler, windowing system, or mature debugger. Graphics programs talk directly to boot services, protocol locators, and a physical framebuffer. AI coding tools can scaffold a modular skeleton quickly, but compiling is not the same as drawing a stable first frame. Split collaboration into review, bring-up, and polish, and drive changes from logs rather than guesses, so structurally decent but non-running code becomes a program you can re-verify.
Scenario and constraints
Typical work in firmware: locate a graphics output protocol, query and set a display mode, obtain framebuffer base and scanline parameters, then in a main loop update game or UI state from input and write pixels. Constraints are stricter than for applications.
Memory is allocated by firmware. After a mode switch, previously returned mode-info pointers may be invalid; using them is a dangling pointer. Protocols are found by GUID, not by name strings; the wrong identifier fails Locate or binds the wrong instance. Input often comes from a simple keyboard protocol with no event queue. Without a console, serial prints are almost the only observability. Builds depend on the firmware SDK’s package description, library references, and target architecture; if the environment is not ready, no algorithm yields a bootable image.
Evaluate AI-assisted work not by how many source files it emitted, but by whether the tree links independently, protocol init completes, refresh continues without exit, state is separated from drawing, and failure paths log.
How to collaborate
Do not prompt “finish the game.” Use three rounds, each adding one class of goal.
Round one is review only. Drop the existing project into a clean build tree. Instruct: list defects by entry, main loop, rules, draw, and input; distinguish compile, link, runtime logic, and undefined behavior; cite the call order for each. Do not rewrite everything before the model understands protocol lifetime.
Round two is “leave observable traces and keep looping.” Add serial logs first: enter entry point, protocol locate success/fail, query-mode status, set-mode status, re-fetch framebuffer info, enter main loop, heartbeat every N frames. Log status-code categories, not just “failed.”
Round three is feel and structure: drop timing, collision, line/block clear, bounds, pause, coordinate mapping after resolution change. Increment features; after each step, serial must still show the program sitting in the main loop.
If you already have a well-modularized tree that does not run, do not throw it away. Review is often more valuable than generating from scratch: when entry, loop, rules, draw, and input are already split, bugs cluster in protocol and memory contracts, not gameplay.
Contracts to watch in review
Write the checklist in call order, not by wandering filenames.
Entry must finish required library and protocol init before any draw API. Graphics-output locate GUID must be distinct from console/text output; firmware has several graphics-related identifiers, and the wrong constant makes later query/set meaningless. Mode-query info must be copied into storage that remains valid across calls; a pointer into firmware’s temporary buffer is dangling after any display reconfiguration. Check set-mode return values; on failure, log-only path—do not keep writing pixels. After a successful set, re-read framebuffer address, bytes per pixel, visible width, and stride. The main loop must not have a branch that always resets or returns to firmware (for example treating one status as fatal and exiting unconditionally). Input must be non-blocking or timed out so a lack of keys does not hang and look like “no picture.” Drawing must write only validated coordinates so a wrong stride does not smash memory and reboot randomly.
A “seven files, clean module boundaries” project still flashes or blacks out if any of these contracts fail. Structure is maintainability, not proof the protocol is used correctly.
Typical failure: crash after SetMode
Logs often show “about to enter main loop,” then reset or firmware prompt, no further heartbeat. That is further than a broken build: the image reached graphics init.
Suspect mode-info lifetime first. QueryMode often returns a pointer to an internal block (resolution, pixel format, scanline). SetMode may rebuild that block or switch scan buffers; the old pointer is gone. Using it afterward to compute framebuffer address and write typically faults on the first draw or first pixel-format read.
Fix: after SetMode succeeds, QueryMode again and copy resolution, pixel format, framebuffer base, and stride into the caller’s own struct. Thereafter read only that copy. Do not stash the protocol pointer in globals across the loop. Sketch of order (status codes are firmware statuses; INFO is the caller’s copy):
/* All returns are firmware status codes; INFO is the caller's own copy */status = QueryMode(gop, mode, &w, &h);if (Failed(status)) { Log("query failed"); return status; }status = SetMode(gop, mode);if (Failed(status)) { Log("set failed"); return status; }status = QueryMode(gop, mode, &w, &h); /* re-query after set */CopyInfo(&local, gop->Mode->Info, gop->Mode->FrameBufferBase);Log("fb ready");
This is order only; real code must use the SDK’s types and calling convention—field names are not universal. Pass criteria: serial must show set success and framebuffer ready, then heartbeats that keep increasing. “No longer exiting immediately” is not enough; a wrong address may reset later.
If re-query still exits, check three things: mode number in the protocol’s reported range; pixel format actually handled (some implementations use bitmasks, not fixed 4 bytes per pixel); writes using bytes per scanline, not visible width times bytes per pixel. When visible width is less than stride, packing by width writes the next line into padding and can corrupt adjacent mappings.
Typical failure: process alive, no graphics
No crash, heartbeats healthy, but the screen is still firmware’s picture or black: usually not game rules—graphics output was never bound, or the mode never took effect.
Locate by GUID only. Text output, pointer, and graphics output have similar names in docs; one field off and locate fails. If failure is ignored and a null check is wrong, draw may be skipped while the loop continues—“alive, no picture.” Locate failure must log clearly and stop drawing; success should log unambiguous fields (revision, mode count) proving you have graphics output, not another protocol.
For modes: read the current mode and try a full-screen solid rectangle. If solid color appears, base, stride, and format are correct; then overlay sprites or tiles. That splits protocol issues from coordinate mapping. If solid color never appears, do not start changing game logic.
Unit-test channel order. Four bytes per pixel can still differ in channel layout. Write known red, green, blue, and white at the top-left four pixels; confirm by eye or capture, then wrap a PutPixel. Tools that fill with desktop-library channel habits produce tinted but healthy loops, easy to misread as “it works.”
Round two polish
Playability only after bring-up. Change one dimension at a time; accept with logs or on-screen counters.
Do not time with empty-loop counts. Instruction throughput varies; busy-wait may be OK in a simulator and unplayable or boot-hogging on hardware. Prefer firmware timer services for monotonic time; accumulate milliseconds for logic frames; draw can be denser or lock-step. After splitting logic and draw, sample input on logic frames to avoid double-fires.
Debounce input. Simple keyboard protocols report current keys, not press edges. Keep the previous frame’s key set; trigger move/rotate only on newly appeared keys. For held horizontal movement, use a separate repeat timer—not one cell every logic frame.
Map all coordinates from a logical resolution to the current mode. Do not hard-code a desktop width in rules. Use integer grid bounding boxes; multiply by cell pixels at draw time so mode changes scale only, not collision.
Prefer a caller-owned back buffer, then blit the whole frame, to reduce tearing and half-frame glitches. If firmware RAM is tight, at least use a line buffer for a clear animation so you do not mix two board states in one frame.
Define an exit path: a key combo returns to firmware, and if the protocol allows, restore the display mode from entry; otherwise later boot menus may stay at the game’s resolution.
Constraining the tool to reduce rework
State invariants in the prompt rather than “fix it again.” Examples: after SetMode, never use any earlier mode-info pointer; no stdout besides serial; check every protocol status; do not write a fixed four bytes until pixel format is confirmed; the main loop must not exit on a single draw failure unless the status is a resource-class fatal error.
Require a list of functions to change and why before edits. If it rewrites modules unrelated to the current fault, stop and narrow scope. Close review items one by one with matching log or screen evidence—not a verbal “should be fine.”
Keep the build reproducible: target architecture, firmware kit version, and package deps in the project notes. Generated references to libraries not in the tree are environment issues—separate round from graphics bugs. Another machine should rebuild from the notes, or you cannot tell code regression from environment drift.
Acceptance checklist
Before delivery, confirm and keep serial excerpts:
Clean tree configures and compiles with no machine-absolute paths. Image locates graphics output; a wrong GUID fails and logs. After set, framebuffer parameters are re-read; full-screen solid color is correct; channel order verified with four color dots. Main-loop heartbeats are continuous; a deliberate out-of-range write in a debug build is rejected rather than resetting. Input edges vs. repeat rate are distinguishable on hardware or emulator. Return-to-firmware path works; mode restore is as designed or explicitly documented as not restored and why.
These turn “the model said it would run” into a checkable result. Different tools can share the same protocol task so comparison means something; different tasks and environments make elapsed time a poor quality metric. With a modular draft, fix contracts rather than rewrite; it usually converges faster.
The hard part of firmware graphics is not tile or block rules—it is protocol identifiers, mode-info lifetime, and the lack of interactive debugging. Lock contracts in review, stage work with serial, prove the framebuffer with solid color, then iterate timing and input. AI tools can then implement and mop up reliably instead of stopping at source that looks clean but never lights the screen.
