Testing General-Purpose LLMs on Legacy Code: A Practical Engineering Retrospective
This article examines how a general-purpose large language model performs on a real engineering task: taking over a low-level graphics program that cannot run out of the box. Across two rounds of work, the model first reviewed the codebase instead of blindly editing it, diagnosed a startup crash caused by a stale framebuffer pointer, then fixed a protocol-instance mismatch that left the screen blank. In a second round it refactored the program for stability and maintainability with an explicit state machine, structured logging, input handling, frame-rate control, and proper resource cleanup. The piece breaks down the evaluation dimensions, the debugging process, the model's strengths and limits, and a reusable, phased workflow for using LLMs as a development copilot on unfamiliar or legacy codebases.
1. Why “Taking Over Legacy Code” Makes a Better Test Sample
Once general-purpose LLMs became available as web services, the question developers cared about was no longer just “can it write code?” but “can it solve problems inside a real project?” Generating a sample program from scratch shows off code organization, but it rarely reflects the true challenges of a complex codebase. Real projects carry large amounts of historical code, implicit conventions, environment dependencies, and cross-module calls. Low-level programs, firmware, and graphics-driver code in particular are prone to initialization-order bugs, resource-lifecycle errors, and interface-mismatch failures.
So this test picked a sample closer to real engineering work: handing a general-purpose LLM a low-level graphics program that could not run as-is. Such programs typically span entry-point initialization, a main loop, a rendering module, input handling, and device-protocol access. A fault in any one of these stages can lead to a compile failure, a runtime crash, a blank screen, or unresponsive input.
Unlike a simple “build a small game” test, the focus here was whether the model could read existing code, locate potential defects, propose actionable fixes, and progressively converge on the root cause through multiple rounds of runtime feedback. This scenario better reflects the value of an AI coding tool in a real development workflow.
2. Test Goals and Evaluation Dimensions
To keep the conclusions rigorous, we did not look only at “did it eventually run.” Instead, we observed the model across several dimensions:
Code comprehension — Could the model quickly grasp the project structure, module responsibilities, and call relationships, rather than fixating on a single function?
Defect identification — Before or without running the program, could it spot latent risks such as null pointers, invalid handles, wrong initialization order, and mismatched protocol identifiers?
Runtime debugging — When the program crashed, went black, or hung, could it propose a troubleshooting path based on logs, symptoms, and code logic?
Fix effectiveness — Were the proposed fixes executable, and could they resolve the problem in one or a few iterations?
Maintainability — Did the modified code stay clear, avoid introducing new complexity, and remain easy to extend and test?
Collaboration style — Was the model suited to a “copilot” role — review first, then modify, then verify — instead of dumping large, hard-to-follow blocks of code?
Seen through these dimensions, the value of a general-purpose LLM is not simply “producing a piece of code” but participating in the full development loop: understand the problem, analyze the cause, propose a fix, verify the result, and improve quality.
3. Round One: From “Won’t Run” to “Basically Runs”
3.1 Review the Code First, Don’t Edit Immediately
The first round did not have the model change code right away. Instead, we asked it to review the whole project and output a list of likely problems. This order matters, because legacy code rarely fails from a single point; it usually results from several factors stacking up. Broad edits made up front tend to break existing logic and make it hard to isolate the root cause.
The model was fairly steady during the review. It analyzed the code layer by layer — entry point, main loop, game logic, rendering module, input handling — and produced a problem list. Typical findings included:
- Resource-acquisition order issues during initialization;
- Memory or handles that may become invalid after a mode switch;
- A protocol-instance lookup that may not match the device environment;
- Input events not being consumed correctly, which could block the loop;
- Unchecked return values that let error states go unnoticed;
- A main loop with no clear exit condition or recovery mechanism.
This stage shows the model has strong static-review ability. It does not depend on the program running; it works from code structure and common systems-programming experience to flag risks early. That is very helpful for maintaining old projects and taking over unfamiliar codebases.
3.2 First Run: The Program Crashes on Startup
After the review, we moved to runtime verification. The program briefly showed initialization output, then exited right before entering the main loop. This symptom indicated the issue was not a compile error but a runtime resource or state anomaly.
Rather than blindly editing business logic, the model suggested adding debug output to observe key variables and call states — logging at key points such as initialization, mode setup, framebuffer acquisition, and main-loop entry. The logs confirmed that the exception happened right after the program accessed a graphics-output interface.
Further analysis led the model to a typical problem: the program fetched framebuffer information before setting the display mode, but after the mode switch, the previous memory mapping or buffer descriptor may already be invalid. Continuing to use that stale pointer could trigger an illegal access and crash the program.
This kind of issue is very common in low-level graphics code. Developers easily fall into the intuition of “query once, then reuse forever.” But under mode switches, device resets, and resource remapping, a pointer obtained earlier is not guaranteed to remain valid.
The model’s fix was: re-query the current framebuffer information after setting the display mode, instead of reusing the pre-switch result. Illustrative code:
void enter_graphic_mode(int width, int height) {int status;status = set_display_mode(width, height);if (status != 0) {log_error("set display mode failed");return;}frame_buffer = query_frame_buffer();if (frame_buffer == NULL) {log_error("frame buffer unavailable after mode switch");return;}screen_width = width;screen_height = height;graphics_ready = 1;}
After the change, the program no longer exited immediately. Small fix, but a critical one. It reflects the model’s grasp of resource lifecycle: in low-level code, handles, pointers, and buffer descriptors are often not permanently valid — they depend on the current runtime state and initialization order.
3.3 No Graphics on Screen: A Protocol-Matching Problem
With the crash resolved, the program could enter the main loop, but no expected graphics appeared on screen. In other words, the program went from “crashes on launch” to “runs but shows nothing.” This class of problem is harder to debug, because nothing errors out and the process stays alive — only the output is wrong.
After re-analyzing the logs and code, the model pointed to the protocol lookup and instance-acquisition step. Its judgment: the code used the wrong protocol identifier or the wrong device handle, so the program kept running but never bound to a usable graphics-output instance.
In low-level graphics programming, the usual approach is to find the corresponding device or service interface through some unified identifier. If the identifier is inconsistent, the instance doesn’t match, or the wrong handle is opened, the program may appear to run normally while being unable to render.
The model’s suggested fixes included:
- Check whether the protocol identifier constants match the current target environment;
- Re-enumerate available devices or interfaces after system startup;
- Look up the corresponding instance from the device handle rather than using a global variable directly;
- Add a state check after acquiring the instance to avoid propagating a null pointer;
- Emit clear error codes to ease later diagnosis.
The illustrative code abstracts to:
status = locate_protocol_instance(&protocol_id, &handle);if (status != 0) {log_error("protocol instance not found");return;}status = open_protocol(handle, &protocol_id, &instance);if (status != 0 || instance == NULL) {log_error("open protocol failed");return;}gfx = instance;
After the fix, the graphics output path gradually recovered and the program reached an observable state. At this point the first-round goal was basically met: the project went from “cannot run directly” to “starts and enters the main loop.”
4. Round Two: From “Runs” to “Can Be Optimized”
Round one solved usability. Round two focused on stability and maintainability. In real engineering, running is only the starting point. A program without logging, error handling, state management, and resource release becomes very expensive to maintain later.
4.1 Introduce a State Machine to Avoid Implicit Flow
The original main loop was fairly linear, with multiple states tangled together in functions. The model suggested introducing an explicit state machine that separates initialization, running, paused, and exit states. This reduces coupling and makes it easier to add new features later.
An illustrative structure:
typedef enum {STATE_INIT = 0,STATE_RUNNING,STATE_PAUSED,STATE_EXIT} app_state_t;void main_loop(void) {while (app_state != STATE_EXIT) {switch (app_state) {case STATE_INIT:if (initialize_all() == 0) {app_state = STATE_RUNNING;} else {app_state = STATE_EXIT;}break;case STATE_RUNNING:process_input();update_game();render_frame();break;case STATE_PAUSED:handle_pause_event();break;default:break;}}}
The value of this refactor: if you later add pause, resume, error restart, or performance degradation, you don’t need to touch the whole main loop — you just add nodes or transition conditions to the state machine.
4.2 Build Out a Proper Logging System
During debugging, logs are the key to locating problems. In the optimization phase, the model suggested splitting logs into levels — error, warning, info, and debug. This avoids excessive logging that hurts performance while keeping essential information in production.
For example:
log_error("display mode switch failed");log_warn("input event dropped: queue full");log_info("game loop entered");log_debug("frame updated, cost=%d ms", cost);
A logging system not only helps with the current problem but also lays the groundwork for future automated monitoring and observability. If the program runs in a controlled environment, logs can also be collected and searched through a general-purpose logging service.
4.3 Add Input Handling and Frame-Rate Control
The original input handling was likely simple and prone to event pile-up, duplicate responses, or blocking the main loop. The model suggested putting input events into a queue and consuming them uniformly each frame, so input processing doesn’t slow down rendering.
It also suggested a frame-rate limit to avoid wasting resources on unbounded redraws. Illustrative logic:
if (elapsed_time >= frame_interval) {render_frame();last_frame_time = now();}
These optimizations may not directly change functionality, but they noticeably improve stability. In low-resource environments especially, frame-rate control and event throttling often matter more than piling on features.
4.4 Strengthen Error Handling and Resource Release
In the second round, the model also added resource release and exception return paths. For example, if initialization fails, handles already opened should be closed and buffers already allocated should be freed to avoid leaks.
A sound initialization flow should be able to roll back:
int initialize_system(void) {if (open_graphics() != 0) {log_error("open graphics failed");return -1;}if (allocate_buffer() != 0) {close_graphics();log_error("allocate buffer failed");return -2;}return 0;}
This adds only a little code but significantly lowers maintenance risk. In low-level programs especially, resource leaks and inconsistent states are often harder to track down than business-logic errors.
5. Notable Highlights
Across these two rounds, the general-purpose LLM showed several clear strengths in a code-engineering context.
5.1 It Starts from the Overall Structure Rather Than Editing in Isolation
The model didn’t begin by modifying a single function. It first reviewed module boundaries, call relationships, and initialization order. This shows a degree of system-level understanding — for complex projects, that ability matters more than simply completing code.
5.2 It Turns Obscure Problems into Verifiable Steps
Startup crashes and black screens are not easy to diagnose directly. By breaking the problem into steps — log output, state observation, pointer checks, protocol matching, runtime verification — the model lowered the debugging difficulty. This decomposition is very helpful for developers, especially when facing an unfamiliar codebase.
5.3 It Has Good Awareness of Low-Level Resource Lifecycles
Whether re-querying the framebuffer after a mode switch or looking up and opening a protocol instance, the model consistently analyzed around the question “is the resource still valid?” Such issues may not stand out in ordinary application development, but they are critical in low-level systems, driver interfaces, and graphics output.
5.4 It Supports Multi-Round Iterative Optimization
The model did not deliver final code in one shot; it kept adjusting based on runtime feedback. The first round addressed runnability, and the second addressed stability, maintainability, and performance. This iterative approach closely matches a real development workflow and shows the model is suited to being a development assistant rather than a simple code generator.
5.5 Its Output Is Somewhat Explainable
When fixing things, the model usually explains the reason, the scope of impact, and how to verify — rather than just handing over code. This matters a lot for team collaboration: developers can judge the risk from the explanation and decide whether to adopt, roll back, or test further.
6. Hands-On Experience and Awareness of the Boundaries
Overall, the general-purpose LLM performed well in code comprehension, defect localization, fix suggestions, and engineering optimization, and it clearly lowered the barrier for developers entering unfamiliar projects. But it shouldn’t be overhyped — it still has clear limits.
6.1 Clear Advantages
For tasks like code review, taking over old projects, log analysis, basic refactoring, and test-script generation, the model can noticeably boost efficiency. It helps developers quickly build an overall understanding of a codebase, and even with missing docs and sparse comments, it can still offer reasonably sound risk hints.
6.2 It Can’t Replace Real Verification
Low-level programs often depend on specific hardware, firmware environments, driver support, and external device states. The model can analyze code logic, but it can’t replace a real runtime environment. Any generated code must go through compilation, execution, log verification, and regression testing before moving downstream.
6.3 Human Developers Must Stay in the Loop
The model may propose solutions that look reasonable but don’t fit the project’s conventions. For example, interface names, initialization order, and resource-management approaches may depend on a specific runtime environment. So developers still own architecture decisions and acceptance. The model is better as a copilot than as an autopilot.
6.4 It’s Sensitive to Context Quality
Give the model too little information and it can only offer generic advice; give it a complete directory structure, error logs, key interface descriptions, and runtime feedback, and its fix accuracy improves markedly. So the key to using the model isn’t “asking one question” — it’s “giving it clearly.”
7. Reusable Engineering Recommendations
Based on this hands-on experience, we can summarize using a general-purpose LLM as a more reliable engineering workflow.
7.1 Define the Task Boundaries
Before calling the model, spell out the target environment, programming language, dependency constraints, no-touch scope, and acceptance criteria. For example:
- Must existing interfaces stay unchanged?
- Are new modules allowed?
- Is only the standard library permitted?
- Must it be compatible with low-resource environments?
- Are logs or error codes required in a specific language?
- Is modifying the startup flow allowed?
The clearer the boundaries, the closer the model’s output is to something usable.
7.2 Provide Minimal Context
Don’t paste just a single function. Provide the module structure, call chain, error logs, and runtime symptoms relevant to the problem. For complex issues, give minimal reproduction steps. The more complete the context, the easier it is for the model to find the root cause.
7.3 Adopt a Phased Strategy
Recommended stages:
- Code review: analyze only, no changes, output a risk list;
- Defect fixing: make minimal changes to key problems;
- Runtime verification: confirm via logs whether the problem is solved;
- Performance optimization: improve efficiency and readability on a stable base;
- Regression testing: confirm the changes introduced no new problems.
This phased approach lowers risk and makes it easier to judge whether each step is effective.
7.4 Ask the Model to Explain Its Changes
After each change, ask the model to state which files it touched, why, which modules might be affected, how to verify, and how to roll back. This makes it easy for developers to audit and for the team to collaborate.
7.5 Build an Automated Verification Mechanism
Manual verification can’t cover every scenario. Combine automated means, such as:
- Automated compile checks;
- Static code analysis;
- Basic unit tests;
- Startup smoke tests;
- Key log assertions;
- Runtime duration and memory-usage observation;
- Abnormal-input replay.
If a company wants to make this workflow a long-term practice, it can build on general-purpose cloud capabilities. For example, archive build artifacts in object storage, collect runtime logs through a logging service, watch for abnormal exits with monitoring and alerting, and reproduce the development environment on a container platform. General-purpose capabilities such as the object storage, logging service, container platform, and monitoring and alerting offered by Baidu AI Cloud can serve as the infrastructure backbone for this kind of engineering workflow.
8. Conclusion: The Model’s Value Lies in Joining the Engineering Loop
This hands-on test shows that beyond code generation, what’s more worth attention is a general-purpose LLM’s ability to participate in the engineering loop. It can not only write code but also review code, analyze logs, locate resource problems, propose fix paths, and drive a program from unrunnable to optimizable through multiple rounds of iteration.
For developers, the significance of this shift is that AI is no longer just a completion tool but is gradually becoming a collaborator for code comprehension, defect localization, and engineering optimization — especially in maintaining legacy code, taking over unfamiliar projects, and debugging low-level programs, where the model helps developers build context quickly and cut repetitive troubleshooting costs.
At the same time, we must stay clear-headed: the model can’t replace real-environment verification, nor can it replace a developer’s engineering judgment. The truly efficient way to use it is to let the model take on analysis and suggestions, while human developers own architecture design, risk control, and final acceptance.
So the sensible positioning of a general-purpose LLM is not “automate everything” but “improve developers’ understanding and efficiency of action.” When it can reliably contribute across code review, problem localization, fix optimization, and engineering collaboration, it already carries real engineering value.
