Refactoring Legacy Systems with Multimodal Agents: Practices and Engineering Value

In recent years, multimodal agents equipped with visual understanding, code generation, GUI operation, and long-term task planning capabilities have been transforming how developers collaborate with computer systems. Previously, AI excelled at single-turn tasks like explaining concepts, completing code, or generating documentation. Today, the technological focus has shifted toward models that can read screen states, understand natural language goals, invoke controlled tools, modify files, execute programs, and continuously adjust their execution paths based on runtime results.

From a technical perspective, this does not imply the systems possess “almighty intelligence.” Rather, it indicates that large model-centric agentic architectures are beginning to cover cross-tool tasks that previously required senior engineers. This capability offers clear engineering value, particularly in areas such as legacy system maintenance, prototype validation, content production, test automation, and DevOps assistance.

1. The Capability Leap: From “Content Generation” to “Task Completion”

Traditional generative models primarily output text, code, or images. Even if a model generates a runnable program, developers must still manually perform the following steps:

1. Create project directories and base configurations;
2. Write code to the correct files;
3. Install dependencies or prepare the runtime environment;
4. Launch the program and observe errors;
5. Analyze exception logs;
6. Modify the implementation and repeat validation;
7. Package the final result into a deliverable artifact.

Agentic systems attempt to chain these discrete steps into a closed loop. The core value is not merely “writing a few more lines of code,” but the ability to manage state and adjust behavior based on feedback.

A comprehensive task loop typically includes:

• Goal Understanding: Translating natural language requirements into executable subtasks;
• Environment Perception: Reading file structures, terminal outputs, UI elements, device states, or log information;
• Plan Generation: Determining the sequence, dependencies, and risk points;
• Tool Execution: Editing code, running scripts, operating UIs, and invoking controlled services;
• Result Verification: Checking build outputs, test results, screen states, or key metrics;
• Reflection and Correction: Identifying causes and iterating when results mismatch expectations;
• Process Auditing: Retaining operation logs, changes, test results, and human approval records.

This workflow can be abstracted into the following pseudocode:

Goal = Parse user requirements()
While acceptance criteria are not met:
    Current State = Get environment state()
    Next Step = Generate plan based on goal and state(Goal, Current State)
    If Next Step involves high-risk operations:
        Request human confirmation()
    Else:
        Execution Result = Execute controlled tool(Next Step)
    Verification Result = Check output(Execution Result)
    If Verification Result fails:
        Log failure reason()
        Adjust plan()
    Else:
        Update task progress()
Output deliverables and audit logs()

The key to this architecture is “verifiability.” If a system can only click through interfaces and generate files but cannot determine if the outcome is correct, it remains an automated demo. Only when it can rely on tests, logs, rules, and human confirmation to evaluate task quality does it possess the foundation to enter real engineering workflows.

2. Legacy Computer Repair: A High-Value Scenario

Restoring an old computer that has been decommissioned for years is often more complex than installing a new system. The problem is rarely a single point of failure but rather a compounding of uncertainties:

• Outdated hardware interface standards and missing compatibility information;
• Aging storage media, bad blocks, or file corruption;
• Boot processes relying on specific sequences or historical configurations;
• Software formats, encoding methods, and driver mechanisms incompatible with current environments;
• Incomplete original documentation, sometimes only scattered paper records;
• Operators unfamiliar with early system commands, file structures, and maintenance methods.

In this context, the value of a multimodal agent should not be misunderstood as “automatically repairing old machines,” but rather as helping experts establish a diagnostic chain more efficiently.

2.1 Information Collection and Fault Hypothesis

The agent can assist in organizing screen prompts, device labels, interface photos, boot logs, and storage directories, categorizing them into:

• Hardware identification information;
• Boot phase information;
• OS or runtime environment information;
• File system status;
• Application dependencies;
• Potential risk warnings.

The system can then establish fault hypotheses based on known phenomena. For example, if a device powers on but stalls during the loading phase, troubleshooting directions might include power stability, boot media readability, configuration parameters, memory status, and display output links. The goal here is not for the model to draw direct conclusions, but to output an “evidence-hypothesis-verification” chain.

An auditable diagnostic record can adopt the following structure:

Phenomena:
- Device boots but stalls during the loading phase
- Storage media read experiences intermittent failures
Hypotheses:
- Aging storage media causes key files to be unreadable
- Boot configuration points to an invalid path
- Peripheral initialization sequence is abnormal
Suggested Verifications:
- Perform a read-only image backup first
- Execute an integrity check on critical directories
- Verify configuration files in an isolated environment
- Prohibit direct writes to the original media

2.2 Protect Data First, Then Attempt Repair

The most important principle in legacy system maintenance is: Original data takes priority over recovery speed. Even if an agent possesses strong operational capabilities, it must be constrained by strict security policies.

Recommended workflows include:

1. Create a read-only copy of the original media;
2. Verify the copy and record the checksum;
3. Analyze the file structure in an isolated environment;
4. Prioritize non-destructive diagnostics;
5. Require human approval for every write operation;
6. Log modification operations as a rollbackable list;
7. Output recovery results, unresolved issues, and risk descriptions.

For enterprises, this principle applies equally to legacy business systems. Many critical operations still rely on long-running applications, databases, or script tasks. Agents can assist in reading historical code, completing documentation, identifying expired dependencies, and generating test cases, but they cannot bypass change management, backup mechanisms, and approval protocols.

3. Interactive Prototype Generation: From Code Snippets to Playable Results

Another compelling direction is the agent’s ability to generate graphical prototypes with interactive logic in a short timeframe. For instance, a simple scene involving flight, locking, movement, collision, and scoring may appear to be purely visual, but it actually involves multiple engineering steps:

• Scene and resource organization;
• Input event handling;
• Entity state management;
• Camera or perspective logic;
• Collision detection;
• Animation and time-stepping;
• Audio and feedback mechanisms;
• Performance control;
• Browser or runtime environment compatibility;
• Repeatable build and deployment.

If a system can generate a runnable initial prototype from a single natural language description and continuously fix it based on console logs and screen results upon failure, its significance lies in reducing the time-to-validation for an idea.

However, development teams must clearly distinguish between “prototype completion” and “product deliverability.”

Prototypes Typically Focus On:

• Whether the core gameplay or interaction path is viable;
• Whether the visual style meets expectations;
• Whether the basic logic is demonstrable;
• Whether users can understand the operation method quickly.

Products Must Also Focus On:

• Resource copyright and source management;
• Security and data protection;
• Performance and compatibility testing;
• Exception handling and degradation strategies;
• Telemetry, monitoring, and observability;
• Multi-platform adaptation;
• Maintainable module partitioning;
• Automated test coverage;
• Version release and rollback capabilities.

Therefore, a more appropriate usage is positioning the agent for “rapid exploration” and “engineering assistance.” It can help teams quickly validate ideas, generate boilerplate code, fill in repetitive logic, construct mock data, and draft tests, but the final architecture design, performance governance, and quality assurance must remain engineer-led.

4. Technical Challenges in 3D Content and GUI Manipulation

When an agent begins to operate complex graphical software, design tools, or development environments, the difficulty increases significantly. GUIs are not inherently designed for automation: button positions may change, windows may overlap, the same action can mean different things in different contexts, and task results are not always expressible via text logs.

To enable an agent to reliably perform tasks like 3D object binding, material adjustment, and scene layout, at least four categories of problems must be solved.

4.1 Interface Semantic Understanding

The system cannot rely solely on coordinate clicks; it must understand “which panel is currently active,” “what object is selected,” and “whether a parameter has taken effect.” Otherwise, window resizing or layout adjustments could lead to operational failure.

4.2 Operation Sequence Planning

Many graphical tasks have strict sequential dependencies. For example, one must confirm the object hierarchy, create relational links, preview, and finally export the asset. Any sequential error can render the result unusable.

4.3 Visual Result Verification

Unlike checking if a compilation passes, graphical results often require multidimensional judgment. For example, checking if a model is clipping, if an animation is natural, if lighting is abnormal, or if materials are missing. This requires the system to combine visual checks with rule validation, rather than relying solely on “command executed successfully.”

4.4 Rollback Mechanisms

Graphical editing typically generates numerous state changes. Before executing an operation, the agent should automatically save a checkpoint; afterward, it should generate a change summary. For batch modifications, it should also support rollback by object, by step, or by version.

5. Security Boundaries for Enterprise Implementation

The stronger the agent’s capabilities, the more critical permission governance becomes. A system capable of reading/writing files, executing commands, operating web pages, and invoking internal services can introduce risks from misoperations, prompt injection, privilege abuse, and data leakage if unconstrained.

Enterprises should establish a governance system across the following dimensions.

5.1 Principle of Least Privilege

Different tasks should use different identities and permissions. A code analysis agent should not have default write access to production environments; a document organization agent should not access sensitive data; DevOps assistance tasks should primarily run in read-only mode.

5.2 High-Risk Action Approval

The following operations should require human confirmation:

• Deleting files or data;
• Modifying production configurations;
• Executing irreversible scripts;
• Accessing sensitive information;
• Sending data to external systems;
• Committing code or releasing versions;
• Adjusting permission policies.

5.3 Sandbox and Isolated Environments

Agents should be prioritized to complete tasks in isolated directories, test environments, temporary containers, or with mock data. Only after passing tests and human review should changes enter the formal workflow.

5.4 Result-Based Acceptance Over Process-Based

Do not assume a task is complete simply because the system “appears to have executed many steps.” Explicit acceptance criteria should be used, such as:

• Did the build succeed?
• Did tests pass?
• Do key metrics meet thresholds?
• Are output files complete?
• Is data consistent?
• Are there security alerts?
• Is there a complete operation log?

5.5 End-to-End Auditing

Enterprises should log what instructions the agent received, what information it read, which tools it called, what changes it made, what exceptions it encountered, and who approved them. This facilitates accountability and helps continuously optimize task workflows.

6. How Development Teams Should Begin Pilots

For teams looking to introduce agentic capabilities, it is not recommended to assign them core production responsibilities initially. Instead, choose low-risk, highly repetitive tasks with low verification costs for piloting.

Scenarios suitable for early trials include:

• Automatically organizing code repository documentation;
• Generating test drafts based on existing APIs;
• Parsing logs and summarizing common fault patterns;
• Generating pre-migration data checklists;
• Scanning configuration files for potential omissions;
• Assisting in the automation of repetitive page operations;
• Reproducing defects in test environments and generating fix suggestions;
• Organizing historical DevOps logs into knowledge base entries.

The pilot process can be advanced according to the following pace:

1. Define task boundaries and acceptance criteria;
2. Prepare isolated environments and desensitized data;
3. Configure limited tool permissions for the agent;
4. Establish manual review checkpoints;
5. Collect failure samples and exception paths;
6. Optimize prompt templates, tool protocols, and test rules;
7. Gradually expand the scope of automation.

7. Conclusion: True Value Lies in Controllable Engineering Loops

The most noteworthy aspect of multimodal agents is not their ability to deliver astonishing demos quickly, but whether they can complete tasks in a controllable, verifiable, and auditable manner within real engineering environments.

For developers, they can reduce the time spent on environment setup, information retrieval, repetitive coding, and initial troubleshooting. For enterprises, they offer the opportunity to connect knowledge scattered across documents, code, logs, interfaces, and historical experience, forming a more efficient collaboration process.

However, agents are not a shortcut to bypass engineering methodologies. The more a task involves legacy systems, complex graphical operations, production DevOps, and sensitive data, the more necessary it is to adhere to backup-first, least privilege, test-driven development, manual gatekeeping, and end-to-end auditing. Only by embedding model capabilities into mature software engineering and security governance systems can we transform “seemingly powerful” automated capabilities into stable, trustworthy, and sustainable productivity.