Architecture Design and Implementation of Agent Harness for Intelligent Programming

As large language model (LLM) applications gradually transition from “conversational Q&A” to “autonomous task execution,” a frequently overlooked issue is that the model itself does not equal a complete intelligent agent.

Models excel at understanding natural language, generating plans, analyzing code, and predicting the next action. However, they typically do not inherently possess file access, command execution capabilities, project knowledge, task state, permission boundaries, or long-term memory. To enable models to genuinely participate in software development, data processing, or DevOps work, a complete runtime environment must be built around the model. This environment is commonly referred to as the Agent Harness, which can also be understood as the “agent’s workbench” or “agent runtime.”

1. What is an Agent Harness?

An Agent Harness is not another larger model, nor is it a fixed set of business processes. Its core role is to provide the model with the capabilities necessary to complete tasks, while simultaneously constraining, recording, and governing the execution process.

It can be represented by the following abstract formula:

Agent Harness =
Tool Capabilities
+ Knowledge Supply
+ Environmental Observation
+ Action Interfaces
+ Permission Governance
+ State Management

Here, tool capabilities determine what the agent “can do”; knowledge supply determines what the agent “knows”; environmental observation determines what the agent “sees”; action interfaces determine how the agent “changes the external world”; permission governance determines “within what boundaries the agent acts”; and state management ensures the agent can continuously complete complex tasks.

Many early intelligent applications focused heavily on prompt design, attempting to make the model strictly follow preset steps through carefully organized instructions. However, as tasks become complex, relying solely on prompts often leads to obvious problems:

1. The model cannot access real project files;
2. The model cannot verify whether its generated modifications are effective;
3. Context is easily lost across multi-turn tasks;
4. There is a lack of recovery mechanisms after task failure;
5. Different operations lack clear permission boundaries;
6. Parallel execution of multiple tasks can cause mutual interference.

Therefore, an excellent agent system does not make all decisions for the model. Instead, it provides a sufficient, reliable, and observable operating space, allowing the model to make decisions within clear boundaries.

2. Starting with the Minimum Agent Loop

Any complex intelligent agent can usually be reduced to a simple loop:

Receive Goal
  ↓
Understand Current State
  ↓
Decide Next Action
  ↓
Call Tool
  ↓
Get Execution Result
  ↓
Update Context
  ↓
Determine if Task is Complete

This can be represented in pseudocode as follows:

def run_agent(user_goal, tools, memory):
    state = initialize_state(user_goal, memory)
    while not state.finished:
        observation = collect_observation(state)
        decision = model_decide(
            goal=user_goal,
            observation=observation,
            history=state.history,
            available_tools=tools
        )
        if decision.type == "tool_call":
            result = execute_tool(
                name=decision.tool_name,
                arguments=decision.arguments,
                permissions=state.permissions
            )
            state = update_state(state, decision, result)
        elif decision.type == "final_answer":
            return decision.content
        else:
            state = handle_invalid_decision(state, decision)
    return state.result

This logic seems simple, but it embodies the most important design philosophy of an agent system: the model proposes the next action, while the runtime executes the action, returns the result, and maintains the state.

In an actual system, the loop also needs to handle timeouts, retries, tool exceptions, user confirmations, interruption recovery, and context length limits. In other words, the Agent Loop is the heart of the agent system, but not its entire body.

3. Tool System: Enabling the Model to Affect the Real Environment

If a model can only output text, it is essentially just a Q&A system. The tool system endows the model with the ability to observe and operate in a real environment.

For intelligent programming agents, tools can generally be categorized into the following classes:

1. Information Reading Tools

Used to view directory structures, read files, search text, analyze configurations, and check version status.

The focus of these tools is not necessarily having more features, but ensuring the returned results are clear, stable, and controllable. For example, reading large files should support range limits, search results should retain file locations, and directory traversal should avoid returning irrelevant content all at once.

2. Content Modification Tools

Includes operations such as creating files, modifying local content, deleting files, and batch replacement.

Modification tools require special attention to precision. Compared to having the model regenerate an entire file, modification based on local diffs is easier to review and reduces the risk of accidental code deletion. Before actual execution, it is also advisable to check if the target content matches expectations, preventing overwrites of other modifications if the file has already changed.

3. Verification and Execution Tools

After modifying code, the agent needs to run tests, static checks, builds, or other verification processes. Without a verification step, the agent can only “guess” whether its work was successful.

The return of verification tools should not just be success or failure; it should also include error locations, output summaries, and potential fix clues. This allows the model to continue iterating based on feedback rather than repeating the same action after a failure.

4. External Information Tools

Some tasks require querying documentation, reading knowledge bases, or accessing controlled external resources. Such tools must set source ranges, access permissions, and content filtering mechanisms to prevent the model from directly treating untrusted information as fact.

4. Knowledge Loading: On-Demand Provision, Not One-Time Stuffing

Complex projects usually contain numerous rules, including directory conventions, code standards, build processes, testing requirements, and security limits. Stuffing all knowledge into the context at once leads to overly long inputs, obscured focus points, and an increased probability of model confusion.

A more reasonable approach is to establish a hierarchical knowledge system:

Global Rules
  ↓
Project Rules
  ↓
Directory Rules
  ↓
Task-Specific Rules
  ↓
Current Files and Execution Results

The agent first loads rules directly related to the current task and only retrieves finer-grained information when needed. This “on-demand loading” mechanism has three benefits:

• Reduces context consumption;
• Minimizes interference from irrelevant information;
• Brings task rules closer to the actual point of operation.

Knowledge loading should also distinguish between “guidance information” and “mandatory constraints.” For example, code style is guidance information, while prohibiting modifications to certain directories is a mandatory constraint. The runtime should handle the two separately to prevent the model from mistaking security limits for general suggestions.

5. Context Management: From Conversation Logs to Task State

Context management is the key to whether an agent system can handle long-term tasks.

Simply saving all historical messages causes the context to continuously expand. A more reasonable approach is to split information into multiple layers:

1. Task Goal: What the user initially wanted to accomplish;
2. Current Plan: Which steps are being executed;
3. Completed Items: What work has passed verification;
4. Key Decisions: Why a certain implementation method was chosen;
5. Environment State: The current state of files, tests, and dependencies;
6. Error Records: Which failed approaches have already been attempted;
7. Long-term Knowledge: Project facts that can be reused for subsequent tasks.

When the context approaches its capacity limit, compression can be performed, but compression should not simply truncate old messages. Effective compression should retain task goals, incomplete steps, key files, failure reasons, and items pending verification.

The compression results can be organized into the following structure:

Task Goal:
Current Progress:
Modified Files:
Passed Verifications:
Unresolved Issues:
Attempted but Failed Approaches:
Next Step Suggestions:

This way, even if early conversations are summarized, the agent can still resume work rather than re-exploring the entire project.

6. Task Planning and Dependencies

When a user requests to “implement a feature,” the agent often needs to complete multiple steps: requirements analysis, code modification, test supplementation, documentation updates, and result verification. Mixing all work into a single loop easily leads to omissions.

Therefore, tasks can be broken down into nodes with dependencies:

Requirements Analysis
  ↓
Interface Design ─────┐
  ↓                   │
Core Implementation   │
  ↓                   │
Test Writing ─────────┘
  ↓
Integration Verification
  ↓
Result Aggregation

The task system should at least record:

• Task name;
• Inputs and outputs;
• Prerequisite dependencies;
• Current status;
• Executing agent;
• Failure reason;
• Retry count;
• Last updated time.

Compared to a simple linear to-do list, a dependency graph is more suitable for complex engineering tasks. Tasks without prerequisites can be executed in parallel, while tasks with unmet dependencies must wait. This not only improves efficiency but also prevents multiple execution units from simultaneously modifying the same unstable state.

7. Multi-Agent Collaboration: Division of Labor Must Be Built on Boundaries

Multi-agent does not mean simply starting multiple model instances. Truly effective collaboration requires clarifying each agent’s responsibilities, inputs, outputs, and modification scope.

Common roles can include:

• Requirements Analysis Agent: Responsible for understanding goals and decomposing tasks;
• Implementation Agent: Responsible for modifying code;
• Verification Agent: Responsible for running tests and checking results;
• Review Agent: Responsible for evaluating code quality and potential risks;
• Aggregation Agent: Responsible for integrating conclusions from each stage.

Collaboration also requires a reliable messaging mechanism to pass task status, error information, and execution results. Messages should be as structured as possible, avoiding the transmission of only vague natural language descriptions.

For example:

{
  "task_id": "task-001",
  "status": "completed",
  "changed_files": ["src/example.py"],
  "verification": {
    "passed": true,
    "summary": "Basic tests passed"
  },
  "notes": "Original interface behavior preserved"
}

The most common issues in multi-agent systems are repetitive modifications, inconsistent states, and blurred responsibility boundaries. Therefore, collaboration design must prioritize traceability rather than blindly increasing the number of agents.

8. Background Tasks and Asynchronous Execution

Some work is time-consuming, such as builds, batch testing, data processing, and dependency scanning. If the agent must wait for the tool to return, the overall interaction experience will significantly degrade.

The background task mechanism can separate task submission, status querying, and result retrieval:

Submit Task → Get Task ID
               ↓
         Query Execution Status
               ↓
         Get Final Result

Background tasks need to consider the following issues:

• How to cancel a running task;
• Whether to automatically retry after task failure;
• How to save standard output and error output;
• How to notify the main agent upon task completion;
• How to recover state after a process exits unexpectedly;
• How to limit resource competition among multiple tasks.

Asynchronous execution is not just about changing synchronous code to background threads; it requires establishing a complete task lifecycle management mechanism.

9. Isolated Environments: The Safe Foundation for Parallel Modifications

When multiple tasks modify the same project simultaneously, directly sharing the working directory introduces severe risks: one task might overwrite another’s files, and it may be impossible to determine which modification produced the test results.

A more robust solution is to create an isolated workspace for each independent task. Each workspace has its own file state, execution log, and set of changes. After the task is completed, the results are aggregated through a review and merge process.

The value of isolated environments is mainly reflected in:

1. Preventing parallel tasks from overwriting each other;
2. Supporting quick rollback of failed tasks;
3. Facilitating the comparison of different implementation approaches;
4. Ensuring clear attribution for test results;
5. Reducing the impact of experimental operations on the main branch.

Isolation applies not only to code but can also be extended to temporary databases, caches, configuration files, and running ports.

10. Permission Governance: Keeping the Agent Always Within Controllable Bounds

The more capabilities an agent possesses, the more important permission governance becomes. Any system capable of modifying files, executing programs, or accessing external resources cannot rely solely on the model’s self-discipline.

Permission design can adopt a tiered approach:

Read-Only Operations
  ↓
Low-Risk Writes
  ↓
Restricted Execution
  ↓
Sensitive Operations Require Confirmation
  ↓
High-Risk Operations Prohibited

Specific governance measures include:

• Setting allowed ranges for file paths;
• Setting whitelists for execution operations;
• Restricting network access;
• Desensitizing sensitive information;
• Requiring manual confirmation for high-risk actions;
• Logging every tool call and its result;
• Setting time, frequency, and resource limits for tasks.

The goal of a permission system is not to prevent the agent from working, but to make risks predictable, auditable, and recoverable.

11. How to Design a Scalable Implementation Roadmap

If building an agent runtime environment from scratch, a progressive roadmap can be adopted:

Phase 1: Minimum Loop

First implement goal reception, model decision-making, tool calling, and result return, ensuring the agent can complete simple reading and modification tasks.

Phase 2: Tool Standardization

Unify tool descriptions, parameter validation, error formats, and execution logs, preventing each tool from having a completely different calling method.

Phase 3: Add Knowledge and Context Management

Implement rule loading, task summarization, history compression, and state recovery, allowing the agent to handle longer, more complex tasks.

Phase 4: Add Task System

Support task decomposition, dependencies, failure retries, and progress tracking, upgrading one-time execution into a manageable engineering process.

Phase 5: Add Parallelism and Isolation

Provide background execution and independent workspaces for independent tasks, improving throughput and reducing mutual interference.

Phase 6: Improve Governance and Observability

Supplement permission control, audit logs, metric monitoring, exception alerts, and manual approvals, equipping the system with the reliability required for production environments.

12. Conclusion

The true value of an Agent Harness lies not in stacking complex modules, but in whether it can connect the model’s capabilities to the real environment in a clear, restrained, and verifiable manner.

A mature intelligent agent system generally follows these principles:

• Use tools to expand the model’s action capabilities;
• Use knowledge loading to reduce invalid exploration;
• Use context management to support long-term tasks;
• Use task dependencies to organize complex processes;
• Use multi-agent division of labor to improve parallel efficiency;
• Use isolated environments to reduce modification risks;
• Use permission governance to ensure controllable actions;
• Use logging and verification mechanisms to ensure trustworthy results.

Future intelligent programming systems will not just be a chat window capable of generating code; they will gradually evolve into engineering runtime environments with perception, planning, execution, verification, and recovery capabilities. Whether deployed on local infrastructure or running in cloud environments like Baidu AI Cloud, the key determinant of their actual value is not the model name itself, but whether the surrounding Harness is reliable, transparent, scalable, and capable of keeping the model working continuously within the correct boundaries.