Coding Agent Harness: What Wins After Model Convergence

Over the past year, the debate over “how many times can AI-assisted development improve efficiency?” has gradually quieted down. A new topic has emerged in developer communities: “Can we build our own Agent Harness?” The underlying architectures of mainstream coding assistants are becoming increasingly similar. At their core is an Agent Loop, which repeatedly iterates through “read/write context — call tools — observe results — make decisions again,” supported by automated tests and permission controls as safeguards. As differences between models narrow, the engineering quality of the Harness is becoming the key variable that determines whether a Coding Agent can truly be adopted.

1. The Question Has Changed: From “How Different Is the Model?” to “What Does the Harness Do?”

In early technology selection, developers first compared models. The capability boundary of the base model largely determined the upper limit of the tools, so fine-tuning, experimentation, and parameter tuning around the model itself were common approaches. Over the past year, multiple mainstream coding models have clearly converged on foundational capabilities such as instruction following, code understanding, multi-file editing, and tool calling. The gap between leading models is often smaller than the gap between engineering implementations.

One tool founder put it bluntly: continuously adjusting and experimenting with different models is yielding diminishing returns. Some developers believe models are approaching a ceiling, and that newer versions may even regress. These judgments are still open to debate, but they all point to one reality: simply swapping models is unlikely to deliver a leap in user experience.

Individual projects are becoming fertile ground for innovation. Much of the innovation in the Harness space comes from individual developers and small teams, rather than platform departments. The authors of these projects come from very different backgrounds. Many projects quickly gained attention after being released in open-source communities, growing from personal experimental tools into active community projects.

One typical example is an open-source project built for deep adaptation to a specific model family. Developers noticed that the training data of mainstream models contained a large amount of game development corpus, so they designed prompts and tool-calling patterns around this model family’s “native language.” Within months of launch, the project had attracted a large number of community followers.

A public beta recruitment call once received more than a thousand responses in a very short time. Many individual developers brought their own Harness projects for self-recommendation. These projects covered coding agents, long-term memory, skill reuse, effect evaluation, and security isolation. The focus of discussion also shifted from “can the tool be used?” to “how should the architecture be designed, how should context be managed, and how should effectiveness be evaluated?”

It is not an exaggeration to describe a Harness with the concept of a “minimal viable kernel”: a while loop, plus the ability to read/write files, execute commands, observe output, and maintain conversation history, is enough to build a coding agent capable of completing simple tasks. The code volume is often only a few hundred lines. But above this shell, there is a series of engineering problems: how to organize context, how to design tools, how to recover from errors, how to control permissions, and how to evaluate effectiveness. These are what truly separate products from one another.

2. The Six-Layer Architecture of a Harness: From the Main Loop to the Security Boundary

A Harness can be understood as the outer structure that carries the model’s runtime: the model is responsible for “thinking,” while the Harness provides information, provides means, constrains behavior, and evaluates results. From an engineering perspective, a complete Coding Agent Harness can be divided into six layers:

Layer	Main Responsibility	Key Design Question
Main loop	Drives the iteration of “reasoning — action — observation”	How to determine termination, prevent idling, and recover from errors
Context layer	Provides task-related information to the model	What to include, how to organize, when to compress, and how to keep it fresh
Tool layer	Allows the model to operate on the codebase and environment	Tool granularity, return format, error expression, and permission boundaries
Memory layer	Retains key information across turns and sessions	What to store, when to write, how to retrieve, and how to handle conflicts
Orchestration layer	Plans, decomposes, schedules, and aggregates work	Single agent or multi-agent, how to parallelize, and how to merge
Safety and evaluation layer	Constrains risk and measures effectiveness	Isolated execution, dangerous operation interception, observability, and regression

2.1 Main Loop: The Minimal Closed Loop and Termination Conditions

The main loop is the heart of a Harness. It repeatedly executes four steps: assemble context, request model reasoning, execute the action chosen by the model, and write the observation result back into the context. This continues until the model no longer requests an action or reaches a preset limit.

Below is an illustrative implementation:

MAX_STEPS = 100          # Hard step limit to prevent infinite loops
MAX_REPEATS = 3          # Tolerance for consecutive failures of the same command
for step in range(MAX_STEPS):
    # 1. Assemble context: system prompt + compressed history + task description + environment information
    messages = context_manager.build(system_prompt=SYSTEM_PROMPT,
                                     history=history,
                                     task=task)
    # 2. Request the model: ask for text or tool calls
    response = model.call(messages, tools=TOOL_SCHEMAS, stream=False)
    # 3. Execute actions: if the model did not request a tool, treat this turn as complete
    if not response.tool_calls:
        task.finish(response.text)
        break
    # 4. Run read-only tools in parallel; run side-effecting tools serially
    for call in response.tool_calls:
        if is_read_only(call):
            results.append(execute_in_parallel(call))
        else:
            results.append(execute(call))  # Writes or commands require permission checks
    history.append(response, results)

Whether a main loop can run reliably depends on three often underestimated designs:

• Termination conditions must be explicit. Typical conditions include the model giving a conclusive reply, all task acceptance items being satisfied, a task-completion signal being produced, and step count, time, or token budget being exhausted.
• Idling must be prevented. If the model repeatedly executes the same command, repeatedly reads the same file, or oscillates between two plans, it wastes budget. It is necessary to perform deduplication detection on the “action — result” sequence and limit the number of consecutive failures of the same command.
• State must be persisted. When a session is interrupted and resumed, the model needs to clearly know which steps have been completed and where it is currently stuck. It is recommended to use a structured task checklist to record state, rather than letting the model “recall” from the conversation history.

2.2 Context Layer: Information Architecture Within a Limited Window

All the information a model can perceive comes from the context. Once the context is filled with irrelevant content, effective information is diluted. Outdated content can also continue to mislead subsequent decisions. Therefore, the primary principle of context management is not “give more,” but “give the right information, give fresh information, and give it compactly.”

Common methods include:

1. Layered loading. The system prompt should retain only stable global constraints. Project-level conventions, such as coding standards and build methods, should be separated into their own section and explicitly labeled with information freshness, so the model does not apply outdated rules.
2. Retrieval on demand. Do not inject the entire codebase at once. Instead, when the model raises a specific question, retrieve relevant code snippets through keyword search or symbol-level search.
3. Phased compression. When the context approaches its limit, summarize early interactions into conclusive summaries, such as which files were modified, which errors were encountered, and which approaches were rejected, then discard the original details.
4. Externalization. Write large pieces of content, such as long plans and test outputs, to files. Keep only the summary and file path in the context, and let the model consult the file when needed. This saves budget and also makes key information easier to version-manage.

The following is an illustrative compression logic:

def compact(history, limit=100_000):
    if estimate_tokens(history) < limit * 0.8:
        return history
    # Keep the system prompt, task checklist, and the most recent few turns; summarize the rest
    summary = summarize(history[:-RECENT_KEEP],
                        focus=["modified files", "verified conclusions", "rejected approaches"])
    return [SYSTEM_PROMPT, summary_message(summary), *history[-RECENT_KEEP:]]

In addition, “dead information” is often more dangerous than “missing information.” After a patch modifies a function, if the context still retains the pre-change content, the model may “fix” a problem that no longer exists. When the code structure changes, the model should be prompted to re-read the relevant files in a timely manner, rather than continuing to use old content.

2.3 Tool Layer: The “Hands” Between the Model and the Environment

Tools are the main means by which the model acts on the codebase and environment. If tools are poorly designed, even the strongest model will frequently make mistakes.

• Moderate granularity. If tools are too coarse, one operation may involve too many changes, making error correction difficult. If tools are too fine, the model may need to make many consecutive calls, increasing the risk of drifting away from the goal. Basic tools can be divided by “find definition — read file — edit patch — run tests — execute command,” and a small number of composite tools can be provided.
• Concise return values. Prioritize returning relevant code snippets, error summaries, and path/line numbers. Avoid dumping an entire directory or lengthy raw logs into the context all at once.
• Errors are information. Error messages should explain the reason for failure and the possible remediation approach, rather than simply returning a non-zero exit code.

Take “finding a symbol in the project source code” as an example. The tool definition can be illustrated as follows:

FIND_SYMBOL = {
    "name": "find_symbol",
    "description": "Find a symbol definition and all references in the project source, returning file paths, line numbers, and context snippets.",
    "parameters": {
        "type": "object",
        "properties": {
            "symbol": {"type": "string", "description": "Function name, class name, or variable name"},
            "max_results": {"type": "integer", "default": 20}
        },
        "required": ["symbol"]
    }
}

For permissions and concurrency, it is recommended to divide tools into three categories:

Tool Type	Typical Operations	Execution Strategy
Read-only tools	Search symbols, read files, view logs	Can be executed in parallel without confirmation
Write tools	Edit files, create branches	Restrict to the working directory, retain change records, and make rollback easy
High-risk tools	Delete data, install dependencies, deploy/release, access external services	Blocked by default, require explicit authorization, and record audit logs

2.4 Memory Layer: Making the Agent Understand the Project Better Over Time

Context is “now,” while memory is “across sessions.” Confusing the two often leads to problems such as “everything resets after restart” or “old conclusions pollute the system for a long time.” Memory can be divided into three layers:

Memory Level	Storage Location	Lifecycle	Typical Content
Short-term memory	Current session context	Single task	Task plan, temporary conclusions, error scenes
Project memory	Convention description files in the repository	Evolves with code versions	Build and test commands, directory conventions, known constraints
Long-term memory	External storage	Across sessions and projects	User preferences, historical decisions and reasons, pitfall records

Several practical experience points:

• Writing should have a threshold. Only record stable and reusable facts, such as build commands, team conventions, and the root cause of a bug. Do not write one-off debugging processes into long-term memory.
• Memory should be verifiable. Each memory item should ideally point to a code location, document, or command, making it easier to verify its freshness next time and prevent “memory hallucination.”
• Conflicts should be resolvable. When new memory contradicts old memory, overwrite the old memory based on time or source credibility, and note the update time in the retrieval result.
• Retrieval should be relevant. Long-term memory should be retrieved on demand, taking only the few items most relevant to the current task. Otherwise, the memory itself becomes context noise.

2.5 Orchestration Layer: Single Agent or Multi-Agent

Multi-agent systems are often regarded as synonymous with “more powerful,” but in fact, they are an architecture choice with clear applicability conditions. The bottleneck of a single agent is that context is easily filled with logs, error messages, and intermediate artifacts, and a single thread also struggles to obtain a second perspective in time. Multi-agent systems are more valuable when tasks can be clearly decomposed, subtasks are relatively independent, multiple approaches need to be explored in parallel, or the “coding” and “review” roles should be separated.

There are three common patterns:

• Master-slave pattern: The master agent is responsible for planning and aggregation, while child agents complete search, implementation, or testing in isolated contexts. Child task results are returned as structured summaries to avoid raw logs polluting the master context.
• Pipeline pattern: Divide roles by “design — implementation — test — review — fix,” and pass artifacts and acceptance criteria step by step.
• Debate pattern: Multiple agents independently provide solutions, and a judge role selects among them based on acceptance criteria.

The costs of multi-agent systems are equally obvious:

• Token consumption usually increases significantly, requiring strict budget control.
• Information distortion can occur between child agents, easily resulting in “working seriously under wrong premises.”
• The debugging chain becomes longer, requiring a unified trace identifier across all subtasks.
• The order of context merging affects the final result, so fixed merging rules are needed.

A pragmatic principle is: first use a single agent to complete the task and quantify the results. Only introduce multi-agent systems after confirming that the bottleneck comes from context capacity or parallel requirements.

2.6 Safety and Evaluation Layer: Making Failures Bearable and Progress Measurable

Because an Agent can directly read/write code and execute commands, its risk is naturally higher than that of traditional code completion tools. The core of safety design is to keep “failure” within a bearable range:

• Isolated execution. Run commands in containers or lightweight virtual machines, limiting network access and file system scope.
• Tiered authorization. Default to read-only. Writes are limited to the working directory. Dangerous operations require manual confirmation. Production and development environments should use different credentials.
• Full auditing. Record every tool call, command, parameter, and result, supporting session replay to facilitate incident review and effectiveness evaluation.
• Injection prevention. The Agent reads external content such as issues, logs, and web pages, which may contain inducing instructions. External content must be treated as untrusted data, explicitly declared in prompts, and the Agent should be restricted from executing high-risk operations based on external content.

Evaluation is the foundation for the continuous evolution of a Harness. It is recommended to build an internally repeatable evaluation set covering typical task types, such as fixing errors, adding features, refactoring modules, and supplementing tests. Fix the model version, prompt, and tool configuration, and compare completion rate, average steps, token consumption, and number of human interventions under different configurations. Without evaluation, optimization easily becomes “tuning by feeling.” If something improves, you may not know why it improved, and if degradation occurs, it is also difficult to detect.

3. Why “Starting With 300 Lines” Works, but Cannot Be Used Directly in Production

A few hundred lines of code are enough to build a runnable coding agent, which shows that the skeleton of a Harness is very thin. The real difficulty is making it stable, safe, and efficient when completing real tasks. The gap between the two is mainly reflected in the following aspects:

Dimension	Prototype Stage	Production Stage
Context	Full conversation history is stuffed in	Layered loading, retrieval on demand, periodic compression, and externalization
Tools	Read/write files and execute commands	Reasonable granularity, concise returns, recoverable errors, and tiered permissions
Termination	Ends when the model stops	Multiple termination conditions, idling detection, and state persistence
Safety	Directly executes on the host machine	Isolated environment, dangerous operation interception, auditing, and replay
Effectiveness	Judged by subjective feeling	Fixed evaluation set, quantifiable metrics, and regression comparison

From a maturity perspective, a Harness can be roughly divided into five stages:

• L0: Runnable. A single-agent loop with basic read/write and command execution capabilities.
• L1: Reusable. Context and tools are abstracted into independent modules, the model can be switched, and project conventions can be configured.
• L2: Observable. Full-chain logs, session replay, token metrics, and time metrics are available.
• L3: Evaluable. An internal evaluation set is established, and any adjustment to prompts, tools, or models has a quantitative basis.
• L4: Governable. Tiered permissions, isolated execution, cost budgets, multi-agent collaboration, and team collaboration norms are fully implemented.

It is worth emphasizing that model replaceability itself is one of the design goals of a Harness. By encapsulating model access as a unified interface and abstracting the standard format of “request — response — tool call — error,” teams can quickly switch and compare models during model iteration, rather than being locked to a specific version.

4. Enterprise Implementation Recommendations and Conclusion

For individual developers, building their own Harness is more about learning and experimentation. For enterprises, the focus is on engineering and platformizing the Harness, and deeply integrating it with the R&D process. The following checklist can be used as a reference:

1. Define evaluation first, then write the loop. Before launch, clarify acceptance criteria and the evaluation set to avoid losing direction during subsequent optimization.
2. Prioritize read-only tools. After read-only tools run stably, gradually open write and high-risk operations.
3. Layer context and persist state. Allow the Agent to continue tasks even after session interruption or compression.
4. Observability first. Every tool call being traceable and replayable is a prerequisite for troubleshooting and evaluating improvements.
5. Budget is a first-class constraint. Set limits for steps, tokens, and duration, and actively wrap up when approaching the limit.
6. Introduce multi-agent systems gradually. After single-agent effectiveness has been quantified, split roles as needed.

At the infrastructure level, enterprises can use Baidu AI Cloud’s object storage to archive session and evaluation data, use its logging service and monitoring/alerting system to achieve observability, and use container services to carry isolated execution environments, providing a stable, secure, and scalable operational foundation for a Harness.

Returning to the initial question: when models gradually converge, Harnesses can be replicated, and common components have been standardized, what else can a Coding Agent rely on to create differentiation? The answer is not in any single “silver bullet” component, but in the engineering details continuously polished around real R&D scenarios: a better way to organize project context, safer execution boundaries, more quantifiable feedback on effectiveness, and an architecture that can absorb model iterations faster. A few hundred lines of code can build the skeleton, but forging that skeleton into a reliable productivity tool is the true value of Harness engineering.