Harness Engineering After Model Convergence: How Execution Frameworks Set the Ceiling for AI Agents

As foundation models gradually approach one another in general capability, the usability of an AI agent is no longer determined only by model parameters or public benchmarks. In public comparisons, the same model can show roughly a 7–8 percentage point difference in task pass rate under different external execution frameworks. After switching to another model, the ranking of accuracy, cost, and speed may also change. This indicates that after model capabilities converge, Harness engineering is becoming a key variable that determines the ceiling of AI agent applications.

Here, Harness is not the model itself, nor is it simply “a better prompt.” It is the complete execution layer between the model and the task. It includes at least system prompts and task prompts, tool definitions and selection, how tool results are returned, the execution loop and stopping conditions, context management, error recovery, permission sandboxing, observability, and evaluation systems. In other words, Harness changes how the model “sees information, can take actions, receives action results, and continues after failure.”

1. Why Gaps Become More Visible After Model Convergence

First, differences in information flow are amplified. As model capabilities become closer, outcomes depend more on whether context is accurate, whether tool results are readable, and whether failure feedback is actionable. A context that is redundant, noisy, and missing evidence references can lead even a strong model to make weak judgments.

Second, differences in action flow are amplified. Tool-call formats, parameter constraints, idempotency, retry strategies, and permission boundaries determine whether a model can reliably complete multi-step tasks. Many failures are not caused by an inability to reason, but by incorrect calls, unavailable data, or an inability to make changes.

Third, errors accumulate. In long tasks, small errors at each step are amplified. Harness reduces accumulated error through checkpoints, validators, rollback, and human takeover, rather than expecting the model to generate a perfect answer in one pass.

Fourth, public self-test data cannot be directly extrapolated. Source-side self-tests usually have specific task distributions, permission configurations, and budget constraints. Repeated trials only show stability under those experimental conditions. Project benefits must be measured again on the team’s own task set, tool set, and cost constraints.

2. Where Harness Matters Most: Long Tasks, Multiple Tools, and Repeated Context

1. Long Tasks: Turning One-Shot Inference into Recoverable Execution

The core of a long task is not single-step correctness, but convergence after many steps. Harness needs to provide subgoal decomposition, state persistence, checkpoints, recovery mechanisms, subtask validators, and context compression. Without state management, the model will lose the goal in a long context. Without validators, errors will propagate all the way to the final result.

2. Multiple Tools: Turning “Callable” into “Composable”

In multi-tool scenarios, tool contracts matter more than tool count. Each tool should clearly define its name, purpose, input and output structures, examples, side effects, idempotency, permissions, and cost hints. The execution layer must also handle tool routing, dependency graphs, serial and parallel choices, conflict resolution, and failure classification. The more tools there are, the more important it is to return “fewer, more accurate” results rather than pushing all raw results back into the model.

3. Repeated Context: Turning “Token Stacking” into “On-Demand Assembly”

Repeated context raises both cost and interference. Harness should perform deduplication, caching, summarization, layered context, and token budget control. A common layered approach is: system policy layer, task goal layer, recent observation layer, long-term memory layer, and tool-result index layer. Each layer is dynamically assembled for the current step rather than concatenated in full every time.

3. Three Ledgers: Accuracy, Cost, and Speed Must Be Accounted for Separately

Accuracy ledger: record at least end-to-end success rate, substep success rate, tool-call correctness rate, error recovery rate, and human takeover rate. Looking only at the final average score hides long-tail failures.

Cost ledger: include input and output tokens, tool-call count, retry count, cache hits, and human review cost. If an accuracy improvement depends on heavy retries and human fallback, it may not be worthwhile.

Speed ledger: include first-token latency, tool wait time, loop rounds, and end-to-end duration. The same engineering approach on different models may be more accurate, cheaper, and slower. Whether that is acceptable depends on the business scenario.

Therefore, Harness optimization should not chase a single metric. It should plot the Pareto frontier: find the combination with the highest accuracy under acceptable cost and latency.

4. Engineering Design: Treat Harness as a State Machine and Contract Layer

A practical Harness usually includes the following modules:

• Tool contract layer: define input and output schemas, permissions, side effects, and idempotency.
• Result normalization layer: structured returns, error classification, evidence references, pagination, truncation, summarization, deduplication, and ranking.
• Context assembly layer: inject system policy, task goals, recent observations, and long-term memory by layer according to budget.
• Execution loop layer: plan–act–observe–reflect, with stopping conditions, maximum rounds, and budget control.
• Validation and recovery layer: subtask validation, failure retry, rollback, and human takeover.
• Observability layer: record each round’s context, actions, results, tokens, latency, and errors; support replay and ablation experiments.

Pseudocode example:

class Harness:
    run(task):
        state = init_state(task)
        while not policy.should_stop(state):
            context = context_builder.build(state)
            action = model.decide(context, tools.schema())
            result = tools.execute(action)
            normalized = result_normalizer.normalize(result)
            state = state_machine.transition(state, action, normalized)
            state = context_compressor.maybe_compress(state)
            budget.consume(action, result)
        return verifier.verify(state)

The point of this pseudocode is not the specific implementation, but the separation of responsibilities: the model is responsible only for decisions, while Harness handles information organization, action execution, result normalization, and state advancement. The clearer the responsibilities, the easier it is to run ablation experiments and locate failures.

5. Model Adaptation Layer: The Same Approach Does Not Mean the Same Results

Different models differ in prompt sensitivity, tool-call format, context window, inference budget, stopping preferences, and error tolerance. The same Harness may perform well on one model but degrade after switching models because prompts are too long, tool descriptions are too detailed, or retry strategies are too aggressive. A model adaptation layer is therefore needed: adjust prompt templates, compress tool descriptions, control result length, set loop limits, and calibrate retry strategies and validation strength.

Adaptation should be done through small-scale A/B tests and ablation experiments. Change only one variable at a time and record accuracy, cost, and speed separately. Public comparison data should be used only as directional reference, not as a project commitment.

6. Rollout Path: From Minimum Closed Loop to Production Guardrails

Step one: select 10–30 representative tasks and define success criteria and budgets. Step two: establish a baseline with a bare model, a single tool, and fixed prompts. Step three: add structured results, validators, context compression, retries, and tool routing one item at a time. Step four: run ablation experiments to confirm the real contribution of each module. Step five: account for accuracy, cost, and speed separately, and plot the Pareto frontier. Step six: before production, add permissions, auditing, logging, monitoring, human takeover, and regression evaluation. Step seven: recalibrate after model upgrades, because the old Harness may no longer be the optimal solution for the new model.

Conclusion

After model convergence, the room for improvement through Harness is mainly concentrated in long tasks, multiple tools, and repeated context. Harness cannot replace model capability, but it can determine whether model capability is reliably converted into application value. Accuracy, cost, and speed must be accounted for separately. Public comparisons are only references, and project benefits must be self-tested. Treating Harness as a state machine, a contract layer, and an evaluation system is the reliable path to raising the ceiling of AI agent applications.