Integrating Third-Party Models in AI Agent Platforms: Cost Control and Harness Co-Design
When major AI agent platforms start supporting third-party large language models, competition shifts from "which model you connect" to "how well you use it." This article explains how to build a unified abstraction layer for systematic multi-model integration, why the same model produces very different task-completion rates, latency, and costs when run on different execution frameworks (the Harness), and how to control those costs. It covers three responsibilities of the abstraction layer, six key adaptation points, seven common cost amplifiers, the four prerequisites for trustworthy cost benchmarking, and how Harness and model can be co-designed so a model reaches its full potential. The goal is a switchable, observable, evaluable multi-model system that lets engineering quality, not unit price, decide the outcome.
Introduction: Once Both Ends Open Up, Competition Returns to the Harness
The competitive focus of agent products is shifting from “which model to connect” to “how to use the model well.”
Model capabilities are mostly exposed through APIs, and mainstream agent platforms have begun supporting integration with third-party large language models. When both model supply and the execution framework (commonly called the Harness) are open, the competitive dimension moves from “did you integrate a model” to “can you bring out the model’s execution ability.”
Open integration is good for users. A user who prefers one vendor’s model but favors another product’s interaction can directly select the target model within their own execution framework. A user who prefers a particular product but has tasks better suited to other models can also switch models inside the product—model capability is no longer a reason to be locked into one product.
But integrating third-party models is far more than “adding one more vendor.” Interface protocols, tool-calling formats, inference parameters, streaming increments, and error semantics all differ. The deeper question is: the same model, running on different Harnesses, can show noticeably different task-completion rates, response speeds, and actual costs. The reason lies not in the model itself, but in how the Harness organizes context, schedules tools, and handles failures and retries.
At the same time, enterprises cannot evaluate options on unit price alone. A service with the same or even lower unit price can, once placed on a specific Harness, have its actual cost multiplied several times by context bloat, cache invalidation, and repeated retries. So a trustworthy cost comparison is essentially a comparison of Harness engineering efficiency.
This article addresses three questions:
- How an agent platform can systematically integrate third-party large models;
- Where the cost differences for the same model come from, and how to control them;
- How Harness and model can be co-designed to truly unlock the model’s potential.
1. Engineering Architecture of Open Integration: The Unified Abstraction Layer
1.1 Three Responsibilities of the Abstraction Layer
The first principle of multi-model integration is that business logic is unaware of the specific model. Core modules such as task planning, tool orchestration, and context management depend only on the platform’s own unified interfaces, with all model differences encapsulated in an adapter layer. The abstraction layer mainly handles three responsibilities:
- Protocol translation: Normalize the request and response formats of different services into the platform’s internal message model.
- Capability negotiation: Identify which capabilities each model supports and adjust prompts, tool sets, and parameters accordingly.
- Usage accounting: Uniformly collect metrics such as tokens, cache hits, and latency, forming the basis for cost accounting and observability.
1.2 Unified Request and Response Model
You can first define a set of vendor-agnostic internal data structures:
// Platform-internal unified message model (illustrative, vendor-agnostic)interface Message {role: 'system' | 'user' | 'assistant' | 'tool';content: string;toolCalls?: ToolCall[]; // only assistant messages may carry thesetoolCallId?: string; // only tool messages carry this}// Platform-internal unified streaming incrementinterface Delta {text?: string; // body text incrementthinking?: string; // reasoning-content incrementtoolCallFragments?: ToolCall[]; // tool-call fragments, stitched by indexdone?: boolean;}interface Provider {readonly id: string;readonly capabilities: Capabilities;// takes platform-internal messages, returns a unified stream of incrementsstream(messages: Message[], options: RunOptions): AsyncIterable<Delta>;// returns a unified usage record after the call finishesusage(): Usage;}interface Usage {inputTokens: number;outputTokens: number;cachedInputTokens: number; // input tokens hit by the prefix cachereasoningTokens: number; // reasoning-content tokens, billed separately by some models}
On top of this, write a separate Provider implementation for each model, converting internal messages into the request format that model’s service needs, and normalizing the response stream back into Delta. Routing, retries, and billing all depend only on the Provider interface, so adding a model requires no changes to the Harness core logic.
1.3 Capability Negotiation: Don’t Let the Abstraction Layer Drag Down the Model
A common pitfall of abstraction layers is to keep only the “smallest common subset” across all models, letting weaker models drag down the overall experience. A more reasonable approach is the smallest common set plus capability declaration:
interface Capabilities {contextWindow: number; // context windowsupportsTools: boolean; // supports function callingsupportsJsonSchema: boolean; // supports output constrained by JSON SchemasupportsVision: boolean; // supports image inputsupportsReasoning: boolean; // supports reasoning modesupportsPromptCache: boolean; // server provides prefix cachingmaxOutputTokens: number; // max output per call}// The scheduler reads capability declarations before building the request and adjusts strategy dynamicallyfunction prepare(model: Provider, task: Task): RunOptions {const cap = model.capabilities;return {tools: cap.supportsTools ? task.tools : [], // fall back to prompt constraints when unsupportedoutputFormat: cap.supportsJsonSchema ? { jsonSchema: task.schema } : 'text',maxOutputTokens: Math.min(task.maxOutput, cap.maxOutputTokens),thinking: cap.supportsReasoning ? task.thinkingLevel : undefined,images: cap.supportsVision ? task.images : [],};}
The scheduling layer dynamically adjusts strategy based on capability declarations:
- Models that don’t support function calling switch to structured text output, parsed by the Harness itself;
- Models that don’t support strict JSON Schema get an extra parse-validation step after output, and on failure the error message is fed back to the model for correction;
- Models that don’t support image input are not assigned multimodal tasks at the routing stage;
- Parameters such as reasoning effort and max output are uniformly clamped to the range the model supports.
This keeps basic functionality available across all models while letting more capable models deliver their full potential.
1.4 Six Key Adaptation Points
1) Protocol differences in tool calling
- Message structures differ: some models require tool results as separate
tool-role messages matched to requests by call ID; others require tool results to be concatenated into the text of a user message. - Reply constraints differ: some models require
contentto be empty when producing a tool call. - Result formats differ: some require arguments to be strictly JSON; others accept natural-language call descriptions.
The adapter layer should abstract tool calls into a standard structure of { name, arguments, result, callId }, with each Provider responsible for conversion to and from its corresponding protocol.
2) Handling reasoning content
Whether the thinking process returned by reasoning models is kept in history varies by service. Discarding it directly loses key decision-making basis; keeping it as-is continuously inflates input tokens. A safer compromise is for the Harness to distill each round’s reasoning into a sentence or two of conclusions and write that back into history. This preserves the decision chain while controlling context size.
3) Stitching streaming fragments
Body increments are usually easy to handle, but stitching tool-call fragments is error-prone: the function name may be split across multiple segments, and the argument JSON may be scattered across multiple increments.
function mergeToolCallFragments(acc: Map<number, ToolCall>, fragments?: ToolCall[]) {for (const f of fragments ?? []) {const cur = acc.get(f.index) ?? { index: f.index, name: '', arguments: '', callId: '' };cur.callId ||= f.callId; // some services return the call ID only in the first fragmentcur.name += f.name ?? ''; // the function name may be splitcur.arguments += f.arguments ?? ''; // argument JSON is stitched fragment by fragmentacc.set(f.index, cur);}}
In addition, some services return only role in the first fragment with no actual content, and some place the end marker in the last increment. A unified adapter layer needs to smooth over these differences.
4) Semantic differences in sampling parameters
- Parameters like
temperatureandtop_pdiffer notably in value ranges and actual effects across models. - In deep reasoning mode, some models ignore or restrict sampling parameters.
- The meaning of max output tokens differs: some include reasoning content, some don’t.
The platform should keep only one set of semantically clear “levels” internally (such as deterministic, balanced, and creative), with the Provider translating them into each model’s concrete parameters, avoiding passing them straight through from the business side.
5) Unifying error semantics
Errors like rate limiting, timeouts, content-moderation blocks, and context overflow carry different identifiers across vendors, but the Harness needs a unified handling strategy: rate limits should trigger exponential-backoff retries; context overflow should trigger compression or summarization rather than blind retries; moderation blocks are usually futile to retry and should terminate immediately with a notice.
6) Explicit declaration of capability boundaries
For models with smaller context windows, no vision support, or limited output, the scheduler should proactively avoid tasks beyond their capability boundaries rather than letting requests fail and repeatedly degrade. The clearer the capability boundaries, the higher the stability of long task chains.
2. Where Cost Differences Come From: Same Model, Different Harness
2.1 Cost Composition
Under the common token-based billing model, the total cost of a task can be written as:
Cost = Σ ( Q_in(t) · P_in + Q_out(t) · P_out )where:Q_in(t) = F + U(t) + A(t) input tokens for request tF fixed overhead: system prompt + tool definitions + environment notesU(t) history context up to turn tA(t) new user input or tool results this turnP_in, P_out input, output unit price (varies by service)If the server supports prefix caching, the cached portion is priced well below the uncached portion:Q_in(t) = C(t) + (1 - r) · (F + U(t)) + A(t)where:C(t) number of cache-hit tokensr cache hit rate, depending on prefix stability and content reuse
From this we can extract the effective cost metric at the Harness level:
cost per successful task = total cost / number of successfully completed tasks
Tokens consumed by failed tasks are billed too. Whether a Harness is “cheap” is not about the unit price of a single request, but about the total number of tokens spent to complete the same task and the reuse efficiency of those tokens.
2.2 Seven Cost Amplifiers
1) Quadratic growth of rounds and context
An agent usually executes a task in a multi-turn loop: the model decides, calls a tool, reads the result, and decides again. Without compression, the input at turn t contains roughly all content from the previous t−1 turns. Assuming each turn adds tokens on the order of A, the total input is approximately:
Σ (F + t · A) ≈ T · F + A · T · (T + 1) / 2
Doubling the number of turns T more than doubles the cost. So controlling the number of turns and compressing history is the first lever for cost reduction at the Harness level.
2) Fixed overhead resent repeatedly
A long system prompt and the complete definitions of dozens of tools are sent with every request. Prefix caching offered by some services can significantly reduce this overhead, but only on the condition that the prefix is byte-for-byte stable.
Inserting timestamps or random session IDs before the fixed prefix, or dynamically reordering the tool list, repeatedly invalidates the cache. In practice, a good approach is to keep the system prompt and tool definitions in a stable order and place dynamic information at the end of the history messages.
3) Tool return content carrying “noise”
When agents read files, search logs, or query monitoring, tools often return far more than the model needs: entire source files, thousands of log lines, verbose API responses. Once in context, this content is resent in every subsequent turn. A more economical approach is to filter at the tool layer:
- Retrieval tools return only the fragments relevant to the question;
- Structured data is trimmed by field first, then expanded on demand;
- Overly long outputs keep only head/tail summaries and key lines.
4) Premature, too-frequent context compression
Context compression is necessary, but it is not free: it usually requires one model call to generate a summary, and the compressed content may need to be re-retrieved later. Summaries are lossy too—key details like line numbers, error stacks, and constraints are easily lost, causing the model to repeat mistakes and rework.
A more reasonable trigger is “predicted remaining budget is insufficient” rather than “compress only when nearing the window limit.” For example, start compression when the remaining budget cannot hold one tool result and one round of thinking:
function ensureBudget(history: Message[],toolDefs: ToolDef[],reserveForToolResult: number,maxOutput: number,window: number,): Message[] {const overhead = countTokens(systemPrompt) + countTokens(toolDefs);const keep = history;let used = overhead + countTokens(history) + reserveForToolResult + maxOutput;// Compress from the back: summarize earlier history first, keeping the most recent turns verbatimwhile (used > window * 0.9 && hasCompressible(keep)) {const seg = pickCompressibleSegment(keep);const summary = summarize(seg); // Harness-generated structured summaryreplaceWithSummary(keep, seg, summary);used = overhead + countTokens(keep) + reserveForToolResult + maxOutput;}return keep;}
In addition, the summary template should retain structured fields such as “task goal, completed steps, remaining steps, key files and commands, assumptions to verify” to reduce information loss.
5) Retries and failure amplification
Timeouts, rate limits, and tool errors all trigger retries, and each retry often resends the entire history. Retries without a cap or backoff easily multiply costs. Handle by error type:
- Retryable errors: exponential backoff with a cap on attempts;
- Context overflow: compress first, then retry;
- Non-retryable errors (such as moderation blocks): terminate immediately.
6) Reasoning content and output redundancy
Reasoning mode improves success rates in complex stages like planning and debugging, but for tasks such as classification, formatting, and simple rewriting, it is often just extra output overhead. A more reasonable approach: have the scheduler choose reasoning effort by task difficulty instead of using the highest tier across the whole chain.
7) “Implicit context”
Some model services inject extra content into the context at positions invisible to users, such as system-level prefixes or tool-usage instructions. This content is usually billed as input tokens too. For cost comparisons, standardize the counting basis: base it on actual request tokens and billing data, not just estimates of the messages you can see.
2.3 Engineering Cost Governance
Recording unified usage per request and per turn is the basis of cost governance:
function estimateCost(u: Usage, price: PriceTable): number {const fresh = Math.max(0, u.inputTokens - u.cachedInputTokens);return (fresh * price.input +u.cachedInputTokens * price.cachedInput +u.outputTokens * price.output +u.reasoningTokens * price.reasoning);}function logTurn(traceId: string, round: number, u: Usage, price: PriceTable) {report({traceId, round,input: u.inputTokens,cached: u.cachedInputTokens,cacheHitRate: u.inputTokens ? u.cachedInputTokens / u.inputTokens : 0,output: u.outputTokens,reasoning: u.reasoningTokens,cost: estimateCost(u, price),});}
On top of this, cost governance can form a closed loop:
- Attribution by task: link usage to task ID, session ID, and business tags to identify high-consumption scenarios.
- Budget guardrails: set token budgets for individual tasks, sessions, and tenants; downgrade the model or terminate the task when a threshold is reached.
- Model routing: assign models by task difficulty. Use more capable models for complex stages like planning and debugging, lightweight models for simple stages like summarizing, formatting, and classification, and automatically escalate and retry when output quality falls short.
3. Four Prerequisites for Trustworthy Cost Benchmarking
Many cost comparisons are really “doing different things in different ways.” For benchmark results to hold up, at least the following four prerequisites must be met.
1) Same task set
- The task set should cover the real business distribution, including different difficulty levels (simple, medium, complex) and long-horizon multi-turn tasks.
- Each task needs a clear success criterion (for example, whether tests pass or output meets constraints).
2) Same model, same parameters
- Specify an identical model version, reasoning effort, sampling parameters, max output length, and context window configuration.
- If testing one side’s Harness default configuration, state it clearly, to avoid misreading default-configuration differences as Harness capability differences.
3) Same tool set and same prompts
- All approaches should be based on the same tool set and permission scope.
- If prompt templates differ, record them as an independent variable; do not mix them into the Harness comparison conclusion.
4) Same environment and same counting basis
- Fix concurrency, timeouts, and retry strategy.
- Uniformly count all input, output, cache, and reasoning tokens, with actual billing as the final reconciliation basis.
For metric design, focus on these dimensions:
- Cost: cost per successful task, average tokens per task, cache hit rate, average rounds;
- Quality: task success rate, manual-review pass rate, rework rate;
- Efficiency: end-to-end latency, time to first token.
Cost must be interpreted alongside quality. Low cost bought by trimming context and compressing reasoning is not a real advantage if it comes at the expense of success rate. Ultimately, all approaches should be compared on the same “cost–quality” curve.
Note as well that cache hit rate rises with usage frequency, so costs measured early may be higher, and results may change as operation stabilizes. Therefore, a benchmark should span a certain operating period and report the statistical interval.
4. Harness and Model Co-Design
4.1 Why Co-Design Is Needed
After the same model is integrated into different Harnesses, its performance differs markedly because whether the model’s potential is unlocked depends on whether the Harness provides a suitable way of working for it:
- Whether the prompt structure is close to the instruction forms the model saw during training;
- Whether the tool-calling format matches the model’s post-training distribution;
- Whether the model can correctly understand the summary after context compression;
- Whether the reasoning budget matches task difficulty.
None of these can be solved by the model itself; they require co-design between Harness and model.
4.2 Four Co-Design Directions
1) Align prompts with model characteristics
The same system prompt does not suit all models. For long-context models, you can put more reference material directly into the context; for models with limited context windows, retrieval and summarization matter more.
2) Align tool-calling formats
Different models encounter different call formats during post-training and prefer different styles—XML-style, JSON-style, function-call structures. The Harness can maintain different prompt templates and call examples for different models in the adapter layer, reducing parse failures and ineffective calls.
3) Co-design the context compression strategy
The post-compression summary is the model’s only basis for subsequent decisions. The summary should structurally retain the task goal, completed steps, remaining items, and key constraints, so the model can continue executing correctly without tracing back through the original history.
4) Feed runtime data back into model optimization
The Harness naturally holds large amounts of real task traces:
- Which prompt changes improved success rate;
- Which tool calls have high failure rates;
- Which task types need stronger reasoning budgets.
After desensitization and compliance processing, these signals can become evaluation sets and training data for joint optimization of model and Harness, forming a positive “run–evaluate–improve” loop.
4.3 Layered Design in an Open Ecosystem
Co-design is not better the deeper it goes. If a Harness over-customizes for one model, it creates an experience gap for the others. A safer approach is layered design:
- General layer: the abstraction layer and general prompt templates, ensuring all models get a stable, consistent baseline experience;
- Optimization layer: targeted tuning for each model’s characteristics, such as dedicated prompt templates, context compression strategies, and reasoning parameter mapping;
- Evaluation layer: validate each layer’s changes with a unified evaluation set, keeping only optimizations that deliver real improvement.
The ideal end state: when users enter an agent platform, they can clearly feel that the same model performs better in this platform than in other execution environments. This advantage comes from long-term accumulation in context management, tool orchestration, and co-optimization—and is the core competitiveness of the Harness in an open ecosystem.
5. Conclusion
- Integrating third-party models has become a baseline capability for agent platforms;
- Releasing model capability depends on the Harness’s engineering level;
- Cost differences essentially come from how efficiently the Harness manages context, cache, rounds, and retries;
- Co-design determines how far the same model can go.
For developers and enterprises, rather than asking “which model is better, which service is cheaper,” it is better to build a switchable, observable, evaluable multi-model system:
- Isolate model differences with a unified abstraction layer;
- See real costs with token-level accounting;
- Support selection decisions with same-condition benchmarking;
- Continuously unlock model potential through co-design.
When both the model and the execution framework stay open, what truly sets them apart will be the engineering quality of the Harness.
