Model-Neutral Agent Frameworks: Multi-Model Integration and Cost Control

    As model providers open up compatible APIs, model replaceability has shifted from a selling point to a baseline capability, pushing Agent competition away from "locking in" a model toward getting the most out of the same one. This article explains why the same model can cost substantially less under one framework than under another, then lays out how to build a model-neutral architecture through unified request/response abstractions, capability declarations with runtime probing, and graceful degradation chains. It covers context budget governance, prefix-cache-friendly orchestration, error normalization and loop protection, cost observability and attribution, and a controlled methodology for benchmarking frameworks fairly. It also addresses security and compliance boundaries and closes with a launch checklist and common anti-patterns. Readers will learn where Agent cost actually comes from and how to make switching models a configuration change rather than a rewrite.

    1. Background: Model Replaceability Is Becoming a Baseline Capability

    For the past two years, the core narrative of Agent products was “end-to-end in-house development”: controlling the model, the tool-calling protocol, and the execution framework. But once major model vendors opened up compatible interfaces, that narrative started to fall apart. Today, any Agent product of meaningful scale already supports several third-party models, broadly covering the main categories of general-purpose LLMs on the market.

    At its core, this is a decoupling of capabilities:

    • The model side handles reasoning, instruction following, and code and tool-calling decisions, supplied via API;
    • The runtime framework side (the harness) handles prompt assembly, tool definition and execution, context management, permission sandboxes, retries and recovery, observability, and billing attribution.

    The direct consequence of this decoupling is that users can call any model from any framework, and model vendors can deliver capabilities to any host. Competition therefore shifts from “binding” to “who uses the same model more intelligently.” Teams that own both models and Agent products still insist on co-designing models and frameworks—not to lock users in, but to make the same model perform better in their own framework: fewer turns, shorter context, and higher task completion rates.

    For developers, this leaves a very concrete technical question: how do you make the framework model-neutral without sacrificing experience and cost in the name of “supporting everything”?

    2. Why the Same Model Costs Differently Across Frameworks

    The most worth-exploring point in the original is the observed result that “a model’s cost after being integrated into a third-party framework can be lower than under its native framework.” This is not an illusion of billing accounting but a reproducible engineering outcome. Let’s make the cost model explicit first.

    The total cost of a task is not as simple as “unit price × token count”:

    1. C_task = Σ_i ( T_in,i · p_in + T_cache,i · p_cache + T_out,i · p_out ) + C_tool + C_infra

    where i is the i-th model call, T_cache is the input tokens that hit the prefix cache, and C_tool is the overhead of external actions such as retrieval, code execution, and browser use. The metrics that actually matter are:

    1. UnitCost = C_task / SuccessRate (cost per successful task)
    2. Latency = P95(end-to-end latency)

    The differences come mainly from six places, all independent of the model and tied only to the framework implementation:

    1. Context engineering. System prompts, tool definitions, and project conventions are repeatedly resent in full; history is only appended, never trimmed; and the same large file is read into context three or four times within a single task. Input tokens are often more than ten times the output, so saving 30% here beats switching models.
    2. Tool return volume and noise. Tools dump raw JSON, full logs, and entire HTML pages back into context, along with heavy indentation and irrelevant fields. Ungoverned tool design is the single largest source of context bloat.
    3. Prefix cache hit rate. If the request prefix isn’t byte-level stable (timestamps, random request IDs, or dynamic retrieval snippets inserted near the front), the prefix cache can’t hit, and input tokens that should be billed at a discount get charged at full price repeatedly. Most services bill cached input tokens at a lower rate; the exact rules depend on each vendor’s documentation.
    4. Retries and failure recovery. Blindly retrying the whole turn, unbounded backoff, giving up early on recoverable errors, or conversely retrying parameter errors repeatedly can all double costs without improving success rates.
    5. Model adaptation quality. Directly applying prompt structures, tool-selection strategies, and reasoning budgets tuned for one model to another forces the model to trial-and-error across multiple turns before completing the task—more turns, linearly rising cost.
    6. Loops and lack of progress. Without a maximum turn count, duplicate-action detection, or a “no new information in two consecutive turns” check, an Agent will spin on the same error and burn its budget on ineffective calls.

    The conclusion: nominal unit price sets the cost floor, and the framework determines how far you are from that floor. A framework that pushes down effective token consumption and raises the cache hit rate can make a more expensive model cheaper on the total bill—and vice versa.

    3. Model-Neutral Architecture: Unified Abstraction and Capability Probing

    Neutrality doesn’t mean “call every model through the same interface.” It means carrying differences through a unified abstraction and expressing differences through capability declarations.

    3.1 Unified Request/Response Model

    Only one normalized message and tool protocol may exist inside the framework; all vendor differences are absorbed at the adapter layer:

    1. from dataclasses import dataclass, field
    2. @dataclass(frozen=True)
    3. class Request:
    4. messages: list[dict] # role: system/user/assistant/tool
    5. tools: list[dict] = field(default_factory=list)
    6. max_output: int = 2048
    7. temperature: float = 0.2
    8. stream: bool = True
    9. # Vendor-specific parameters go in a controlled "extra" dict; business code must not write them directly
    10. extra: dict = field(default_factory=dict)
    11. @dataclass
    12. class Response:
    13. text: str
    14. tool_calls: list[dict]
    15. usage: dict # input/output/cached/reasoning tokens
    16. stop_reason: str # end_turn | tool_use | max_tokens | content_filter
    17. raw_meta: dict

    The key constraint: business logic must not contain vendor branches. Any “if it’s this vendor’s model then…” check is future technical debt that becomes impossible to maintain.

    3.2 Capability Declaration + Startup Probing

    Don’t hardcode “which model supports what.” Give each adapter a capability declaration, and during startup or a canary phase, verify with minimal-cost probes whether the declarations match production reality:

    1. @dataclass(frozen=True)
    2. class Capabilities:
    3. system_role: bool = True
    4. streaming: bool = True
    5. parallel_tool_calls: bool = False
    6. structured_output: str = "prompt_only" # native | schema_constrained | prompt_only
    7. reasoning_budget: bool = False
    8. max_context: int = 128_000
    9. vision: bool = False
    1. PROBES = {
    2. "streaming": lambda a: a.stream_probe_ok(),
    3. "parallel_tool_calls":lambda a: a.tool_probe_ok(parallel=True),
    4. "structured_output": lambda a: a.json_probe_ok()["ok"],
    5. "max_context": lambda a: a.probe_ctx(advertised := 32_000)["ok"],
    6. }
    7. def resolve(adapter) -> Capabilities:
    8. caps = adapter.declared
    9. measured = {k: bool(fn(adapter)) for k, fn in PROBES.items()}
    10. return replace(caps, **measured) # measured results win; declarations are just defaults

    Cache the probe results and record their version; they also serve as the routing basis for the degradation chain.

    3.3 Use a Degradation Chain Instead of “Capability Intersection”

    If you take the intersection of all models’ capabilities, high-end model capabilities get wasted. The right approach is declared capabilities plus stepwise degradation:

    1. Structured output: native schema constraint → schema-constrained mode → prompt constraint + output validation + one repair retry
    2. Parallel tool calls: native parallelism → serial multiple calls (merged into one follow-up request)
    3. Reasoning budget: native parameter → prompt the model to "reason briefly before answering"
    4. Long context: native extended window → retrieval recall + context compression

    3.4 Model Version Is Runtime Configuration

    A model is a replaceable resource, not a build-time constant. Put it in a configuration center that supports routing by scenario, tenant, and canary ratio, and keep both shadow traffic (the new model only runs without taking effect, for comparison) and fallback routing (automatically switching to a backup when the primary model is unavailable). Example:

    1. {
    2. "route": "coding.agent",
    3. "primary": { "adapter": "vendor_a", "model_ref": "chat-large", "weight": 0.7 },
    4. "secondary": { "adapter": "vendor_b", "model_ref": "chat-pro", "weight": 0.3 },
    5. "fallback": { "adapter": "vendor_b", "model_ref": "chat-fast" },
    6. "policy": { "max_turns": 25, "context_budget": 96000, "cache_friendly": true }
    7. }

    4. Context Budget Governance: The First Cost Lever

    Give context a “budget ledger” and manage tokens like you manage memory:

    1. def plan_context(system, tool_defs, history, dynamic, budget):
    2. fixed = count(system) + count(tool_defs) # paid every turn
    3. reserve = budget.output + budget.margin
    4. room = budget.total - fixed - reserve
    5. if room <= 0:
    6. tool_defs = prune_tools(tool_defs, budget=budget.total // 4) # cut the tool set first
    7. keep, dropped = fit_history(history, room - count(dynamic))
    8. summary = summarize(dropped) if dropped else ""
    9. return build(system, tool_defs, summary, keep, dynamic)

    Four high-frequency optimization points:

    • Tool descriptions themselves must be budgeted. The full descriptions of dozens of tools can eat up a considerable amount of input. Build a dynamic tool set grouped by intent: each turn exposes only the subset relevant to the current task, with the rest loaded via retrieval.
    • Tool returns must be compressed. Truncate + summarize + provide a retrievable reference, rather than feeding them back as-is:
    1. def shrink(name, payload, limit=4000):
    2. text = stable_json(payload)
    3. if len(text) <= limit:
    4. return payload
    5. return {
    6. "truncated": True,
    7. "omitted": len(text) - limit,
    8. "preview": text[:limit],
    9. "ref": {"kind": "artifact", "key": digest(payload), "tool": "fetch_artifact"},
    10. }
    • Avoid repeated reads. Register files/pages already read by content hash; on subsequent access, return only a change summary rather than re-reading the full content.
    • Compression should be layered. Sliding window (keep the most recent N turns verbatim) + key-point summary (conclusions of the trimmed parts) + retrieval replay (recall originals on demand when details are needed). Compression strategy and thresholds should be configurable and evaluated jointly on “compression ratio / task success rate,” not on tokens saved alone.

    5. Prefix-Cache-Friendly Orcheation

    Making the prefix cache effective requires no vendor-specific tricks—only ordering discipline:

    1. [stable prefix] identity and global policy → tool definitions (fixed order) → long-term memory/project conventions
    2. [semi-stable] project structure summary, coding conventions, dependency manifest (infrequent changes)
    3. [volatile tail] current time, task description, dynamic retrieval snippets, this turn's user input

    Three hard rules:

    1. Timestamps, random IDs, and request tracing numbers must never appear in the prefix; put them at the tail or send them in response headers;
    2. Keep tool definition order and serialization format fixed (same key order, same whitespace handling), and serialize with deterministic JSON;
    3. Append each turn’s incremental content only to the tail; don’t go back and rewrite history messages.

    Also bring cached_input_tokens into routine monitoring and track hit rates by scenario. A sudden drop in the hit rate usually means a prompt-template change broke prefix stability—the kind of regression that’s hard to catch by human review but immediately exposed by metrics.

    In addition, for latency-insensitive background tasks (batch code review, offline data processing), consider an asynchronous batch channel. These channels usually have a lower unit price at the cost of latency; whether to use them depends on your SLA.

    6. Retries, Loop Protection, and Failure Recovery

    Errors must be normalized first, then a strategy decided:

    1. @dataclass
    2. class NormError(Exception):
    3. kind: str # rate_limited|timeout|server|invalid_request|content_policy|unknown
    4. retryable: bool
    5. retry_after: float | None = None
    6. def call(policy, fn):
    7. for attempt in range(policy.max_attempts):
    8. try:
    9. return fn()
    10. except NormError as e:
    11. if not e.retryable or attempt == policy.max_attempts - 1:
    12. raise
    13. base = e.retry_after or min(policy.cap, policy.base * 2 ** attempt)
    14. sleep(base * uniform(0.8, 1.2)) # exponential backoff + jitter to avoid synchronized retry storms

    Key points:

    • Rate limits/timeouts/server errors → back off and retry; invalid_request and content_policy → don’t retry, feed the readable error back to the model so it self-corrects, which is far cheaper than re-issuing the whole turn;
    • Circuit breaking: when the same model fails consecutively past a threshold, switch to the fallback route and preserve the scene;
    • The loop-protection trio: a maximum turn cap, duplicate-action fingerprint deduplication (abort if the same tool with the same arguments appears consecutively), and no-progress detection (force a summary and converge if the tool result of two consecutive turns is identical to the previous one);
    • Budget guardrails: per-task token and dollar caps plus a timeout cap; on exceeding them, return partial results rather than failing silently.

    7. Cost Observability and Attribution

    Without attribution, there is no optimization. At minimum, log one structured record per task:

    1. {
    2. "task_id": "...", "scene": "code_review", "tenant": "...",
    3. "adapter": "vendor_a", "model_ref": "chat-large", "model_version": "2026-xx",
    4. "turns": 7, "input_tokens": 182_400, "cached_tokens": 151_200,
    5. "output_tokens": 9_800, "tool_calls": 13, "tool_bytes_in": 1_240_000,
    6. "retries": 1, "compressed": true, "success": true, "latency_ms": 41_500,
    7. "cost_est": 0.0
    8. }

    Core metrics:

    Metric Definition Use
    Cost per successful task Total cost / number of successful tasks The only credible cost KPI
    Cache hit rate cached / input Diagnose orchestration and prompt stability
    Token efficiency effective output / total input Diagnose context and tool bloat
    Average turns Model calls to complete a task Diagnose adaptation quality and loops
    Retry rate Retries / total calls Diagnose stability and rate limiting
    Human takeover rate Share of tasks needing human intervention Experience fallback metric

    On the engineering side, thread the trace ID through the entire call chain, use a logging service and monitoring/alerting for a real-time dashboard, forward detailed samples to object storage, and drive offline accounting and reconciliation with a message queue, forming a closed loop of “detect anomalies in real time—locate root causes offline—verify regressions with load tests.” When building such a system on Baidu AI Cloud, general capabilities such as a model access gateway, logging service, monitoring and alerting, object storage, and message queues can cover the entire chain above, without relying on any proprietary implementation.

    8. Selection and Load Testing: How to Reach a Credible “Cheaper” Conclusion

    Any conclusion like “Framework A is cheaper than Framework B” must control variables; otherwise you’re just comparing configurations rather than framework quality. I recommend a two-dimensional cross load test:

    Fixed items: the same real task replay set (sampled from desensitized production traces, covering typical/tail/failure scenarios), the same concurrency and latency requirements, the same accounting and billing method, and the same test time and region.

    Variable matrix:

    • Dimension A: candidate models (2–3)
    • Dimension B: framework configuration (context compression on/off, tool description trimming on/off, cache-friendly ordering on/off, retry strategy aggressive/conservative)

    The report must state clearly: the context length distribution, whether cache discounts are counted, whether the cost of failed tasks is included in the denominator, and latency percentiles (P50/P95). Then look at three numbers together: success rate, cost per successful task, and P95 latency. Optimizing only one usually worsens another—for example, extreme context compression saves tokens but lowers the success rate on multi-file refactoring tasks, raising the cost per successful task instead.

    9. Security and Compliance Boundaries of a Multi-Model Architecture

    Neutrality enlarges the attack surface and amplifies compliance complexity:

    • Data classification and egress control. Different models differ in where they run, their data retention policies, and their training-use policies. Do data classification, field masking, and cross-border policies uniformly at the gateway layer, rather than relying on callers’ discretion;
    • Key and tenant isolation. Manage credentials centrally and issue them as short-lived on demand; never push keys down to the execution sandbox;
    • Prompt-injection protection doesn’t change with the model. Treat tool returns, retrieval snippets, and web content uniformly as untrusted data, define boundaries at the prompt level, and require secondary confirmation on the gateway side for sensitive actions (deletion, external sending, permission changes);
    • Auditability and reproducibility. Record the model identifier and version, prompt-template version, and tool-set version for every call. Otherwise production issues can’t be reproduced and cost anomalies can’t be attributed;
    • Replaceability is itself a disaster-recovery capability. Multiple models plus fallback routing mean a single vendor’s rate limiting or outage won’t take the whole product down—also a practical consideration in enterprise procurement.

    10. Launch Checklist and Common Anti-Patterns

    Self-check before launch:

    1. Are there any vendor condition branches left in business code? (Should be 0.)
    2. Are there capability declarations and startup probes, updated with model versions?
    3. Is there a context budget ledger, with a clear trimming order on overflow (tool set → history → dynamic snippets)?
    4. Do tool returns all go through truncation/summarization/referencing?
    5. Is the prefix ordering stable, and are timestamps and random IDs moved out of the prefix?
    6. Are errors normalized, with retryable and non-retryable cases separated?
    7. Are there a maximum turn count, duplicate-action detection, no-progress detection, and budget guardrails?
    8. Are tokens and costs logged by task_id and drillable by scenario/model?
    9. Does the load-test report give all three of success rate, cost per successful task, and P95 latency?
    10. Is there shadow traffic and one-click rollback?

    High-frequency anti-patterns:

    • Writing prompts to the “lowest common capability,” wasting strong models’ capabilities and having to compensate with more turns;
    • Selecting models on token unit price alone, ignoring the multiplier effect of turns and retries;
    • Piling on more and more tools without governing descriptions, letting input costs run out of control as features grow;
    • Pursuing compression ratio when compressing context without evaluating the change in success rate;
    • Splitting cost and quality metrics: someone watches the bill, but no one attributes the success rate;
    • Reassembling the system prompt on every request with unstable ordering, so the cache can never hit.

    Back to the original judgment: when models become replaceable resources, an Agent product’s moat falls on the runtime framework—context governance, tool design, cache-friendly orchestration, failure recovery, and cost attribution. This work is independent of which model you use, yet it directly determines how much of a given model’s capability you actually realize and how much you truly pay to complete each task. Model neutrality isn’t about flattening everything to the worst experience; it’s about making “switching models” a configuration change, and continuously investing engineering effort in the places that won’t break no matter who you switch to.