Multi-Model Integration and Cost Control for AI Agent Office Products

    AI Agent products built on large language models are moving from conversational Q&A toward end-to-end task delivery, yet a single model struggles to cover every office scenario well. This article explains how to integrate multiple models behind a unified gateway and keep costs under control. It covers a layered architecture (adaptation, abstraction, routing, and observability), a declarative model-spec design that replaces hardcoded model choices, a three-layer routing strategy with a composite scoring function, a quota and billing system based on consumption multipliers with tiered free usage and off-peak discounts, reliability safeguards such as health checks, circuit breaking, and fallback, and a reference implementation of the plan-route-execute-settle pipeline. It closes with quality assurance, gray-release strategy, and the cloud capabilities that support such a system.

    1. Introduction: Why Office Agents Need “Multi-Model”

    AI Agent products built around large language models are being adopted quickly. A user might ask to “compile this week’s project status report,” “turn a meeting recording into structured minutes,” or “generate charts from sales data and write up the conclusions.” The product then autonomously handles understanding, decomposition, tool invocation, and delivery of the final result. The category has moved beyond conversational Q&A toward end-to-end task delivery.

    As task complexity grows, a single model can rarely cover everything, and some scenarios expose clear weaknesses:

    • Complex reasoning tasks: such as multi-step data analysis or comparing contract terms, require stronger reasoning models, which tend to cost more.
    • Real-time interaction tasks: such as message polishing or quick replies, where users care more about time-to-first-byte and interaction smoothness.
    • Long-document tasks: such as summarizing a hundred-page bid document or extracting information across multiple documents, need a large enough context window and stable information recall.
    • Format-sensitive tasks: such as generating tables, schedules, and code snippets, require the model to strictly follow structured-output constraints.

    If a product binds itself to just one model, it usually has to compromise overall between quality and cost. Either a high-end model handles everything and wastes resources on simple tasks, or a low-cost model handles everything and shortchanges the critical tasks.

    For this reason, bringing in several third-party models alongside an in-house model, forming a model pool that is tiered by capability and dynamically routable, has become the mainstream architectural choice for AI Agent products. One easily overlooked but decisive piece is the user-facing quota and billing system. Helping users clearly understand the relative cost of different models, choose on demand, and plan their usage sensibly directly determines whether the product can balance quality, experience, and commercial sustainability.

    This article discusses the design and implementation of multi-model access for AI Agent office products across five layers: architecture, routing, billing, service guarantees, and engineering implementation.

    2. Overall Architecture: A Layered, Decoupled Model Integration System

    Multi-model integration is far more than simply wiring several model APIs together. Different model services vary significantly in interface protocol, parameter definitions, authentication, streaming response format, and error-code semantics. Calling them directly in business code quickly produces a large amount of branching logic and tightly coupled code. A layered architecture of “integration adaptation → unified abstraction → intelligent routing → observable operations” is recommended, with each layer taking on distinct responsibilities:

    Layer Core Responsibility Key Design Points
    Integration adaptation layer Connect to each model service Protocol conversion, credential management, stream aggregation
    Unified abstraction layer Hide model differences Unified message format, parameter mapping, error normalization
    Intelligent routing layer Select a model for each request Rules combined with scoring, multi-factor decisions
    Observability & operations layer Manage quality, cost, and stability Call-chain tracing, quality evaluation, cost control

    1. Integration adaptation layer. This layer handles protocol adaptation, converting each model’s requests and responses into an internal standard format, while also managing credential custody, key rotation, and rate-limit isolation. Each model’s error-code semantics differ—for example, invalid parameters, content-safety blocks, or rate-limit exceedances—and these need to be normalized into a unified set of error types so that upper layers can make retry and fallback decisions.

    2. Unified abstraction layer. The core here is modeling common capabilities such as “messages, tools, output formats, and streaming events” as internal standard structures, then describing each model’s characteristics through capability declarations: supported context length, whether function calling is supported, whether structured output is supported, the per-request rate ceiling, and so on. The business layer programs only against the unified interface, and model differences are contained within the adapters.

    3. Intelligent routing layer. Based on task characteristics, user configuration, the real-time state of each model (such as remaining rate-limit headroom and average latency), and the cost strategy, this layer selects the most suitable model for each request. The routing layer is where the trade-off among quality, speed, and cost is resolved, and it is discussed separately below.

    4. Observability & operations layer. This layer records the model used for each task, tokens consumed, response time, success rate, and user feedback, forming a “call → evaluate → optimize” loop that provides data support for bringing models online or offline, adjusting routing strategies, and changing billing rules.

    3. Unified Model Abstraction: Capability Declarations Instead of Hardcoding

    The practical form of unified abstraction is a “model specification” that describes each model’s attributes through declarative configuration. An example follows:

    1. from dataclasses import dataclass
    2. from enum import Enum
    3. class ModelTier(Enum):
    4. LIGHT = "light" # light tier: fast response, low cost
    5. STANDARD = "standard" # standard tier: balanced
    6. PRO = "pro" # pro tier: complex reasoning
    7. @dataclass(frozen=True)
    8. class ModelSpec:
    9. id: str # unified model identifier, e.g. "lite", "pro"
    10. tier: ModelTier # capability tier
    11. max_context: int # context window
    12. supports_tools: bool # whether function/tool calling is supported
    13. supports_json_schema: bool # whether structured output is supported
    14. quota_rate: float # quota consumption multiplier
    15. free_quota: bool = False # whether free quota is provided

    The point of introducing a unified model identifier is this: users see capability tiers with clear semantics, business code routes through the unified identifier, and decisions about which specific models are wired in and how they map are made through operations configuration. This lets you adjust the model composition flexibly while avoiding tightly coupling business logic to any specific model.

    At the interface level, the unified gateway exposes a standard message format, tool definitions, and output constraints externally, then performs per-model parameter conversion internally according to each model’s capability declaration:

    1. def adapt_request(spec: ModelSpec, task: TaskSpec) -> dict:
    2. """Convert a standard task description into the parameters a target model API expects."""
    3. params = {
    4. "messages": task.to_messages(),
    5. "temperature": task.temperature,
    6. "max_tokens": task.max_tokens,
    7. }
    8. if task.tools and spec.supports_tools:
    9. params["tools"] = [t.to_schema() for t in task.tools]
    10. if task.response_schema and spec.supports_json_schema:
    11. params["response_format"] = {"type": "json_schema",
    12. "json_schema": task.response_schema}
    13. return params

    4. Intelligent Routing: Sending Each Task to the Right Model

    Once the model pool is in place, the next key question is: which model should actually handle a given request? A common engineering approach is “layered routing + composite scoring.”

    4.1 Three-Layer Routing Decision

    • Layer 1: Task-type triage. Use rules or a lightweight classifier to determine task intent—long-document summarization, code generation, information extraction, conversational polishing, complex analysis, and so on—then match it against models’ capability tags.
    • Layer 2: Complexity grading. Assess the depth of reasoning the task requires, including number of steps, number of dependent documents, and output-structure complexity. When a threshold is exceeded, the task is automatically escalated to a pro-tier model, avoiding the situation where a lightweight model repeatedly fails on a complex task and multiple retries end up costing more.
    • Layer 3: Real-time state and cost constraints. Combine each model’s real-time load and remaining rate-limit headroom, the user’s remaining quota and selected tier, whether it is a discount window, and other conditions to make the final decision.

    4.2 Composite Scoring Model

    Multi-factor trade-offs can be formalized as a scoring function. Compute a composite score for each candidate model and pick the highest. An example follows:

    1. from datetime import datetime
    2. def route(task: TaskSpec, candidates: list[ModelSpec],
    3. quota: UserQuota, now: datetime) -> str:
    4. eligible = [m for m in candidates if fits(m, task)]
    5. if not eligible:
    6. return task.fallback_model # fallback model
    7. def score(m: ModelSpec) -> float:
    8. # 1. Quality score: based on historical eval pass rate (0~1)
    9. quality = eval_stats.pass_rate(m.id, task.type)
    10. # 2. Cost score: the lower the consumption multiplier, the higher the cost score
    11. cost = 1.0 / max(m.quota_rate, 0.1)
    12. # 3. Latency score: the lower the real-time P90 latency, the higher the score
    13. latency = 1.0 / max(metrics.p90_latency(m.id) / 1000, 0.1)
    14. # 4. Discount bonus: raise priority when inside the model's discount window
    15. discount = 1.2 if m.quota_rate < 1.0 and in_discount_window(m.id, now) else 1.0
    16. w_quality = quota.preference_weights["quality"] # user preference weights
    17. w_cost = quota.preference_weights["cost"]
    18. w_speed = quota.preference_weights["speed"]
    19. return discount * (w_quality * quality + w_cost * cost + w_speed * latency)
    20. return max(eligible, key=score).id

    Note that the weights and coefficients above are illustrative only; actual values should be calibrated through offline replay and online A/B experiments.

    4.3 Routing Engineering Details

    • Cost estimability. When a task requires multiple model calls (for example, “plan first, then execute, then proofread”), routing should make decisions based on the total cost of the entire chain rather than a single call in isolation.
    • Context-length constraints. In long-document scenarios, candidate models must satisfy both the context-window requirement and the summarization/chunking strategy.
    • Capability-boundary protection. For scenarios with strict structured-output requirements, prefer models that explicitly support structured-output constraints, and validate outputs on the gateway side; when validation fails, automatically switch to a candidate model and retry.
    • Strategy explainability. Every routing decision should record its reason—which rule was hit, how each candidate scored. This helps both troubleshoot quality issues and explain to users “why this model was chosen.”

    5. Quotas and Billing: Making Cost and Value Transparent

    Multiple models bring cost differences: strong-reasoning models cost more per call, while lightweight models are significantly cheaper. If all models consumed the same quota, high-cost models would be overused and the product would be hard to sustain. The core goal of the billing system is therefore to balance three things: understandable to users, cost-controllable for the business, and predictable in usage.

    5.1 Unified Metering Based on Consumption Multipliers

    In practice, a “quota consumption multiplier” can serve as a relative cost coefficient: take one baseline model’s consumption as 1.0, and convert other models relative to the baseline according to their actual cost. The quota consumed by a task is the product of the following factors:

    • Input token count;
    • Output token count;
    • The multiplier of the model used;
    • Additional overhead such as tool calls or multi-turn loops.

    This way, users do not need to understand the specific price of each model—they only need to watch the multiplier to develop an intuition for relative cost.

    5.2 User-Facing Transparency Design

    • Show the multiplier at selection. The model selection entry clearly displays each model’s multiplier along with helper labels such as “saves quota” or “stronger quality.”
    • Estimate and confirm. For complex tasks that take longer and cost more, show an estimated consumption before execution and let the user confirm.
    • Task-level billing. After a task completes, show actual consumption, the model used, and the routing reason, forming a “choose → execute → review” loop.
    • Quota and usage alerts. Proactively notify users when quota approaches a threshold or when abnormal consumption appears, helping them adjust their usage strategy in time.

    5.3 Tiered Free Usage and Off-Peak Discounts

    • Tiered free usage. Provide free quota for simple, low-cost tasks. This lowers the barrier for new users while filtering out low-value, high-cost calls.
    • Off-peak discounts. During business off-peak hours, lower the multiplier for participating models—effectively using a price signal to guide users toward off-peak usage. This both smooths demand and improves overall utilization of compute resources, and lets users enjoy lower cost on non-urgent tasks.

    Multipliers are operational parameters and should be centrally configured, support hot updates, and be linked with the routing strategy. However, safety constraints must be set between the two to prevent strategy changes from causing cost overruns or sharp drops in service quality. An example follows:

    1. MIN_RATE, MAX_RATE = 0.2, 3.0 # safe multiplier range to prevent misconfiguration
    2. def apply_quota_policy(model: str, base_rate: float, ctx: QuotaContext) -> float:
    3. rate = base_rate
    4. if ctx.in_discount_window(model): # hits a time-window discount
    5. rate *= ctx.discount_factor[model]
    6. # rate must not drop below the safe floor even after discount
    7. return max(rate, MIN_RATE)

    6. Service Guarantees: The Key to Stable Multi-Model Deployment

    After multiple models are introduced, system stability no longer depends on a single model service alone. Fluctuations in any one model affect the overall task success rate. The following safeguards are recommended:

    • Health checks and automatic circuit breaking. Continuously monitor each model’s success rate and latency; on anomalies, automatically reduce weight or temporarily remove it, then gradually ramp traffic back up after recovery.
    • Fallback and retry. When a task fails, switch to backup models in a predefined order; for timeout-type errors, use retries with backoff to avoid a retry storm.
    • Streaming output. In long-text generation scenarios, use streaming responses to improve time-to-first-byte and interaction experience, and to catch anomalies early during generation.
    • Cache reuse. For high-frequency, highly similar requests, build result caches or semantic caches to reduce repeated consumption.
    • Context compression. For long documents, summarize before reasoning, or compress conversation history, to reduce token consumption.
    • Output validation. Validate structured output against format and business rules; on non-compliance, retry or fall back, avoiding “hidden rework caused by format errors.”

    7. Engineering Implementation: Key Actionable Code

    Following the main line of “task planning + model routing + unified execution + quota settlement,” the core flow can be simplified as:

    1. class AgentOffice:
    2. def __init__(self, gateway: ModelGateway, registry: ModelRegistry,
    3. ledger: QuotaLedger):
    4. self.gateway, self.registry, self.ledger = gateway, registry, ledger
    5. def run(self, user: str, goal: str, ctx: TaskContext) -> TaskResult:
    6. ledger = self.ledger.for_user(user)
    7. # 1. Plan: break the goal into executable subtasks (default in-house planner)
    8. plan = self.gateway.chat(
    9. model="default_planner",
    10. messages=[{"role": "user", "content": f"Please break down and plan the steps for the goal: {goal}"}],
    11. )
    12. tasks = parse_plan(plan)
    13. for task in tasks:
    14. # 2. Pre-check: abort early and notify if user quota cannot cover the estimate
    15. estimate = estimate_cost(task)
    16. if ledger.remaining < estimate:
    17. return TaskResult(status="insufficient_quota",
    18. message="Insufficient quota, please adjust and retry")
    19. # 3. Route: pick a model using task features, model capabilities, and real-time state
    20. model = route(task, self.registry.candidates(task.type), ledger, now())
    21. spec = self.registry.get(model)
    22. # 4. Execute: unified gateway call, auto-fallback to a backup model on failure
    23. try:
    24. output = self.gateway.chat(
    25. model=model,
    26. messages=build_messages(task),
    27. tools=task.tools,
    28. stream=task.stream,
    29. )
    30. except ModelError:
    31. output = self.gateway.chat(
    32. model=task.fallback_model, messages=build_messages(task)
    33. )
    34. # 5. Settle: deduct quota by token usage × multiplier
    35. usage = self.gateway.last_usage()
    36. ledger.charge(usage.input_tokens, usage.output_tokens, spec.quota_rate)
    37. # 6. Validate: check structured output against format and business rules
    38. task.result = validate(task, output)
    39. return TaskResult(status="completed", tasks=tasks)

    This example reflects three key design points:

    1. Business code depends only on logical identifiers such as the “default planner model” and “fallback model,” so changes to the backend model composition do not affect business logic;
    2. The quota pre-check runs before task execution, avoiding failures caused by running out of quota halfway through;
    3. Routing and settlement are decoupled, making it easy to iterate on the routing strategy and billing rules independently.

    8. Quality Assurance and Launch Strategy

    The dynamic nature of the model pool demands a more rigorous quality-assurance system:

    1. Build an office-task evaluation set. Cover typical scenarios such as document processing, table analysis, coding assistance, and information extraction, and annotate reference answers or acceptance criteria to serve as a regression baseline for every model-configuration change.
    2. Establish multi-dimensional metrics. Include task completion rate, output adoption rate, average time, average consumption per task, user retention, and so on, to avoid making judgments on a single metric alone.
    3. Gray release and A/B experiments. When bringing a new model online or adjusting the routing strategy, start with a small traffic slice, compare quality and cost metrics, then gradually ramp up.
    4. Feedback loop. Collect user behavior signals such as “accept / modify / regenerate” on results to continuously optimize routing weights and the evaluation set.
    5. Model offline and rollback mechanisms. When a model shows persistent quality or stability issues, be able to quickly switch back to an alternative to ensure business continuity.

    9. Cloud Support Capabilities

    Bringing a multi-model Agent product to life can typically draw on foundational capabilities provided by a cloud platform. Taking Baidu AI Cloud as an example:

    • Model service access: large-model services generally provide a unified interface form, making it convenient to integrate multiple models uniformly;
    • Elastic compute: containers and elastic compute resources can handle traffic fluctuations across different time periods;
    • Data and messaging: object storage can hold unstructured data such as office files, and message queues can make long-running tasks asynchronous;
    • Service guarantees: monitoring, alerting, and logging services are used to track call chains, success rates, and cost changes.

    Combined with the routing and billing design described in this article, this makes it possible to build a multi-model Agent system with controllable cost and a reliable experience.

    10. Conclusion

    Multi-model integration for AI Agent office products is, at its core, a continuous, dynamic trade-off among “task quality, response speed, and usage cost.” The key engineering points can be summarized in five items:

    • Layered architecture: use unified abstraction to hide model differences and decouple business from models;
    • Intelligent routing: triage by task type and complexity, balancing quality, latency, and cost;
    • Transparent billing: quantify relative cost with consumption multipliers, and use tiered free usage and off-peak discounts to guide sensible use;
    • Robust guarantees: ensure stability in a multi-model environment through health checks, circuit breaking and fallback, caching, and validation;
    • Iteration loop: drive ongoing strategy optimization with evaluation sets, gray releases, and user feedback.

    Only by refining all of these areas together can a product deliver high-quality task results across diverse office scenarios while maintaining economical and sustainable resource usage.