Multi-Model Integration for AI Office Assistants: Unified Routing and Cost Governance
This article presents a reusable engineering framework for integrating multiple large language models into AI office assistants. It examines four core challenges—inconsistent API protocols, differing capability boundaries, non-uniform cost structures, and varying upstream stability—and describes a unified abstraction layer, quota metering with idempotent ledger design, routing and fallback strategies, evaluation-driven model selection, and security and compliance controls. It also outlines a phased implementation path so teams can expand multi-model capabilities while keeping cost, quality, and risk under control.
1. Background: Office Scenarios Enter a Multi-Model Phase
AI office applications are moving from simple question-answering tools toward agent-like systems that can understand tasks, call tools, and deliver results. These products often connect both in-house models and multiple external models, allowing users to choose based on task difficulty, response speed, and budget. Multi-model access adds flexibility, but it also shifts the complexity of integration, billing, and quality management to engineering teams.
This article does not evaluate specific products. Instead, it focuses on the general technical problem of multi-model integration and presents a reusable engineering framework, including a unified abstraction layer, quota metering, routing and fallback, and quality evaluation.
2. Four Core Problems in Multi-Model Integration
1. Inconsistent API protocols
Different model services vary in request body structure, streaming output format, function-calling fields, and error codes. If business code directly integrates with each service, every new model requires changes in multiple places, and maintenance costs rise quickly.
2. Different capability boundaries
Long-context processing, code generation, Chinese semantic understanding, and multimodal input can differ significantly across models. The same task may produce very different output quality on two models, so routing strategies must be based on actual evaluation data rather than model names or marketing descriptions alone.
3. Non-uniform cost structures
Most models charge by the number of input and output tokens, but unit prices and metering rules differ. Platforms often assign each model a relative multiplier and convert consumption across models into a common metering unit. For the same request, a high-multiplier model may consume several times the quota of a low-multiplier model.
4. Different upstream stability
Upstream services may experience rate limiting, timeouts, or short-term unavailability. The system needs automatic retries, fallback switching, and timeout circuit breaking so that a single model failure does not directly affect user experience.
3. Unified Abstraction Layer: Isolate Differences and Stabilize Upper Layers
The unified abstraction layer is the foundation of multi-model integration. Its goal is to let business code depend only on one internal interface, while all external differences are handled inside adapters.
The design usually includes three parts:
- Standardized requests and responses: Define a unified internal data structure and hide differences in field naming across providers.
- Adapter implementations: Each model has an adapter responsible for protocol conversion, authentication, and error mapping.
- Capability declarations: Each adapter declares the capabilities it supports, such as streaming output, function calling, multimodal input, and maximum context length.
Example code:
from abc import ABC, abstractmethodfrom dataclasses import dataclass@dataclassclass ChatRequest:prompt: strmax_tokens: int = 1024stream: bool = False@dataclassclass ChatResult:text: strinput_tokens: intoutput_tokens: intmodel_id: strclass ModelAdapter(ABC):"""Unified interface for all model adapters."""model_id: strcost_multiplier: float # Quota multiplier relative to the baseline modelsupports_stream: bool = Falsemax_context_tokens: int = 8192@abstractmethoddef chat(self, req: ChatRequest) -> ChatResult:"""Convert an internal request to the external protocol and return a unified result."""raise NotImplementedError
Inside adapters, external errors should be mapped to a limited set of internal exceptions, such as rate limiting, timeout, authentication failure, and invalid parameters. Upper-layer retry and fallback logic depends only on these exception types and does not need to understand provider-specific error codes.
4. Quota Metering: Multiplier Conversion and Ledger Design
4.1 Meaning of the multiplier
Assume the platform uses a baseline model with a multiplier of 1.0. If a model has a multiplier of 2.5, then processing the same number of tokens consumes 2.5 times the quota of the baseline model. The numbers below are only for illustrating the calculation method and are not real pricing for any model.
This design lets users see a unified “quota” without needing to understand each model’s original unit price. The interface should clearly label each model’s multiplier so users understand cost differences when making choices.
4.2 Metering formula
A common way to calculate consumed quota is:
consumed_quota = ceil((input_tokens + output_tokens) * multiplier / unit)
Here, unit is the conversion unit, for example 1 quota point per 1,000 tokens. The rounding method must be explicit in product rules. A common approach is to round up to avoid negative balances or fractional balances.
4.3 Ledger and idempotency
Billing must be traceable and non-duplicative. It is recommended to generate a unique request identifier for each call and use “request identifier + action type” as the unique key for ledger records. When a network timeout causes a client retry, the server uses the unique key to determine whether the request has already been charged, avoiding duplicate deductions.
import mathdef calc_quota(input_tokens: int, output_tokens: int,multiplier: float, unit: int = 1000) -> int:"""Convert the quota consumed by this call using the multiplier, rounded up."""raw = input_tokens + output_tokensreturn math.ceil(raw * multiplier / unit)def charge_once(ledger: dict, request_id: str, quota: int, balance: int) -> int:"""Ensure idempotent charging by request ID and return the balance after charging."""if request_id in ledger: # Already charged; return directly without deducting againreturn balanceif balance < quota:raise ValueError("Insufficient quota")ledger[request_id] = quotareturn balance - quota
In production, the ledger is usually stored in a database that supports transactions and is periodically reconciled with model call logs, ensuring that every quota change can be explained transaction by transaction.
5. Routing and Fallback: Trade-offs Among Quality, Speed, and Cost
Routing determines which model handles a request. Common routing dimensions include task type, user choice, budget constraints, and service health status.
5.1 Three types of routing rules
- Route by task type: Simple rewriting and summarization tasks can go to low-multiplier models; complex reasoning and long-document analysis can go to more capable models.
- Route by user intent: When the user explicitly specifies a model, respect that choice first and show the estimated quota consumption.
- Fallback by health status: When a model’s error rate or latency exceeds a threshold, automatically switch to a backup model and record the fallback event.
5.2 Fallback logic example
from typing import Listdef route_with_fallback(req: ChatRequest,candidates: List[ModelAdapter],is_healthy) -> ChatResult:"""Try models in candidate order and switch to the next on recoverable errors."""last_error = Nonefor adapter in candidates:if not is_healthy(adapter.model_id):continueif len(req.prompt) > adapter.max_context_tokens:continuetry:return adapter.chat(req)except (TimeoutError, ConnectionError) as exc:last_error = exc # Record the error and try the next candidate modelraise RuntimeError("All candidate models are unavailable") from last_error
The order of the candidate list should be determined by the business. A common pattern is “among models that meet capability requirements, sort by multiplier from low to high.” Two points require attention during fallback: the output style of the switched model may differ, so the product layer should provide a notice; fallback counts should be monitored, because persistent fallback often indicates a problem with the primary model or network path.
6. Model Selection and Evaluation: Use Data Instead of Impressions
Model selection should not rely on subjective experience. A reliable evaluation process usually includes the following steps.
- Build an evaluation set: Extract hundreds of representative samples from real business scenarios, covering major task types and edge cases, and have business experts label reference answers or scoring criteria.
- Define metrics: Common metrics include task correctness, format compliance rate, time to first token, overall response time, and unit task cost. Unit task cost equals single-task quota consumption multiplied by the reciprocal of the task completion rate, providing a more intuitive comparison of cost-effectiveness.
- Run offline evaluation: Run all candidate models on the same evaluation set, record each metric, and generate a comparison table.
- Run online canary tests: Select a small percentage of traffic for A/B testing and observe user satisfaction, retry rate, and complaint rate.
- Re-evaluate regularly: Model versions update frequently, so re-evaluation is recommended every quarter or after each major version change.
Evaluation results should be linked to routing rules. For example, if a model clearly leads in format compliance rate, it can be used for tasks that require structured output.
7. Security and Compliance: Additional Constraints in Multi-Model Scenarios
Multi-model integration expands the scope of data flow and requires stricter management measures.
- Data classification: Clearly define which data can be sent to external models and which must only be processed by internal models. Content involving personal information or trade secrets should be redacted before calling external models.
- Permission isolation: Different departments or tenants may have different model scopes, requiring access control at the gateway layer.
- Audit logs: Record the model identifier, time, quota, and redacted request summary for each call to support later tracing. Logs themselves also need protection; plaintext sensitive information should not be stored.
- Compliance filing: When providing generative services to the public, organizations may need to complete filings and security assessments required by applicable regulations. Engineering teams should maintain a model inventory, version records, and content safety policies, and ensure that filing information matches the models actually running in production. This requirement also applies to vertical domains such as embodied intelligence.
8. Implementation Recommendations
Multi-model integration can be advanced in phases in the following order:
- Build the unified adapter layer first, connect only one or two models, and validate whether the abstraction is reasonable.
- Establish the quota ledger and idempotency mechanism, and open user model selection only after billing accuracy is ensured.
- Complete evaluation set construction and use data to determine the default model and fallback order.
- Launch routing and fallback strategies, and configure health checks and alerts.
- Improve audit, redaction, and compliance processes, then gradually expand the number of models.
The core principle is: the business layer depends only on internal interfaces, costs are presented as unified quota, quality is based on evaluation data, and risk is controlled through classification and auditing. Following this framework, multi-model capabilities can continue to expand within a controlled scope.
