Multi-Model Integration and Intelligent Routing for Office AI Agents: Architecture and Design Practice

    Office AI agent products are evolving from integrating a single model to managing a model pool. This article explains why office agents need multiple models, how to structure a decoupled five-layer architecture, and how to implement a unified ChatModel interface with per-model adapters and a structured model catalog. It also covers scoring-based intelligent routing, transparent credit metering with multipliers, multi-vendor failover and quality governance, and security and compliance controls. The article concludes with a four-step rollout plan and common pitfalls. The goal is to turn model selection into a measurable, governable decision rather than an ad hoc engineering choice.

    AI Agent office products are moving from integrating a single model to managing a model pool. Recently, an AI Agent office product announced support for multiple third-party large models. In addition to its own lightweight and enhanced models, the product now lets users select external models on demand, and it labels the credit consumption multiplier for each model.

    From a technical perspective, this capability relies on a multi-model integration and routing governance system. Such a system must address four core problems:

    • Unified invocation
    • Differences in model capabilities
    • Balancing cost and quality
    • Stability and compliance

    This article is organized around these four areas.

    1. Why Office Agents Need Multiple Models

    1. Capability Complementarity

    No single model performs best at every task:

    • Long-document summarization, contract clause review, and multi-turn meeting analysis demand large context windows.
    • Table extraction and structured data generation depend more on instruction following and output format stability.
    • Lightweight tasks such as classification, headline rewriting, and keyword extraction are not very complex, so large models offer little marginal benefit.

    2. Significant Cost Differences

    Large model calls are typically billed separately for input and output tokens. Output tokens usually require sequential, token-by-token generation, so they cost more and carry higher prices. Office workloads are high-volume, with clear peaks and troughs. In this setting, routing simple tasks to low-cost models is a key cost control lever.

    3. Availability and Supply Chain Resilience

    Rate limits, queuing, or outages from a single provider can disrupt the business directly. A multi-model pool lets the platform fail over automatically when the primary model becomes unavailable. Fault isolation then contains the impact of any single failure.

    4. Compliance and Data Boundaries

    Model integration is subject to constraints such as data residency, security assessments, and filing requirements. The architecture therefore needs to restrict the set of available models according to data classification, tenant region, and task type.

    2. Overall Architecture: Five Decoupled Layers

    1. ┌─────────────────────────────────────────────┐
    2. │ Application: docs / sheets / meetings / │
    3. │ search / workflow automation │
    4. ├─────────────────────────────────────────────┤
    5. │ Orchestration & Routing: │
    6. │ intent detection → selection → retry │
    7. ├─────────────────────────────────────────────┤
    8. │ Unified Access: ChatModel interface │
    9. │ + per-model Adapters │
    10. ├─────────────────────────────────────────────┤
    11. │ Governance: model catalog / quota & billing │
    12. │ / observability / security │
    13. ├─────────────────────────────────────────────┤
    14. │ Models: in-house (light / enhanced) │
    15. │ + third-party models │
    16. └─────────────────────────────────────────────┘

    Decoupling delivers three benefits:

    • Business code targets a single unified interface, so models are pluggable.
    • Policies are separated from implementations, so routing rules can change without shipping business code.
    • Governance policies are managed centrally rather than scattered across call sites.

    3. Unified Access Layer: Abstract Interfaces and Capability Metadata

    1. Unified Interface

    Across the industry, conversational model calls generally follow the same pattern: a message array with roles, a system prompt, and generation parameters. Streaming output, tool calling, and structured output are common extensions. A platform can define this pattern as a unified interface and write an adapter for each model’s differences:

    1. from dataclasses import dataclass
    2. from typing import Iterable, Protocol
    3. @dataclass(frozen=True)
    4. class ChatRequest:
    5. messages: list[dict] # [{"role": "system"|"user"|"assistant", "content": ...}]
    6. tools: list[dict] | None = None
    7. response_format: str | None = None # "text" | "json_schema"
    8. max_output_tokens: int = 1024
    9. temperature: float = 0.2
    10. @dataclass
    11. class ChatResult:
    12. text: str
    13. finish_reason: str
    14. input_tokens: int
    15. output_tokens: int
    16. tool_calls: list[dict]
    17. model_id: str
    18. class ChatModel(Protocol):
    19. def chat(self, req: ChatRequest) -> ChatResult: ...
    20. def chat_stream(self, req: ChatRequest) -> Iterable[str]: ...
    21. def build_adapter(profile: "ModelProfile") -> ChatModel:
    22. """Instantiate adapters dynamically from the model catalog;
    23. onboarding a new model only requires registering its config and adapter."""
    24. ...

    Adapters typically need to smooth over these differences:

    • System prompts: Some models accept the system prompt as a separate message, while others take it as a parameter. The adapter handles the conversion.
    • Streaming formats: Text deltas, tool call deltas, and end markers can have different structures. The adapter normalizes them into a unified event stream.
    • Structured output: Reuse the unified JSON Schema constraint wherever possible. For models without native structured output, fall back to prompting plus output validation.
    • Maximum output length: Limits vary by model. The adapter layer enforces consistent parameter capping and truncation.

    2. Capability Metadata (Model Catalog)

    The unified interface only answers how to call a model. Choosing which model to use requires a structured model catalog:

    1. {
    2. "model_id": "agent-pro-v3",
    3. "vendor": "self",
    4. "context_window": 128000,
    5. "max_output_tokens": 8192,
    6. "capabilities": ["tool_use", "structured_output", "long_context"],
    7. "credit_multiplier": 2.0,
    8. "latency_p50_ms": 1200,
    9. "region": ["cn-north"],
    10. "data_level_max": 2,
    11. "version_status": "stable"
    12. }
    • model_id: The stable identifier used for routing, decoupled from underlying model versions.
    • capabilities: Used to filter candidate models.
    • credit_multiplier: The basis for billing (see Section 5).
    • latency_p50_ms: An input to routing scores.
    • region / data_level_max: Security constraint fields (see Section 7).

    3. Unified Accounting and Observability

    Every request is logged with its request ID, model ID, prompt version, token usage, latency, failure reason, and cost estimate, regardless of which model handled it. These records underpin cost accounting, quality attribution, and troubleshooting.

    4. Intelligent Routing: From Rules to Scoring Functions

    1. Routing Stages

    A typical routing flow has four stages: task classification → candidate filtering → scoring and ranking → invocation with fallback.

    2. Task Profiling

    Task features can be extracted with a lightweight classification model or a set of rules:

    • Task type
    • Input length
    • Whether tool calls are needed
    • Whether structured output is needed
    • Latency requirements
    • Data level

    When onboarding a new model, start with rule-based policies to ship quickly. Then iterate on task profiling using production data.

    3. Scoring Model

    Routing decisions can be expressed as a score:

    Score = Q (quality score) − α · C (cost score) − β · L (latency score)

    Here, α reflects the user’s cost preference tier, and L captures deviations from real-time latency.

    The candidate set is filtered from the model catalog:

    1. def choose_model(task: Task, catalog, budget_left: int) -> str:
    2. # 1) Hard constraint filtering: capabilities, context window, data level
    3. candidates = [
    4. m for m in catalog
    5. if m.version_status == "stable"
    6. and task.required_caps <= set(m.capabilities)
    7. and task.input_tokens + task.max_output_tokens <= m.context_window
    8. and m.data_level_max >= task.data_level
    9. and m.credit_multiplier * task.est_tokens <= budget_left
    10. ]
    11. if not candidates:
    12. raise NoAvailableModel(task)
    13. # 2) Quality score + cost constraint
    14. def score(m):
    15. latency_penalty = max(0.0, m.latency / 3000 - 1.0)
    16. return m.quality[task.type] - task.alpha * m.credit_multiplier - 0.2 * latency_penalty
    17. return max(candidates, key=score).model_id

    4. Dynamic Feedback

    Adapters report time to first token (TTFT) and total latency in real time. The routing layer maintains a real-time health score for each model using a sliding window. When latency rises or error rates increase, the router lowers that model’s weight and directs requests to other models.

    5. Model Combinations for Complex Tasks

    An agent task often spans several stages, such as planning, tool calling, code execution, and summarization. Each stage can be routed separately: send planning to a strong reasoning model, use a low-cost model for format conversion, and aggregate all results through the unified interface.

    1. pipeline = [
    2. Step("plan", Task(type="reasoning", required_caps={"tool_use"})),
    3. Step("extract", Task(type="structured", required_caps={"structured_output"})),
    4. Step("summarize", Task(type="chat")),
    5. ]
    6. for step in pipeline:
    7. model_id = router.choose(step.task, catalog, budget_left)
    8. result = models[model_id].chat(step.task.to_request())
    9. step.result = result

    5. Credit Metering and Cost Transparency

    1. Why Multipliers Instead of Absolute Prices

    Models differ in unit price, tokenization, and output speed. Showing users a price per 1,000 tokens is hard to compare across models. It can also lead users to assume that pricier models are always a bad deal.

    A more common approach is to set consumption for one token of the in-house base model as the baseline of 1.0. Other models are then converted into multipliers based on their relative cost. This standardizes metering while giving users an intuitive reference.

    2. Billing Formula and Display

    Cost = (input tokens × input weight + output tokens × output weight) × multiplier

    It is advisable to estimate the cost before a call and ask the user to confirm it. After the call, show a detailed breakdown of actual consumption.

    3. Governance Considerations

    • Pricing per entry point: Each multiplier is tied to a specific call entry point. Promotions affect only what users are charged, not how usage is measured.
    • Multi-level quotas: Set daily and monthly limits per user, budget caps per department, and caps per task. Check quotas before each request and deduct usage in real time while the request runs.
    • Preventing multiplier arbitrage: The same request must not be billed twice. Internal helper calls, such as context compression and multi-turn rewriting, must also be counted and listed separately.
    • Reconciliation and audit: Reconcile platform bills against vendor usage line by line to catch metering discrepancies quickly.

    6. Multi-Vendor Failover and Quality Governance

    1. Fault Isolation

    Each adapter is configured with its own connection pool, timeouts, and concurrency limit. This keeps a single vendor’s rate limiting or outages from affecting other models. Active health checks run periodically and remove unhealthy nodes quickly.

    2. Request-Level Retries and Fallback

    Each request can be configured with an ordered fallback chain: preferred model → backup model → last-resort model. To keep the experience consistent across retries, compress conversation history into a summary before retrying. This avoids the extra cost and latency of repeatedly sending a growing context.

    1. FALLBACK_CHAIN = ["agent-pro-v3", "agent-flash-v2", "agent-lite"]
    2. def chat_with_fallback(req: ChatRequest) -> ChatResult:
    3. summary = summarize_history(req.messages) # Compress context before retry to prevent token inflation
    4. last_err = None
    5. for mid in FALLBACK_CHAIN:
    6. try:
    7. return models[mid].chat(req.with_messages(summary))
    8. except (TimeoutError, RateLimitError, ProviderError) as err:
    9. last_err = err
    10. raise AllModelsFailed(last_err)

    3. Fallback Strategies

    • For parallelizable fan-out tasks, such as multi-document summarization, distribute subtasks across different models to avoid overloading a single model.
    • When a fallback is used, automatically label the response with “served by a backup model” so users know where the result came from.

    4. Quality Evaluation

    Build a regression benchmark that covers core office tasks: document summarization, table extraction, meeting notes, and email polishing. Evaluate models against each other and run regression tests on a regular basis. Keep the quality scores in the routing catalog up to date to catch quality regressions after model upgrades.

    7. Security, Compliance, and Cost Management

    1. Data Classification and Routing Constraints

    Assign data sensitivity levels. Highly sensitive data may only be processed by models that fall within the approved compliance scope. At the request egress point, detect and redact sensitive information such as ID numbers, bank card numbers, and phone numbers.

    2. Compliance Review

    Many jurisdictions regulate generative AI services through filing and algorithm registration requirements. Before integrating an external model, verify its compliance credentials, data usage policies, and data processing locations. Record the findings as constraint fields in the model catalog. Cross-border data transfers require a dedicated data handling process.

    3. Closing the Cost Loop

    • Budget monitoring: Track consumption by department or tenant. Set threshold alerts and automatic circuit breaking.
    • Capacity planning: Use peak and trough curves to forecast call volume and cost for each model. Smooth out demand spikes through routing ratios.
    • Cache reuse: Add semantic or result caching for frequent, similar requests to reduce call volume directly.

    8. Rollout Path and Common Pitfalls

    1. Unified interface: Build the unified interface and model catalog, then integrate one or two models.
    2. Failover readiness: Add backup models and implement the fallback chain and fault isolation.
    3. Intelligent routing: Launch task profiling and scoring-based routing, together with multiplier billing and quota controls.
    4. Granular governance: Introduce an evaluation system, cost dashboards, and compliance policies to complete the governance loop.

    Common Pitfalls

    • Relying only on leaderboards: Public benchmark scores can differ greatly from real office scenarios, so your own evaluation set should be the source of truth.
    • Tight coupling: When business code calls a model’s proprietary parameters directly, switching models requires changing business code.
    • Ignoring output costs: Focusing only on input prices overlooks the actual expense of tasks that generate long outputs.
    • Missing fallbacks: Switching models without compressing the context can increase cost instead of reducing it.

    In essence, multi-model integration turns “which model should we use” from an engineering problem into a measurable, governable decision. A unified interface ensures models can connect. A model catalog makes model selection accurate. Multiplier billing makes costs clear. Failover and compliance ensure resilience. Together, these four elements form a scalable model foundation for office AI agents.