Multi-Model Integration and Routing: Engineering Practices for an AI Office Platform

    This article presents a practical engineering approach to multi-model access in AI office platforms. It covers a unified abstraction layer, model metadata management, task-aware routing, cost accounting, reliability through fallback chains, and security/compliance controls. It includes architecture guidance and code examples for teams building or operating multi-model AI services.

    1. Introduction: Multi-Model Access Is Foundational Engineering for AI Office Platforms

    Publicly reported updates to mainstream AI office platforms show that, in addition to self-developed lightweight and high-performance models, platforms also integrate multiple third-party models and expose credit consumption multipliers so users can choose based on their needs. AI office tasks are highly diverse—long-document reading, meeting minutes, spreadsheet and presentation assistance, and cross-application operations—so a single model cannot simultaneously optimize capability, cost, and response speed.

    The real challenge is not merely “being able to call multiple models,” but engineering them effectively: hiding interface differences, selecting models by task, measuring and controlling cost, providing smooth fallbacks during failures, and meeting compliance requirements. This article presents practical practices around these areas.

    2. Core Challenges: Four Types of Heterogeneity Increase Architectural Complexity

    The first-principles problem of a multi-model platform is heterogeneity. The following table summarizes four types of heterogeneity and their architectural countermeasures.

    Heterogeneity type Specific differences Architectural countermeasure
    Interface heterogeneity Different message structures, parameter naming, tool-calling conventions, and streaming formats Unified abstraction layer plus adapter normalization
    Capability heterogeneity Different context windows, reasoning depth, and tool-calling stability Capability labels in a model metadata center plus capability-driven routing
    Cost heterogeneity Large differences in input/output token prices requiring normalization Unified credit baseline plus consumption multipliers
    Stability and compliance heterogeneity Differences in throughput, latency, error rates, data residency, and regulatory status Circuit breaking/isolation plus chain-based fallback plus compliance verification

    If business code uses if-else statements to decide which model to call, it quickly turns into a “condition hell.” The correct approach is to contain heterogeneity within a unified abstraction layer and move selection logic upward into a configurable routing layer.

    3. Overall Architecture: Five-Layer Separation

    • Access layer: Provides a unified chat interface supporting synchronous, streaming, and asynchronous calls.
    • Abstraction layer: Implements an adapter for each model provider to normalize provider-specific protocols into internal standard requests and responses.
    • Routing layer: Selects models and fallback chains based on task type, context length, cost budget, and data sensitivity.
    • Execution and governance layer: Handles invocation, retries, circuit breaking, rate limiting, billing, budget control, and end-to-end logging.
    • Model metadata center: Maintains the model catalog as data rather than code, including capabilities, versions, costs, and compliance information.

    The value of layering is that adding a model usually only requires adding one adapter and registering one metadata record, without changing business code.

    4. Unified Abstraction: One Interface to Hide Differences Across Providers

    1. from dataclasses import dataclass, field
    2. from typing import Protocol
    3. from decimal import Decimal
    4. @dataclass
    5. class ChatRequest:
    6. trace_id: str
    7. tenant_id: str
    8. task_type: str # chat / summarize / extract / code
    9. messages: list[dict]
    10. tools: list[dict] = field(default_factory=list)
    11. temperature: float = 0.7
    12. max_tokens: int = 2048
    13. need_json: bool = False
    14. data_level: str = "normal" # normal / sensitive
    15. @dataclass
    16. class Usage:
    17. input_tokens: int
    18. output_tokens: int
    19. cached_tokens: int = 0
    20. @dataclass
    21. class ChatResponse:
    22. content: str
    23. tool_calls: list[dict] = field(default_factory=list)
    24. usage: Usage = None
    25. model_id: str = ""
    26. class ChatProvider(Protocol):
    27. def supports(self, req: ChatRequest) -> bool: ...
    28. def chat(self, req: ChatRequest) -> ChatResponse: ...
    29. def stream(self, req: ChatRequest): ...
    30. def estimate_tokens(self, req: ChatRequest) -> int: ...

    Each adapter is responsible for translating requests and normalizing responses. It must handle three main differences: roles and message structures, tool-calling protocols, and streaming protocols. Streaming responses should be unified into three standard event types: incremental text, tool-call deltas, and completion events.

    Errors must also be normalized. The system should distinguish between “retryable errors” such as rate limiting, timeouts, and temporary failures, and “non-retryable errors” such as invalid parameters, content moderation blocks, and insufficient credits.

    5. Model Metadata Center: Describing Models as Data

    1. {
    2. "model_id": "fast-general-v1",
    3. "provider": "vendor_a",
    4. "version": "2026-xx",
    5. "context_window": 131072,
    6. "capabilities": ["tools", "stream", "long_context", "json"],
    7. "input_price": 0.8,
    8. "output_price": 2.0,
    9. "cached_price": 0.16,
    10. "cache_pricing_rule": "Input tokens that hit the prefix cache are charged at cached_price, typically 10%-25% of input_price",
    11. "credit_multiplier": 1.0,
    12. "latency_budget_ms": 3000,
    13. "compliance": {"filed": true, "region": "cn"},
    14. "fallback_chain": ["cheap-general-v2", "local-rerank-fallback"],
    15. "enabled": true,
    16. "canary_weight": 0
    17. }

    The metadata center has three key responsibilities:

    1. Version pinning: New versions are introduced gradually through canary_weight, and rollback is supported.
    2. Capability-driven routing: Candidate models are filtered based on capabilities and context_window, not by model names.
    3. Unified cost and compliance semantics: Credit multipliers, cached-input pricing rules, and compliance status are maintained in the same metadata source.

    6. Intelligent Routing: From Rule Matching to Multi-Factor Decisions

    Routing has four steps: intent recognition, candidate filtering, candidate scoring, and selecting a primary model plus a fallback chain.

    1. ROUTE_TABLE = [
    2. {"task": "long_doc", "min_ctx": 60000, "prefer": ["pro-v2", "mid-v3"]},
    3. {"task": "code", "caps": ["tools"], "prefer": ["pro-v2", "fast-v1"]},
    4. {"task": "chat", "caps": [], "prefer": ["fast-v1", "lite-v1"]},
    5. ]
    6. def route(req: ChatRequest, registry: dict) -> tuple[str, list[str]]:
    7. rule = next((r for r in ROUTE_TABLE if r["task"] == req.task_type), None)
    8. est_ctx = sum(len(m["content"]) for m in req.messages) // 2
    9. candidates = []
    10. for mid in (rule["prefer"] if rule else []):
    11. m = registry[mid]
    12. if not m["enabled"]:
    13. continue
    14. if rule and "min_ctx" in rule and m["context_window"] < max(rule["min_ctx"], est_ctx):
    15. continue
    16. if rule and "caps" in rule and not set(rule["caps"]).issubset(m["capabilities"]):
    17. continue
    18. if req.need_json and "json" not in m["capabilities"]:
    19. continue
    20. if req.data_level == "sensitive" and not m["compliance"]["filed"]:
    21. continue
    22. candidates.append(mid)
    23. return (candidates[0], candidates[1:]) if candidates else ("", [])

    Multi-factor scoring can be expressed as:

    score(m, req) = w_q·Q(m,task) - w_c·Cost(m,req) - w_l·Lat(m)

    Here, Q comes from offline evaluation scores, Cost is the expected token usage multiplied by unit price, and Lat is the measured latency percentile. Weights depend on business goals. For example, consumer-facing products may prioritize speed, while enterprise batch processing may prioritize cost.

    Routing checklist: Each routing decision should verify task type, context length, cost ceiling, data level, tool capabilities, canary weights, and whether the fallback chain is complete. Routing rules should remain configurable, gradually releasable, and auditable. Logs should record the candidate set, selected model, decision reason, and actual model used.

    7. Cost Accounting: Making Consumption Multipliers Understandable and Controllable

    Define a unified credit baseline. The baseline model has a multiplier of 1.0, and other models are priced upward based on relative cost. User credit consumption is approximately:

    token count × multiplier

    The accounting principles are:

    • Internal consistency: Credits are derived from actual costs and reviewed periodically.
    • Dimension alignment: Input and output tokens are separated, and multimodal conversion rules are explicit.
    • Estimation transparency: Provide an estimate range before invocation and show real-time usage during streaming.

    First, define an independent token estimation function. In production, it is recommended to replace it with a real tokenizer.

    1. def estimate_prompt_tokens(req: ChatRequest) -> int:
    2. text = "".join(m.get("content", "") for m in req.messages)
    3. if req.tools:
    4. text += "".join(json.dumps(t, ensure_ascii=False) for t in req.tools)
    5. return max(1, len(text) // 2)
    6. def estimate_credits(model: dict, req: ChatRequest) -> Decimal:
    7. in_tok = estimate_prompt_tokens(req)
    8. out_tok = req.max_tokens
    9. in_price = Decimal(str(model["input_price"]))
    10. out_price = Decimal(str(model["output_price"]))
    11. mult = Decimal(str(model["credit_multiplier"]))
    12. return (in_tok * in_price + out_tok * out_price) * mult
    13. def apply_billing(model: dict, resp: ChatResponse) -> Decimal:
    14. # Convert to Decimal before arithmetic to avoid errors from mixing float/int with Decimal
    15. in_price = Decimal(str(model["input_price"]))
    16. out_price = Decimal(str(model["output_price"]))
    17. cached_price = Decimal(str(model["cached_price"]))
    18. mult = Decimal(str(model["credit_multiplier"]))
    19. uncached_in = resp.usage.input_tokens - resp.usage.cached_tokens
    20. cost = (uncached_in * in_price
    21. + resp.usage.cached_tokens * cached_price
    22. + resp.usage.output_tokens * out_price)
    23. return cost * mult

    At the governance level, set daily or monthly quotas and per-user circuit-breaking thresholds by tenant, user, and feature. When the budget is insufficient, the system should automatically switch to a lower-multiplier model or queue the request, and clearly notify the user rather than failing directly. Results for high-frequency similar requests can be cached. When a model is retired or its price changes, the platform can update the default model and multiplier through the metadata center without releasing a new version.

    8. Reliability: Fallback Chains and Fault Tolerance

    The following types are provided to support the execution logic. They are illustrative definitions rather than a complete runnable module.

    1. @dataclass
    2. class RoutePlan:
    3. primary: str # Primary model ID
    4. fallbacks: list[str] # Fallback chain
    5. @dataclass
    6. class ChatResult:
    7. response: ChatResponse | None = None
    8. fallback_answer: str | None = None
    9. # provider_registry: dict[str, ChatProvider] — model_id -> adapter instance
    10. # billing: billing service, charge(req, model, resp) is only called after successful invocation
    11. # emit_event(name, **kwargs): sends structured events to the observability/audit pipeline

    The execution layer supports retries, circuit breaking, and fallback:

    • Tiered retries: Retry only retryable errors, using exponential backoff with jitter.
    • Circuit breaking and isolation: Each model has an independent circuit breaker and semaphore.
    • Chain-based fallback: Try models in sequence according to the fallback chain.
    1. def invoke(req: ChatRequest, plan: RoutePlan) -> ChatResult:
    2. chain = [plan.primary] + plan.fallbacks
    3. for i, model in enumerate(chain):
    4. try:
    5. resp = provider_registry[model].chat(req) # Adapter includes tiered retries
    6. billing.charge(req, model, resp) # Charge according to the actual model used
    7. emit_event("model_used", trace_id=req.trace_id, model=model, rank=i)
    8. return ChatResult(response=resp)
    9. except NonRetryableError:
    10. break # Parameter errors, content blocks, etc. should not be bypassed by switching models
    11. except Exception as e:
    12. if i == len(chain) - 1:
    13. return ChatResult(fallback_answer=fallback_answer(req, error=e))
    14. return ChatResult(fallback_answer=fallback_answer(req))

    Key points:

    • Failures must not cause duplicate charges. Billing happens only after success, and interrupted requests should be reconciled using idempotency keys.
    • Fallback should be transparent to users. The interface should indicate the actual model used.
    • Fallback must not bypass security. Content moderation is non-retryable, and switching models should still be subject to the same controls.
    • Streaming fallback should be smooth. Previously generated content should be preserved, and an explanatory message can be appended.
    • Tool calls must be idempotent. Retries should carry the same idempotency key to prevent duplicate side effects.

    9. Security and Compliance: More Models Do Not Have to Mean More Risk Exposure

    Model-agnostic moderation on the platform side: The platform should detect prompt injection, sensitive information leakage, and prohibited content at the input stage, and harmful content or leakage at the output stage. Moderation should be implemented consistently at the platform level rather than relying on the built-in safety capabilities of any single model. All connected models should pass the same set of policies.

    Data classification and residency verification: Prompts and uploaded files should be classified by sensitivity. Highly sensitive data should only be processed in owned or compliant environments. Before routing, the system should verify the candidate model’s data residency region (compliance.region) and regulatory status (filed). Non-matching models should be filtered out.

    Third-party compliance verification and recordkeeping: For public-facing generative AI services, applicable laws and regulations may require security assessments, algorithm registration, or filing. Platforms should verify the compliance qualifications of connected models and retain records, including qualification documents, verification time, verifier, and validity period. When new forms such as embodied intelligence are involved, additional safety and responsibility requirements arise from training data, control interfaces, and physical execution. These should be evaluated during model selection.

    Security policies cannot be bypassed through fallback: Content moderation blocks and restrictions on sending sensitive data outside approved environments are hard constraints. They must remain effective no matter which model in the fallback chain is selected. Switching models must not be used as a way to bypass security policies.

    Auditability and explainability: Each invocation should retain the call chain, model version, routing rationale, and billing details. If a content dispute arises, it should be possible to quickly identify which model and which version generated the output.

    10. Evaluation and Continuous Operations

    • Offline evaluation: Build standard evaluation sets for each task type, and evaluate models and prompt versions separately.
    • Online metrics: Define metrics explicitly.
      • First-token latency: Time from request dispatch to the first renderable token. Track P50/P95 using a 5-minute sliding window.
      • Error rate: Failed calls divided by total calls, measured by model.
      • Fallback rate: Number of times the fallback path is entered divided by total calls.
      • Adoption rate: Outputs copied, edited, or accepted as final documents divided by total outputs.
      • Recommended alert thresholds: Error rate above 5% for 3 minutes, fallback rate above 10%, or first-token latency P95 exceeding twice the budget.
    • Feedback loop: Treat user adoption and edits as implicit feedback. Use periodic regression evaluation to adjust routing weights and default models.

    Prompt normalization is often an overlooked lever. Maintain separate configurations for “task semantic prompts” for each task type and “rendering templates” for each model. When switching models, the corresponding template can be applied automatically.

    11. Cloud Deployment: Reusing General-Purpose Cloud Capabilities

    Mainstream cloud vendors usually provide several general-purpose capabilities. This article uses Baidu AI Cloud as an example:

    • Container platform: Deploy gateways and adapters, and scale elastically to handle peak traffic.
    • Object storage and logging services: Store call records, evaluation samples, and audit logs.
    • Monitoring and alerting: Monitor latency, error rate, and credit consumption.
    • Message queues: Asynchronously decouple time-consuming tasks such as long-document processing.

    Self-developed model services should support standardized invocation protocols and version management so they can be integrated into a unified metadata center.

    12. Conclusion

    The abstraction layer solves “how to connect models.” The metadata center solves “how to manage them.” The routing layer solves “how to choose the right model.” Metering and budgeting solve “how to account for cost.” Fallback and security solve “how to remain resilient and compliant.” All five are necessary.

    As models evolve from text toward multimodal capabilities and eventually embodied intelligence, this architecture—centered on abstraction, routing, and governance—remains applicable. What needs to be extended is mainly the capability-label system and the compliance evaluation dimensions.