Multi-Model Office Platform Architecture: Capability Tiers, Gateway Design, and Intelligent Scheduling

    This article explains how to build a multi-model foundation for enterprise office platforms. It covers capability tiering and model evaluation, a unified API abstraction layer and adapter-based gateway, context and cache management, intelligent routing with degradation and circuit-breaking rules, infrastructure selection, and data compliance and security controls. The goal is to help teams route each request to an appropriate model while keeping latency, cost, quality, and risk under control.

    Multi-Model Office Platform Architecture: Capability Tiers, Gateway Design, and Intelligent Scheduling

    Introduction: The Core Conflict That a Multi-Model Matrix Must Solve

    Once an enterprise office platform integrates large models, it quickly encounters a conflict. Complex reasoning requires a capable but slow heavyweight model, while input autocomplete and email completion require responses within a few hundred milliseconds, and cost must remain controllable. A single model can hardly satisfy both requirements at the same time, so a multi-model matrix has become a common approach.

    The key to a multi-model matrix is to ensure that each call lands on the right model. The number of models itself is not the goal. This article explains how to build this foundation from four perspectives: capability tiering, gateway architecture, routing strategy, and infrastructure selection.

    1. Capability Tiering and Selection Evaluation

    Integrated models can generally be divided into three capability tiers:

    1. Heavy reasoning models (vendor flagship versions such as Max and Pro): These have large parameter scales and are good at multi-step logical reasoning, complex code review, and long-form planning and writing. Their per-call cost and latency are the highest, making them suitable for tasks such as in-depth analysis of financial reports and drafting cross-department business plans.
    2. Ultra-fast response models (vendor Flash and Lite versions): These reduce time to first token through distillation, quantization, and other methods. They are suitable for interactive scenarios such as input autocomplete, email completion, and short-text translation.
    3. Specialized-capability models (long-context models): These can process hundreds of thousands of tokens of input and are suitable for retrieving entire policy manuals and generating comprehensive summaries of dozens of documents.

    After tiering, selection evaluation is still needed. The table below lists recommended comparison dimensions. Each dimension should be tested with your own business data rather than relying only on vendor-published figures.

    Dimension Evaluation Method Considerations
    Time to first token Measure P50 and P95 for typical inputs Use P95 for interactive scenarios
    Context window Test truncation and information recall with real longest documents A nominal window does not equal a usable window
    Inference cost Convert to cost per thousand calls Include input, output, and retry overhead
    Code capability Score with internal code review samples Record pass rate and manual modification effort
    Summarization and analysis Use a fixed sample set and two independent reviewers Scoring criteria must be aligned in advance
    Rate limits and availability Stress-test QPS limits and record 429 and timeout ratios Determines whether a fallback model is needed

    2. Multi-Model Gateway Architecture

    2.1 Unified API Abstraction Layer and Capability Tags

    Business services should not directly specify vendors and model versions. Instead, they should pass capability tags. Tags must be predefined as an enum, and each tag must be configured with a primary model and fallback rules. Tags outside the enum should be rejected. This prevents business teams from passing arbitrary values and making routing results unpredictable.

    Recommended tag examples:

    • quick_summary: fast summarization, with an ultra-fast model as primary and fallback to a Lite version.
    • complex_analysis: complex analysis, with a heavy reasoning model as primary and fallback to a secondary flagship version.
    • long_context: ultra-long document processing, using a long-context model.
    • inline_completion: input autocomplete, with no fallback to a heavy reasoning model to avoid uncontrolled latency.

    2.2 Adapter Pattern

    Each vendor implements an Adapter responsible for converting internal standard requests into that vendor’s protocol and converting responses, error codes, and streaming data back into the internal format. The code below is the core skeleton, including authentication injection, timeout retries, error code mapping, and SSE streaming handling.

    1. import json
    2. import logging
    3. import time
    4. from dataclasses import dataclass
    5. from typing import Iterator, Optional
    6. import httpx
    7. logger = logging.getLogger("ai_gateway")
    8. class ModelError(Exception):
    9. def __init__(self, code: str, message: str, retryable: bool):
    10. super().__init__(message)
    11. self.code = code
    12. self.retryable = retryable
    13. # Map vendor HTTP status codes to internal error codes
    14. ERROR_MAP = {
    15. 400: ("BAD_REQUEST", False),
    16. 401: ("AUTH_FAILED", False),
    17. 429: ("RATE_LIMITED", True),
    18. 500: ("UPSTREAM_ERROR", True),
    19. 503: ("UPSTREAM_UNAVAILABLE", True),
    20. }
    21. @dataclass(frozen=True)
    22. class ModelConfig:
    23. model_id: str # Vendor-side model version
    24. base_url: str
    25. api_key: str # Injected by the secret service at startup; never hard-coded or stored in config files
    26. timeout_s: float = 30.0
    27. max_retries: int = 2
    28. class BaseAdapter:
    29. def __init__(self, cfg: ModelConfig):
    30. self.cfg = cfg
    31. self.client = httpx.Client(
    32. base_url=cfg.base_url,
    33. timeout=cfg.timeout_s,
    34. headers={"Authorization": f"Bearer {cfg.api_key}"},
    35. )
    36. def build_payload(self, prompt: str, **kwargs) -> dict:
    37. raise NotImplementedError
    38. def parse_stream_chunk(self, data: str) -> Optional[str]:
    39. raise NotImplementedError
    40. def _raise_for_status(self, status: int, body: str):
    41. code, retryable = ERROR_MAP.get(status, ("UNKNOWN", False))
    42. raise ModelError(code, body[:200], retryable)
    43. def invoke(self, prompt: str, **kwargs) -> str:
    44. payload = self.build_payload(prompt, **kwargs)
    45. last_error: Optional[ModelError] = None
    46. for attempt in range(self.cfg.max_retries + 1):
    47. try:
    48. resp = self.client.post("/v1/chat", json=payload)
    49. if resp.status_code != 200:
    50. self._raise_for_status(resp.status_code, resp.text)
    51. return resp.json()["output"]
    52. except httpx.TimeoutException as e:
    53. last_error = ModelError("TIMEOUT", str(e), retryable=True)
    54. except ModelError as e:
    55. last_error = e
    56. if not e.retryable:
    57. raise
    58. if attempt < self.cfg.max_retries:
    59. time.sleep(0.5 * (2 ** attempt)) # Exponential backoff
    60. logger.warning("model call failed, code=%s", last_error.code)
    61. raise last_error
    62. def stream(self, prompt: str, **kwargs) -> Iterator[str]:
    63. payload = self.build_payload(prompt, stream=True, **kwargs)
    64. with self.client.stream("POST", "/v1/chat", json=payload) as resp:
    65. if resp.status_code != 200:
    66. body = resp.read().decode("utf-8", "ignore")
    67. self._raise_for_status(resp.status_code, body)
    68. for line in resp.iter_lines():
    69. if not line.startswith("data:"):
    70. continue
    71. data = line[len("data:"):].strip()
    72. if data == "[DONE]":
    73. break
    74. token = self.parse_stream_chunk(data)
    75. if token:
    76. yield token
    77. class FlashAdapter(BaseAdapter):
    78. def build_payload(self, prompt, **kwargs):
    79. return {
    80. "model": self.cfg.model_id,
    81. "input": prompt,
    82. "speed_mode": "fast",
    83. "max_tokens": kwargs.get("max_tokens", 512),
    84. "stream": kwargs.get("stream", False),
    85. }
    86. def parse_stream_chunk(self, data):
    87. return json.loads(data).get("delta")
    88. class MaxAdapter(BaseAdapter):
    89. def build_payload(self, prompt, **kwargs):
    90. return {
    91. "model": self.cfg.model_id,
    92. "query": prompt,
    93. "reasoning_depth": "max",
    94. "max_tokens": kwargs.get("max_tokens", 4096),
    95. "stream": kwargs.get("stream", False),
    96. }
    97. def parse_stream_chunk(self, data):
    98. return json.loads(data).get("delta")
    99. # Capability tag -> (primary model, fallback model); fallback None means no fallback
    100. ROUTE_TABLE = {
    101. "quick_summary": ("flash", "lite"),
    102. "complex_analysis": ("max", "pro"),
    103. "inline_completion": ("flash", None),
    104. }
    105. class ModelRouter:
    106. def __init__(self, adapters: dict):
    107. self.adapters = adapters
    108. def route(self, capability: str, prompt: str, **kwargs) -> str:
    109. if capability not in ROUTE_TABLE:
    110. raise ValueError(f"unsupported capability: {capability}")
    111. primary, fallback = ROUTE_TABLE[capability]
    112. try:
    113. return self.adapters[primary].invoke(prompt, **kwargs)
    114. except ModelError as e:
    115. if fallback is None:
    116. raise
    117. logger.info("fallback %s -> %s, reason=%s", primary, fallback, e.code)
    118. return self.adapters[fallback].invoke(prompt, **kwargs)

    This code is only a core skeleton. The interface paths and field names are placeholders and must be replaced with the actual definitions in vendor documentation. A production environment also needs circuit breakers, rate limiting, data masking, and audit logs. In addition, once streaming output has started being pushed to the user, it cannot be retried. Retries are allowed only before the first token is returned.

    2.3 Context and Cache Management

    In a multi-model architecture, session context must be maintained centrally by the platform rather than determined by the session state of any one model. When a user switches from an ultra-fast model to a heavy model within the same session, the platform must pass historical messages and system instructions to the new model in a unified format and truncate or summarize them when they exceed the target model’s window.

    Semantic caching can reduce the cost of repeated queries, such as high-frequency requests like “help me write a leave request email.” When using caching, four points must be clarified:

    • Hit determination: The similarity threshold should be set based on evaluation results, and model version, system instruction hash, and tenant identifier must also be compared. If any of the three differs, the cache should not be hit.
    • Invalidation strategy: Set a validity period and proactively clear related caches when model versions are upgraded, prompt templates change, or the knowledge base is updated.
    • Tenant isolation: Cache keys must include the tenant identifier. Otherwise, one tenant’s answer may be returned to another tenant, causing cross-tenant data leakage.
    • Privacy control: Requests containing personal information, financial data, or other sensitive content should not be written to cache.

    3. Intelligent Scheduling and Routing Strategy

    The core of routing strategy is to map business scenarios to model tiers and define timeout, degradation, and circuit-breaking rules for each tier. The table below is an example rule set; the values need to be adjusted based on your own stress test results.

    Task Type Entry Condition Default Tier Timeout Degradation Order
    Input autocomplete, email completion Interactive scenario, time-to-first-token target within 300 ms Ultra-fast 2 s Ultra-fast; after timeout, return an empty result without blocking input
    Text correction, short-sentence translation Input no more than 200 characters, intent confidence no lower than 0.9 Ultra-fast 5 s Ultra-fast, then Lite
    Long-document summarization Input exceeds the configured token threshold General-purpose 30 s General-purpose, then segment summarization followed by aggregation
    Complex analysis, code review User selects deep thinking, or intent is multi-step reasoning with confidence no lower than 0.8 Heavy reasoning 60 s Flagship, then secondary flagship, then queue and notify the user
    Ultra-long document retrieval Input exceeds the long-context threshold Long-context 120 s Long-context, then retrieve chunks and pass them to general-purpose

    Routing rules also need to handle the following situations:

    1. Uncertainty in intent recognition: When the confidence of a lightweight classification model is below the threshold, route to the middle tier by default rather than the lowest tier. Dropping directly to the lowest tier can easily reduce the quality of complex tasks, and users may hardly notice.
    2. Cost tiering: The per-call cost of a heavy reasoning model is usually several times that of an ultra-fast model. Casual chat in non-core business should be fixed to a low-cost tier. Heavy reasoning models should be called only when users explicitly choose them or when a task exceeds a preset threshold.
    3. Circuit-breaking conditions: If a model’s error rate in the last 60 seconds exceeds 50% and the sample count is at least 20, break the circuit for 30 seconds. During the circuit-breaking period, requests go directly to the degradation path. After the circuit-breaking period ends, allow a small amount of probe traffic first, and restore full traffic only after probes succeed.
    4. Queueing and notification: Tasks entering a queue need to inform users of the estimated wait time and allow them to cancel. When the queue length exceeds the upper limit, return a failure directly instead of accumulating indefinitely.

    After the routing system goes live, continuously monitor the following metrics: P95 time to first token, P95 end-to-end latency, timeout rate, 429 ratio, degradation trigger count, cost per thousand calls, cache hit rate, and manual sampling-based routing misjudgment rate. When troubleshooting routing issues, check the misjudgment rate and degradation count first, then examine the error code distribution of specific models.

    4. Enterprise Implementation: Infrastructure Selection

    A multi-model gateway requires stable underlying infrastructure, including asynchronous task scheduling, log aggregation, monitoring and alerting, and elastic scaling. Enterprises can choose public cloud, private deployment, or a hybrid approach. When selecting, compare the following indicators:

    • Heterogeneous compute scheduling capability: Whether different GPU or NPU models can be managed uniformly and allocated based on load.
    • SLA and availability zones: Service availability commitments and whether cross-availability-zone deployment is supported.
    • Compliance certifications: Whether industry-specific requirements such as multi-level protection and data localization are met, and whether corresponding audit materials can be provided.
    • Cost metering: Whether bills can be split by tenant, model, and business line to facilitate cost allocation.
    • Observability: Whether logs, metrics, and distributed tracing are provided and can integrate with the existing monitoring system.
    • Migration and lock-in risk: Whether the API is compatible with standard protocols and how much code must change when switching model vendors.

    These indicators should be determined after a trial run with real business load, not only through paper evaluation.

    5. Data Compliance and Security

    Multi-model calls send enterprise data to different external services, so security measures must be enforced at the gateway layer. The following six items are mandatory checks before launch:

    1. Data classification: Classify data into four levels—public, internal, sensitive, and confidential—and define the model scope allowed for each level. Confidential data may be sent only to privately deployed models.
    2. Masking location: Masking is completed at the gateway entry before requests leave the enterprise network. Fields such as phone numbers, ID numbers, and bank card numbers are replaced with placeholders. Model responses also need to be checked to confirm that sensitive information has not been restored.
    3. Tenant isolation: Session context, cache keys, log records, and retrieval scope must all carry the tenant identifier and be validated on every access.
    4. Key management: API keys for each vendor are stored in a key management service, authorized separately by model, and rotated regularly. Keys must not enter code repositories or be written to logs.
    5. Audit logs: Record the caller, tenant, model version, masking actions, and timestamps. Logs themselves need to be stored according to data classification and have access restricted. Logs should not retain sensitive content from the original text.
    6. Sensitive-word interception: Both inbound requests and outbound responses must pass sensitive-word checks. On a hit, reject the request or route it to manual review.

    The measures above can significantly reduce data leakage and compliance risks, but they cannot eliminate them. Security requires continuous operations, including regular penetration testing, permission reviews, and compliance audits.

    Conclusion

    Moving from a single model to a multi-model matrix changes the calling model, not merely the number of models. Unified capability tags free business teams from worrying about vendor differences. Adapters isolate protocol differences in one place. Explicit routing tables, degradation orders, and circuit-breaking rules keep cost and latency within a controllable range. On this foundation, the platform can run low-latency interactive tasks and high-quality deep tasks at the same time and continue to expand capabilities within security and compliance boundaries.