Implementing Multi-Model Routing: Architecture Practices for Office Agents
This article explores the architectural practices for implementing multi-model routing in office agents. It covers the necessity of dynamic routing based on task complexity, unified interface design using adapters, intent recognition, context window management, cost-performance balancing, degradation strategies, data compliance, and a launch checklist to ensure system stability and efficiency.
Recently, several collaborative office platforms have begun integrating third-party large language models (LLMs), allowing users to select models based on task complexity, response speed, and token cost. This shift indicates that the underlying architecture of office agents is transitioning from “binding a single model” to “dynamic routing based on tasks.”
Some platforms once integrated two external models but took them offline within days, eventually reverting to different tiers of a single model. This case highlights that the routing layer must support gradual rollouts and quick rollbacks; otherwise, the risks of integrating new models fall directly on the users.
1. Why Office Agents Need Multi-Model Routing
Different models exhibit distinct capabilities. Some excel at complex reasoning, others are suited for processing long texts, and some offer fast response times with lower costs per token. Tasks in office scenarios are highly diverse, including daily Q&A, document summarization, code generation, and data analysis.
Routing all requests to a flagship model will cause token costs to spike rapidly, and queuing will slow down responses. Conversely, routing everything to a lightweight model cannot guarantee the output quality for complex tasks. The goal of multi-model routing is to match each type of task with the most appropriate model.
2. Unified Interface Layer: Shielding Model Differences with Adapters
The API formats of third-party models vary. Developers need to add a unified interface layer between the business layer and the models. Common practices include:
- Defining a unified message structure, such as mandatorily including fields like
role,content, andtool_calls. - Writing an adapter for each model, responsible for request parameter conversion, response parsing, and error code mapping.
- Standardizing the streaming output protocol so the business layer only handles one event format.
With this layer, switching models only requires modifying the routing configuration, leaving the business code untouched.
3. Intent Recognition and Task Distribution
Routing operates in two modes. When a user explicitly specifies a model, the system forwards the request directly. When a user selects “auto,” the system scores the request before deciding which tier to dispatch it to.
1. Complexity Scoring
The scoring consists of four features, each normalized between 0 and 1:
- Task type: Determined by a lightweight classifier, e.g., chat, summary, analysis, code.
- Keyword hits: The number of hits for words like “code,” “reasoning,” “proof,” or “debug,” capped at 3.
- Token count: Counted using the target model’s tokenizer; character count cannot be used as a substitute.
- Tool requirements: Whether retrieval or tool invocation is needed.
The weights are empirical values. Before launch, they should be evaluated offline based on hundreds of manually labeled requests and adjusted accordingly. A score of 0.7 serves as the initial threshold, which can be tuned based on the flagship model’s quality and cost requirements.
from dataclasses import dataclass@dataclassclass RouteFeatures:task_type: str # chat / summary / analysis / codekeyword_hits: int # Number of keyword hitsprompt_tokens: int # Token count measured by the target model's tokenizerneeds_tools: bool # Whether tools or retrieval are neededTASK_WEIGHT = {"chat": 0.0, "summary": 0.2, "analysis": 0.5, "code": 0.6}def complexity_score(f: RouteFeatures) -> float:score = TASK_WEIGHT.get(f.task_type, 0.3)score += min(f.keyword_hits, 3) * 0.1score += 0.2 if f.needs_tools else 0.0score += min(f.prompt_tokens / 8000, 1.0) * 0.1return min(score, 1.0)class ModelRouter:FLAGSHIP_THRESHOLD = 0.7def __init__(self, registry, tokenizer):self.registry = registry # Model registry, providing tier and context window infoself.tokenizer = tokenizer # Tokenizer consistent with the target modeldef route(self, prompt, task_type, keyword_hits, needs_tools,user_preference="auto"):if user_preference != "auto":return self.registry.get_model(user_preference)tokens = self.tokenizer.count(prompt)feats = RouteFeatures(task_type, keyword_hits, tokens, needs_tools)score = complexity_score(feats)# If exceeding the long-context model's window, the caller should truncate or summarize history firstif tokens > self.registry.max_context("long_context"):return self.registry.get_model(tier="long_context")if score >= self.FLAGSHIP_THRESHOLD:return self.registry.get_model(tier="flagship")return self.registry.get_model(tier="lite")
2. Context Window Boundaries
Before routing, you must compare the token count with the model’s window, not the character count. In Chinese, roughly 1 to 2 characters correspond to 1 token. A rough estimate can be obtained by dividing the character count by 1.5, but actual tokenizer measurements must be used before launch.
When the request exceeds the long-context window, the system processes it in the following order: retain the system prompt and the most recent conversation turns, compress earlier messages into a summary, and if it still exceeds the limit, prompt the user to start a new session.
4. Balancing Cost and Performance
1. Cost Calculation Metrics
Cost-benefit analysis must be compared under the same metrics: the same batch of request samples, the same billing cycle, and the same price list. The cost of a single request is calculated as:
Single Cost = Input Token Count × Input Unit Price + Output Token Count × Output Unit Price
Routing benefits equal the baseline bill minus the routed bill. Without traffic distribution data, any fixed savings ratio is unreliable.
Here is a hypothetical scenario. Assume 10,000 requests per month, all routed to the flagship model. If the single cost is 0.02 units, the total is 200 units. After routing, 70% of requests go to the lightweight model at 0.002 units per request, and 30% go to the flagship model. The total is:
10000 × (0.7 × 0.002 + 0.3 × 0.02) = 74 units
This saves approximately 63%. Actual savings depend on task distribution, price differences, and cache hit rates, which need to be verified using shadow traffic or A/B testing.
2. Response Speed
Record the P50 and P95 latencies for each tier separately. If the P95 latency of the lightweight model is significantly lower than that of the flagship model, high-frequency short requests should be prioritized for it.
5. Degradation and Circuit Breaking
Relying on third-party models means availability is affected by external services. The routing layer needs to handle timeouts, rate limiting, and service outages.
- Timeouts: Set first-byte timeout and overall timeout separately. The following values are just starting points and must be adjusted according to the business: first byte 3 seconds, overall chat 30 seconds, long-form generation 120 seconds.
- Retries: Only retry on 429 and 5xx errors, up to 1 time, with backoff starting at 0.5 seconds and adding random jitter. Global retries must not exceed 10% of total requests to prevent amplifying failures.
- Circuit breaking: Open the circuit breaker when there are at least 20 requests in the last 60 seconds and the error rate exceeds 50%. After 30 seconds, enter the half-open state, allowing 3 probe requests; it fully recovers only if all succeed.
- Backup models: Configure a whitelist by tier, prioritizing the same tier. Long-context tasks cannot be downgraded to models with smaller windows.
def call_with_fallback(registry, prompt, primary, backups):for name in [primary, *backups]:try:return registry.call(name, prompt, timeout=30)except (TimeoutError, RateLimitError, ServerError):continueraise ServiceUnavailable("All available models failed")
When switching models, inform the user rather than silently replacing it. For tasks with side effects, such as code execution or data modification, user confirmation should be required before switching.
6. Data Compliance
After integrating third-party models, request content may be transmitted externally or processed by the service provider. You must carefully verify the following:
- The service provider’s data processing agreement, including data storage locations and whether data is used for training.
- Routing based on data classification. Requests involving personal information or trade secrets must only be routed to audited models or privately deployed models.
- Log desensitization. Do not log sensitive information in prompts, such as API keys or ID numbers.
- Audit logs. Record the model used, timestamp, and routing rationale for each call to facilitate post-incident investigation.
7. Launch Checklist
- Gradual rollout: Open gradually at 5%, 20%, and 100%, observing error rates and latency at each stage.
- Cost dashboard: Track token usage and costs by model and task type, comparing against the baseline.
- Context compatibility testing: Switch models during multi-turn conversations to check if history messages and formats function normally.
- Rollback switch: Retain a one-click configuration to revert to a single model, effective immediately if anomalies occur.
The core of multi-model routing is placing model capabilities, costs, and stability into a single set of measurable rules. After the rules go live, you must continuously calibrate the thresholds and weights using real-world data.
