Multi-LLM Routing for Office AI Assistants: How to Select and Route Models
When an office AI assistant connects to multiple general-purpose LLMs at once, the challenge shifts from "which model to pick" to a broader engineering problem. What really shapes the office experience is not how many models are integrated, but whether each task lands on the most suitable model. This article breaks the design of a multi-model architecture into four layers—routing strategy, capability abstraction, cascade scheduling, and cache/budget governance—and turns each into actionable engineering guidance, covering capability cards, rule-, feature-, and policy-based routing, caching, evaluation-driven quality assurance, and security and compliance.
An office AI assistant that connects to multiple general-purpose LLMs at the same time has turned “model selection” from a dilemma into an engineering problem. When a platform offers flagship, lightweight, long-context, and reasoning-oriented models of different tiers all at once, what truly determines the office experience is not “how many models are integrated” but “whether each task is routed to the most appropriate model.” This article organizes the key design points of a multi-model office assistant architecture across four layers—routing strategy, capability abstraction, cascade scheduling, and cache/budget governance—and provides actionable engineering recommendations.
The Technical Background of Multi-Model Setups
The range of tasks an office assistant must cover is expanding quickly:
- Light tasks such as polishing and rewriting, email replies, translation, and title/summary generation are high-frequency, short-answer, and latency-sensitive.
- Long-context tasks such as long-document parsing, contract clause comparison, and multi-turn dialogue are strongly affected by context-window length.
- Hard tasks such as complex reasoning, structured data generation, and coding assistance place heavy demands on a model’s reasoning depth and instruction-following ability.
- Multimodal tasks such as dialect recognition, image understanding, and audio/video transcription require dedicated capabilities.
The industry has broadly observed that no single model is simultaneously best on every dimension, which makes multi-model coexistence an inevitable choice. A mature office assistant typically tags models along four dimensions:
- Capability: general dialogue, long context, reasoning, code, and multimodality.
- Performance: context window, maximum output length, first-token latency, overall throughput, and support for tool calling and structured output.
- Cost: input and output are priced separately, so the cost of a single request is roughly the number of input tokens times the input price, plus the number of output tokens times the output price.
- Stability: service-level agreements, rate limits, and outage history.
Together these dimensions form the basis for selection. One point must be clear: a growing number of models brings scheduling complexity, not product competitiveness. Integrating ten models is only a capability reserve; what actually improves the user experience is a scheduling system that makes the right choice for each task.
Overall Architecture: Four Decoupled Layers
It is recommended to divide the system into four layers:
- Capability access layer: adapts uniformly to the interfaces of different model services and hides protocol differences.
- Model abstraction layer: builds a unified “capability card” for each model that describes its capabilities and constraints.
- Routing and scheduling layer: decides which model to use based on task characteristics and policy, and handles fallback and defaults.
- Evaluation and governance layer: handles offline evaluation, online feedback, cost monitoring, and alerting.
User request│▼Access gateway (auth · quota · audit)│▼Intent recognition / task classification ──────► Model capability cards (capability tags · pricing · context length · availability)│▼Routing decision (rules / policy / budget constraints)│├──► Flagship model (general · long context)├──► Lightweight model (high frequency · low cost)├──► Reasoning model (high difficulty)└──► Multimodal model (image / audio)│▼Unified response normalization ──► Streaming return ──► Cache ──► Telemetry & evaluation
With responsibilities cleanly separated across the layers, adding or retiring a model only requires changing the capability card and routing configuration, with no changes to business code. This is also the prerequisite for platformizing a multi-model system.
Routing Strategies: From Static Rules to Hybrid Scheduling
Routing is the core of a multi-model architecture and can evolve from simple to sophisticated.
1. Rule-based routing
Directly mapping by task type is simple and reliable, making it a good cold-start approach:
def route_by_rule(task_type):return {"rewrite": "chat-standard", # polishing / rewriting"translate": "chat-standard", # translation"summary": "chat-long", # long-document summary"reasoning": "chat-reasoning", # complex reasoning"ocr_vision": "vision-model", # image + text understanding}.get(task_type, "chat-standard")
2. Feature-based routing
Intent recognition is often noisy, so feature checks can be layered on top:
- When the text length exceeds a threshold (for example, 20,000 characters), route directly to a long-context model to avoid truncation.
- When code fences, tables, JSON, or other structured content are detected, prefer a model with strong structured-output capability.
- When a user has repeatedly been dissatisfied with answers to a certain type of task, dynamically adjust the default model for that task.
3. Policy-based routing
When task difficulty is hard to judge in advance, the “cascade” idea can be borrowed (for example, the approach proposed by FrugalGPT): a lightweight model answers first, then a lightweight classifier judges whether the answer is trustworthy; if not, it escalates to the flagship model.
def route_with_cascade(task, policy):m = policy.get(task_type=task.type)if m.kind == "cascade":ans = call_model("chat-lite", task) # a low-cost model answers firstif classify_confidence(ans) < m.threshold: # escalate if confidence is too lowans = call_model("chat-max", task, context=task + [ans])return ansreturn call_model(m.model, task)
This design turns model capability into “configuration” rather than “hard-coded” logic; the business side only needs to care about the task type.
Capability Cards: The Key Abstraction for Unified Model Access
The most easily underestimated part of multi-model integration is the cost of “interface normalization.” Different models differ markedly in context length, tool-calling format, streaming protocol, error codes, and rate-limiting policy. In engineering practice, a capability card is usually maintained for each model and managed through declarative configuration:
models:- id: chat-maxcapability: [general, reasoning, long_context]max_input_tokens: 262144tools: truestream: truepricing: {input: "...", output: "..."} # refer to actual billing- id: chat-litecapability: [general, rewrite, translate]max_input_tokens: 131072tools: falsestream: truepricing: {input: "...", output: "..."}
Based on these cards, a unified gateway does three things:
- Pre-request validation, for example whether tool calling is available and whether the context exceeds the limit; if it does, it summarizes and compresses first.
- Adapter selection at call time according to the capability card, converting the unified request format into the target model’s interface format.
- Post-response normalization, mapping each model’s streaming chunks and error codes into a unified structure so the upper layers can handle them consistently.
Take streaming returns as an example: the “typewriter effect” of an office assistant depends heavily on first-token latency, so the gateway layer must support forwarding chunks as they arrive rather than waiting for the complete response:
def stream_chat(model_id, messages, on_chunk):with client.chat(model_id=model_id, messages=messages, stream=True) as resp:for delta in resp.iter_deltas(): # incremental texton_chunk(delta.content)usage = resp.usage # used for cost and rate-limit accountingreturn usage
In addition, failover must be considered: when the primary model is rate-limited or times out, the system can automatically switch to an equivalent-capability alternative model marked in the capability card, preserving availability in office scenarios.
Cost Governance: Caching, Cascades, and Budgets
Once multi-model support is rolled out, cost is the easiest thing to lose control of. Office scenarios have high request volume and repetitive patterns, so cost can be reduced through the following means:
- Prefix caching: in office scenarios, system prompts, templates, and knowledge-base fragments are often fixed, and most inference services automatically hit the cache for repeated prefixes, so keeping the prompt prefix stable reduces input cost.
- Semantic caching: for high-frequency requests such as “polish this text” or “rewrite this email in a formal tone,” a cache is built keyed on a text fingerprint and a parameter hash, returning directly on a hit.
- Model downgrading: non-critical tasks use lightweight models, such as format cleanup, field extraction, and simple classification.
- Batch processing: tasks that can be batched asynchronously—summarization, translation, and tag generation—go through a batch interface rather than a real-time one; this is usually cheaper per unit and suits offline scenarios that do not require instant feedback.
- Budget constraints: set token quotas per department and per application, automatically switching to a lower-cost model or queuing when the limit is exceeded.
It is worth noting that caching is not free: cache hit rates for content-oriented requests are often low, and a reasonable expiration time must be set to avoid returning stale answers.
Quality Assurance: Evaluation-Driven, Not Gut-Feel Selection
The most common mistake in a multi-model environment is selecting models based on “subjective impression.” It is recommended to build three types of evaluation sets:
- Scenario evaluation set: sampled from real office corpora, covering typical tasks such as meeting minutes, contract review, spreadsheet processing, and email drafting.
- Adversarial evaluation set: including edge cases such as extremely long documents, ambiguous instructions, multi-turn follow-ups, and format constraints.
- Regression evaluation set: run automatically after a model version upgrade or switch to guard against quality regressions.
Evaluation dimensions include task completion rate, factual accuracy, format-compliance rate, average latency, and per-call cost. After launch, results are validated through canary releases and A/B testing, and user “thumbs up / thumbs down” feedback is collected as online signal.
Security and Compliance: Multi-Model Access Should Not Become a Data-Leak Channel
After multi-model integration, data flows become more complex, and the following deserve particular attention:
- Data redaction: identify and redact sensitive fields such as ID numbers, phone numbers, and bank card numbers before upload.
- Network and permission isolation: configure model access permissions by business domain to prevent test models from being called in production.
- Audit trails: record the model identifier, token consumption, latency, and status code of each request for traceability.
- Content safety: add sensitive-word and compliance filtering on the output side; in office scenarios, factual and citation accuracy deserves special care.
Conclusion
Multiple models sharing the stage is an inevitable result of capability upgrades for an office assistant, but an increase in the number of models does not automatically translate into a better experience. For enterprises, the more worthwhile investment is the scheduling system: using capability cards to complete access abstraction, using routing strategies to distribute tasks, using cascades and caching to control cost, and using evaluation sets and canary releases to guarantee quality. Baidu AI Cloud (Qianfan) provides unified access to and invocation of a range of general-purpose LLMs; combined with the methodology above, enterprises can quickly build a manageable and controllable office AI assistant and turn “multi-model” into real business value.
