Multi-Model Integration and Unified Routing for Office Workbenches
This article examines the engineering approach to moving an office collaboration workbench from a single built-in model to multi-model supply. It covers a unified access layer, model registry and capability model, policy-based routing, evaluation and canary release, security auditing, and cost governance. Readers will learn how to normalize requests and responses, design routing trade-offs, build regression evaluation, and avoid common pitfalls in multi-model adoption.
1. Background: From a Single Model to Multi-Model Supply
Office collaboration platforms are moving from a single built-in model to multi-model supply. As a workbench adds multiple external models, users can choose different capabilities for scenarios such as document drafting, meeting minutes, spreadsheet analysis, coding assistance, and knowledge Q&A. On the surface, this is an expansion of the model list. From an engineering perspective, it means the system must handle authentication, billing, rate limiting, context-length differences, output-format differences, content safety policies, and tenant isolation at the same time.
If different upstream APIs are simply forwarded one by one to the frontend, the system can go live in the short term, but over time it will create hard-to-govern model silos: each model has its own set of keys, error codes, and billing rules; evaluation data cannot be compared across models; failures cannot be quickly degraded; and audits cannot answer which tenant called which type of model in which scenario and how much cost was consumed. Therefore, the core of multi-model integration is not adding as many models as possible, but establishing a unified model access layer and an operable routing governance system.
2. Overall Architecture: A Unified Access Layer as the Middle Tier
A recommended design divides the system into four layers: a unified access layer, a model registry and capability layer, a routing and policy layer, and an operations governance layer. The unified access layer provides business applications with a consistent invocation protocol, error codes, streaming responses, and structured output conventions. The model registry and capability layer maintains each model’s context length, structured output support, tool-calling support, multimodal capabilities, language support, rate-limit rules, and billing granularity. The routing and policy layer selects models based on scenario, tenant, data classification, budget, and health. The operations governance layer is responsible for evaluation, canary release, auditing, monitoring, and cost analysis.
For deployment, the access layer should be stateless to support horizontal scaling. Keys should be managed by a unified key management capability and must not enter business code. Invocation logs, evaluation sets, and audit records can be stored in object storage and log services, with routing health observed through monitoring and alerting. For cloud deployment, Baidu AI Cloud can provide general cloud service categories such as object storage, log services, and monitoring and alerting, but specific selection should be evaluated against enterprise compliance, network, and cost requirements. The access layer should not be bound to a particular type of infrastructure.
3. Unified Abstraction: Normalize Before Routing
The first engineering action in multi-model integration is to define a unified abstraction. The business side submits only common fields such as scenario, messages, expected output structure, data classification, and budget cap, without directly perceiving upstream differences. The access layer converts the unified request into prompts and tool descriptions acceptable to each model, and then normalizes responses into a unified structure. Particular attention should be paid to streaming responses, timeouts, retries, cancellation, and idempotency so that the frontend does not need separate logic for each model.
The unified abstraction should also include unified error codes, such as invalid parameters, rate limiting, timeout, content safety block, upstream unavailable, and insufficient budget. Only after error codes are normalized can the routing layer perform consistent degradation. Prompt adaptation should not be overlooked either: different models vary in their sensitivity to system prompts, tool-calling descriptions, and structured output constraints. The access layer should maintain versioned adaptation templates and write the template version into invocation logs to support issue tracing.
# Pseudocode: unified invocation entry pointfunction handle_request(tenant_id, scene, messages, policy):request = normalize(messages)candidates = registry.list(scene=scene, tenant=tenant_id)candidates = filter_by_policy(candidates, policy) # data classification, region, budgetmodel = router.choose(candidates, request, policy)if model is None:return fallback_response("The current scenario is temporarily unavailable")try:raw = model.invoke(adapt(request, model.capabilities))result = normalize_response(raw, model.capabilities)audit_log.write(tenant_id, scene, model.id, policy, result.usage)metrics.record(scene, model.id, result.latency, result.usage)return resultexcept Timeout:router.mark_unhealthy(model)return retry_or_degrade(candidates, request)
4. Routing Strategy: Balancing Quality, Cost, Latency, and Compliance
Routing should not be hardcoded simply by model name; it should be based on capability registration and evaluation results. Common strategies include quality-first, cost-first, latency-first, compliance-first, and hybrid scoring. Hybrid scoring can weight quality score, unit cost, P95 latency, availability, and data-classification match, with circuit breaking and a degradation chain. For highly sensitive data, candidate models should first be filtered by data classification, and performance and cost optimization should then be performed on the remaining set.
Routing also needs to distinguish between session-level and request-level routing. Session-level routing keeps the model stable within the same session to reduce style shifts in context. Request-level routing can dynamically select models by task type; for example, long-document summarization may prioritize large-context capability, while structured extraction may prioritize stable structured output. During canary release, traffic can be split by tenant, scenario, or traffic ratio, and the reason for each routing hit should be logged to avoid black-box decisions.
5. Evaluation and Canary Releases: No Regression Set, No Routing
Multi-model integration must establish an evaluation system. Offline evaluation covers accuracy, structured output success rate, tool-calling correctness, long-context stability, content safety block rate, and average consumption. Online evaluation focuses on user adoption rate, retry rate, manual switch rate, latency, and cost. Evaluation sets should be stratified by scenario, with a failure case library retained. Regression tests should be run whenever a new model is added or a prompt template is adjusted.
For canary release, a recommended cadence is small-traffic validation, scenario expansion, and full switch. Open low-risk scenarios first and then gradually cover core business. At the same time, set automatic rollback conditions, such as error rate, timeout rate, or cost exceeding a budget threshold. During canary release, compare metric differences between the old and new routing rather than only looking at average scores, so that long-tail scenarios are not hidden by averages.
6. Security, Auditing, and Cost Governance
For security and compliance, the access layer should implement tenant isolation, key custody, data masking, content safety checks, and audit logs. Audit logs should record at least the tenant, scenario, model identifier, request time, consumption, routing policy version, and block reason, with a retention policy. For cross-region or sensitive-data scenarios, data residency and classification filtering should be completed before routing, not remediated after invocation.
Cost governance requires budget quotas, usage dashboards by tenant/project/scenario, maximum output limits, caching strategies, and degradation strategies. Caching must distinguish between cacheable and non-cacheable content; sensitive data should not enter a shared cache. Cost optimization should not come at the expense of quality. It is advisable to use “cost per successful task” as a metric rather than looking only at the unit price per call.
7. Rollout Path and Common Pitfalls
Rollout can be divided into four steps. First, unify the access abstraction and error codes to eliminate multiple sets of invocation logic on the business side. Second, establish a model registry and basic routing to support manual switching and circuit-breaker degradation. Third, complete evaluation, canary release, and auditing to form a rollback-capable release process. Fourth, establish a closed loop for cost and quality operations and continuously optimize routing weights.
Common pitfalls include: only forwarding APIs without normalizing responses; routing based only on price without considering task success rate; frequently switching models without regression evaluation; scattering keys across business services; ignoring differences in context length and streaming timeouts; and missing audit logs that make issue tracing impossible. The long-term value of multi-model integration lies in turning model supply into governable service capability, rather than stacking up a list of models.
