Multi-Model Routing for Office Agents: Balancing Capability and Cost

    As large language models evolve rapidly, relying on a single model for office agents makes it difficult to balance performance, cost, and response speed. This article explores practical strategies for implementing multi-model routing, covering task-model matching, unified gateway design, intelligent routing rules, cost management systems, and enterprise deployment practices to help developers build robust and cost-effective AI office products.

    Introduction: From Single Model to Multi-Model Collaboration

    Over the past year, large language models have updated rapidly, with cloud vendors and research institutions releasing models with larger parameters and stronger reasoning capabilities. For office agents, choosing a single model has become increasingly difficult. Strong models incur high invocation costs, while lightweight models often exhibit unstable quality when handling complex tasks. Relying on just one model makes it hard to simultaneously balance effectiveness, cost, and response speed.

    Therefore, multi-model access and intelligent routing are becoming common practices in AI office products. Instead of binding to a single model, the system selects the appropriate model based on task characteristics and switches when necessary. This article outlines a set of actionable practices in the order of “how to connect models, how to allocate tasks, how to manage costs, and how to ensure stability.”

    1. Task and Model Matching in a Multi-Model Environment

    1.1 Model Capability Tiers

    In practice, models can be divided into three tiers based on capability and price:

    • Flagship Tier: Offers the best performance in complex reasoning, long-document understanding, code generation, and multi-step planning, but incurs the highest cost per call.
    • Balanced Tier: Suitable for most office writing, summarization, and translation tasks, offering a good balance between quality and cost.
    • Lightweight Tier: Features low latency and low cost, ideal for structured tasks like classification, tagging, keyword extraction, and format conversion.

    Tier division should be determined based on your own evaluation set, rather than solely relying on vendor marketing parameters.

    1.2 Office Task Classification List

    Requests in office scenarios are often fragmented. The following classification can serve as an initial mapping, but actual values need to be refined through evaluation:

    Task Type Typical Examples Recommended Tier Notes
    Text Classification & Tagging Email categorization, ticket routing Lightweight Fixed output categories; errors can be batch-reviewed
    Key Information Extraction Extracting amounts and dates from contracts Lightweight or Balanced Requires structured output; recommend combining with rule validation
    Format Conversion Table to Markdown, meeting notes formatting Lightweight Clear rules; easy to check quality
    Document Summarization Summarizing long reports, weekly roundups Balanced Requires understanding the overall text structure
    Translation & Comparison Bilingual alignment, clause difference comparison Balanced Sensitive to professional terminology
    Multi-step Planning Project plan breakdown, workflow orchestration Flagship Planning errors affect all subsequent steps
    High-Risk Content Contract risk assessment, financial conclusions Flagship + mandatory manual review Model output should only serve as a supplementary opinion

    2. Unified Gateway and Model Metadata

    2.1 Gateway Responsibilities

    The upper-layer business logic requires a unified model access gateway. The gateway exposes fixed request formats, response formats, and error codes externally. Internally, it writes independent adapters for each model, handling protocol conversion, error mapping, and standardized return values. When adding or retiring models, only the corresponding adapter needs modification, leaving the business logic untouched.

    The gateway must also define the following strategies:

    • Timeout: Set first-byte timeout and overall timeout separately. For example, set the first-byte timeout to 8 seconds, and the overall timeout based on task type (e.g., 60 seconds for summarization tasks).
    • Retry: Only retry idempotent requests, up to 2 times, using exponential backoff. Non-idempotent requests should return a failure directly upon timeout, letting the upper layer decide the handling method.
    • Rate Limiting: Set rate limits per tenant and per model to prevent a single tenant from exhausting a specific model’s quota.
    • Error Code Mapping: Unify vendor errors into internal error codes (e.g., rate limit, timeout, content interception, auth failure) to help the routing layer decide whether to switch models.
    • Idempotency Key: Each request carries a request_id. The gateway and billing system use this for deduplication, ensuring a single request is billed only once.

    While a gateway reduces integration complexity, it cannot guarantee that requests will always succeed. Failure handling must be clearly defined during the design phase.

    2.2 Model Metadata Center

    In a multi-model environment, the models themselves are resources that need to be registered and managed. The metadata center should record the following fields:

    • Capability Tags: e.g., code, Chinese writing, long text, domain tags (legal, finance, medical).
    • Context Window: Maximum acceptable input token count.
    • Billing Info: Input/output unit prices, as well as cache billing rules.
    • Latency Features: P95 values for first-byte latency and overall latency.
    • 7-Day Success Rate: Statistics broken down by error type.
    • Compliance Level: Whether it is allowed to process sensitive data and meets data residency requirements.

    The routing service first reads the metadata center to filter candidate models that meet compliance, context length, and domain requirements, then makes the final selection based on cost and latency.

    3. Intelligent Routing Strategies and Implementation

    3.1 Rule-Based Static Routing

    Static routing is the foundation of a multi-model system. Rules are maintained in a tabular format for easy review by both product and operations teams. The thresholds and unit prices in the table below are examples and need to be calibrated against actual pricing and evaluation results:

    Priority Condition Example Threshold Preferred Tier Max Cost per Call (Example) Fallback Strategy
    1 Involves sensitive data Trigger on hit Models meeting compliance only No limit If no compliant model is available, return failure and prompt manual processing
    2 Domain is legal or financial Trigger on hit Flagship models passing domain eval $0.50 If no qualified model, escalate to manual; do not downgrade to lightweight
    3 User SLA is real-time First-byte latency < 1.5s Lightweight or Balanced $0.05 On timeout, return cached result or a brief prompt
    4 Task is code generation or logic reasoning Trigger on hit Flagship $0.30 If flagship unavailable, switch to backup in same tier; if still failing, downgrade to Balanced and flag quality
    5 Input tokens exceed long context threshold Over 32k Models supporting long context $0.40 If no long context model available, segment and summarize first
    6 Task is classification, extraction, conversion Trigger on hit Lightweight $0.01 Retry once on failure; if still failing, upgrade to Balanced

    Rules are matched sequentially from highest to lowest priority. Compliance checks must take precedence over cost checks; otherwise, sensitive data might be sent to non-compliant, low-cost models.

    3.2 Confidence-Based Dynamic Upgrade

    Static rules cannot cover all scenarios, so dynamic evaluation after results are returned is necessary. The system can use the following three sources of confidence, chosen based on availability:

    1. Model token-level log probabilities (logprobs). Supported by some models, this allows calculating the average log probability of the generated content. Lower probability indicates the model is less certain about its output.
    2. Self-consistency check. Sample the same input twice and compare if the results are consistent. This costs about twice as much as a single call, so it is only used for high-value tasks.
    3. Rule and small model validation. For example, checking if JSON can be parsed, if required fields are present, or if numbers and dates match the source text.

    Note that confidence only indicates the model’s certainty about its output, which is not equivalent to factual correctness. Its purpose is to decide whether to upgrade, not to prove the result is reliable.

    Downgrade/upgrade rules are as follows:

    • If the model does not return logprobs, use rule validation only.
    • If rule validation fails, upgrade directly to a flagship model for reprocessing and log the sample into the offline evaluation set.
    • Each request can be upgraded at most once to avoid exponential cost increases from continuous upgrades.
    • Results that still fail validation after an upgrade are flagged as “requires manual confirmation” and are not shown directly to the customer.

    3.3 User-Controllable Cost Prompts

    Beyond automatic routing, users should be aware of the relative costs of different models and be able to choose for themselves. The UI can display a multiplier: setting the standard model as 1.0x, other models are calculated based on the weighted price of input and output tokens. For example:

    “This request will use the ‘Advanced’ model, with a relative cost of approximately 3.0x the standard model.”

    The prompt should clarify two points: the multiplier only reflects relative cost, not the actual billing amount; actual billing is subject to the backend statement, and cache hits are billed according to cache rules. This prevents users from misunderstanding the multiplier as the unit price.

    4. Cost Management System

    4.1 Three-Tier Budget and Overlimit Actions

    After integrating multiple models, costs can spike rapidly due to abnormal calls from a specific application. Therefore, tiered budgets must be set. The following values are examples and should be set based on historical usage:

    • Tenant monthly budget: $10,000.
    • Department monthly budget: $2,000.
    • Application daily budget: $300.

    Budget alerts are divided into three thresholds:

    • Reaching 70%: Send a notification to the application owner.
    • Reaching 90%: Notify the department head and log the source of current high-cost calls.
    • Reaching 100%: Execute overlimit actions.

    Overlimit actions differ by tier:

    • Application daily budget exceeded: The application’s calls are downgraded to a lightweight model; other applications are unaffected.
    • Department monthly budget exceeded: Retain flagship calls only for applications marked as high priority; downgrade the rest.
    • Tenant monthly budget exceeded: Block non-core calls and enter an approval process.

    To prevent toggling back and forth near the threshold, once a downgrade is triggered, it will not switch back to the original model for at least 30 minutes. Daily budgets reset at midnight. Increasing a budget requires department head approval and the reason must be logged.

    4.2 Cost Dashboard Metrics

    Only data with unified definitions can be used for decision-making. The dashboard should include the following metrics:

    • Input and output token consumption.
    • Call counts, separated by success and failure.
    • Cache hit rate and the amount saved due to caching.
    • Retry counts and final failure rate after retries.
    • Average cost per task.

    Cost per task is defined as the sum of costs for all calls under the same task_id, including retries after failures. This clarifies that a model that “seems cheap but requires frequent retries” is actually not cheap. Dimensionally, costs should be attributable by model, task type, tenant, department, and application.

    4.3 Cost Minimization Under Quality Constraints

    The goal of cost optimization is to reduce total costs while meeting quality requirements, rather than blindly choosing the cheapest model. Graded processing can be adopted:

    • Internal testing and prototype validation can use lightweight models, as results are for internal reference only.
    • Customer-facing outputs must be generated by models that pass quality evaluation.
    • Before switching models, use the same evaluation set to compare quality, latency, and cost per task.

    5. Enterprise Implementation Practices for Multi-Model Architecture

    5.1 Canary Release and Comparative Evaluation

    When launching a new model, you can gradually ramp up traffic proportionally and run it in parallel with existing models. The recommended process is as follows:

    1. Phase 1: Route 5% of traffic to the new model and run continuously for 3 days.
    2. Monitor the following metrics: Quality score (from manual spot checks or evaluation sets), P95 latency, cost per task, and failure rate.
    3. Criteria for expanding traffic: Quality score not lower than 98% of the baseline for 3 consecutive days, failure rate not higher than 1 percentage point above the baseline, and cost per task within budget.
    4. Criteria for rollback: Failure rate exceeds 2 percentage points above the baseline, or quality score declines for two consecutive cycles.

    The above values are examples and should be adjusted based on business risk.

    5.2 Continuity in Failure Scenarios

    A multi-model architecture can reduce the impact of a single model’s failure on office workflows, but it cannot guarantee uninterrupted processes. The actual design should include the following mechanisms:

    • Health Check: Probe the availability of each model every 30 seconds.
    • Circuit Breaking: If a model times out 5 consecutive times or the error rate exceeds 20%, break the circuit for 60 seconds, during which no requests are sent to it.
    • Failover: After circuit breaking, requests switch to a same-tier model based on the backup order in the metadata center.
    • Fallback Processing: If backup models are also unavailable, the system saves user input and enters a queue state, automatically retrying upon recovery; for urgent tasks, provide a manual processing entry.

    During the switch, the same request might be submitted multiple times, so deduplication must rely on request_id. Service Level Agreements (SLAs) should clearly define availability targets, and regular disaster recovery drills should be conducted to verify the effectiveness of failover and fallback processes.

    5.3 Hybrid Model Workflows

    When office agents execute multi-step tasks, different steps