Multi-Model Routing and Cost Control for Office AI Agents

    This article explains why a single LLM is insufficient for office AI agents, how a unified model gateway and three-tier routing can be implemented, and how to handle permissions, timeouts, retries, fallbacks, and audit logging. It also covers quota multiplier calculations and billing policy considerations for cost control.

    Why a Single Model Is Not Enough

    A single model quickly shows its limits in office scenarios. A complex financial analysis requires strong reasoning, while converting meeting notes into a table only needs a lightweight model. If every task is sent to a high-parameter model, simple work such as format conversion still pays the same compute cost, wasting a large part of the budget. Conversely, if all tasks are sent to a lightweight model, reasoning quality on long documents falls short.

    Teams therefore usually connect self-developed models and several third-party models, assigning each to tasks of different difficulty. Models differ significantly in context length, reasoning depth, code capability, and multilingual understanding, so selection must be based on the task.

    How to Implement a Multi-Model Gateway

    The core is a unified model access gateway. It converts interfaces from different vendors into the same call format, so the upper-layer agent only needs to connect to one entry point. The gateway also handles authentication, rate limiting, billing records, and logging.

    Implementation can proceed in three steps:

    1. Run offline evaluations for each candidate model and record accuracy, average latency, and per-call cost.
    2. Classify models into three tiers: lightweight, advanced, and reasoning.
    3. Implement routing, timeouts, retries, and fallbacks in the gateway, and write an audit log for every call.

    How to Choose Routes

    The routing function takes task intent, complexity score, and user preference as inputs. The complexity score needs to be defined separately. A common approach is to use a small classifier that outputs a score between 0 and 1, combining three features: input length, number of required tool calls, and whether multi-step reasoning is needed. Thresholds should be calibrated with offline evaluation data rather than hard-coded from experience.

    The following example shows routing with permission checks, timeouts, and fallbacks:

    1. import logging
    2. from dataclasses import dataclass, field
    3. logger = logging.getLogger("model_router")
    4. @dataclass
    5. class RouteRequest:
    6. task_intent: str
    7. complexity_score: float # 0.0 to 1.0, provided by the task classifier
    8. prefer_speed: bool
    9. user_id: str
    10. allowed_models: set = field(default_factory=set)
    11. # Fallback order when the primary model is unavailable
    12. FALLBACK_CHAIN = {
    13. "third_party_reasoning": ["self_developed_pro", "self_developed_lite"],
    14. "third_party_fast": ["self_developed_pro"],
    15. "self_developed_pro": ["self_developed_lite"],
    16. "self_developed_lite": [],
    17. }
    18. def pick_primary_model(req: RouteRequest) -> str:
    19. if not 0.0 <= req.complexity_score <= 1.0:
    20. raise ValueError("complexity_score must be between 0 and 1")
    21. if req.complexity_score < 0.3 and req.task_intent in {"chat", "format_transform"}:
    22. return "self_developed_lite"
    23. if req.complexity_score > 0.7 and req.task_intent in {"code_generation", "data_analysis"}:
    24. return "third_party_fast" if req.prefer_speed else "third_party_reasoning"
    25. return "self_developed_pro"
    26. def call_with_fallback(req: RouteRequest, invoke, max_retries: int = 1, timeout_s: float = 30.0):
    27. primary = pick_primary_model(req)
    28. candidates = [primary] + FALLBACK_CHAIN.get(primary, [])
    29. for model in candidates:
    30. if model not in req.allowed_models:
    31. # Skip models without permission; do not call them or incur cost
    32. logger.info("skip model=%s user=%s reason=not_allowed", model, req.user_id)
    33. continue
    34. for attempt in range(max_retries + 1):
    35. try:
    36. result = invoke(model, timeout=timeout_s)
    37. logger.info("ok model=%s user=%s attempt=%d", model, req.user_id, attempt)
    38. return result
    39. except TimeoutError:
    40. logger.warning("timeout model=%s user=%s attempt=%d", model, req.user_id, attempt)
    41. except Exception:
    42. logger.exception("error model=%s user=%s", model, req.user_id)
    43. break # Do not retry non-timeout errors on the same model; fall back directly
    44. raise RuntimeError("All candidate models failed")

    There are three points to note in this code. First, permission checks are completed before the call, and models the user is not allowed to use are skipped directly. Second, fallback is triggered only for availability issues such as timeouts, rate limiting, and service errors. When falling back to a weaker model, the user should be notified that result quality may degrade. Third, logs record only the user identifier, model name, and latency, not the original prompt text, to avoid sensitive data entering the logging system.

    How to Calculate Quota Multipliers

    Platforms usually convert each call into user quota. One usable calculation method is:

    Base quota = input tokens × input coefficient + output tokens × output coefficient

    Actual deducted quota = base quota × model multiplier

    The input and output coefficients are set by the platform based on underlying cost. The output coefficient is generally higher because generating tokens consumes more compute. The model multiplier is used for relative conversion between different models.

    For example, assume a task has 2,000 input tokens and 800 output tokens, with an input coefficient of 0.5 and an output coefficient of 2. The base quota is 2,000 × 0.5 + 800 × 2 = 2,600. The table below shows estimated results for three model types on the same task. The multipliers and quota values are illustrative only and do not represent the actual configuration of any platform.

    Model type Typical use Illustrative multiplier Quota deducted per call Response speed Main limitation
    Lightweight model Format conversion, routine Q&A 1.0 2,600 Fast Weaker long-text reasoning
    Advanced model Document summarization, general writing 2.0 5,200 Medium Still gaps on complex reasoning
    Deep reasoning model Code generation, data analysis 5.0 13,000 Slow High cost, longer queuing time

    Within the platform’s free quota and rate limits, incremental cost is close to zero. Beyond that range, usage is billed by multiplier, and the interface should clearly state this.

    Billing rules also require policy points to be defined in advance:

    1. Input tokens that hit the cache are usually billed at a discount, and the discount ratio must be specified.
    2. Whether failed retries are charged.