Multi-Model Routing and Cost Accounting for AI Agents

    This article explains how to balance capability and cost in AI agents by tiering models, routing tasks, accounting for quota consumption and raw cost, and validating the design before launch. It provides model-selection criteria, a routing example with code, cost formulas, a hypothetical savings calculation, architecture principles, a launch checklist, and key risks.

    Multi-Model Routing and Cost Accounting for AI Agents

    The purpose of connecting multiple models is to find a balance between capability and cost. A single model can cover most scenarios, but it tends to be too expensive for high-frequency simple tasks and may fall short on complex reasoning or very long documents. This article follows four steps: tiered model selection, task routing, cost accounting, and launch checks. Each step provides rules and calculation methods that can be applied directly.

    1. Why Agents Need Multiple Models

    A traditional question-answering assistant only needs to answer one question. A modern AI agent must complete multi-step work around a goal. A typical flow includes understanding intent, decomposing subtasks, calling tools such as search or code execution, reading and writing documents, and finally delivering a result. Different steps have very different requirements for models. Subtask decomposition and format conversion are sensitive to speed. Code generation and data analysis are sensitive to reasoning capability. Contracts or reports dozens of pages long are sensitive to context length.

    If every step is assigned to the same model, two problems arise. First, simple steps also use an expensive model, which raises the cost per task. Second, complex steps use a low-cost model, which raises the error rate and requires manual rework. The value of a multi-model architecture is that each step uses a model that is just sufficient for the job.

    2. Model Tiers and Selection

    In practice, models are usually divided into three tiers plus one dedicated layer. The table below is a reference framework organized by capability dimension. Specific values must be based on the official documentation of the platform in use and your own evaluation results. The quota multipliers in the table are example values, and later calculations use these values.

    Tier Typical use Latency Context Reasoning Quota multiplier (example)
    Lite / ultra-fast Format conversion, short-text polishing, intent classification Low Short Weak 0.3
    General enhanced Long-form writing, knowledge Q&A, multi-turn dialogue Medium Medium Medium 1.0
    Reasoning / specialized Complex calculation, deep coding, data analysis High Medium Strong 2.0
    Long-context Analysis of hundreds of pages, cross-file retrieval High Very long Medium 2.5

    When selecting a model, use the following order: first confirm whether the task requires long context, then confirm whether it requires multi-step reasoning, and only then consider speed and price. This avoids assigning a long-document task to a lite model with insufficient context simply because the task looks simple.

    Selection also requires offline evaluation. Prepare a batch of historical tasks with ground-truth answers, run each model once, and record the pass rate, average token consumption, and P95 latency. The cheapest model whose pass rate meets the business threshold becomes the default model for that task type.

    3. Task Routing Design

    The router takes task features as input and outputs a model name. Common features fall into four categories.

    1. Input token count, meaning the length of the user input and any attached documents.
    2. Whether tool calls are needed, such as search, code execution, or file read/write.
    3. Estimated number of steps, which can be provided by the task decomposition module or taken from the historical average for similar tasks.
    4. Task-type keywords, such as “code,” “data analysis,” or “report.”

    The complexity score can be calculated as a weighted sum:

    complexity_score = 0.4 × min(input_tokens / 30000, 1) + 0.3 × needs_tools (0 or 1) + 0.3 × min(steps_estimate / 5, 1)

    The weights are initial values and need to be fitted and adjusted with offline evaluation data. The score ranges from 0 to 1. Scores below 0.3 are treated as simple tasks, and scores of 0.6 or above are treated as complex tasks.

    Keyword matching must support Chinese. An early version matched only English keywords, so a Chinese request meaning “help me write a piece of code” could be incorrectly routed to the general model. The fix is to maintain both Chinese and English keywords and place the keyword table in a configuration file for easy addition and removal. Keywords are only an initial filter. If a keyword is hit but complexity is very low, the task can still go to the lite model. If many expressions are not covered by keywords, a lightweight classifier can replace keyword matching, but keywords should remain as a fallback.

    The revised routing example below removes the original single-English-match logic and adds a budget check.

    1. from dataclasses import dataclass
    2. MULTIPLIER = {"lite": 0.3, "general": 1.0, "reasoning": 2.0, "long_context": 2.5}
    3. CODE_KEYWORDS = ["代码", "编程", "函数", "脚本", "bug", "debug", "code", "sql"]
    4. ANALYSIS_KEYWORDS = ["数据分析", "统计", "报表", "透视表", "data analysis"]
    5. @dataclass
    6. class TaskFeatures:
    7. text: str
    8. input_tokens: int
    9. needs_tools: bool
    10. steps_estimate: int
    11. base_tokens: int # Estimated per-call token consumption under the base measurement
    12. def complexity_score(f: TaskFeatures) -> float:
    13. score = 0.4 * min(f.input_tokens / 30000, 1.0)
    14. score += 0.3 * (1.0 if f.needs_tools else 0.0)
    15. score += 0.3 * min(f.steps_estimate / 5, 1.0)
    16. return round(score, 3)
    17. def pick_model(f: TaskFeatures) -> str:
    18. text = f.text.lower()
    19. score = complexity_score(f)
    20. if f.input_tokens > 30000:
    21. return "long_context"
    22. if any(k in text for k in CODE_KEYWORDS + ANALYSIS_KEYWORDS) or score >= 0.6:
    23. return "reasoning"
    24. if score < 0.3 and f.input_tokens < 2000 and not f.needs_tools:
    25. return "lite"
    26. return "general"
    27. def route_with_budget(f: TaskFeatures, budget_tokens: float) -> str | None:
    28. model = pick_model(f)
    29. if f.base_tokens * MULTIPLIER[model] <= budget_tokens:
    30. return model
    31. # If over budget, downgrade to general. Log the downgrade and flag quality risk.
    32. if model != "long_context" and f.base_tokens * MULTIPLIER["general"] <= budget_tokens:
    33. return "general"
    34. # If still over budget, return None. The upper layer decides whether to abort or ask the user.
    35. return None

    The example code is only a skeleton. Before launch, the following mechanisms must be added.

    Timeout and retry: set a timeout for each model call and retry once after a timeout. If it still fails, upgrade to a stronger model or return a failure state. Retry count must have an upper limit; otherwise retry loops can make cost spiral.

    Output validation: validate the format and fields of lite-model output. If validation fails, upgrade to the general model, and upgrade only once.

    Per-task cost cap: set a budget for each task. If the budget is exceeded, downgrade or abort according to the logic above, and record the reason for later troubleshooting.

    4. Quota Multipliers and Cost Accounting

    Measurement definitions must be unified first. This article uses the following three formulas.

    1. Single-task quota consumption = base token count × model quota multiplier × time-window coefficient
    2. Model raw cost = input tokens / 1000 × input unit price + output tokens / 1000 × output unit price
    3. Daily quota consumption = Σ (number of tasks per type × single-task quota consumption)

    Platform quota and raw cost are two different measures. Quota is the platform’s metering unit for users, while raw cost is the settlement price between the platform and model providers. Do not mix them in cost analysis.

    The following hypothetical scenario illustrates the effect of tiered scheduling. Assume a team processes 1,000 tasks per day and the base token count is uniformly calculated as 4,000. This is illustrative data. Of these tasks, 60% are simple and use the lite model, 30% use the general model, and 10% use the reasoning model.

    • All tasks use the general model: 1000 × 4000 × 1.0 = 4,000,000 quota
    • Tiered scheduling: 600 × 4000 × 0.3 + 300 × 4000 × 1.0 + 100 × 4000 × 2.0 = 720,000 + 1,200,000 + 800,000 = 2,720,000 quota
    • Savings ratio: (4,000,000 − 2,720,000) ÷ 4,000,000 = 32%

    This result holds only under two conditions. First, the lite model meets the required pass rate for those 600 tasks. Second, the upgrade and retry ratio is low. If 10% of lite-model tasks need to be upgraded to the general model, the savings ratio will drop noticeably and must be recalculated.

    Time-window discounts are calculated in a similar way. Assume the off-peak time-window coefficient is 0.8. For the same 300 general-model batch tasks, if they are executed during off-peak hours, consumption is 300 × 4000 × 1.0 × 0.8 = 960,000, which is 240,000 less than the peak-hour figure of 1,200,000.

    A quota multiplier reduction can be estimated in the same way. Assume the lite-model multiplier drops from 0.3 to 0.2. Then the consumption of the 600 lite tasks decreases from 720,000 to 480,000, a reduction of 240,000.

    The time-window coefficient and multiplier are platform billing rules. Before calculation, confirm the applicable time window, applicable models, and expiration date. Do not assume that a discount will remain valid long term.

    Metering should be recorded by model, task type, and time window, and reconciled with the platform bill every day. Base token count should be calculated from the actual total input and output tokens consumed, not replaced by an estimate.

    For cost anomaly troubleshooting, use the following order. First check whether any single-task consumption exceeds the P99 value. Then check whether retry loops exist. Then check the number of failed downgrade tasks. Finally check whether there is a sudden consumption spike in a particular time window or task type.

    5. Architecture Selection and Implementation Notes

    In a multi-model architecture, three design principles deserve priority.

    Unified access layer: all models are wrapped behind the same calling interface, and business code does not directly depend on a vendor SDK. When replacing or adding a model, changes are limited to the access layer.

    Observability: each call records the model name, task type, input and output token counts, latency, success status, and whether a downgrade occurred. These data are the basis for adjusting routing thresholds and multiplier rules.

    Configuration: routing thresholds, keyword tables, and multiplier parameters are stored in a configuration center and support hot updates. After a rule change, a switch can roll it back quickly.

    When choosing a model service platform, evaluate it from the following dimensions: whether it provides a unified calling interface, whether it provides usage details split by model and time window, whether billing rules are transparent, whether service-level agreements are clear, and how high the migration cost is. This article does not recommend specific vendors; evaluation conclusions should be based on actual test data.

    6. Launch Checklist

    1. Measurement definitions are unified and documented in the team documentation.
    2. Each model has offline evaluation results for the corresponding task types.
    3. Routing thresholds are supported by canary data, with the canary percentage starting at 5% and gradually expanding to 20%.
    4. Timeouts, retries, and downgrades all have upper limits.
    5. A budget cap has been set for each task.
    6. Daily budget alerts are configured, preferably at 70% and 90% consumption.
    7. The troubleshooting process for anomalous bills is defined.
    8. The rollback switch for routing rules works properly.
    9. The expiration time and applicable scope of platform discounts are recorded.

    7. Risks and Boundaries

    Routing errors affect both quality and cost. Downgrading complex tasks to lite models saves money in the short term, but as rework increases, long-term cost may be higher.

    Platform quota policies may change. Cost models that rely on promotional prices should reserve margin and be recalculated periodically using the latest billing rules.

    Time-window scheduling changes the user experience. Non-urgent batch tasks are suitable for delayed execution, while real-time conversations are not.

    Keywords and classifiers both produce misjudgments. Real samples should be periodically sampled and reviewed, and the keyword table and weights should be updated based on misjudgment cases.

    Multi-model scheduling does not automatically lead to low cost. It requires three conditions at the same time: tasks can be tiered by complexity, consumption can be accurately metered, and rules can be rolled back quickly when problems occur. With all three conditions in place, tiered scheduling can stably reduce unit task cost while keeping result quality from declining.