AI Agent Multi-Model Routing: Model Selection and Cost Control in Practice
This article explains how an AI agent can route tasks across multiple LLMs and expose model cost through a unified quota multiplier. It covers capability tagging, adapter-layer routing, rate caps, budget thresholds, off-peak pricing, free quotas, and the operational and compliance considerations needed to keep model selection transparent and cost-effective.
Tasks in office scenarios vary widely. Polishing a paragraph takes only a few seconds and can be handled by a lightweight model. Cross-file code review or multi-step logical reasoning requires a larger reasoning model, and the cost per call can be several times higher. A single model can hardly balance quality, response speed, and price at the same time. Therefore, after integrating multiple models, the system must solve two problems: which model should handle this task, and how much quota should this call deduct. This article explains how to implement these two things from a developer’s perspective.
1. Task Tiers and User Requirements
When users call an agent to deliver results, their requirements usually fall into three dimensions:
- Task quality: logical accuracy and delivery quality for complex tasks.
- Response speed: concurrency handling and time to first token for simple tasks.
- Cost-effectiveness: reducing token call costs as much as possible while meeting quality requirements.
These three requirements constrain one another. A model with stronger reasoning ability usually has higher latency and a higher price. Therefore, the goal of the routing layer is not to find the strongest model, but to find the cheapest model that meets the quality requirement for each task.
2. Model Pool and Capability Tags
When a model is registered, it needs to declare capability tags, context window, and cost baseline. The table below is a simplified model pool example. It lists only relative multipliers; context windows and cost baselines need to be filled in according to each vendor’s actual specifications.
| Model | Type | Regular rate | Suitable tasks | Latency characteristics |
|---|---|---|---|---|
| self_lite | Fast | 0.5 | Summarization, format conversion, simple Q&A | Low time to first token, high throughput |
| third_party_flash | Fast | 0.8 | Medium-length text, information extraction | Low latency |
| self_pro | Balanced | 1.0 | Regular writing, medium reasoning | Medium latency |
| third_party_pro | Reasoning | 1.5 | Code review, multi-step reasoning, long-document analysis | High latency, high cost per call |
3. Unified Adapter Layer and Routing Logic
Different vendors have different request body structures and streaming response protocols. The adapter layer converts the internal unified request format into the format required by each model, while the routing layer only works with the internal format. The following code illustrates routing selection, rate-cap filtering, and fallback logic:
from dataclasses import dataclassfrom typing import Callable, Optional@dataclassclass RouteDecision:model: strrate: float # Actual rate used for this callfree: bool # Whether the call is freeclass ModelRouter:# Model pool: rate is the regular rate, discount_rate is the off-peak ratePOOL = {"self_lite": {"rate": 0.5, "discount_rate": 0.4},"third_party_flash": {"rate": 0.8, "discount_rate": 0.6},"self_pro": {"rate": 1.0},"third_party_pro": {"rate": 1.5},}# Candidate order for each complexity level: try in order, fall back to the next on failureCHAIN = {"simple": ["self_lite", "third_party_flash"],"medium": ["self_pro", "third_party_flash"],"complex": ["third_party_pro", "self_pro"],}def effective_rate(self, name: str, in_discount: bool) -> float:spec = self.POOL[name]if in_discount and "discount_rate" in spec:return spec["discount_rate"]return spec["rate"]def route(self,complexity: str,free_quota_ok: bool,in_discount: bool,rate_cap: Optional[float] = None,is_healthy: Callable[[str], bool] = lambda name: True,) -> RouteDecision:chain = self.CHAIN.get(complexity)if chain is None:raise ValueError(f"Unknown task complexity: {complexity}")for name in chain:rate = self.effective_rate(name, in_discount)if rate_cap is not None and rate > rate_cap:continue # Exceeds the enterprise rate cap, skipif not is_healthy(name):continue # Timeout, rate limiting, or circuit breaking; fall back to the nextif name == "self_lite" and free_quota_ok:return RouteDecision(name, 0.0, True)return RouteDecision(name, rate, False)# All candidates are unavailable: let the caller queue, retry, or return an errorraise RuntimeError("No healthy model available within the rate cap")
Several points in the code require attention.
First, the budget check is completed before routing. Requests with insufficient quota should be intercepted before entering model selection.
Second, the caller retries timeout errors at most twice and uses exponential backoff. Before retrying, confirm that the request can be safely retried.
Third, define in advance whether failed requests deduct quota. The recommended rule is that failures do not deduct quota; otherwise, users pay for an unavailable service.
Fourth, complex tasks are blocked when the rate cap is low. This is the expected result of the policy. The caller should return a clear message to the user instead of silently falling back to a model with insufficient capability.
4. Quota Multipliers and Conversion Method
The multiplier mechanism unifies billing differences across vendors into one internal quota unit. Users only need to look at the multiplier and do not need to compare each vendor’s token unit price one by one. The conversion formula is:
Quota deduction = base cost × model multiplier × platform coefficient
Base cost is the unit count of input and output tokens for the request after conversion using a unified base unit price. The platform coefficient defaults to 1.0 and can be used for capacity protection or peak regulation. Suppose the base cost of a request is 200 units:
- self_pro (multiplier 1.0) deducts 200 units.
- third_party_pro (multiplier 1.5) deducts 300 units.
- third_party_flash during a discount period (multiplier 0.6) deducts 120 units.
The lower the multiplier, the fewer units are deducted and the lower the cost. The higher the multiplier, the more units are deducted and the higher the cost.
Enterprise customers can configure two types of constraints. They serve different purposes and should not be mixed:
- Rate cap: limits the highest multiplier of models that can be called. The value range is 0.5 to 1.5. Below 0.5, no model is available. Above 1.5, it is equivalent to no limit.
- Budget threshold: limits the quota available to a department or individual within a period. The value must be no less than 0.
For example, if a low-priority department is configured with a rate cap of 0.8, the system will call only self_lite and third_party_flash. self_pro and third_party_pro will be skipped, so low-priority tasks use only low-cost models.
5. Cost Optimization Strategies
1. Free quota for simple tasks
Low-compute tasks such as text polishing and short-text translation can preferentially call in-house lightweight models. Provide a certain amount of free calls each day; after that, bill at the 0.5 multiplier. The free quota must have an upper limit to avoid being occupied by batch scripts.
2. Off-peak discount periods
GPU compute demand usually has obvious peaks and valleys. The platform can lower the multipliers of some models during agreed low-demand periods to guide users to shift usage. The purpose is to improve overall cluster utilization and spread fixed costs. The discount effect needs to be verified with metrics. It is recommended to monitor the following four items:
- GPU utilization
- P95 latency
- Failure rate
- Token consumption distribution
The alert thresholds are examples only and need to be adjusted according to business baselines. For example, alert when P95 latency exceeds 3 seconds for 5 consecutive minutes, alert when the failure rate exceeds 2%, and if GPU utilization stays below 30% for 30 consecutive minutes during a discount period, it indicates that the discount is not attracting traffic.
3. Lower multipliers for lightweight models
After lowering the multipliers of lightweight models, users’ quotas last longer on tasks such as information extraction and simple Q&A. These models have low latency and high throughput. When the proportion of lightweight tasks rises and concurrency capacity is sufficient, call volume and daily active users may grow accordingly. This hypothesis needs to be verified through A/B testing and capacity testing and cannot be used directly as a pricing basis.
6. Infrastructure and Security Compliance
Multi-model routing requires stable inference services, an API gateway, and monitoring. Mature cloud vendors’ model inference platforms and API gateways usually provide load balancing and rate limiting. Logging and monitoring components can record each model’s call success rate, average response time, and token consumption. These data are the basis for adjusting multipliers.
Security and compliance need to be designed separately:
- Store multi-vendor API keys in a key management service, not in code or logs.
- Mask logs and do not record users’ original text.
- When cross-border data transfer is involved, confirm each provider’s data storage locations and compliance requirements.
- The adapter layer isolates vendor differences and reduces vendor lock-in risk. When switching models, only the model pool configuration needs to be changed.
Conclusion
The core of multi-model integration is two things: selecting models by task complexity and using a unified multiplier to show cost transparently to users. With time-based discounts and free quotas, the platform can balance quality, speed, and cost.
Multi-model routing also has applicable boundaries. If task volume is not large, or the quality differences among models are not obvious, a single model plus a fixed rate table is usually sufficient. Introducing a complex gateway too early increases development and maintenance costs. When choosing an architecture, first validate routing benefits with real task data, and then decide whether to expand.
