Cost Calculation and Routing for Multi-Model Agents

    This article explains how to estimate available token capacity for different model tiers using input and output pricing, route agent tasks by cognitive depth, and handle failures and budget thresholds. It covers average-cost formulas, cache-hit calculations, tiered routing rules, reference Python code, and a cost audit checklist. The pricing figures are illustrative and should be replaced with platform-specific rates.

    Cost Calculation and Routing for Multi-Model Agents

    In agent projects, model costs often rise faster than expected. Sending the same task to the strongest model can produce a bill dozens of times higher than sending it to a lightweight model. This article covers three topics: how to estimate the available token volume for each tier using a formula, how to route tasks to the appropriate model, and what rules to apply when calls fail or exceed budget.

    1. Data Sources and Assumptions

    The prices in this article are illustrative numbers, not official quotes from any platform. The four tiers are grouped by model scale and pricing range. Tiers A through D correspond to a heavy-parameter model, a mid-sized reasoning model, a lightweight Flash model, and a minimal Lite model. Readers should use their own platform’s price list and replace the numbers before recalculating.

    Tier Input price (credits per million tokens) Output price (credits per million tokens)
    A Heavy-parameter 80,000 400,000
    B Mid-sized reasoning 32,000 112,000
    C Lightweight Flash 4,000 8,000
    D Minimal Lite 2,000 2,000

    With this illustrative price set, Tier A’s output price is 200 times that of Tier D.

    2. Calculating Average Cost with a Formula

    Let the input share be r_in and the output share be r_out, where the two sum to 1. The average cost per million tokens is:

    Average cost = r_in × input price + r_out × output price

    In agent scenarios, a common input-to-output ratio is about 8:2, meaning r_in = 0.8 and r_out = 0.2. On that basis:

    Tier Calculation Average cost (credits per million tokens)
    A 0.8×80,000 + 0.2×400,000 144,000
    B 0.8×32,000 + 0.2×112,000 48,000
    C 0.8×4,000 + 0.2×8,000 4,800
    D 0.8×2,000 + 0.2×2,000 2,000

    3. Available Token Volume Under a Budget

    Assume the 5-hour quota is 60,000 credits. The available token volume is calculated as:

    Available token volume = quota ÷ average cost × 1,000,000

    Tier Available token volume per 5 hours
    A About 417K
    B About 1.25M
    C About 12.5M
    D About 30M

    Tier C’s available volume is 30 times Tier A’s, and Tier D’s is 72 times Tier A’s.

    The weekly quota is 600,000 credits, equal to ten 5-hour quotas. If calls run at full load during all periods, only ten full windows can be used within a week, and the weekly quota will be exhausted before the time windows are. For Tier A, the weekly cap is about 4.17M tokens. Budget planning should consider both the 5-hour window and the weekly quota limits.

    4. Cache Hit Benefits and Prerequisites

    Take Tier C as an example. When a cache is hit, the input price drops to 80 credits per million tokens, which is 2% of the normal price. Let the cache hit rate be h. The average cost becomes:

    Average cost = 0.8 × [h×80 + (1−h)×4,000] + 0.2 × 8,000

    Cache hit rate Available token volume per 5 hours
    0% About 12.5M
    50% About 18.56M
    90% About 30.34M
    100% About 36.06M

    The premise here is that the input price is illustrative and the output portion receives no cache discount. Actual platforms differ in discount ratio, minimum cache length, and expiration time.

    There are three common cache issues. First, caching depends on an exactly matching prefix. If the system prompt begins with a timestamp or random ID, the prefix changes every time and the hit rate will be close to 0. Second, cache keys should be isolated by user or tenant; otherwise, one user’s context may be hit by another user. Third, sensitive information should not be placed in a cacheable prefix.

    5. Tiered Routing Rules

    Tasks are assigned to tiers by cognitive depth:

    • Planning and review: plan, review → Tier A
    • Complex branch judgment: reason, branch → Tier B
    • Code and tool calls: code_gen, tool_call → Tier C
    • Classification, cleaning, and masking: classify, clean, mask → Tier D

    Handle failures with the following rules.

    1. If Tier D output fails validation, retry up to the limit, then upgrade to Tier C.
    2. If Tier C output fails validation, retry up to the limit, then upgrade to Tier B.
    3. If Tier A or B fails, hand off to human handling and do not upgrade automatically.
    4. When weekly usage reaches 80% of quota, block Tier A and B calls; Tier C and D may continue.
    5. When weekly usage reaches 95%, only Tier D calls are allowed.
    6. For call timeouts, retry with exponential backoff; after the limit, apply rules 1 through 3.

    The thresholds in these rules are starting points and should be adjusted based on your own failure rates and budget fluctuations.

    6. Routing Code

    The following code demonstrates the control flow. The client must be implemented separately, and TimeoutError should be replaced with the timeout exception used by your SDK.

    1. import time
    2. TIER_BY_TASK = {
    3. "plan": "A", "review": "A",
    4. "reason": "B", "branch": "B",
    5. "code_gen": "C", "tool_call": "C",
    6. "classify": "D", "clean": "D", "mask": "D",
    7. }
    8. MODEL_BY_TIER = {
    9. "A": "heavy_model_v1",
    10. "B": "mid_model_v2",
    11. "C": "flash_model_v3",
    12. "D": "lite_model_v1",
    13. }
    14. # Upgrade to the next tier after failure. A and B failures go to human handling.
    15. NEXT_TIER = {"D": "C", "C": "B"}
    16. class BudgetExceeded(Exception):
    17. pass
    18. class Router:
    19. def __init__(self, client, weekly_quota, max_retries=2, timeout_s=30):
    20. self.client = client # Must implement invoke(model, prompt, context, timeout, cache)
    21. self.weekly_quota = weekly_quota
    22. self.used = 0
    23. self.max_retries = max_retries
    24. self.timeout_s = timeout_s
    25. def _allowed(self, tier):
    26. ratio = self.used / self.weekly_quota
    27. if ratio >= 0.95:
    28. return tier == "D"
    29. if ratio >= 0.80:
    30. return tier in ("C", "D")
    31. return True
    32. def _call(self, tier, prompt, context, validate):
    33. for attempt in range(self.max_retries):
    34. try:
    35. resp = self.client.invoke(
    36. model=MODEL_BY_TIER[tier],
    37. prompt=prompt,
    38. context=context,
    39. timeout=self.timeout_s,
    40. cache=len(context) > 2000,
    41. )
    42. except TimeoutError:
    43. time.sleep(2 ** attempt)
    44. continue
    45. self.used += resp["cost"] # Cost should come from the billing return value
    46. if validate(resp["text"]):
    47. return resp["text"]
    48. return None
    49. def run(self, task_type, prompt, context="", validate=lambda t: bool(t.strip())):
    50. if task_type not in TIER_BY_TASK:
    51. raise ValueError(f"Unregistered task type: {task_type}")
    52. tier = TIER_BY_TASK[task_type]
    53. while True:
    54. if not self._allowed(tier):
    55. raise BudgetExceeded(f"Weekly usage threshold reached; tier {tier} is blocked")
    56. text = self._call(tier, prompt, context, validate)
    57. if text is not None:
    58. return text
    59. if tier not in NEXT_TIER:
    60. raise RuntimeError(f"Tier {tier} failed; hand off to human handling")
    61. tier = NEXT_TIER[tier]
    62. class FakeClient:
    63. def invoke(self, model, prompt, context, timeout, cache):
    64. return {"text": f"[{model}] completed", "cost": 1200}
    65. if __name__ == "__main__":
    66. router = Router(FakeClient(), weekly_quota=600_000)
    67. print(router.run("classify", "Determine log level: ERROR connection reset"))

    Unregistered task types raise an error directly instead of being assigned to the cheapest tier by default. If unknown tasks were silently sent to Tier D, output quality issues would surface later.

    7. Cost Audit Checklist

    • Count input tokens, output tokens, and credit consumption by task type.
    • Measure the cache hit rate weekly; if it is below expectations, check whether the prefix changes frequently.
    • Alert when a single task’s cost exceeds 3 times the median for tasks of the same type.
    • Notify the owner when weekly usage reaches 80% and 95%.
    • Sample and reconcile bills monthly, comparing the formula results in this article with actual charges.

    8. Limitations

    The illustrative prices do not represent any platform. The formula assumes a fixed input-to-output ratio, while actual task lengths vary considerably; the estimates are only a planning reference. Cache discounts and billing rules may change, so the official price list should be used. The review quality of heavy-parameter models cannot be measured by cost numbers and requires periodic manual spot checks.

    The core of multi-model routing is to reserve expensive compute for the stages that must use it. First use formulas to estimate the budget cap, then use failure rules and an audit checklist to control actual spending. Both are necessary.