画板 1

    LLM Inference Routing and Cost Governance with Tiered Model Supply

    4 hr ago0 views

    This article explains how application teams can design LLM inference routing when upstream model supply is tiered by capability, price, and quota limits. It covers capacity estimation, budget gates, adaptive tier escalation, quota-exhaustion handling, observability metrics, and safe tier migration, with practical formulas and code examples.

    1. Read version gaps as capacity signals, not marketing noise

    A model release cadence is often interpreted as product marketing. From an engineering perspective, it is closer to a capacity-scheduling report.

    A common timeline looks like this: a cost-effective tier is released first, often marketed as having “capability close to the previous flagship at a lower price.” The flagship tier was released two months ago and has not been updated since. The actual capability ceiling is served by a separate premium line: pay-as-you-go top-tier models and high-end models available only through specific channels. This creates a three-tier structure:

    • Top tier: the capability ceiling; high unit price; usually billed by usage or through a dedicated channel rather than a monthly quota.
    • Middle tier: sufficient capability and controlled pricing; often the default workhorse for applications.
    • Bottom tier: very low cost or free quota; serves as a traffic-entry layer for long-tail tasks.

    The timeline above is a common pattern, but the exact cadence must be validated against the rate-limit curves and quota-exhaustion curves you observe yourself. When the middle tier has a gap of more than two months, it often means that limited compute for training and inference is being reallocated: some is reserved for the high-cost top tier that has not yet recovered its cost, and some is used to compensate users pushed out of the previous tier. You cannot observe the upstream provider’s capacity plan, but you can observe which tier is unstable during a given period, which tier has longer queues, and which tier exhausts its quota earlier. Those are the signals that application architecture must absorb.

    Therefore, this article is not about “who wins.” It focuses on a more practical technical question: when upstream supply is tiered and each tier’s availability fluctuates over time, how should the application be designed so that cost remains controlled and user experience does not collapse?

    2. Translate model capability into schedulable resources: three dimensions of a tier matrix

    Most teams select models by looking at only two fields: unit price and capability score. That may be enough for a single-layer invocation, but it is far from sufficient under tiered supply. The constraints that actually determine cost are three orthogonal dimensions:

    Dimension Meaning Engineering consequence
    Capability ceiling Upper bound of complex reasoning, long-context understanding, and code generation Determines which tasks must use the top tier
    Unit price Billing difference per million input/output tokens Input/output pricing is often very different; in long-context scenarios, input-side cost usually dominates
    Obtainable quota Monthly quota, requests per minute (RPM), tokens per minute (TPM), burst quota Determines effective capacity and often becomes a bottleneck earlier than unit price

    The third dimension is the easiest to overlook. A tier that looks “three times cheaper” on paper may become expensive in practice if its TPM is only one-tenth of the flagship tier. During peak periods, it will hit rate limits first, pushing traffic back to a more expensive tier or forcing requests into retry queues. The final bill and the latency perceived by users will both move toward the more expensive side.

    This requires a concept: effective unit cost, not listed unit cost.

    1. effective_unit_cost = total_actual_bill / successfully_delivered_tasks

    The numerator includes all consumption caused by retries, rate-limit backoff, and degradation compensation. The denominator counts only results actually accepted by users. This ratio is almost impossible to estimate without instrumentation; it must be measured.

    3. Capacity estimation: put monthly quota and pay-as-you-go budget into the same formula

    The most typical mixed-pricing scenario is tiered supply: some traffic uses the monthly quota, and overflow traffic spills into pay-as-you-go billing. In that case, “is this month enough?” must be quantified.

    First, unify the definitions (inconsistent definitions are the easiest place to make mistakes in this section):

    • Use a single currency consistently, and keep p aligned with the tax-inclusive or tax-exclusive definition used in the bill.
    • Use one calendar month as the statistical period (engineering approximation: 30 days); Q and B must belong to the same month.
    • Q (effective monthly quota, in tokens) includes only the official monthly quota available for that tier in the current month; it does not include free grants, trial quota, or expired temporary quota that has not been cleared.
    • B (pay-as-you-go budget cap, in currency) is used only to purchase invocation capacity that overflows into another tier.
    • The unit price p for the overflow portion is the unit price of the overflow target tier, not the unit price of the primary tier.
    • Overflow traffic does not enjoy the primary tier’s monthly quota, but it is still subject to the overflow target tier’s RPM/TPM constraints. This directly determines real capacity.

    Let I and O be the average input and output tokens per task, h be the prefix-cache hit rate (applied only to reusable input portions), and r be the retry amplification factor. Then:

    1. T_task   = (I × (1 - h) + O) × (1 + r)
    2. N_quota  = Q / T_task
    3. N_ovf    = (B / p) / T_task
    4. N        = N_quota + N_ovf          # theoretical task count

    Next, consider time distribution. Let S be the effective request window in seconds for the month, d be the average latency per task, and k be the peak-concentration factor (peak QPS relative to average QPS). Then the required concurrency upper bound is:

    1. Concurrency = (N_real × d / S) × k

    An empirical value of k = 3~5 applies only to read-heavy, write-light online workloads with clear daily traffic peaks and valleys. It must be calibrated with load-test results or historical monitoring data: use P99 minute-level QPS / monthly average QPS as the initial value, then adjust upward according to business event calendars. For offline batch workloads, k may be close to 1; using 4 directly will overestimate concurrency requirements.

    The key point: the theoretical task count N is inflated. Real capacity must be obtained by separately truncating each tier with its own RPM/TPM ceiling and then summing the results.

    1. from dataclasses import dataclass
    3. MTOK = 1_000_000                 # tokens per million, for converting currency-per-million-token prices into token counts
    4. MONTH_SECONDS = 30 * 24 * 3600   # statistical period: one calendar month (approximated as 30 days)
    5. MONTH_MINUTES = 30 * 24 * 60
    8. @dataclass
    9. class TierCapacity:
    10.     name: str
    11.     price_per_mtok: float    # price per million tokens (for example, CNY)
    12.     quota_tokens: int        # effective monthly quota (tokens), effective in the calendar month; excludes free/trial/expired temporary quota
    13.     rpm: int                 # requests per minute
    14.     tpm: int                 # tokens per minute
    17. def estimate(primary: TierCapacity,
    18.              overflow: TierCapacity,
    19.              in_tokens: int,
    20.              out_tokens: int,
    21.              cache_hit: float = 0.0,
    22.              retry_factor: float = 1.0,
    23.              avg_latency_s: float = 2.0,
    24.              peak_factor: float = 4.0,
    25.              overflow_budget: float = 0.0) -> dict:
    26.     """Return planned capacity, true capacity after rate-limit truncation, and cost per task for a tier combination.
    28.     Conventions:
    29.       - Use a single currency and a calendar month (approximated as 30 days).
    30.       - Q = primary.quota_tokens (only the primary tier's monthly quota).
    31.       - B = overflow_budget (currency per month), used only to buy overflow capacity on the overflow tier.
    32.       - p = overflow.price_per_mtok (overflow traffic is billed at the overflow tier price).
    33.       - Overflow traffic does not consume the primary tier's monthly quota, but is subject to the overflow tier's RPM/TPM limits.
    34.       - Assume 1 task = 1 request; retry amplification is included in retry_factor.
    35.     """
    36.     per_task = (in_tokens * (1 - cache_hit) + out_tokens) * retry_factor
    38.     # Convert both budgets into tokens, then into task counts.
    39.     quota_tasks = primary.quota_tokens / per_task
    40.     overflow_tokens = overflow_budget / overflow.price_per_mtok * MTOK
    41.     overflow_tasks = overflow_tokens / per_task
    43.     # Rate-limit ceilings: quota portion uses primary limits; overflow portion uses overflow limits.
    44.     quota_ceiling = min(quota_tasks,
    45.                         primary.rpm * MONTH_MINUTES,
    46.                         primary.tpm * MONTH_MINUTES / per_task)
    47.     overflow_ceiling = min(overflow_tasks,
    48.                            overflow.rpm * MONTH_MINUTES,
    49.                            overflow.tpm * MONTH_MINUTES / per_task)
    51.     real_tasks = quota_ceiling + overflow_ceiling
    52.     concurrency = (real_tasks / MONTH_SECONDS) * avg_latency_s * peak_factor
    54.     bill = (quota_ceiling * per_task / MTOK * primary.price_per_mtok
    55.             + overflow_ceiling * per_task / MTOK * overflow.price_per_mtok)
    57.     bottlenecks = []
    58.     if quota_tasks > quota_ceiling:
    59.         bottlenecks.append(f"{primary.name}: limited by RPM/TPM")
    60.     if overflow_tasks > overflow_ceiling:
    61.         bottlenecks.append(f"{overflow.name}: overflow portion limited by RPM/TPM")
    63.     return {
    64.         "per_task_tokens": round(per_task, 1),
    65.         "planned_tasks": int(quota_tasks + overflow_tasks),   # theoretical value
    66.         "real_tasks": int(real_tasks),                        # true value after rate-limit truncation
    67.         "bottleneck": bottlenecks or ["quota/budget reached first"],
    68.         "peak_concurrency": round(concurrency, 1),
    69.         "bill": round(bill, 2),
    70.         "cost_per_task": round(bill / real_tasks, 5) if real_tasks else None,
    71.     }

    If the bottleneck list contains rate-limit items, it means this tier is not a money-saving tier; it is a queueing tier. Real capacity is locked by TPM, overflow traffic is bounced back to a more expensive tier, and the bill is settled at the expensive tier price. This is one of the most common direct causes of bill spikes during tier changes.

    4. Tiered routing: translate tiers into invocation policies

    Once you have a capacity profile, the next step is to change routing from “call the flagship tier for everything” to “classify by task difficulty.”

    4.1 Classify task difficulty first, and keep the classification itself cheap

    1. Rule pre-check: structured input, template-based rewriting, and fixed-format extraction can go directly to the lowest tier or even a pure-rule implementation.
    2. Lightweight classifier: use a small model or local classifier to produce a difficulty score between 0 and 1.
    3. Adaptive escalation: try the low tier first, then use verifiable signals (JSON parsing success, field validation, unit tests, confidence threshold) to decide whether to escalate and retry on a higher tier.

    The third method has the best cost/benefit ratio because it turns “model selection” into “acceptance testing,” rather than relying on prior judgment about model capability.

    4.2 Routing skeleton with budget gate and spend tracking

    1. import random
    2. import time
    3. from enum import IntEnum
    4. from typing import Callable, Protocol
    7. class InferenceBackend(Protocol):
    8.     name: str
    9.     def invoke(self, payload: dict) -> dict: ...   # return value must include a `cost` field (currency units)
    10.     def probe(self) -> bool: ...                   # health probe: can a minimal request return 200?
    13. class RateLimited(RuntimeError):
    14.     """Map upstream 429 or rate-limiting errors: transient congestion; backoff and retry are useful."""
    17. class QuotaExhausted(RuntimeError):
    18.     """Map upstream quota exhaustion, credit exhaustion, or account arrears: retrying only burns money faster; must propagate."""
    21. class BudgetExceeded(RuntimeError):
    22.     """Expected failure caused by the local budget gate or upstream quota; should not be silently swallowed."""
    25. class TaskLevel(IntEnum):
    26.     TRIVIAL = 0     # template / extraction / classification
    27.     STANDARD = 1    # routine conversation, summarization
    28.     HARD = 2        # multi-step reasoning, long-form rewriting
    29.     CRITICAL = 3    # complex code refactoring, high-risk decisions
    32. class Router:
    33.     def __init__(self, backends: dict, budget: float,
    34.                  budget_period_s: int = 24 * 3600,
    35.                  gate_ratio: float = 0.9,
    36.                  clock: Callable[[], float] = time.time):
    37.         self.backends = backends            # TaskLevel -> InferenceBackend
    38.         self.budget = budget                # maximum spend per budget slice (currency units)
    39.         self.budget_period_s = budget_period_s   # 86400 = daily gate; 3600 = hourly gate
    40.         self.gate_ratio = gate_ratio        # trigger downgrade at 90% utilization
    41.         self.clock = clock
    42.         self.spent = 0.0
    43.         self._slice_start = clock()
    45.     def _roll_slice(self):
    46.         """Reset the spend counter when crossing a budget slice."""
    47.         now = self.clock()
    48.         if now - self._slice_start >= self.budget_period_s:
    49.             self._slice_start = now
    50.             self.spent = 0.0
    52.     def effective_level(self, level: TaskLevel) -> TaskLevel:
    53.         self._roll_slice()
    54.         if self.spent >= self.gate_ratio * self.budget and level > TaskLevel.STANDARD:
    55.             return TaskLevel.STANDARD       # near the limit: step down; do not keep burning top-tier capacity
    56.         return level
    58.     def _charge(self, cost: float):
    59.         """Charge regardless of whether verification passes; failed attempts also incur bills."""
    60.         self.spent += cost
    61.         if self.spent >= self.budget:
    62.             raise BudgetExceeded(f"budget slice of {self.budget} has been exhausted")
    64.     def run(self, level: TaskLevel, payload: dict,
    65.             verify: Callable[[dict], bool] | None = None,
    66.             escalate_to: TaskLevel | None = None,
    67.             max_attempts: int = 2,
    68.             cost_of: Callable[[dict], float] = lambda r: float(r.get("cost", 0.0))) -> dict:
    69.         self._roll_slice()
    70.         if self.spent >= self.budget:
    71.             raise BudgetExceeded("the current budget slice is exhausted; trigger a business-side fallback")
    73.         last_err = None
    74.         for attempt in range(max_attempts):
    75.             level = self.effective_level(level)
    76.             backend = self.backends[level]
    77.             backoff = min(2 ** attempt + random.random(), 8.0)   # jitter is required
    78.             try:
    79.                 result = backend.invoke(payload)
    80.             except RateLimited as e:                # 429: transient congestion
    81.                 last_err = e
    82.                 time.sleep(backoff)
    83.                 continue
    84.             except QuotaExhausted as e:             # quota exhausted: not retryable
    85.                 raise BudgetExceeded(str(e)) from e
    87.             self._charge(cost_of(result))           # record spend before verification
    89.             if verify is None or verify(result):
    90.                 return result
    92.             # Verification failed: escalate and retry. If already at the highest level, stop;
    93.             # retrying the same tier has very low expected value.
    94.             last_err = ValueError("verification failed")
    95.             if escalate_to is not None and level < escalate_to:
    96.                 level = escalate_to
    97.                 time.sleep(backoff)
    98.             else:
    99.                 break
    101.         raise RuntimeError(f"routing failed after {attempt + 1} attempts") from last_err

    Key design points:

    • Handle 429 and quota exhaustion separately. A 429 is transient congestion; backoff and retry are effective. Quota exhaustion is a budget problem; retrying only burns money faster, so it must be propagated and trigger a business-side fallback: cache fallback, asynchronous recomputation, or an explicit user-facing failure with a suggested retry time.
    • Retries must include jitter. Synchronous retries create a spike immediately after upstream recovery, which can knock the just-recovered tier down again.
    • Verification failure should trigger escalation, not same-tier retry. Same-tier retry has very low expected value and mostly amplifies cost.
    • Charge before verification. Failed attempts still generate bills. Counting only successful requests makes spent systematically too low and turns the budget gate into decoration.

    5. Model “quota exhaustion” as an expected failure class

    Traditional web services include timeouts and connection failures in their failure model, but they rarely include quota exhaustion. Under tiered supply, quota exhaustion will happen, and its timing is often predictable: month-end, promotional events, or overlapping batch jobs.

    A minimal viable handling combination:

    • Budget gate: distribute budget by day or hour to avoid burning the entire quota at the beginning of the month.
    • Tenant-level isolation: in multi-tenant scenarios, give each tenant an independent quota so that one tenant’s batch jobs cannot consume all capacity.
    • Priority queue: prioritize online interactive requests over offline batch jobs; batch jobs can be paused, delayed, or re-run on a lower-cost tier.
    • Degradation ladder: top tier → primary tier → low tier → cache/rule-based result → explicit failure with a suggested retry time.
    • Visibility: expose remaining quota ratio and current consumption in response headers or response bodies so callers can adapt instead of waiting for a 429.

    When deploying on mainstream public cloud environments, these capabilities can often be evaluated through managed services rather than built from scratch: container platforms can provide elastic scaling and concurrency isolation, message queues can handle peak shaving and delayed retries for batch jobs, and logging/monitoring/alerting services can provide time-series views of quota consumption. Making the budget gate configurable rather than hard-coded is one of the most valuable changes during tier transitions.

    6. Observability: calculate “total cost is lower” correctly

    To compare the real cost of different tiers, at least the following fields must be collected; if any one is missing, the conclusion may reverse:

    • tier: the tier actually hit (record the actual value, not the planned value)
    • input_tokens / output_tokens / cached_tokens
    • attempts: total attempts for the task, including cross-tier escalation and failed attempts
    • escalated: whether escalation occurred
    • latency_first_token / latency_total
    • verified: whether verification passed
    • outcome: success / rate_limited / quota_exhausted / verification_failed
    • tenant_id, task_type, trace_id

    Define four metrics based on these fields:

    1. cost_per_task       = Σ(tier tokens × tier unit price) / task_count
    2. cost_per_success    = total_cost / successful_task_count          # denominator excludes failures and abandoned tasks
    3. escalation_rate     = escalated_task_count / total_task_count      # indicates insufficient capability of the primary tier
    4. effective_cache_gain = cached_tokens / (input_tokens - cached_tokens + actual_paid_input_tokens_after_cache_conversion)

    The metric definitions must state three things clearly, otherwise they are easy to misread:

    • Cache billing: whether cached portions are billed at a discounted price or full price determines the denominator of “effective.”
    • Failures and retries: whether tokens consumed by failed attempts are included in the numerator (recommended, otherwise cost is systematically underestimated).
    • Abandoned tasks: requests cancelled by users or abandoned due to timeouts are included in the numerator but not the denominator. This is why cost_per_success is higher than cost_per_task.

    To judge whether a tier is worth switching to, look at cost per successful task. Consider a concrete example: the primary tier costs 10 currency units per million tokens, the low tier costs 4 currency units per million tokens (60% cheaper), and each task consumes 10,000 tokens (0.01 million tokens).

    • Escalation rate 30%, with an average of 1 attempt before escalation: a low-tier successful task costs 0.04; an escalated task costs 0.04 + 0.10 = 0.14; the average is 0.7 × 0.04 + 0.3 × 0.14 = 0.07, which is 30% cheaper than 0.10, not 60% cheaper.
    • Escalation rate 60%, with an average of 1.6 attempts before escalation: a low-tier successful task costs 0.04; an escalated task costs 1.6 × 0.04 + 0.10 = 0.164; the average is 0.4 × 0.04 + 0.6 × 0.164 = 0.114, which is 14% more expensive than the primary tier.

    The gap between listed price savings and actual savings is mediated by escalation rate and attempt count.

    7. Four architecture actions for tier transitions: make switching configuration, not a project

    1. Unified inference adapter layer. Business code depends only on an abstract interface; tier switching is completed through configuration.
    1. class Registry:
    2.     def __init__(self):
    3.         self._routes: dict[TaskLevel, list[InferenceBackend]] = {}
    5.     def bind(self, level: TaskLevel, backends: list[InferenceBackend]):
    6.         self._routes[level] = backends   # ordered by priority; naturally supports failover
    8.     def chain(self, level: TaskLevel) -> list[InferenceBackend]:
    9.         return self._routes.get(level, [])
    1. Regression baseline. Prepare 50–200 labeled examples with standard answers for each task type. Before any tier change, run the baseline, compare pass rate, average token usage, and P95 latency, then decide whether to increase traffic. “It feels better” is not a valid switching basis without a baseline.
    2. Shadow traffic and canary. Let a new tier receive 1%–5% mirrored traffic for real requests, record metrics only, and do not return results to users. Observe cost and latency distributions, then gradually increase traffic.
    3. Contract tests. When upstream changes response structures, adds fields, or adjusts default values, contract tests should catch the changes in CI to prevent online parsing failures in the middle of the night.

    8. Four frequently asked questions

    Q: When a new tier launches, how do I decide whether it becomes the primary tier or a low-cost tier?

    A: Do not look at unit price first. Follow three steps: use the regression baseline to measure the acceptance pass rate for your task distribution; use the formula in Section 3 to calculate real_tasks after rate-limit truncation and peak concurrency; and use shadow traffic to measure cost_per_success. Only increase traffic if all three are better than the current tier.

    Q: How can I avoid a sudden bill doubling during a tier transition?

    A: Bill spikes are almost never caused by unit price alone. They usually come from three factors: the primary tier’s TPM is exhausted first and traffic is bounced to a more expensive tier; verification failures cause same-tier retries that amplify invocation volume; and a missing budget gate allows the monthly quota to be burned at once. Corresponding actions: include real_tasks in the capacity dashboard, change retries to “escalate on failure, do not retry the same tier,” and make budget gates configurable as daily/hourly slices.

    Q: Should I switch to a low-cost tier as soon as the monthly quota is exhausted?

    A: Look at the task distribution for the remaining period. If the remaining tasks are mostly TRIVIAL/STANDARD, switching to a low tier is reasonable. If they are concentrated in HARD/CRITICAL, switching will produce a high escalation rate and may become more expensive. A safer approach is to follow the degradation ladder: first defer non-critical batch jobs, then downgrade tiers, and only then fail explicitly and tell users when to retry.

    Q: What are common mistakes?

    A: They are concentrated in five areas, described in the next section.

    9. Five common pitfalls

    1. Only looking at input/output unit prices. Ignoring rate-limit thresholds and effective quota makes a tier look cheap on paper but expensive in practice.
    2. Defaulting to the top tier. The difficulty distribution of most online tasks is heavy-tailed; 90% of tasks can pass verification with the primary or even low tier.
    3. Ignoring prefix caching. System prompts, tool definitions, and few-shot examples are stable prefixes. Improving cache hit rate often reduces input-side spending more than switching tiers.
    4. Treating longer context as a lossless operation. Input tokens are approximately linearly related to cost and first-token latency. Dumping untrimmed retrieval results into the prompt is a common hidden waste.
    5. Switching tiers without equivalence validation. Different tiers can differ significantly in structured-output stability and instruction following. Before switching to a new tier, compare three hard metrics with the regression baseline before increasing traffic: structured-output success rate, parsing failure rate, and P95 latency. If any metric degrades beyond an agreed threshold (for example, success rate drops by 2 percentage points or P95 rises by 30%), roll back even if the unit price is lower.

    10. Implementation checklist

    • Label the difficulty level for each task type and measure the real distribution.
    • Unify currency and statistical-period definitions for Q/B/p, and clarify whether overflow traffic is subject to RPM/TPM constraints.
    • Use the formula in Section 3 to calculate the rate-limit-truncated real task capacity and peak concurrency for each tier.
    • Calibrate the peak-concentration factor k with load tests or historical P99/average QPS.
    • Distinguish 429 from quota exhaustion, implementing backoff retry and propagation/degradation respectively.
    • Launch budget gates (daily/hourly slices, including spent accumulation and cross-slice reset) and tenant-level quota isolation.
    • Complete the unified inference adapter layer and make tier switching configuration-driven.
    • Instrument attempts / escalated / cached_tokens / outcome, and clearly define cache billing and failure accounting.
    • Establish a regression baseline and shadow-traffic canary process for each task type.
    • Use “cost per successful task” as the sole decision metric.

    Tiering and gaps on the supply side will always exist. They are natural results of capacity allocation, not something application teams can optimize directly. What the application side can optimize is flattening both the cost curve and the experience curve under uncertain supply: tiered routing decides where money is spent, budget gates decide when to stop spending, and observability decides whether the next switch is based on data or intuition. Once these three things are in place, your system only needs to change one configuration file no matter how the upstream changes tiers.

    09040904

    Top articles