Multi-Model Routing in Practice: Cutting Token Costs at Equivalent Quality

    This article presents a practical multi-model routing workflow for reducing token costs while maintaining equivalent quality. It covers how to establish an evaluation baseline, tier models by capability and cost, define routing thresholds and fallback paths, implement routing and cost tracking, and apply context trimming, prompt shortening, and caching. It also discusses post-launch monitoring, cost reconciliation, and when the approach is not a good fit.

    1. Where the 85% Figure Comes From

    In an internal comparison, we tested four model configurations on the same data-processing task. The task required generating standardized Excel table data under given constraints, and every output was checked against preset format and logic validation rules. All four configurations passed every validation item on that test set. Under this condition, the multi-model collaboration and automatic routing approach reduced token consumption by about 85% relative to a single flagship-model baseline.

    This number depends on several preconditions: the task set was fixed, model versions were fixed, the baseline called the same flagship model directly for every request, and the measurement included both input and output tokens. With a different task, a different set of prompts, or a different group of users, the result could be completely different. The method below can be reused, but readers need to measure it again with their own data.

    2. Establish an Evaluation Baseline

    Cost reduction starts with knowing how much you currently spend and where it goes. We recommend building a baseline in three steps.

    Step 1: Fix the task set. Sample representative requests from production logs, covering simple structured output, medium-complexity text generation, and requests that require multi-step reasoning. A sample size of at least several hundred requests is recommended, with stratified sampling by category.

    Step 2: Fix model versions and parameters. Record the exact version number, temperature, and other parameters for each candidate model. After a vendor updates a version, previous conclusions need to be revalidated.

    Step 3: Write quality standards as programmatically checkable items, such as field count, data type, required fields, and whether formula references are correct. For parts that cannot be checked programmatically, arrange manual spot checks.

    3. Model Tiers and Selection

    Divide models into three tiers and record the applicable boundary of each tier. Prices and latency should follow the vendor’s current announcements; the table only shows relative levels.

    Tier Typical capabilities Latency Relative price Suitable tasks Scenarios not recommended
    Flagship Complex reasoning, long context, code generation Higher Highest Multi-step reasoning, cross-document analysis, solution review Batch extraction with fixed formats
    Balanced Medium-complexity generation and Q&A Medium Medium Daily Q&A, summarization, general rewriting Tasks that require strict logical deduction
    Lightweight Classification, extraction, structured output Low Lowest Intent recognition, field extraction, CSV or JSON generation Open-ended creation, complex reasoning

    A suggested selection threshold is: when a tier’s validation pass rate on the baseline task set is below the target value (for example, 98%), that tier should not handle this type of task. Selection should be based on validation pass rate; parameter scale and brand recognition should be references only.

    4. Intelligent Routing: Decisions, Thresholds, and Fallback

    The routing layer first performs lightweight feature analysis on the request, then decides the target model. The table below is a starting point and needs to be calibrated with baseline data.

    Decision condition Routing target Notes
    Structured output is required, and template fields are known Lightweight Requires spot-check validation
    Contains signals such as multi-step reasoning or cross-document comparison Flagship Signals can come from keywords or a small classifier
    Other cases Balanced Default tier

    Keyword rules are only examples. Words such as “Excel” and “JSON” can misclassify requests that only mention a format but actually need explanation. In production, it is better to use a small classifier or train a decision model on historical labeled data, and periodically sample misrouted requests for evaluation.

    The fallback mechanism has three layers. First, if a lightweight model’s output fails format validation, retry once automatically. Second, if it still fails, escalate to the balanced tier. Third, perform quality spot checks by proportion, for example, randomly sample 1% to 5% of results each day and send them to humans or a stronger model for review. When the spot-check failure rate exceeds the threshold, the system automatically raises the routing tier for that task type and notifies the owner.

    5. Code Implementation Notes

    The code below shows the structure for routing, timeouts, retries, and cost tracking. It is still a simplified version and cannot be used directly in production. _post is a placeholder function for connecting to a specific vendor SDK.

    1. import os
    2. import time
    3. from dataclasses import dataclass
    4. @dataclass
    5. class ModelConfig:
    6. endpoint: str
    7. input_price: float # Price per thousand input tokens, subject to vendor announcements
    8. output_price: float # Price per thousand output tokens, subject to vendor announcements
    9. MODEL_POOL = {
    10. "lightweight": ModelConfig("api/model-light", 0.0, 0.0),
    11. "balanced": ModelConfig("api/model-balance", 0.0, 0.0),
    12. "flagship": ModelConfig("api/model-flagship", 0.0, 0.0),
    13. }
    14. API_KEY = os.environ["MODEL_API_KEY"] # Read the key from an environment variable; do not hard-code it or write it to logs
    15. def call_model(cfg: ModelConfig, prompt: str, timeout: float = 20.0, retries: int = 2) -> dict:
    16. """Call the model and retry with exponential backoff after a timeout. The return value must include the usage fields provided by the vendor."""
    17. for attempt in range(retries + 1):
    18. try:
    19. return _post(cfg.endpoint, prompt, api_key=API_KEY, timeout=timeout)
    20. except TimeoutError:
    21. if attempt == retries:
    22. raise
    23. time.sleep(2 ** attempt)
    24. def estimate_cost(cfg: ModelConfig, input_tokens: int, output_tokens: int) -> float:
    25. """Cost is priced separately for input and output. Token counts should come from the usage field returned by the vendor."""
    26. return (input_tokens / 1000) * cfg.input_price + (output_tokens / 1000) * cfg.output_price

    Two points need attention. First, do not estimate tokens by dividing character count by 4. That kind of estimate is only suitable for rough pre-checks; formal accounting must use the usage data returned by the vendor. Second, cost includes both input and output, and calls generated by retries must also be counted.

    6. Other Cost-Reduction Measures

    1. Context Trimming

    Many developers are used to putting all historical conversations and background documents into the prompt. This increases input tokens and can also distract the model. A better approach is to use vector retrieval or caching and inject only the fragments highly relevant to the current task. For generation tasks with clear templates, it is usually enough to provide clear field definitions and rules; lengthy background descriptions are often unnecessary.

    2. Prompt Shortening

    Provided the model can still correctly understand the intent, prompts should be as short as possible. Remove greetings and repeated constraints, then use A/B tests to compare validation pass rates and token consumption across versions. A shortened version that lowers the pass rate must not go live.

    3. Cache Reuse

    For highly repetitive requests, introduce a cache layer, such as Redis. The cache key design should include three parts: the normalized prompt (unified whitespace and casing), the model version number, and key parameters. When the cache is hit, the model token consumption for that request is zero.

    Caches also need a time-to-live (TTL) and invalidation rules. Market data may be cacheable for only a few minutes, while static rule-based content can be cached longer. When the prompt template or model version changes, related caches must be invalidated. Content involving user privacy should be redacted first or simply not cached. Cache hit rate fluctuates with business traffic, so monitor it daily and track spending separately for hits and misses.

    7. Post-Launch Monitoring and Risks

    • Price changes: After a vendor adjusts prices, the cost model must be updated in sync; otherwise reports will be inaccurate.
    • Cache hit rate fluctuations: When the hit rate drops, actual costs will rise again, so track it separately.
    • Quality fluctuations caused by degradation: Backup models may have lower validation pass rates. Record quality metrics during degradation and set alerts, for example, trigger an alert when the validation pass rate stays below the target for 10 consecutive minutes.
    • Cost data verification: Reconcile gateway statistics with vendor bills every month. When the difference exceeds the set proportion, first check the measurement scope, then check the routing rules.

    8. Applicable Boundaries and Implementation Checklist

    This approach suits tasks whose outputs can be programmatically validated, whose request types are relatively concentrated, and whose call volume is large. It is not suitable for open-ended creation, tasks without validation standards, or scenarios that require extremely high single-output quality and cannot be spot-checked.

    Implementation can proceed in the following order:

    1. Build a baseline task set and record input and output tokens and validation pass rates.
    2. Define applicable tasks and pass-rate thresholds for each model tier.
    3. Launch routing rules while keeping fallback mechanisms and quality spot checks.
    4. Add caching and define cache keys, TTLs, and invalidation rules.
    5. Reconcile costs weekly and review routing performance monthly, updating the baseline as needed.