Multi-Model Routing in Practice: Quantifying and Scheduling LLM API Cost Reduction

    This article explains how to reduce large language model API costs through task-aware multi-model routing. Using an Excel table generation task as an example, it compares the cost differences between model types, breaks down API billing components, and describes a routing system built on task classification, model capability profiles, validation, fallback, and retries. It also covers infrastructure-level tactics such as prompt slimming, batching, structured output constraints, and caching, plus a monitoring method based on the cost per valid task rather than the cost per API call.

    Introduction: Cost Is Now a Core Variable in Model Selection

    Large language model APIs are billed by usage. The cost of a single call may look small, but high-frequency tasks such as batch structured data extraction, table generation, and document classification can magnify unit costs to the project level. Model selection and calling strategy directly determine marginal cost, and therefore whether a product can be profitable.

    This article uses an Excel table generation task as an example to explain how to quantify cost differences across models, how to design a routing strategy, and how to use monitoring data for continuous adjustment. Any cost reduction method must be premised on output quality meeting the required standard. If cost falls but the validation pass rate also falls, the overall outcome is worse.

    1. Benchmark: Bill Differences for the Same Task

    Experimental design: send exactly the same instructions to multiple models, asking them to generate an Excel table with conditional formatting and summary formulas based on given sales data. All models use default parameters without additional tuning.

    Quality results: the outputs of all four models passed checks on table structure, formula correctness, and formatting details. According to the acceptance criteria for this task, all four had comparable usability.

    Cost results use the “automatic routing mode” as the baseline, recorded as 1:

    Model type Relative bill multiplier Share of the most expensive option
    Top reasoning model 6 100%
    Long-context model 4 About 67%
    General flagship model 2 About 33%
    Lightweight fast model 0.15 2.5%

    The lightweight fast model costs about 1/40 of the most expensive option and 85% less than the automatic routing mode. This ratio comes from 0.15 divided by 6.

    This test has several limitations that should be understood before citing the results:

    • The candidate model set and decision rules for the automatic routing mode are not disclosed. The multipliers are valid only for this test and cannot be directly extrapolated to other routing systems.
    • The results come from a single task. A lightweight model may require multiple retries on complex reasoning tasks, and retry costs can offset its unit-price advantage.
    • The test did not record model versions, number of runs, or billing dates. These fields should be added when reproducing the test, because model pricing changes frequently.

    This data illustrates a fact that is easy to overlook: strong model capability does not mean high cost efficiency on a particular task.

    2. What Makes Up the Cost

    API bills mainly consist of three parts.

    Input tokens: every request prompt is counted as input. Format instructions, example data, and constraints are billed repeatedly on every call. Input unit prices can differ significantly across models, and the current price list of each platform should be used as the reference.

    Output tokens: all generated content is billed. For the same task, output length can vary widely across models, and some models add lengthy explanations before the answer. This part should be measured with your own samples rather than estimated from impressions.

    Caching and concurrency add-ons: some platforms offer discounts for context cache hits and set tiered pricing for high-concurrency requests. The calling method changes the bill.

    Beyond pricing, the match between task and model also determines cost. Using a flagship model for a simple structured task means paying for reasoning capability that is not needed. Using a lightweight model for a complex reasoning task may lead to repeated retries because outputs fail validation, which can increase overall spending. A reasonable approach is to create a graded match between task difficulty and model capability.

    3. Three Components of a Routing Strategy

    A usable routing system usually contains three layers.

    1. Task classification

    A classifier assigns requests to different complexity levels, such as simple extraction, structured transformation, complex reasoning, and creative generation. There are two classification methods: rule engines and small models.

    The minimum viable approach is to start with rule matching. For example, if a request contains “table,” “Excel,” or “field mapping” and does not contain keywords such as “reasoning,” “proof,” or “planning,” it is classified as structured transformation. Requests not covered by rules are then passed to a small model for classification. When classification confidence is below a threshold (for example, 0.7, which should be calibrated on business samples), the request is sent directly to a mid-tier model rather than being aggressively downgraded. This fallback strategy prevents misclassification from sending complex tasks to a lightweight model.

    2. Model capability profiles and scoring

    Each connected model needs a dynamic profile, including historical validation pass rates by task type, median response latency, and price per token. When a new request arrives, the routing engine combines the classification result, profile data, and current load to calculate a composite score:

    Composite score = quality weight × expected pass rate − cost weight × expected per-call cost (after normalization)

    The two weights are determined by the business. For scenarios with low tolerance for errors, such as finance and healthcare, the quality weight should be clearly higher than the cost weight.

    3. Validation, fallback, and retries

    Routing decisions are not one-time. Outputs need to pass machine-checkable validation, such as whether JSON can be parsed, whether fields are complete, and whether formulas can be calculated. When validation fails, the request is escalated to a stronger model. To prevent infinite loops, each request should have a maximum escalation count, such as two. After that, it should be sent to manual handling or returned with a failure status.

    4. Infrastructure-Level Cost Reduction Methods

    1. Prompt slimming

    Put fixed rules and format requirements in the System Prompt to avoid describing them repeatedly on every request. Remove redundant examples and keep only samples that can change the output. For frequently used fixed templates, use the platform’s prompt caching capability where available. The effect of slimming should be measured by comparing token counts before and after on your own sample set, rather than assuming a certain amount of savings.

    2. Batching and asynchronous processing

    Large numbers of similar requests can be submitted together to a platform’s batch interface. Most platforms offer discounts for this, but response times are longer. For tasks with low real-time requirements, such as overnight data cleaning and weekly report generation, requests can be placed in an asynchronous queue and executed off-peak to avoid peak-hour premiums.

    3. Output structure constraints

    Using JSON Schema, function calling, or JSON mode can restrict output freedom and reduce explanatory text. The following example shows one approach:

    1. SYSTEM_PROMPT = """
    2. You are a data organization assistant. Output only JSON that matches the structure below, with no explanatory text.
    3. {
    4. "summary": {"total_revenue": number, "yoy_growth": string},
    5. "rows": [{"product": string, "revenue": number}]
    6. }
    7. """

    The reduction in output length depends on the strength of the constraints and the task type. It is recommended to use your own task samples to compare output token counts and field completeness rates before and after constraints, and then decide whether to roll them out broadly.

    4. Result caching

    For scenarios such as the same report being viewed by multiple users or the same document being classified repeatedly, results can be cached at the application layer and returned directly on cache hits.

    Caching needs to address three issues:

    • Cache keys should be normalized, for example by removing extra spaces and unifying case, so that equivalent requests do not miss the cache.
    • Semantic caching can match similar but not identical instructions, but it may return incorrect results. A similarity threshold should be set, and periodic sampling should be performed.
    • Data security. Cached values should not contain personal information such as phone numbers or government ID numbers. When necessary, data should be desensitized first. Caches need a time-to-live (TTL), tenant-isolated access permissions, and encrypted storage.

    5. Joint Monitoring of Quality and Cost

    The effect of cost reduction can only be confirmed through monitoring. After each task is completed, it is recommended to record the following fields:

    • Task ID and routing model identifier
    • Input and output token counts and cost
    • Validation rule results and retry flag
    • User-side feedback, such as whether regeneration was requested
    • End-to-end latency

    The core metric should be the cost per valid task, not the cost per API call. The calculation is:

    Cost per valid task = sum of costs for all calls for the task ÷ number of tasks that ultimately pass

    If one call fails validation and the user requests a retry, two calls are generated. Both calls must be included in the cost.

    The following are illustrative numbers, not measured results. Suppose 100 tasks are first sent to a lightweight model at a cost of 0.15 per call, and 40 of them fail validation and are retried with a flagship model at a cost of 2 per call. The total cost is 100 × 0.15 + 40 × 2 = 95, the number of successful tasks is 100, and the cost per valid task is 0.95. If all tasks are sent to the flagship model, the total cost is 200, and the cost per valid task is 2.

    Thresholds also need data support. For example, if a lightweight model’s validation pass rate on a certain task type is below 80% for 7 consecutive days with a sample size of at least 500 calls, reduce its priority for that task type and route traffic to a more stable mid-tier model. The 80% figure is only an example and should be calibrated according to business error tolerance and the statistical confidence of the samples.

    6. Trend: Multi-Model Collaboration Will Become a Standard Architecture

    No single model dominates across all tasks and all price ranges. A common layering approach is: lightweight models handle simple tasks, mid-tier models provide fallback, and flagship models handle difficult tasks. Model gateways are also evolving toward task routing, budget control, and canary releases.

    For teams, two capabilities have long-term value: cross-model architecture design and data-driven cost governance. Prompt techniques for a single model will become obsolete as models are updated, while scheduling systems and monitoring data can be reused continuously.

    Conclusion

    The unit cost of large language model calls is determined jointly by model pricing, scheduling strategy, and task matching. The test in this article shows that for the same task, cost differences between models can reach dozens of times. However, this difference is meaningful only when quality meets the required standard.

    Teams are advised to start with three things: curate about 200 real business tasks as a benchmark set; record the cost and validation result of each call and calculate the cost per valid task; set classification rules, routing thresholds, and escalation conditions based on data, and review them once a month.