Reducing LLM API Costs: Model Selection, Routing, and Caching in Practice
This article explains how to lower large language model API costs through pricing tier analysis, model selection with an internal evaluation set, dynamic model routing, context caching, and cloud-native deployment. It also covers the cost and compliance trade-offs of self-hosting open-weight models, and the metrics to track whether cost-reduction measures are working.
As large language model (LLM) applications move from proof of concept to large-scale production, inference cost has become a problem most companies cannot avoid. Some models cost far less than flagship models yet perform close to them on routine tasks. This gives companies a chance to rethink the cost structure of their AI infrastructure.
This article reviews practical ways to reduce costs across four areas: pricing, model selection, deployment, and operations. The prices and practices described here are general examples. Before you act on them, check each vendor’s current published pricing and your own business data.
1. Understanding API Pricing Tiers: Start with the Price List
Most LLM APIs charge by token, usually quoted per million tokens, with input and output priced separately. Based on capability and price, the market falls roughly into three tiers, plus a group of low-cost open-weight models.
| Tier | Typical use cases | Illustrative price (USD per million tokens, input/output) | Selection notes |
|---|---|---|---|
| Flagship | Complex reasoning, long-document analysis, difficult code generation | About 5 / about 30 | Suited to core steps that need high accuracy. Highest unit price. |
| Balanced | Customer support, content summarization, routine Q&A | About 2.5 / about 15 | Balances response speed and cost. Often the starting point for most applications. |
| Lightweight | Intent classification, text rewriting, simple Q&A | About 1 / about 6 | Suited to high-frequency, low-complexity requests. |
| Low-cost open-weight | Batch processing, high-frequency calls | Input about 1.25 / output about 4.25. Input about 0.15 on cache hit. | Confirm caching conditions, rate limits, and service stability. |
The figures in the table are illustrative ranges. They do not reflect any vendor’s current price. Vendors often change prices and apply different billing rules based on input length, caching conditions, and batch discounts. Check the official pricing page before you buy, and record the date you checked.
A price list shows unit prices, not total cost. A task that needs more conversation turns or longer outputs can cost more overall than a model with a higher unit price. Compare models by the total cost of completing one business task.
2. Model Selection: Use Your Own Evaluation Set Instead of Leaderboards
A common mistake in model selection is looking only at public leaderboards. Leaderboards measure general capability, which may not match your specific business needs. A more reliable approach is to build your own evaluation set:
- Sample 200 to 500 requests from real business logs, covering both common questions and hard cases.
- Define a verifiable reference answer for each request, or have domain experts write scoring rules.
- Run the candidate models on the set, and record accuracy, response latency, and cost per request.
- Among the models that meet your quality bar, choose the one with the lowest unit cost.
The benefit is that the results are reproducible. Suppose a lightweight model passes 95% of the evaluation set, while a flagship model is only 2 percentage points higher. Then routing most requests to the lightweight model is usually the better choice. If a certain task type falls clearly short of the quality bar, keep the flagship model for that type.
3. Open-Weight Models: Benefits and Costs of Self-Hosting
Open-weight models let companies download the model and run it in their own environment or a designated cloud environment. This gives more room for customization and data control.
The main benefits fall into three areas:
- Data control: Sensitive data can stay within an environment the company controls, which reduces the risk of data leaving the organization. This matters especially in finance, healthcare, and government.
- Customization: After fine-tuning on industry corpora or adding domain knowledge, a model may outperform general-purpose models on vertical tasks. Verify this with your evaluation set rather than relying on intuition.
- Long-term cost: For high-frequency, stable workloads, a self-built or rented GPU cluster may cost less than pay-per-token pricing. The break-even point depends on call volume, GPU utilization, and operations staffing.
The costs must also be part of the decision:
- Operations: GPU purchase or rental, driver and inference framework maintenance, and incident response all require dedicated staff and budget.
- Model updates: Open-source models release new versions quickly. Each upgrade requires re-evaluation, and you may need to handle compatibility with fine-tuned weights.
- Safety alignment: Self-hosted models need their own content filtering, access control, and defenses against prompt injection.
- Licensing: License terms vary widely between models. Legal review should confirm permitted commercial use and any requirements for publishing derivative models.
- Data governance: Private deployment does not automatically meet compliance requirements. Access control, encryption, audit logging, and data retention policies must each be implemented.
Private deployment can reduce the risk of data leaving your environment, but it does not replace a complete security and compliance program.
4. Cost Reduction Architecture in Practice
After model selection, engineering changes often produce more direct savings. The following three practices are common.
1. Dynamic Model Routing
An API gateway can send each request to a different model based on its complexity. Simple requests go to a lightweight model, and complex requests go to a flagship model. The complexity assessment should be explainable and adjustable, and the system needs a fallback mechanism for failures.
import loggingHARD_KEYWORDS = ("推导", "证明", "多步", "代码重构", "风险评估") # Chinese keywords for derivation, proof, multi-step, code refactoring, and risk assessmentCOMPLEXITY_THRESHOLD = 0.6 # Calibrate with your own evaluation set. Do not fix it by intuition.TIMEOUT_SECONDS = {"flagship": 20, "lightweight": 8}def evaluate_complexity(query: str, history_len: int) -> float:"""Score complexity by rules. Returns a value between 0 and 1."""score = 0.0if len(query) > 300:score += 0.4 # Long inputs usually require more context understandingif history_len > 5:score += 0.2 # Multi-turn conversations need consistent contextif any(k in query for k in HARD_KEYWORDS):score += 0.4 # Keywords for reasoning and code tasksreturn min(score, 1.0)def route_model(user_query: str, history_len: int = 0, budget_ok: bool = True):score = evaluate_complexity(user_query, history_len)target = "flagship" if (score >= COMPLEXITY_THRESHOLD and budget_ok) else "lightweight"try:result = call_api(target, user_query, timeout=TIMEOUT_SECONDS[target])except (TimeoutError, RateLimitError) as exc:logging.warning("模型 %s 调用失败,降级处理:%s", target, exc)# Fallback: if the flagship model fails, use the lightweight model. If the lightweight model also fails, use the backup model.fallback = "lightweight" if target == "flagship" else "backup"result = call_api(fallback, user_query, timeout=TIMEOUT_SECONDS.get(fallback, 8))logging.info("route=%s score=%.2f", target, score)return result
A few points matter in practice:
- Set the threshold with your evaluation set. If the lightweight model still meets the quality bar on medium-complexity requests, you can raise the threshold to send fewer requests to the flagship model.
- Rule-based scoring can misjudge requests. You can add a small classifier on top of the rules, but count the classifier’s own call cost in your total cost.
- Sample downgraded results for quality review, either manually or with automated scoring.
2. Context Caching
In multi-turn conversations, and in scenarios with long system prompts or document context, much of the input stays the same from one request to the next. Context caching, also called prompt caching, is offered by many vendors and can reduce billing for this repeated content. When a cache hit occurs, input costs are often much lower than the uncached price.
Keep these conditions in mind when using caching:
- **The prefix must
