Large Language Model Selection and Cost Optimization: A Token Consumption Perspective
This article explains why popularity rankings and cost leaderboards are insufficient for choosing large language models. It introduces tokens as a shared unit for pricing and context capacity, breaks down input, output, and call-frequency costs, and proposes a selection framework based on task capability, latency and throughput, context length, and total cost of ownership. It also presents practical optimization strategies such as prompt compression, result caching, tiered routing, batch processing, and output constraints, along with a repeatable evaluation loop and production architecture recommendations.
1. Why Rankings Are Not Enough for Model Selection
Many teams start by checking rankings of API calls or spending when choosing a large language model. These rankings can show overall usage popularity, but they do not answer a specific question: which model is best for my use case? Raw usage data often shows two patterns. First, call volume is concentrated on a small number of models. Second, the top of a spending ranking is often dominated by premium flagship models with higher unit prices, and a single model may account for a significant share of total expenditure. This suggests two points:
- Usage popularity is not the same as fit. A model that is popular for general-purpose scenarios may not be suitable for your specialized task.
- High spending concentration is not necessarily waste. If a high-priced model is used in a critical path, its cost may be fully justified.
A more rational approach is to return to your own business requirements and evaluate with measurable indicators instead of following rankings blindly. Model selection is essentially a multi-objective optimization problem: find the best solution under three constraints: capability, latency, and cost.
2. Understanding Tokens: A Shared Unit of Cost and Capability
A token is the basic unit used by large language models to process text. One token roughly corresponds to a word or subword. In Chinese, a character or word may be split into one or more tokens. Tokens determine two key questions:
- Cost: almost all commercial LLMs are priced per token, with separate rates for input prompts and output completions. These prices are usually different, and output is often more expensive.
- Effectiveness and context: the model’s context window, or the maximum number of tokens it can handle in one request, determines how much reference material, code, or conversation history you can include.
A simplified cost estimation model can be written as follows:
# Example: estimate the cost of a single calldef estimate_cost(input_tokens, output_tokens, price_in, price_out):# price_in / price_out are prices per 1,000 tokensreturn (input_tokens / 1000) * price_in + (output_tokens / 1000) * price_out# Monthly cost = cost per call × number of callsdef monthly_cost(single_call_cost, calls_per_month):return single_call_cost * calls_per_month
This simple model reveals two key optimization levers: reduce token consumption per call and reduce unnecessary calls.
3. Three Major Cost Components
Breaking down the bill reveals three main sources of cost, each requiring different optimization methods:
- Input cost: system prompts, context, retrieved reference material, and conversation history all count toward input tokens. In long conversations and high-frequency calling scenarios, repeatedly carrying long context is a hidden cost driver.
- Output cost: the length of generated content directly determines output tokens. Asking the model for long-winded answers is often unnecessary. Clearly defining output format and length can reduce costs significantly.
- Call frequency cost: repeated batch calls, retries after failures, and lack of caching can multiply expenses.
There are also indirect costs to consider. Higher latency slows down user experience and consumes more concurrency capacity. A larger context window usually comes with a higher unit price. These factors should also be included in a comprehensive assessment.
4. Four Dimensions for Model Selection
Setting aside brand preferences, a reusable model selection framework should include the following dimensions:
- Task capability: evaluate with real business data instead of generic benchmarks. Different tasks, such as classification, extraction, generation, code completion, and multi-turn reasoning, have very different capability requirements.
- Latency and throughput: interactive applications care about time to first token (TTFT), while batch applications care about throughput and total cost. These scenarios often prefer different model tiers.
- Context length: long-document summarization and code repository understanding require larger windows, while short question-answering tasks do not need to pay for them.
- Cost and stability: unit token price, rate-limiting policies, service availability, and elasticity all affect the total cost of ownership in production.
A practical approach is tiered routing: send simple high-frequency tasks to lightweight, low-cost models, and send complex critical tasks to stronger models, with an orchestration layer coordinating the routing.
5. Practical Cost Optimization Strategies
- Prompt compression: remove redundant system prompts and examples, and compress context. When necessary, use a structured summary instead of carrying the full original text.
- Result caching: use semantic caching for identical or similar requests. If a cache hit occurs, return the result immediately and avoid consuming tokens again. Example logic:
cache = {}def call_model(prompt):key = hash(prompt) # For production, semantic similarity matching is recommendedif key in cache:return cache[key]result = llm_client.invoke(prompt)cache[key] = resultreturn result
- Tiered routing: choose different model tiers based on task complexity, and avoid using flagship models for simple tasks.
- Batch processing and asynchronous execution: offline, non-real-time tasks can use batch interfaces, which usually reduce costs. Asynchronous execution can also smooth concurrency pressure.
- Output constraints: limit the maximum number of output tokens and require structured output, such as JSON. This saves money and makes downstream parsing easier.
6. Build Your Own Evaluation Loop
Do not rely on a one-time selection decision. Instead, build a sustainable evaluation loop:
- Create an evaluation set that covers real scenarios, including typical inputs and expected outputs.
- Run candidate models in batches and record accuracy/satisfaction, average latency, token consumption per call, and cost per call.
- Rank the candidates with a weighted scoring model. For example:
def score(quality, latency, cost, w=(0.6, 0.2, 0.2)):# Normalize each metric to [0, 1] firstreturn w[0]*quality + w[1]*(1-latency) + w[2]*(1-cost)
- After deployment, use logs and monitoring to continuously track real call volume, token distribution, and spending trends. Run periodic regression evaluations to prevent quality and cost drift caused by model upgrades or business changes.
7. Recommendations for Production Architecture
In a real system, model invocation is only one part of the chain. Use message queues to smooth peaks and valleys, and make batch tasks asynchronous. Use container platforms to host the inference orchestration and routing layer and enable elastic scaling. Use object storage to store raw corpora and evaluation sets. Use logging services to record inputs, outputs, tokens, and latency for each request. Use monitoring and alerting to set thresholds for sudden spending increases, latency degradation, and rising error rates. Together, these general capabilities help make LLM applications stable, controllable, and cost-transparent. On cloud platforms such as Baidu AI Cloud, these building blocks are typically offered as standard service categories, making it easier for teams to build observable and governable application architectures.
Conclusion
There is no standard answer to “which model is most convenient.” The real question is “which model is most suitable for your scenario and budget.” By changing model selection from “following rankings” to “following data,” and changing cost management from “discovering it at the end of the month” to “monitoring it in real time,” you can find your own balance among capability, latency, and cost. What ultimately determines the experience is not the model’s name, but how well you understand tokens, cost structures, and evaluation methods.
