How to Evaluate LLM Serving Costs and Choose Models by Scenario
This guide explains why price per million tokens is not enough to estimate the cost of LLM applications. It covers input and output pricing, context length, retries, tool calls, caching, batching, and operations overhead. It also presents a cost model, a multi-dimensional selection framework, scenario-based strategies, optimization techniques, and budget controls for developers and enterprise teams.
When deploying large language models, cost estimates often reduce to a single figure: the price per million tokens. This comparison is intuitive, but it can hide the real structure of your bill. Input and output are priced differently, context length affects resource use, retries and tool calls add extra consumption, cache hit rates determine how much reuse saves, and output length limits set the upper bound on spending. For developers and enterprises, a better approach is to treat cost as an engineering metric that you can observe, model, and govern, rather than as a static price list.
This article covers three general questions: how to evaluate LLM serving costs, how to choose a model by scenario, and how to control budgets through engineering practices. It does not depend on any specific third-party product. When platform capabilities are mentioned, it uses only general categories. In practice, you can use general capabilities from Baidu AI Cloud, such as Object Storage, Log Service, and Monitoring and Alerting, to store evaluation datasets, record call metrics, and set budget thresholds. Verify specific services and pricing against the official documentation and your own evaluation results.
- Why Price per Million Tokens Is Not Enough
First, input and output tokens are usually priced differently. Most services set separate prices for input and output tokens, and output tokens are often more expensive. If your application mainly produces long answers, code, or reports, the output share is high. A option with a low input price but a high output price may not have the lowest total cost. Start by measuring the input-to-output ratio of your real requests, then apply it to the pricing formula.
Second, context length adds hidden costs. Long-document Q&A, multi-turn conversations, and code repository analysis often resend conversation history on every call. Even when a single call is inexpensive, long contexts combined with many turns can drive total consumption up quickly. Separate the context you must send from the context you can trim. Use summarization, retrieval, chunking, and citation mechanisms to reduce repeated transmission.
Third, retries, failures, and tool calls do not appear in the unit price. Network jitter, rate limiting, outputs that do not match the required format, and tool call timeouts can all trigger retries. Each retry can consume new input and output tokens. Without idempotent design, backoff strategies, and result caching, the actual bill will exceed your theoretical estimate.
Fourth, caching and batching change the cost curve. You can cache or reuse stable prompts, system instructions, fixed knowledge snippets, and common question-answer pairs. Offline batch processing can lower the unit cost. Include cache hit rate, batch ratio, and request merging strategy in your cost model, rather than looking only at the online price per call.
Fifth, operations and evaluation costs are significant. Building evaluation datasets, manual spot checks, log storage, monitoring and alerting, gateway forwarding, rate limiting and circuit breaking, data masking, and compliance auditing all consume resources. For enterprise applications, these costs can sometimes deserve more attention than model call fees.
- Build an Explainable Cost Model
A practical cost model should include at least input, output, retries, caching, tool calls, and operations. You can express it with the following general formula:
Total cost C = (input tokens / 1,000,000 × input price + output tokens / 1,000,000 × output price) × (1 + retry rate) + tool call cost + storage and logging cost + allocated operations cost
If you use caching, split the input cost further:
Effective input cost = input tokens × (1 − cache hit rate) × input price + input tokens × cache hit rate × cache price
Enter price parameters from your procurement terms or the platform’s public information. Confirm cache prices and hit rules against the service documentation. The pseudocode below shows only the calculation structure. It does not represent any real pricing:
function estimate_cost(input_tokens, output_tokens, input_price, output_price, retry_rate, cache_hit_rate, cache_price):effective_input_cost = input_tokens / 1000000 * ((1 - cache_hit_rate) * input_price + cache_hit_rate * cache_price)output_cost = output_tokens / 1000000 * output_pricecall_cost = (effective_input_cost + output_cost) * (1 + retry_rate)return call_cost
After you build the model, record the input tokens, output tokens, latency, retry count, cache hits, tool call count, and business scenario tag for each request. Store these in a cost dashboard. Without usage attribution, you cannot tell which feature, tenant, or request type causes a cost increase.
- Selection Dimensions: Quality, Latency, Context, Stability, and Cost
Model selection should not rely on price ranking alone. Build a multi-dimensional scoring matrix. Consider at least the following dimensions:
- Task quality: Measure accuracy, format compliance, code execution rate, and summary faithfulness with your own business evaluation set, rather than relying only on public leaderboards.
- Latency and throughput: Distinguish time to first token from total generation time. Online interactive and offline batch workloads have different latency requirements.
- Context capability: For long-document scenarios, check how effectively the model uses context, along with your chunking strategy and citation accuracy.
- Stability and rate limits: Evaluate error rates, rate limiting policies, timeout behavior, and retry costs.
- Compliance and data governance: If you handle sensitive data, assess masking, auditing, regional requirements, and access control.
- Cost predictability: Consider pricing transparency, output length control, and support for caching and batching.
In practice, use a “quality threshold first, then cost ranking” method. Set the minimum quality level your business can accept, then choose the lowest-cost option that meets it. For critical tasks, keep a high-quality option as a fallback. For non-critical tasks, use a lighter option.
- Scenario-Based Strategies: Different Tasks Need Different Solutions
Everyday coding and simple tasks: Use a lightweight model, limit the maximum output length, and enable template-based prompts and result caching. For repeated code explanations, simple completions, and format conversion, a small model is often sufficient. In low-budget learning stages, use the free tiers of integrated development environments, the free quotas of general-purpose coding assistants, and small practice projects first, and save your budget for complex tasks.
Complex multi-file projects: Use a workflow of segmented understanding, plan generation, local modification, and test verification. Do not load the entire codebase at once. Build the necessary context through retrieval, dependency analysis, and summaries. Use advanced models for architecture design and difficult fixes, and lightweight models for comments, renaming, and simple refactoring.
Chinese-language understanding and long documents: Use chunking, summarization, retrieval augmentation, and citation localization. A long context window does not mean unlimited context, so you still need to control input size. For contracts, reports, and manuals, extract the table of contents and key sections first, answer questions section by section, and then summarize the results.
Heavy reasoning and complex architecture: Advanced models suit complex reasoning, design trade-offs, and cross-module analysis. Pair them with budget gating, human review, and result caching. Set per-request limits, daily budgets, and approval thresholds for high-cost requests to prevent uncontrolled spending.
- Practical Cost Optimization Techniques
First, compress and structure prompts. Remove redundant background, use structured fields, and state clear output constraints to reduce wasted input and overly long output. Cache or reuse fixed system instructions.
Second, control output length. Set a maximum output token count. Ask for the conclusion first and details second. Avoid unbounded generation. For classification and extraction tasks, use short output formats.
Third, route requests by tier. Select a capability tier based on task complexity, user level, and historical success rate. Make routing rules configurable and reversible, and log the reason for each decision.
Fourth, use caching and batching. Build caches for stable questions, knowledge snippets, and tool call results. Merge offline tasks into batches. Set expiration and invalidation policies for caches to avoid stale results.
Fifth, govern retries. Use backoff, idempotency keys, and failure classification to avoid useless retries. Fix format errors first with prompt corrections and validators, not by resending requests blindly.
Sixth, set budget alerts and quotas. Set daily and monthly budgets and alert thresholds by tenant, feature, and environment. When a threshold is reached, downgrade to a lighter model or hand off to a human. You can build a budget dashboard with general capabilities such as monitoring and alerting from Baidu AI Cloud, but set specific metrics and thresholds based on your business.
Seventh, run offline evaluation and regression testing. Build a small evaluation set that covers core scenarios. Before and after each model change, compare quality, latency, and cost. Cost savings that come with quality losses that you have not measured are not real savings.
- Engineering Example: Routing and Budget Gating
The pseudocode below shows a general skeleton for routing and budget gating. Price parameters and thresholds are placeholders. Replace them with your actual configuration before use:
config = {"light": {"input_price": placeholder, "output_price": placeholder, "max_output": 800},"balanced": {"input_price": placeholder, "output_price": placeholder, "max_output": 2000},"advanced": {"input_price": placeholder, "output_price": placeholder, "max_output": 4000}}function select_tier(task_type, complexity, remaining_budget):if remaining_budget < low_budget_threshold:return "light"if task_type in ["simple extraction", "format conversion"]:return "light"if complexity is high and remaining_budget > high_budget_threshold:return "advanced"return "balanced"function pre_call_check(estimated_cost, remaining_budget):if estimated_cost > per_call_limit:return reject and suggest splitting into subtasksif estimated_cost > remaining_budget:return downgrade to light tier or hand off to a humanreturn allow
The key point of this skeleton is not the code itself. It is making the task label, complexity, budget, and fallback strategy explicit. Log every routing decision so you can later analyze both cost and quality.
- Checklists for Low-Budget Projects and Team Rollouts
For personal learning or small team validation, follow this checklist: Complete a proof of concept with free quotas and a lightweight model first. Build 10 to 50 core evaluation cases. Record tokens and latency for each call. Set a maximum output length and a daily budget. Optimize prompts and caching before you change models. Keep human review for complex tasks. Clean up logs and temporary data regularly.
For enterprise rollouts, add the following: attribute costs to tenants and features, set budget alerts and approval workflows, mask sensitive data, assess vendors and compliance, prepare fallback plans for outages, version your evaluation sets, and build rollback mechanisms for model changes. The goal is not the lowest unit price. The goal is a total cost that is predictable, explainable, and sustainable, at a level of quality and latency your business can accept.
