Evaluating LLM API Pricing and Cost-Effectiveness: A Practical Framework

1. Why Billing Structure Matters When a Model Goes Commercial

When a model moves from preview to general availability, developers typically care about three things rather than whether it works at all: whether capabilities are stable, what the service commitments are, and whether pricing is predictable. Pricing is especially critical—the cost of experimenting during preview and the bill in a large-scale production environment can differ by orders of magnitude. If the general availability release keeps the same pricing structure as the preview and maintains clear, transparent billing rules, the migration cost for existing users drops significantly. Conversely, frequent changes to prices and billing rules directly increase the evaluation burden on engineering teams.

Most mainstream LLM APIs today use three-tier billing: input tokens, output tokens, and cached tokens are priced separately. Understanding what these three tiers mean and how they relate is a prerequisite for evaluating whether any model’s pricing is reasonable.

2. Breaking Down the Three-Tier Billing Structure

1. Input tokens (request cost)

This refers to everything sent to the model in a request, including system prompts, retrieved context, conversation history, and code snippets. Input volume is often underestimated in engineering practice—a coding assistant request carrying full code context may have an input token count more than ten times its output.

2. Output tokens (generation cost)

Generation requires token-by-token autoregressive computation, and inference overhead is far higher than encoding the input, so output unit prices are typically several times input unit prices. This is an industry-wide phenomenon determined by the inference mechanism, not a pricing strategy unique to any particular vendor.

3. Cached tokens (discount for repeated context)

Most mainstream platforms offer prompt caching: when the prefix of a request matches a recent request, that portion of input can hit the cache and be billed at a substantial discount. For scenarios with fixed system prompts and highly repetitive context (coding assistants, customer service bots, retrieval-augmented question answering), the cache hit rate almost directly determines the order of magnitude of cost.

A set of illustrative numbers shows the scale relationship: suppose a flagship model charges 6 CNY per million input tokens, 18 CNY for output, and 0.3 CNY for cache hits, with all three tiers sharing the same billing basis. The cached unit price is then only 5% of regular input. For the same 100,000-token fixed context, a cache miss costs 0.6 CNY per call, while a cache hit costs just 0.03 CNY per call; at ten thousand calls per day, that is the difference between thousands of yuan and hundreds. The value of caching is obvious.

3. Three Common Pitfalls in Unit Price Comparisons

Pitfall 1: Looking only at output unit price. Output price is the most intuitive indicator of a “flagship” feel, but total cost = input volume × input price + output volume × output price + cached volume × cached price. The input-to-output ratio varies enormously by scenario: code completion typically has long input and short output; long-form writing is the opposite. Discussing unit prices without a workload profile is meaningless.

Pitfall 2: Ignoring currency and billing rules. Some platforms price in USD, others in CNY; comparing the raw numbers is meaningless—you must convert to a common currency first. At the same time, each vendor’s billing boundaries may differ (whether cache writes are charged separately, whether failed requests are billed, whether context storage is charged). Refer to the official documentation rather than relying on secondhand information.

Pitfall 3: Ignoring discount structures. Long-term contracts, batch inference, and committed usage often come with discounts, and prices may be adjusted when a preview becomes generally available. Evaluations should leave a margin for fluctuation on top of list prices.

4. Modeling Against Real Workloads: A Practical Estimation Method

A four-step process is recommended:

1. Sample real traffic: measure the average input tokens, output tokens, and cache hit rate per request over a period of time—let the data speak;
2. Model by scenario: model completion, conversation, and batch processing separately, since input-to-output ratios differ greatly across scenarios;
3. Simulate billing: replay historical requests through the pricing logic, as in the sample code below;

# pricing unit: CNY per million tokens (illustrative values)
PRICING = {"input": 6, "output": 18, "cached": 0.3}
def estimate_cost(requests):
    total = 0.0
    for r in requests:
        cached_cost = r.prefix_tokens / 1e6 * PRICING["cached"]
        input_cost  = (r.input_tokens - r.prefix_tokens) / 1e6 * PRICING["input"]
        output_cost = r.output_tokens / 1e6 * PRICING["output"]
        total += cached_cost + input_cost + output_cost
    return total

1. Set budgets and alerts: continuously correct model parameters using actual bills.

Here is a concrete example. A typical coding assistant request: 40,000 input tokens, of which 35,000 are a fixed prefix (system prompt + repository context) and 5,000 are new content; output is 1,000 tokens. Plugging in the illustrative unit prices above:

• Cached portion: 35000/1e6 × 0.3 ≈ 0.0105 CNY
• New input: 5000/1e6 × 6 = 0.03 CNY
• Output portion: 1000/1e6 × 18 = 0.018 CNY
• Total per call ≈ 0.059 CNY

If caching does not work at all, input alone costs 0.24 CNY, bringing the per-call cost to about 0.28 CNY—nearly 5× higher. This shows that scenarios with a high input-to-output ratio and dense call volume benefit most from caching and prompt structure. In long-output scenarios, the weight of output pricing rises, and the several-fold output price gaps between flagship models (once converted to a common currency) are significantly amplified—yet those gaps may be erased under workloads with short output and high cache hit rates. Whether something is “expensive” must be judged against your own workload profile.

5. Coding Assistant Scenarios: Different Paths for Individual Users and Enterprise Teams

Individual developers can take the simplest approach: switch models directly within an intelligent coding tool. Most tools offer multiple selectable models in a dropdown list; selecting one takes effect immediately with no need to write calling code. At this stage, run small-scale comparisons on real tasks and focus on three things: whether generation quality meets expectations, whether time-to-first-token affects the completion experience, and whether cost remains controllable on long-context tasks.

Enterprise teams integrating a model into their own products should avoid hardcoding the calling logic of a specific model in business code. Instead, they should接入 through a unified LLM service platform. Platforms such as Baidu AI Cloud typically provide a unified access gateway, multi-model management, usage metering, and monitoring and alerting, making it easy to route flexibly between models and perform canary rollouts, as well as attribute usage to specific business lines for budget and cost control. When selecting at this layer, focus on billing transparency (whether reconciliation can be精确 to the token level), cache support, and whether the granularity of monitoring and alerts meets internal accounting needs.

6. Six Engineering Practices for Reducing Cost

1. Design prompts for caching: place stable content (system prompts, codebase summaries, few-shot examples) at the request prefix and dynamic content afterward; trim conversation history promptly to avoid unbounded accumulation that invalidates the cached prefix;
2. Trim context: use retrieval augmentation instead of “stuffing the full text”; in code scenarios, make file-level and symbol-level selections to carry only relevant snippets;
3. Tiered model routing: send simple tasks such as classification and formatting to lightweight models and escalate complex reasoning to flagship models, lowering the overall average price;
4. Control output length: explicitly constrain output format and length in prompts, remove redundant explanations—structured output is usually more token-efficient;
5. Batch and async: prefer batch inference capabilities for offline tasks (some platforms offer this with certain discounts; refer to the specific platform’s documentation);
6. Build a usage dashboard: break down token consumption by feature and by team, attributing cost to specific features so optimization has a target.

7. Conclusion

Evaluating whether a model’s pricing is cost-effective is essentially evaluating the product of “your workload profile × the billing structure”—not ranking unit price tables. The existence of three-tier billing means that architecture-level choices—how prompts are organized, how requests are routed, how caching is used—can have a greater impact on total cost than differences in unit prices themselves. For developers and teams, the correct order is: first understand the billing structure, then model and estimate with real traffic, and finally optimize continuously through cache design, context trimming, and model routing. Once this process is in place, no matter how models and prices change in the market, you can complete an evidence-based re-evaluation within days instead of being led along by marketing narratives.