How to Evaluate a Newly Released LLM: Starting with API Pricing

New foundation models are being released at an increasingly rapid pace, and “official version now available” has almost become monthly industry news. For developers and technical decision-makers, benchmark rankings are important, but what usually determines whether a model can enter production comes down to more practical questions: How does API billing work? How much does a single conversation actually cost? How much can caching save? And how do different roles integrate the model?

This article moves beyond reviewing a single product and instead breaks down “how to evaluate a new model” into a reusable analysis framework, with API pricing and cost as the central focus.

Three billing tiers: input, output, and cache hits

Most mainstream LLM APIs charge by token, and pricing is typically divided into three tiers that share the same unit of measurement, usually per million tokens:

• Input price: the cost of the tokens the model reads, covering the system prompt, conversation history, retrieved documents, tool definitions, and all other context.
• Output price: the cost of the tokens the model generates.
• Cache hit price: when the prompt prefix of a request exactly matches a recent request and the provider can reuse previously computed intermediate results, the repeated prefix is billed at a much lower rate than the standard input price.

Why is output more expensive than input? This is determined by the inference characteristics of autoregressive architectures. The input phase, also known as prefill, can be processed in parallel with high compute utilization. The output phase, also known as decode, must generate tokens one by one. Every generated token requires a complete forward pass, and the bottleneck is memory bandwidth rather than raw compute, making it a typical memory-bound workload. As a result, the service cost per token is significantly higher for output. In common industry pricing, output rates are roughly 2 to 5 times input rates.

Why can caching be an order of magnitude cheaper? In multi-turn conversations, fixed system prompts, RAG retrieval, and agent tool definitions, prompt prefixes are highly repetitive. Providers retain the computed KV cache for a period of time. When a request hits the cache, the repeated prefill computation is skipped, and the saved compute is passed back to the caller as a discount. Cache hit pricing is often only a fraction, or even a few percent, of the standard input price. For context-heavy applications with stable prefixes, this tier is often the real cost-saving lever in the final bill.

Build your own cost estimation model

Unit price is only one part of the picture. What developers really need to know is how much their application will cost per month. The minimum estimation formula is:

Cost per call = input tokens × input unit price + output tokens × output unit price

When caching is involved, the input portion must be split into cached and non-cached segments and priced separately. The following sample code estimates cumulative cost for a multi-turn conversation:

def estimate_cost(turns, sys_tokens, user_per_turn, reply_per_turn,
                  p_in, p_out, p_cache=0.0, hit_ratio=0.0):
    """Estimate cumulative multi-turn conversation cost by turn.
    p_in / p_out / p_cache: unit price per million tokens.
    hit_ratio: prefix cache hit rate (0~1)."""
    cost, history = 0.0, 0
    for _ in range(turns):
        prompt = sys_tokens + history + user_per_turn  # full context is resent each turn
        hit = prompt * hit_ratio
        cost += ((prompt - hit) * p_in + hit * p_cache + reply_per_turn * p_out) / 1e6
        history += user_per_turn + reply_per_turn
    return cost
# Example: 10 turns, 2000-token system prompt, 300 user input tokens and 500 output tokens per turn
print(estimate_cost(10, 2000, 300, 500, p_in=2, p_out=8, p_cache=0.2, hit_ratio=0.8))

This code reveals a commonly overlooked fact: multi-turn conversation costs do not grow linearly. Because the full conversation history must be resent as input in every turn, cumulative input tokens grow roughly quadratically with the number of turns. The deeper the conversation and the longer the context, the greater the impact of cache hit ratio on the final bill. Placing stable content such as system prompts and tool definitions at the beginning of the prompt and dynamic content later is a basic technique for improving cache hit rates.

Four common mistakes when comparing prices across models

1. Currency and units are not aligned. Models from different providers may be priced in different currencies. After currency conversion, the price gap between models can be several times larger or smaller than it first appears. Always normalize to the same unit, such as per million tokens, before comparing.
2. Lower unit price does not mean lower total cost. If one model requires longer outputs, more retries, or additional clarification turns to complete the same task, the actual overall cost may exceed that of a model with a higher sticker price. A more reliable approach is to select a set of representative real-world tasks and compare task-level completion costs.
3. Discount mechanisms are ignored. Batch APIs, non-real-time tasks, cache hits, and tiered volume discounts can significantly change the effective unit price. Evaluation should be based on the real price after deployment, not the public list price.
4. Only looking at the first monthly bill. During prototyping, call volume is small and contexts are short, so the bill is naturally low. After entering production, context length expands and concurrency increases, and the cost structure may change completely. Estimates should be extrapolated to production scale.

Integration paths for different roles

The same set of models can have very different adoption barriers depending on the user’s role:

• General users: switch models directly in the model selector of a mainstream AI chat product or coding assistant. Authentication, billing, and context management are all handled by the product, and users only need to focus on output quality.
• Application developers: obtain an API key through a model platform or cloud-based model service, then replace the endpoint and model name to migrate. Key areas to evaluate include streaming output, timeout handling, retry behavior, error codes, and usage query APIs.
• Enterprise users: also need to assess private deployment, dedicated compute, audit and compliance capabilities, and build a complete inference pipeline using standard cloud components such as logging services, monitoring, and alerting.

Cloud platforms typically package model access, usage tracking, monitoring, alerting, and content safety into a unified model service. For teams without in-house infrastructure capabilities, this can significantly reduce integration and operational overhead.

Practical checklist for model selection and cost reduction

1. Task-based routing: send complex reasoning tasks to flagship models, while routing lightweight tasks such as classification, extraction, and formatting to smaller models. Overall cost can drop by an order of magnitude.
2. Compress output: explicitly require concise responses or structured output such as JSON in the prompt. Since output token pricing is higher, compressing output yields immediate cost savings.
3. Maximize cache hits: place stable content first and dynamic content later. Reuse the same prefix template for fixed workflows to keep the hit rate consistently high.
4. Establish usage observability: tag usage by business line, configure budgets and anomaly alerts, and prevent issues such as infinite retry loops from unexpectedly driving up the bill.
5. Maintain the ability to switch models: wrap model calls behind a unified abstraction layer to support rapid switching between multiple models. This avoids vendor lock-in and leaves room for low-cost evaluation of future models.

Conclusion

When evaluating a newly released model, benchmark scores are only the starting point. Understanding the inference cost logic behind input, output, and cache-hit pricing, building your own estimation model, and combining that with integration requirements and migration costs are what enable rational, verifiable, and reversible technical decisions in a fast-moving model landscape.