How to Evaluate and Choose a Low-Cost LLM
The cheapest LLM per token is not necessarily the cheapest model to run in production. This guide explains how to compare token pricing, cache performance, latency, throughput, and task quality using real workloads. It also outlines a testing process and shows how cost per successful task can inform model selection and routing.
1. Look Beyond the Price per Token
A new generation of large language models (LLMs) emphasizes low cost, high throughput, and long context windows. Many use sparse activation: the model has a large total parameter count, but only a subset of those parameters is active during inference. These models may also support multimodal inputs, very long contexts, or prompt caching.
Attractive input and output rates tell you only part of what a model will cost in production. A useful comparison accounts for:
- Input and output prices per token
- The actual ratio of input to output tokens
- The context cache hit rate
- Average response length
- Time to first token and end-to-end response time
- Concurrency limits and rate-limiting policies
- Task accuracy and the cost of human review
- Whether capabilities such as multimodal input and long context actually meet business needs
The key question is not which model charges less per token, but which model delivers an acceptable result at the lowest cost.
2. Understand Input, Output, and Cache Pricing
LLM services typically bill by the token. Input tokens include prompts, conversation history, documents, and tool results. Output tokens are the content the model generates.
A simple cost estimate is:
Total cost =Input tokens × input price per token+ Output tokens × output price per token- Cache savings
If a request includes 100,000 tokens of fixed instructions, knowledge base material, or document text, while the user’s question is only a short paragraph, the cache hit rate can have a significant effect on the final cost.
Repeated content is not automatically free. The savings depend on:
- Whether the repeated content is in a cacheable part of the request
- Whether the request prefix stays consistent across calls
- How long cached content remains valid
- How the provider bills cached input tokens
- Whether dynamic content prevents a cache hit
In customer support, code review, long-document question answering, and multi-turn assistants, the full input can be much larger than the user’s visible message. In these workloads, cached input pricing may matter more than the standard input rate.
For summarization, short-form rewriting, and simple classification with brief inputs, output can account for a larger share of the bill. In those cases, pay closer attention to the output rate and average generation length.
3. Fewer Active Parameters Do Not Guarantee Lower Total Cost
Sparse models have two distinct measures of size: total parameters and active parameters. The total count reflects the model’s overall capacity, while the active count is more closely related to how much computation the model uses during inference. All else being equal, activating fewer parameters can reduce compute costs and improve throughput.
Neither count, however, determines how well a model will perform. Quality also depends on:
- The scale and quality of the training data
- The data mix and deduplication strategy
- Coverage of domain-specific material
- Routing during inference
- Alignment training and instruction following
- Attention efficiency over long contexts
- Multimodal encoders and how information is combined across modalities
- Whether the model uses additional reasoning or planning during inference
A model with 12 billion active parameters is not necessarily worse than one with 49 billion. For structured extraction, code completion, and text classification, a smaller active parameter count may be sufficient. For complex mathematical reasoning, multi-step planning, specialized analysis, and synthesis across documents, training quality, inference strategy, and context handling may matter more.
4. Put Claims of an Improved Model Version in Context
When a provider says a new version is slightly better overall than its predecessor, ask three questions before treating the claim as a reason to switch.
Which tasks does “overall performance” cover?
Aggregate scores usually combine several benchmarks, but those benchmarks may not reflect your workload. A model can excel at knowledge questions while offering no advantage in code changes, table interpretation, or long-document retrieval.
Evaluate the tasks your business actually needs the model to perform:
- Extract key clauses from contracts accurately
- Generate database queries in a required format
- Identify edge cases in code
- Locate the right supporting evidence in a long document
- Decline to answer when the available information is insufficient
- Consistently produce structured outputs that satisfy interface constraints
Is the improvement larger than normal test variation?
If two models score close to each other, repeat the evaluation to see whether the gap holds up. Track average and lowest scores, task success rates, format compliance, hallucination rates, and stability across different prompts and temperature settings.
Were the test conditions consistent?
For each head-to-head comparison, keep the system prompt, input documents, maximum output length, temperature, tool access, timeouts, retry behavior, and scoring rules consistent. Otherwise, a difference attributed to the model may come from the test configuration.
5. Compare Prices Using Workload Profiles
Pricing periods, discounts, and caching rules can differ by provider. Promotional rates can make a short trial cheaper, but they should not be the basis for a long-term architecture decision. Instead, estimate costs for representative workloads.
Short Questions and Answers
When both inputs and outputs are brief, caching is unlikely to drive the decision. Compare average cost per request, time to first token, throughput under concurrent load, and accuracy on straightforward questions.
Long-Document Question Answering
Sending large documents with each request can quickly increase input costs. Evaluate cache hit rates, recall across long contexts, and how document chunking and retrieval affect results. Check whether the model overlooks relevant passages in the middle of a document, and account for the cost of retrying a failed request.
Code Generation and Review
Coding tasks often involve long outputs and multiple turns. In addition to output pricing, check whether generated code compiles, handles exceptions and edge cases, preserves existing behavior, and follows project constraints. During several rounds of fixes, see whether the model loses track of earlier context.
Multimodal Processing
Images, tables, and scanned documents may be represented as additional input tokens or separate visual features. Do not estimate these workloads using text-only rates. Measure the cost of the complete request flow.
6. Measure Cost per Successful Task
A more useful production metric than nominal price per request is:
Cost per successful task =Total inference cost ÷ Number of successfully completed tasks
Suppose Model A costs less per request but succeeds 70% of the time, while Model B costs more per request but succeeds 90% of the time. If failed tasks require human rework, Model B may be cheaper overall.
For a broader comparison, include the costs that arise outside the initial model call:
Total operating cost =Model call costs+ Retry costs for failed requests+ Human review costs+ Business losses caused by errors
An awkward phrase in marketing copy may only need a quick edit. An error in finance, legal work, code deployment, or data processing can cost far more than the model call itself. The acceptable trade-off between cost and accuracy depends on the risk of the task.
7. Run a Repeatable Evaluation
Step 1: Collect representative samples
Do not rely solely on public benchmarks. Use de-identified requests from production logs and include common tasks, less frequent cases, and unusual or malformed inputs.
Step 2: Group requests by task
Useful categories include knowledge question answering, information extraction, content generation, coding, long-document analysis, multimodal understanding, tool calling, and appropriate safety refusals.
Step 3: Define automated metrics
Choose measures appropriate to each task, such as accuracy, recall, format compliance, compilation success, citation correctness, hallucination rate, and average output length. Track operational metrics as well: time to first token, end-to-end response time, and cost per 1,000 requests.
Step 4: Repeat the tests
Run the same requests multiple times to observe variation between runs. In production, consistent performance often matters more than the best score from a single run.
Step 5: Roll out to a small share of traffic
Start with low-risk workloads where you can roll back easily. Observe actual costs, failure patterns, and user feedback before increasing traffic.
8. Choose the Model That Fits the Workload
A model with favorable input, output, and cache pricing can be a good fit for high-volume calls, long-text processing, and applications with high concurrency, provided it meets the accuracy requirements of the target tasks.
For complex reasoning, code generation with strict correctness requirements, or high-risk decisions, a lower price is not enough to justify replacing the existing model. A tiered routing strategy can offer a better balance:
- Use a low-cost model for simple classification, rewriting, and summarization.
- Route complex reasoning and high-risk tasks to a more capable model.
- Add validation, retrieval, or human review for high-value requests.
- Use caching, batching, and prompt compression to reduce overall costs.
Base the decision on consistent test conditions, representative workloads, full cost accounting, and performance observed over time. The best model for a workload is rarely the one with the most parameters or the lowest listed price. It is the one that delivers the right balance of quality, latency, cost, operational control, and fit with the rest of the system.
