LLM Cost-Effectiveness: Performance, Token Efficiency, and Cost Optimization

Introduction: Do Not Choose LLMs by Per-Million-Token Price Alone

When many teams evaluate APIs for large language model (LLM) calls, their first instinct is to compare the quoted price per million tokens. The conclusion is clear: looking at unit price alone can easily misjudge the real cost.

What should be compared is not “how much does 1,000 tokens cost,” but “how much does it cost to complete one successful task?”

The actual bill for LLM calls is usually determined by several factors working together:

• First-call success rate
• Token efficiency
• Retry count
• Context length
• Human fallback cost
• Billing model

A model with a lower unit price but a high failure rate, verbose outputs, frequent retries, or the need for repeated human correction may actually end up with a higher total cost.

This article presents an engineering-executable evaluation framework: first filter for performance, then calculate token efficiency, then apply engineering cost-reduction methods, and finally lock down the budget based on the billing model.

1. First Ask Whether Performance Meets the Threshold: The Denominator of Cost-Effectiveness Is Effective Output

Cost-effectiveness is not “price per token.” It is “the cost paid per unit of effective output.”

Therefore, the first step in evaluating LLM cost-effectiveness is not to list prices, but to confirm whether performance meets the task threshold.

Here, performance does not mean a vague claim that a model is “strong.” It means an acceptable task success rate that can be verified. For example:

• Code generation tasks: Does the output pass unit tests?
• Structured output tasks: Is the JSON valid, and are the required fields complete?
• Question-answering tasks: Do the answers meet manual sampling standards?
• Agent tasks: Are tool calls correct, and is the task ultimately completed?

If a model has a low first-call success rate, it will require more retries, longer contexts, and more human correction. These extra actions can quickly erase any unit-price advantage.

The reasonable order is therefore:

1. First filter out candidate models whose performance meets the threshold.
2. Then compare prices.

Putting clearly unqualified models into a unit-price comparison table creates a flawed comparison baseline from the start.

2. Token Efficiency: Convert Unit Price into Cost per Successful Task

Once a candidate set has been established, the core metric should shift from “unit price” to “total tokens needed to complete a task.”

Token consumption for one task usually includes three categories:

1. Input tokens: Prompts, system instructions, conversation history, retrieval snippets, and tool return results.
2. Output tokens: Generated answers, function call parameters, and JSON content.
3. Retry tokens: Repeated calls caused by format errors, timeouts, or unacceptable content.

The simplified calculation function below first assumes that input and output tokens have the same unit price. Note: all unit prices in the examples are assumed values and do not represent real quotes.

def task_cost_effectiveness(unit_price, tokens_used, success_rate, retry_factor=1.0):
    """
    Simplified calculation: input and output tokens have the same unit price.
    unit_price: price per 1,000 tokens (values in the example are assumptions, not real quotes)
    tokens_used: average tokens consumed by one task (input + output)
    success_rate: first-call success rate, range 0~1
    retry_factor: additional retry amplification unrelated to success rate
                  (timeouts, rate limiting, link jitter, etc.), default 1.0;
                  retries caused by failures are already covered by 1 / success_rate.
    Returns: cost per successful task (CNY/successful task).
    """
    if success_rate <= 0:
        return float("inf")
    retry_factor = max(1.0, retry_factor)
    effective_calls = retry_factor / success_rate
    effective_tokens = tokens_used * effective_calls
    return unit_price * effective_tokens / 1000.0
model_a = task_cost_effectiveness(unit_price=0.03, tokens_used=800, success_rate=0.95)
model_b = task_cost_effectiveness(unit_price=0.015, tokens_used=1000, success_rate=0.5)

The calculations are as follows:

• Model A: 0.03 × 800 ÷ 0.95 ÷ 1,000 ≈ 0.02526 CNY/successful task.
• Model B: 0.015 × 1000 ÷ 0.5 ÷ 1000 = 0.03000 CNY/successful task.

Therefore, although Model B has a lower unit price, its lower first-call success rate makes the effective cost per successful task higher.

If input and output token prices are different, the calculation needs to be split further. After input and output are priced separately, the formula becomes:

Effective model cost = (input tokens × input unit price + output tokens × output unit price) × expected number of calls ÷ 1,000.

Expected number of calls = (1 ÷ first-call success rate) × retry_factor.

If the task also involves human fallback, human cost can be added:

Human cost = human intervention rate × (1 − first-call success rate) ÷ (1 − retry_factor × first-call success rate) × cost per intervention.

When the denominator is less than or equal to 0, it means that after combining failed retries with additional link instability, the cost can no longer be extrapolated using a simple linear formula. In that case, a separate fault-tree analysis should be performed, or the worst case should be estimated.

def estimate_task_cost(
    unit_price_in,
    unit_price_out,
    tokens_in,
    tokens_out,
    success_rate,
    retry_factor=1.0,
    human_intervention_rate=0.0,
    human_cost_per_intervention=0.0,
):
    """
    Returns: cost per successful task (CNY/successful task).
    Input and output tokens are priced separately.
    success_rate: first-call success rate.
    retry_factor: additional retry amplification unrelated to success rate, default 1.0;
                  retries caused by failures are already covered by 1 / success_rate.
    Human cost formula:
        human intervention rate
        × (1 - first-call success rate)
        ÷ (1 - retry_factor × first-call success rate)
        × cost per intervention.
    If the denominator is less than or equal to 0, the link is considered unstable
    and requires separate evaluation.
    """
    if success_rate <= 0:
        return float("inf")
    retry_factor = max(1.0, retry_factor)
    effective_calls = retry_factor / success_rate
    model_cost = (
        tokens_in * unit_price_in + tokens_out * unit_price_out
    ) * effective_calls / 1000.0
    human_cost = 0.0
    if human_intervention_rate > 0 and human_cost_per_intervention > 0:
        denom = 1.0 - retry_factor * success_rate
        if denom > 0:
            human_cost = (
                human_intervention_rate
                * (1.0 - success_rate)
                / denom
                * human_cost_per_intervention
            )
        else:
            # Conservative handling: count at least one human intervention
            # and trigger a link-stability review.
            human_cost = human_intervention_rate * human_cost_per_intervention
    return model_cost + human_cost

It is important to note that the formulas above assume each retry is independent. If failures are concentrated on specific inputs, such as fixed-format JSON parsing errors, retries may not save the task. In this case, effective cost should be measured separately by failure type, and retries should be given clear limits and timeout budgets.

3. Engineering Cost-Reduction Methods: What Can Directly Lower the Bill

Selecting the right model and billing model alone may not be enough to reduce costs. Engineering practices can also reduce spending through call patterns, context management, and routing strategies.

Cost-reduction method	Implementation	Suitable scenarios	Expected benefit
Result caching	Cache results for identical requests; use semantic caching for similar requests	High-frequency repeated questions, customer support, knowledge-base Q&A	Reduce repeated token consumption
Batch or asynchronous APIs	Move non-real-time tasks to batch, offline, or asynchronous processing	Data labeling, summarization, report generation	Obtain discounts and reduce peak load
Context trimming	Keep only necessary history; use summaries instead of long conversations	Multi-turn dialogue, long-document Q&A	Reduce input tokens and latency
Prefix caching	Reuse stable prefixes such as system prompts and knowledge-base snippets	Agents, fixed prompts, RAG	Reduce repeated input costs
Tiered routing	Send simple tasks to cheaper models and complex tasks to stronger models	Mixed-traffic workloads	Avoid using high-cost models for all tasks
Streaming output	Return results progressively and stop invalid output from continuing to generate	Long-text generation, code completion	Reduce invalid tokens and user waiting time
Timeout and retry limits	Set maximum retry counts, timeout limits, and degradation strategies	APIs, agents, tool calls	Prevent a single failure from becoming a billing incident

Among these, caching, context trimming, and tiered routing usually provide the most obvious benefits.

• Caching is suitable for repeated traffic.
• Context trimming is suitable for long conversations and RAG.
• Tiered routing is suitable for systems where task complexity varies significantly.

4. API and Agent Scenarios: Medium-to-Heavy Tasks Amplify Performance Differences

Ordinary chat may be a lightweight task, but API integration and agent scenarios are often medium-to-heavy tasks. They are more sensitive to model performance, token efficiency, and cost structure for three reasons:

1. Multi-step reasoning and tool calls: One agent task may involve dozens of model calls. Every call adds more tokens.
2. Long-context maintenance: To keep the task consistent, conversation history, tool return results, and retrieval content are continuously added to the context.
3. Structured output requirements: JSON, function call parameters, and tool protocols must be strictly correct. Otherwise, parsing failures can break the entire pipeline.

In agent scenarios, failures accumulate along the pipeline. If the success rate of a single step is p, and the task requires n steps, the end-to-end task success rate is approximately p^n.

Therefore, agent model selection cannot focus only on single-call success rate. It must also consider end-to-end task success rate.

As the number of calls increases, performance differences are amplified. A model with higher single-step accuracy often reduces retries, shortens context, and lowers human intervention, thereby reducing overall cost.

5. Billing Models: Subscriptions vs. Pay-as-You-Go Depends on Usage Predictability

In addition to the model itself, the billing model can also materially affect cost. There are two common types:

Billing model	Suitable scenarios	Advantages	Risks
Pay-as-you-go	Highly variable usage, early validation, low-frequency calls	No prepayment, flexible, pay only for what you use	Higher unit price at large scale and fluctuating bills
Subscription or resource package	Stable usage, predictable consumption	Lower unit price and predictable cost	Waste if over-purchased and less flexibility during the commitment period

Decision recommendations:

1. First use pay-as-you-go to validate the business and collect real usage data, including daily call volume, QPS, and input/output token distribution.
2. Once usage is stable and can cover the subscription threshold, switch to a subscription or resource package.
3. Most mainstream model service platforms typically provide usage statistics, logs, and alerting capabilities. It is recommended to establish usage observability first, then decide on the billing strategy.

In addition, regression test samples and prompts may contain business data or user data. Before sending them to a model API, follow the principle of minimum necessity, complete anonymization or data masking, and pay attention to data retention, regional compliance, and the data policies of third-party model services.

6. A Practical Evaluation Process: Build a Closed-Loop Selection System in Six Steps

It is better to turn LLM cost-effectiveness evaluation into a repeatable process rather than relying on experience alone.

1. Define the task and acceptance criteria: Clarify what counts as one successful output, such as passing unit tests, producing valid JSON, or meeting manual sampling standards.
2. Build a business regression test set: Use real business samples, not generic leaderboards.
3. Collect key metrics: Record first-call success rate, average token consumption, average latency, and retry amplification factor.
4. Calculate effective cost: Convert unit price into cost per successful task.
5. Layer in the billing model: Combine usage predictability to estimate the monthly budget.
6. Establish a continuous optimization loop: Model versions, prompts, context strategies, and prices all change, so regular regression is required.

The recommended definition of first-call success rate is:

First-call success rate = number of tasks that pass acceptance criteria on the first call ÷ total number of tasks.

For single-turn tasks, single-call success rate can be used. For multi-turn or agent tasks, end-to-end task success rate should be used.

It is recommended to use at least 200 real samples for each candidate model and provide confidence intervals to avoid misjudgment caused by samples that are too small.

The evaluation record table can be structured as follows:

Model	Input tokens	Output tokens	First-call success rate	Input unit price	Output unit price	Effective cost	Average latency	Notes
Model A	600	200	0.95	Assumed value	Assumed value	0.02526	To be collected	Single-call metric
Model B	700	300	0.50	Assumed value	Assumed value	0.03000	To be collected	Failure reasons need attention

7. FAQ: Common Questions About LLM Cost Optimization

How should an LLM be selected?
First define acceptance criteria, then filter models whose performance meets the threshold, and finally calculate effective cost using a business regression set. Do not compare unit prices directly.

How is API call cost calculated?
It should include at least input tokens, output tokens, retry tokens, and human fallback cost. If input and output prices are different, they should be priced separately.

How should agents be selected?
Focus on end-to-end task success rate, tool-call correctness, average number of steps, context length, and failure accumulation effects.

What are common mistakes when looking only at unit price?
Ignoring first-call success rate, retry tokens, human intervention cost, and the amplification effects caused by long contexts and multi-step agent calls.

How should pay-as-you-go and subscriptions be chosen?
Use pay-as-you-go during early validation or when usage is highly variable. Consider subscriptions or resource packages when usage is stable, predictable, and reaches the subscription threshold.

What factors can cause cost to be underestimated?
Human fixes, repeated retries, long contexts, tool failures, structured output errors, model version drift, and additional calls caused by latency and rate limiting.

Conclusion: Make Decisions with a Quick-Reference Table

LLM cost-effectiveness is not just a unit-price comparison table. It is a composite function determined by performance, token efficiency, task complexity, engineering optimization, and billing model.

Metric	Definition	Purpose
First-call success rate	Percentage of tasks that pass acceptance criteria on the first call	Determine whether the model is usable
End-to-end task success rate	Percentage of multi-step tasks that are ultimately completed	Determine whether an agent is reliable
Token efficiency	Average token consumption per unit task	Estimate bill size
Effective cost	Total cost per successful task	Compare models horizontally
Human cost	Extra spending caused by human fallback	Avoid underestimating total cost

The rational path is:

1. Ensure performance meets the threshold.
2. Use token efficiency to reduce effective cost.
3. Use caching, batch processing, context trimming, and tiered routing for engineering optimization.
4. Use an appropriate billing model to lock down the budget.

For teams working on API integration and agent systems, making this evaluation process repeatable provides a more long-term and controllable cost advantage than chasing a single low price.