Large Language Model API Selection: How to Evaluate Cost and Performance

1. Why a Higher Price per Token Does Not Always Mean Higher Costs

Model prices are typically listed per million tokens, with separate rates for input and output. But the actual cost of a business task depends on how many tokens it takes to complete and what retries or human reviews are needed when a response is unusable.

The cost of a single call can be expressed as:

Cost per call = input tokens × input price + output tokens × output price

This formula captures only the cost of one request. If a model produces a usable result on the first try, a higher per-token price may still be cheaper overall than a lower-priced model that requires repeated adjustments. When comparing models, focus on the cost to complete a task, not just the price per million tokens.

2. Four Variables That Affect Cost

1. Input and output pricing

Output tokens are often priced higher than input tokens. This means that generating 1,000 tokens can cost more than processing 1,000 tokens. For tasks that produce long responses, output length can be the main driver of the bill.

2. Context windows and repeated input

A model’s context window determines how much content it can accept in one request. A larger window does not mean you need to fill it. If every request includes the same long document, those input tokens may be billed repeatedly.

Common ways to reduce unnecessary input include:

• Retrieve only the passages relevant to the question before sending them to the model.
• Summarize a long document first, then use the summary as context for follow-up requests.
• If the provider supports prefix caching, place fixed system instructions and documents at the beginning of the request to reduce repeated computation.

Longer contexts can also increase processing time, which should be included in latency comparisons.

3. Maximum and actual output length

The maximum output length in API documentation is a limit, not a prediction of actual usage. Base cost estimates on average output token counts and percentile distributions from production logs, rather than on the maximum. You can also reduce output variability by specifying a response format and setting limits for individual fields.

4. Output speed and concurrency costs

Output speed is usually measured in tokens per second. It affects both how long users wait and how many concurrent requests a server can handle. Slower generation means each instance can process fewer requests in a given period, potentially requiring more compute resources during peak demand.

End-to-end latency can be estimated as follows:

def estimated_latency(first_token_s: float,
                      output_tokens: int,
                      tokens_per_s: float) -> float:
    """End-to-end latency = time to first token + output tokens / generation speed"""
    if tokens_per_s <= 0:
        raise ValueError("tokens_per_s must be greater than 0")
    return first_token_s + output_tokens / tokens_per_s

In production, record time to first token, generation speed, and output length separately. Use the formula to estimate latency distributions across different requests.

3. Establish a Consistent Evaluation Method

Comparisons are meaningful only when models are tested under the same conditions. Use a consistent evaluation process for all candidates.

1. Build a fixed test set

Sample real tasks from your use case. A test set of at least 200 examples is recommended, covering three categories:

• Simple tasks, such as classification and key-field extraction
• Complex tasks, such as multi-step reasoning and long-document summarization
• Edge cases, such as malformed input or missing information

Once established, keep the test set fixed so each evaluation uses the same examples.

2. Define effective cost per task

Looking only at the cost of a single call understates the impact of failures. A more useful measure accounts for retries or human review. If a failed task is retried until it succeeds, the expected cost per task is:

Effective cost = cost per call ÷ first-pass success rate + review cost per failure × (1 ÷ first-pass success rate − 1)

The first-pass success rate is the share of tasks that produce a usable result on the initial call.

The following code calculates this measure so you can compare multiple candidate models:

from dataclasses import dataclass
@dataclass
class ModelProfile:
    name: str
    input_price: float    # per million input tokens, based on provider pricing
    output_price: float   # per million output tokens, based on provider pricing
    pass_rate: float      # first-pass success rate, from 0 to 1
    p95_latency_s: float  # 95th-percentile end-to-end latency in seconds
def cost_per_call(p: ModelProfile, in_tok: int, out_tok: int) -> float:
    return (in_tok * p.input_price + out_tok * p.output_price) / 1_000_000
def effective_cost(p: ModelProfile, in_tok: int, out_tok: int,
                   review_cost: float = 0.0) -> float:
    """Assumes failed tasks are retried until success, with one review cost per failure."""
    if not 0 < p.pass_rate <= 1:
        raise ValueError(f"{p.name} pass rate must be between 0 and 1")
    base = cost_per_call(p, in_tok, out_tok)
    expected_calls = 1 / p.pass_rate
    expected_failures = expected_calls - 1
    return base * expected_calls + review_cost * expected_failures

3. Interpret results using illustrative values

The following values demonstrate how the calculation works. They do not represent the actual prices or performance of any model.

candidates = [
    ModelProfile("Model A", input_price=2.0, output_price=8.0,
                 pass_rate=0.95, p95_latency_s=6.0),
    ModelProfile("Model B", input_price=1.0, output_price=4.0,
                 pass_rate=0.70, p95_latency_s=4.0),
]
for m in candidates:
    c = effective_cost(m, in_tok=3000, out_tok=800, review_cost=0.5)
    print(f"{m.name}: effective cost {c:.4f}, P95 latency {m.p95_latency_s} seconds")

In this example, Model B costs less per call but has a lower first-pass success rate. With a human review cost of 0.5 per failure, its effective cost is substantially higher than Model A’s. When review costs greatly exceed model call costs, first-pass success rate can become the dominant factor. This conclusion applies only to the illustrative values above; use your own test set to make a real selection.

4. Do Not Skip Quality Checks

Cost estimates are meaningful only if the output is usable. Quality checks should have two layers:

• Automated validation: Check structured output for valid formats, such as parseable JSON, complete fields, and values within acceptable ranges. For text output, use keyword checks or other rules.
• Human review: Have subject-matter experts score a fixed sample of results, and periodically recalibrate automated rules.

After any model version update or prompt change, rerun the test set and compare changes in success rate and cost. This can be incorporated into your release process as regression testing.

5. A Practical Model Selection Workflow

1. Define the task type. Distinguish among classification, extraction, generation, and reasoning, since each task has different capability and output-length requirements.
2. Build a test set and fix evaluation conditions. Version the samples, prompts, parameter settings, and scoring criteria.
3. Standardize cost calculations. Apply the same effective-cost formula to every candidate, and include latency percentiles in the comparison.
4. Run a small canary test. Route a small share of real traffic to the candidate model, then compare its production success rate and latency with offline results.
5. Set cost and latency alerts. Monitor daily spend, average tokens per task, and P95 latency. Investigate anomalies promptly.
6. Review regularly. Prices, rate limits, and model versions can change, so rerun the evaluation at least quarterly.

6. Common Mistakes

• Comparing only per-token prices: This ignores output length and the cost of retries, and can make a seemingly cheap option more expensive in practice.
• Focusing only on context-window size: A larger window does not need to be filled. Sending unnecessary long passages increases input costs.
• Treating a few examples as representative: Small samples can exaggerate random variation in success rates. Use a sufficiently large test set.
• Ignoring output-length variation: Averages can hide long-tail requests. Track P95 or P99 as well.
• Ignoring concurrency and rate limits: If throughput constraints are excluded from the cost model, real-world availability and costs during peak periods may differ from estimates.

Key Takeaways

To compare the value of LLM APIs, measure the effective cost of completing a task rather than focusing only on the listed price per token. Evaluate cost alongside latency and output quality. A fixed test set, consistent calculations, and canary testing make model selection more reproducible and easier to audit.