LLM Evaluation in Practice: How to Trade Off Accuracy, Latency, Token Usage, and Cost

    A practical guide to evaluating large language models across four dimensions: accuracy, latency, token consumption, and inference cost. Using an illustrative generation-to-generation comparison, it shows how to break results down by domain to spot regressions, how to build a reproducible cost formula from token usage, how to compare models within the same cost tier, and how to turn evaluation into an engineering process with monitoring and cost-regression alerts. It also answers common questions on model selection, version upgrades, and cost reduction.

    1. Why Accuracy Alone Is Not Enough

    In model selection and version upgrades for large language models, the most common mistake is “accuracy above all”: looking at a single leaderboard score and using it to decide whether to ship or upgrade. Production environments are never a single-metric contest. They are a joint trade-off among accuracy, latency, token consumption, and cost.

    A model with slightly higher accuracy but slower responses and heavier token usage is not necessarily the better choice in cost-sensitive or latency-sensitive workloads. Conversely, a model with slightly lower accuracy that is stable, cheap, and fast may be a better fit for high-concurrency scenarios.

    A reusable evaluation framework should therefore cover at least four dimensions:

    • Accuracy: the share of correct answers, which reflects the model’s capability ceiling;
    • Response latency: the average duration of a single call, which directly affects user experience and throughput;
    • Token consumption: average input and output tokens per call, the hidden bill behind capability;
    • Inference cost: unit price combined with actual token consumption, converted into real spend at a fixed call volume.

    Only by putting all four metrics into one table and analyzing them together can you tell whether an upgrade is a pure gain or a trade-off gain that buys accuracy with cost and latency.

    2. How Do You Compare Versions Across Multiple Dimensions?

    The example evaluation case below illustrates the analysis framework. All figures are constructed illustrative values for explanatory purposes, not measured results from any specific product. Suppose a new-generation model is compared against the previous generation on the same Chinese-language comprehensive evaluation set of roughly 15,000 questions:

    Metric Previous generation New generation Change
    Overall accuracy 68.1% 71.5% +3.4pp
    Average latency 59s 96s +63%
    Average tokens per call 2,851 3,922 +37.6%
    Cost per 1,000 calls (simplified estimate) CNY 39.9 CNY 54.9 +37.6%

    Looking only at the first row, the upgrade appears positive. Looking at the remaining three rows together reveals that this is an upgrade that trades latency and cost for accuracy. The “cost per 1,000 calls” column here uses a simplified approach based on output price alone, in order to illustrate the parallel movement of token growth and cost growth. A real bill must price input and output separately and account for factors such as caching and discounts; the method is covered in section 3.1.

    The values such as 59s/96s and 2,851/3,922 are illustrative as well. Actual latency and token consumption depend on concurrency, input length distribution, region, decoding parameters, prompt templates, and tool-call chains. If you want to reproduce an evaluation, fix the test conditions and record the full context.

    2.1 Breaking Results Down by Domain: How Do You Locate Gains and Weak Spots?

    The 3.4-percentage-point gain in overall accuracy is not the result of uniform improvement across every domain. Splitting the evaluation set by capability dimension reveals a more detailed picture (illustrative data):

    • Reasoning and mathematics: 66.7% → 72.7%, +6.0pp
    • Education: 46.9% → 52.0%, +5.1pp
    • Healthcare and mental health: 77.6% → 83.6%, +6.0pp
    • Finance: 77.7% → 84.2%, +6.5pp
    • Language and instruction following: 64.6% → 69.3%, +4.7pp
    • Agent and tool calling: 69.9% → 68.1%, -1.8pp

    Broad improvement across the first five categories suggests the new version may have made systematic progress on knowledge question answering, reasoning and calculation, and specialized text tasks in finance and healthcare. The single regression, however, appears in tool calling and agent capability.

    This pattern of “improvement in many places, regression in one” is not unusual in model iteration. One possible explanation is that in order to handle harder problems, the model generates longer intermediate reasoning, which may crowd out the output budget reserved for strict format compliance, exact function signatures, and deterministic workflow execution. This, however, is a hypothesis based on illustrative data, not a causal conclusion established by measurement. If your workload depends on agent workflows, structured output, function calling, or strict JSON responses, re-test that dimension separately on your own evaluation set even if the overall score rises, and pay particular attention to format failure rate, tool-call success rate, and workflow interruption rate.

    3. How Do You See the Hidden Bill Behind Token Consumption?

    Another easily overlooked signal in the case is that average tokens rose from 2,851 to 3,922, an increase of roughly 37.6%. This is usually more than the model simply becoming verbose; it may reflect longer reasoning, more complex context handling, or longer tool-call chains.

    Token consumption matters because it pulls on two ends at once:

    1. The cost side: tokens are the core basis for billing;
    2. The latency side: more output tokens generally means longer autoregressive decoding, which is one important reason average latency increases.

    So an “improvement in reasoning capability” often shows up in the ledger simultaneously as higher token usage, higher latency, and higher cost. When planning capacity and budget, you cannot estimate only “unit price × number of calls”; average input and output tokens must be part of the model.

    3.1 How Do You Build a Reproducible Cost Model?

    A cost formula that prices input and output separately is preferable:

    1. Cost per 1,000 calls =
    2. (avg input tokens per call × input price + avg output tokens per call × output price) × 1000 / 1,000,000

    Unit prices are typically expressed as currency per million tokens.

    The corresponding code example:

    1. def cost_per_1k_calls(
    2. avg_input_tokens: int,
    3. avg_output_tokens: int,
    4. input_price: float,
    5. output_price: float,
    6. ) -> float:
    7. """
    8. Calculate the cost per 1,000 calls, in CNY.
    9. Price unit: CNY per million tokens.
    10. """
    11. return (
    12. avg_input_tokens * input_price
    13. + avg_output_tokens * output_price
    14. ) * 1000 / 1_000_000
    15. # Separate input/output pricing example:
    16. # 300 input tokens on average at an input price of CNY 1 per million tokens
    17. # 600 output tokens on average at an output price of CNY 14 per million tokens
    18. print(cost_per_1k_calls(300, 600, 1, 14)) # 8.7 CNY per 1,000 calls

    If you only care about scenarios where input and output prices differ but token volume is fixed, this formula is more accurate than “total tokens × output price”. With 300 input and 600 output tokens, the cost is:

    1. (300 × 1 + 600 × 14) × 1000 / 1,000,000 = 8.7 CNY

    In the example table in this article, a simplified approach based on output price alone was used in order to illustrate the relationship between token growth and cost growth:

    1. def simplified_cost_per_1k_calls(avg_total_tokens: int, output_price: float) -> float:
    2. return avg_total_tokens * output_price * 1000 / 1_000_000
    3. print(simplified_cost_per_1k_calls(2851, 14)) # ≈ 39.9
    4. print(simplified_cost_per_1k_calls(3922, 14)) # ≈ 54.9

    This simplification is fine for quick comparison but unsuitable for a formal budget, because:

    • If input tokens are also billed, actual cost may exceed the simplified estimate;
    • If cache hits, batch discounts, or tiered pricing apply, actual cost may fall below the simplified estimate;
    • If input and output unit prices differ substantially, the simplified approach may produce a noticeable deviation.

    Before a production launch, replace it with the real billing basis and include “input tokens, output tokens, input price, output price” in the evaluation record.

    4. How Do You Compare Models Horizontally at the Same Cost Level?

    Beyond vertical version comparison, horizontal model selection also needs a multi-dimensional view. The common approach is to bucket models by measured cost first, then compare accuracy and latency within the same cost bucket.

    Using the example data: one model costs roughly CNY 54.9 per 1,000 calls under the simplified approach, placing it in the “CNY 55 tier”; another costs roughly CNY 56 per 1,000 calls, which is close to the same tier. Both have overall accuracy of about 71.5%, but the first has average latency of 96s while the second has 72s. At this point, if accuracy is a tie, the decision tips toward the model with better latency.

    Recommended steps for same-cost-tier selection:

    1. First define cost tiers based on the business budget to narrow the candidate set;
    2. Within a tier, compare accuracy first and eliminate models that fall short on capability;
    3. When accuracy is close, use response latency as a secondary sort;
    4. Finally, make the decision based on performance in your specific domain.

    For example, if your business is financial question answering, you cannot look only at overall accuracy but must look at the finance dimension; if your business is an agent workflow, focus on tool calling, structured output, and workflow stability; if your business is high-concurrency customer support, latency and cost may matter more than accuracy differences beyond the decimal point.

    Also watch for language and domain bias in the evaluation set. A model that excels on Chinese-language general tasks is not necessarily equally strong in English, multilingual, or specialized vertical domains. The evaluation set should therefore stay as close as possible to your real workload distribution rather than relying on public leaderboards alone.

    5. How Do You Turn Evaluation into an Engineering Process?

    To make the above approach reusable, evaluation must move from a one-off experiment to an engineering process. At minimum, put these five practices in place:

    1. Fix the evaluation set and random seeds. Once the question set, prompt templates, decoding parameters, temperature, top_p, and maximum output length are set, version them so that vertical comparisons remain reproducible.

    2. Collect all four data points together. Log correctness, latency, input/output tokens, and billing details for every call, so that you avoid backfilling data later and ending up with inconsistent measurement definitions.

    3. Instrument and aggregate by domain. Do not output only a total score; aggregate separately per capability dimension. Only then can you locate weak spots and spot regressions instead of having the overall average hide the problem.

    4. Connect monitoring and logging. Use the logging and monitoring/alerting capabilities of your cloud platform to continuously collect and visualize metrics such as latency, token consumption, error rate, and tool-call failure rate, producing long-term trends rather than a single snapshot.

    5. Set up cost regression alerts. Trigger an automatic alert when an upgrade pushes average tokens, cost per 1,000 calls, or average latency above a threshold compared with the previous period. For example, trigger a manual review when average tokens rise more than 15% period over period, or when cost per 1,000 calls rises more than 15%.

    A minimal but workable evaluation record structure looks like this:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class EvalRecord:
    4. task_id: str
    5. category: str # Domain label: reasoning / education / agent ...
    6. correct: bool # Whether the answer was correct
    7. latency_s: float # Duration of a single call
    8. input_tokens: int
    9. output_tokens: int
    10. @property
    11. def total_tokens(self) -> int:
    12. return self.input_tokens + self.output_tokens

    With structured records like these, you can go further: per-domain statistics, cost regression analysis, latency percentile statistics, and version comparison dashboards.

    6. FAQ: How Do You Choose, Upgrade, and Reduce Cost?

    Q: How do you actually choose a large language model?

    A: Do not start from the leaderboard total score; start by defining the business constraints. Set the cost budget and latency ceiling first, then compare accuracy within the same cost tier, and finally decide based on performance in your business domain. For high-concurrency scenarios, prioritize latency and stability; for complex reasoning scenarios, prioritize accuracy and controllable token usage; for agent scenarios, evaluate tool calling and structured output separately.

    Q: What is the most common evaluation mistake when upgrading versions?

    A: Seeing overall accuracy rise and assuming the upgrade is necessarily beneficial. In reality, you need to check whether token usage, latency, and cost deteriorate at the same time. Tasks involving agents, function calling, JSON output, and strict format compliance are especially prone to a situation where the overall score rises while a critical scenario regresses.

    Q: How do you reduce token and inference costs?

    A: Several levers are available: shortening prompts, reducing repeated context, splitting complex tasks, limiting maximum output length, choosing more suitable decoding parameters, caching frequent questions, and routing simple tasks to a lighter model. During evaluation, include the “cost reduction measures” in the same set of metrics to confirm whether accuracy, latency, and cost all improve in the same direction.

    In short, evaluating a large language model is fundamentally about finding a business-acceptable balance point across accuracy, latency, token consumption, and cost. Building per-domain breakdowns, per-price-component cost conversion, same-cost-tier horizontal comparison, and an engineered evaluation process is what enables teams to make rational, quantifiable, reproducible, and defensible decisions during version iteration and model selection.