How to Quantitatively Evaluate LLMs: Accuracy, Latency, and Cost

    Choosing and upgrading a large language model should not come down to "it feels stronger". This article explains why qualitative claims are not enough for engineering decisions and how to build a quantitative evaluation around four core metrics: accuracy, average response time, token consumption, and cost per 1,000 calls. It walks through a sample result set, breaks accuracy down by domain, compares a new version against its predecessor, and highlights the special challenges of benchmarking agent capabilities. It also includes illustrative Python code for turning benchmark numbers into a weighted selection score, along with practical advice on building your own test set, fixing the evaluation pipeline, and tracking cost structure over time.

    1. Why “It Feels Better” Is Not a Substitute for Quantitative Evaluation

    Large language model (LLM) releases come out frequently, and vendor announcements tend to emphasize capability labels such as larger context windows, stronger reasoning, and better agent capabilities. These qualitative claims, however, leave several questions that matter to engineers unanswered: how much did accuracy actually improve after the upgrade? Is response time better? How many more tokens does each call consume? What does that mean for cost per 1,000 calls?

    You cannot answer these questions without a consistent benchmark that turns “better” into comparable numbers, and then use that benchmark to decide whether an upgrade truly fits your business requirements. This step is easy to overlook when selecting or upgrading a model, yet it has one of the biggest effects on long-term cost.

    2. Evaluation Metrics: Four Core Dimensions

    A complete evaluation should cover at least the following four types of metrics:

    1. Accuracy: the proportion of correct model outputs. It is typically measured over a large, structured test set and reflects a model’s upper bound of capability.
    2. Response time (average latency): the average delay per call. It directly shapes the interaction experience, especially in scenarios such as AI-assisted programming and real-time question answering.
    3. Token consumption: the average number of tokens consumed per call. It affects both response speed and billing.
    4. Call cost: combine token consumption with unit price to derive the cost per 1,000 calls. This is a key metric for assessing cost-effectiveness at large-scale production use.

    One version, evaluated on a test set of about 15,000 questions, might produce the following results: accuracy of about 68.7%, average latency of about 59 seconds, average consumption of about 2,851 tokens, and a cost of about CNY 37.6 per 1,000 calls. These four numbers form the baseline for all subsequent comparisons.

    3. Breaking Results Down by Domain to See Where Capability Lies

    A single overall accuracy score can hide structural differences in a model’s capabilities. A more rigorous approach is to break results down by domain and examine where the model is strong or weak across tasks. Common categories include:

    • Law and public administration: accuracy in understanding legal provisions and making compliance judgments;
    • Education: correctness in explaining concepts and solving problems;
    • Finance: comprehension of data and handling of domain-specific terminology;
    • Healthcare and mental health: rigor and safety;
    • Reasoning and math: multi-step logical derivation;
    • Language and instruction following: adherence to formatting, tone, and stated constraints;
    • Agents and tool calling: ability to invoke external tools in complex tasks.

    A domain-level comparison might reveal that a version makes large gains in a specific domain, such as law and public administration, while regressing in agent and tool-calling tasks and in language and instruction following. This kind of trade-off often reflects how a model balances competing capabilities. It also serves as a reminder: model selection should be driven by your actual business scenarios, not by an overall score alone.

    4. Comparing Versions: Was the Upgrade Worth It?

    The most direct way to evaluate an upgrade is to put the new and previous versions side by side. Comparing a version with its predecessor reveals a few typical trends:

    • Steady overall improvement: accuracy rose from 66.6% to 68.7%, an increase of about 2.1 percentage points, along with a notable gain in ranking;
    • Stronger in specialized domains: gains in law and public administration reached up to 7.6 percentage points, with education, healthcare, reasoning, and others also improving considerably;
    • Faster responses: average latency dropped from 98 seconds to 59 seconds, an improvement of nearly 40%, which has a major impact on interaction quality;
    • Lower token and cost footprint: token consumption per call fell by about 3.2%, and the cost per 1,000 calls dropped by about 3.1%.

    One engineering judgment is important here: improvements in response speed and reductions in token consumption often affect the real-world experience and cost structure more than a small gain in overall accuracy. When evaluating an upgrade, therefore, do not focus on accuracy alone. Put latency and cost into the same comparison table.

    5. What Makes Agent Evaluation Different

    Vendor announcements often emphasize that a model supports tool calling during reasoning or performs better as a search-based agent. Evaluating agents, however, is more complex than evaluating ordinary question answering, because it involves multi-turn interactions, tool selection, and the orchestration of calls. A single run is hard to reproduce reliably.

    When evaluating agent capabilities, consider the following:

    • Use repeatable multi-step task scenarios rather than single-turn question answering;
    • Track metrics such as tool call success rate, task completion rate, and average number of call steps;
    • Record the token overhead of tool calls, which is often much higher than that of ordinary question answering.

    It is also worth noting that a version may regress on agent benchmarks. That suggests that vendor claims of “enhanced capabilities” and the results on a particular test set can differ, so you should verify against your own business data.

    6. The Engineering View: Balancing Cost and Performance

    In real engineering work, model selection is essentially a trade-off. The following simplified example code shows how to turn evaluation results into a decision:

    1. # Illustrative: weighted model selection based on benchmark metrics
    2. candidates = [
    3. {"name": "Candidate A", "accuracy": 0.687, "latency_s": 59, "cost_per_1k": 37.6},
    4. # ... other candidate models
    5. ]
    6. # Set weights based on your business scenario
    7. weights = {"accuracy": 0.5, "latency_s": 0.2, "cost_per_1k": 0.3}
    8. def score(m, norm):
    9. return (
    10. weights["accuracy"] * m["accuracy"]
    11. - weights["latency_s"] * (m["latency_s"] / norm["latency_s"])
    12. - weights["cost_per_1k"] * (m["cost_per_1k"] / norm["cost_per_1k"])
    13. )
    14. # Use the maximum across candidates as the normalization factor
    15. norm = {
    16. "latency_s": max(c["latency_s"] for c in candidates),
    17. "cost_per_1k": max(c["cost_per_1k"] for c in candidates),
    18. }
    19. for c in candidates:
    20. print(c["name"], round(score(c, norm), 4))

    The key takeaway is that the weights should be tailored to your business. For real-time interaction, give latency more weight. For offline batch processing, focus more on accuracy and cost.

    7. Practical Tips for a Reusable Evaluation Process

    • Build your own test set: public benchmarks often diverge from your business scenarios, so construct a test set that reflects your own data.
    • Fix the evaluation process: use the same questions, temperature setting, and scoring rules so that different versions and different models can be compared side by side.
    • Use a multi-dimensional scorecard: put accuracy, latency, tokens, and cost in the same table to avoid being misled by a single metric.
    • Run version regressions: run the historical test set after every upgrade to catch capability regressions promptly.
    • Watch the cost structure: as long as the output unit price stays the same, a reduction in token consumption alone improves cost-effectiveness.

    If your team already has cloud infrastructure in place, general-purpose cloud products such as logging services, monitoring and alerting, and object storage can handle the collection, storage, and visualization of evaluation data without building an entire system from scratch. For example, with Baidu AI Cloud’s general-purpose cloud products, you can quickly set up a reusable evaluation pipeline.

    8. Conclusion

    A model version upgrade should never be judged on the feeling that things are “better”. Only by quantifying accuracy, latency, token consumption, and cost across these four dimensions, and then adding domain-level breakdowns and old-versus-new comparisons, can you truly answer whether an upgrade was worth it. For individual developers, this means getting usable capabilities at lower cost. For enterprise teams, it means turning model selection from an intuition-driven decision into an engineering decision grounded in evidence.