LLM Model Selection: Evaluating Accuracy, Latency, and Cost

    This article explains how to evaluate large language models for production use. It covers measuring task accuracy, end-to-end latency, token usage, and cost together under the same test conditions. It also shows how to build a domain-structured test set, report metrics reliably, compare versions and models within cost tiers, and set up a repeatable evaluation pipeline.

    Each model release can change reasoning depth, output length, and serving latency at the same time. If only a single accuracy score is published, teams cannot easily tell whether a new version is more useful, or whether it earned a higher score by using more tokens and longer wait times. For Chinese-language use cases, model selection should be a repeatable engineering process, not a one-time reading of a leaderboard.

    Define “good” before comparing scores

    In production, “good” means at least four things.

    1. Task correctness: Set a verifiable standard for each question type, so that fluent but wrong answers do not count as passing.
    2. Latency tolerance: Synchronous chat, batch quality review, and overnight offline analysis tolerate very different response times.
    3. Context and generated tokens per call: Long reasoning chains and multi-turn tool calls both increase token usage.
    4. Actual spend per thousand calls: Even with unchanged unit prices, higher usage raises the bill.

    Compare these four dimensions on the same question set, with the same decoding parameters and the same measurement method. After you change the prompt template, temperature, or maximum output length, earlier numbers cannot be compared directly.

    Build the test set around capability structure, not sheer volume

    The test set must be large enough to reduce random noise, but domain coverage matters more. Report at least the following categories separately:

    • Reasoning and math: multi-step derivations, unit conversion, and constraint satisfaction. Check whether intermediate steps are consistent.
    • Education: concept explanation, step-by-step walkthroughs, and error correction. Check for invented formulas or skipped steps.
    • Medicine and mental health: information organization, risk warnings, and the boundaries of advice to seek care. Model output must not be treated as a diagnosis.
    • Finance: consistency of definitions, calculation process, and disclaimers. Estimates must not be written as certain conclusions.
    • Language and instruction following: format, length, forbidden items, and multiple conditions that must hold at once.
    • Agents and tool use: whether the model calls a tool when it should, whether parameters are complete, and whether it can correct itself using tool returns after a failure.

    When the overall score rises, still check the sub-scores. A common pattern is that reasoning, domain Q&A, and instruction following improve together while tool calling slightly declines. This suggests that the training or inference strategy may have given capacity to longer internal reasoning, while the stability of the tool protocol did not improve alongside it. If your product depends heavily on function calling, a higher overall score should not hide this regression.

    For medical and financial questions, run a separate safety spot check. Check whether the model avoids overreaching advice, cites no invented regulations, and does not present probabilities as facts. Do not pass an answer just because it contains the right keywords.

    Calculate metrics so they are not misleading

    Use the same judge for every accuracy score. For open-ended questions, prefer rule-based scoring with manual spot checks, rather than relying on a single uncalibrated model as the only judge. Report overall and per-domain results, and state whether the questions are public and whether they may overlap with training data.

    Measure latency end to end, from sending the request to receiving a complete, readable result, not only time to first token. Long tails pull averages up, so report P50 and P95 side by side. In interactive use, P95 often describes the user experience better than the mean.

    Track input and output tokens separately. A rise in average token usage usually means longer reasoning chains, more self-checking, or a longer system prompt. Higher token use may come with higher accuracy, but it is not free.

    Convert cost using the actual billing unit. If pricing is per million tokens, use:

    Cost per 1,000 calls ≈ (average input tokens × input price + average output tokens × output price) × 1,000 / 1,000,000

    If usage rises while unit prices stay the same, the cost per 1,000 calls still increases. When comparing models in the same price tier, use measured cost per 1,000 calls, not list prices. When two models have similar accuracy, the faster one often has an advantage in interactive products. A slower model with stronger sub-scores may only suit offline or asynchronous tasks.

    Describe trade-offs when comparing versions, not just gains

    When you interpret an old and new version, record these four points:

    • How much the overall accuracy changed. Relative ranking is only a reference, because leaderboard composition changes.
    • Which sub-scores improved most, and whether those gains apply to the question types that matter most for your business.
    • Whether any sub-scores declined, and whether the drop exceeds your acceptable threshold.
    • Whether the increases in latency and tokens are proportionate to the accuracy gain. If latency and cost rise substantially while accuracy improves only slightly, route traffic in tiers rather than switching everything to the new version.

    Also check whether difficulty is balanced across the test set. If the new version only answers a few more easy questions correctly, the higher total score adds little value for critical workloads. Use stratified sampling: keep easy, medium, and hard questions at fixed proportions, and report accuracy for each stratum.

    Compare models within cost tiers

    Compare models within cost bands instead of only ranking them on a leaderboard. The steps are:

    1. Run all candidates on the same question set.
    2. Record accuracy, P50 and P95 latency, average tokens, and cost per 1,000 calls.
    3. Group models into cost tiers by cost per 1,000 calls, and compare only within each tier.
    4. When accuracy is close within a tier, break ties using latency and the retry rate on failed requests.
    5. If the business has hard requirements, such as financial calculations must exceed a minimum score on a sub-category, or tool calling must not regress, filter on these first and then compare cost.

    A small ranking gap combined with a noticeably slower response matters differently for an online customer service bot than for a math grading tool. Conclusions from a Chinese-language evaluation should not be extended to other languages or niche fields. State the scope of the scenarios in your report.

    A repeatable evaluation pipeline

    The evaluation jobs can run on a container platform. Store questions and results in object storage, send run logs to a logging service, and connect accuracy, latency, and cost metrics to monitoring and alerts. If you use Baidu AI Cloud, treat it only as a general cloud environment for compute, storage, and observability. The evaluation logic remains in your own scripts and does not depend on any undisclosed proprietary evaluation API.

    The following sketch shows the flow. The parameters are placeholders and should be replaced to match your actual gateway.

    1. def evaluate(samples, call_model):
    2. stats = []
    3. for s in samples:
    4. t0 = now_ms()
    5. resp = call_model(s.prompt, max_tokens=s.max_tokens, temperature=0)
    6. latency = now_ms() - t0
    7. ok = judge(s, resp.text) # Rule-based or manual review, using one consistent scoring basis
    8. stats.append({
    9. "domain": s.domain,
    10. "ok": ok,
    11. "latency_ms": latency,
    12. "tokens": resp.usage.total,
    13. "cost": price(resp.usage),
    14. })
    15. return aggregate(stats) # Per-domain accuracy, P50/P95 latency, cost per 1,000 calls

    Write these run constraints as a checklist:

    • Fix the random seed and decoding parameters.
    • Record the model version and the date of the run.
    • Report failed requests as a separate error rate instead of quietly removing them from the denominator.
    • Do not repeat the same question so often that cache hits look like real capability.
    • Raise the manual review ratio for domains with higher risk.

    Making the deployment decision

    Traffic can be split by routing. Send high-risk, reasoning-heavy requests that can run asynchronously to the version that prioritizes quality. Keep latency-sensitive requests, or requests that depend on a stable tool chain, on the faster version or the one with stronger tool-use sub-scores. Test the change with a small traffic canary, and watch real support tickets for retries, human handoffs, and complaints, not only offline total scores.

    When accuracy, latency, tokens, and cost per 1,000 calls are reported together, and you can identify which domains improved and which regressed, version selection becomes an engineering decision rather than a slogan. Evaluation helps you link every unit of cost and every second of waiting to verifiable task gains, instead of declaring a general-purpose model better across the board.