How to Evaluate New LLM Versions: Metrics, Methodologies, and Practices

1. Why Official Claims Are Not Enough: The Need for Independent Evaluation

The iteration speed of Large Language Models (LLMs) is accelerating, with vendors frequently releasing “the most powerful” versions yet. While official metrics serve as an important reference, making selection decisions based solely on marketing claims carries significant risk for developers and enterprise users. There are three main reasons for this:

1. Capability Structure Shifts: Version upgrades rarely offer across-the-board improvements. Instead, they often involve trade-offs between different categories: reasoning capabilities might improve while instruction following lags behind; the total score might increase, while performance on long-tail tasks regresses. Only itemized data can reveal where the actual improvements occurred.
2. Production Readiness Depends on Multiple Metrics: Beyond accuracy, response time, token consumption, and API costs directly impact the online user experience and billing. A model with higher accuracy but double the latency could be a negative optimization for online scenarios.
3. Marketing Metrics Are Not Reproducible: The value of independent evaluation lies in its reproducibility. Using identical question sets, fixed parameters, and clear scoring rules, anyone should be able to verify the conclusions.

Therefore, a scientific evaluation methodology is essentially a quantification process that translates “feels stronger” into “where is it stronger, by how much, and at what cost.”

2. Evaluation Metrics Framework: Four Core Dimensions

1. Accuracy: Both Total and Itemized Scores Are Essential

The total score reflects overall capability and is suitable for coarse-grained comparisons between versions. However, itemized scores truly guide decision-making. It is recommended to cover at least the following categories:

• Complex Reasoning and Mathematical Calculation: Multi-step deduction, symbolic computation, and process rigor.
• Language Understanding and Instruction Following: Whether the model strictly outputs according to required formats, constraints, and styles.
• Code Generation and Debugging: Functional correctness and runnability.
• Agent and Tool Calling: Task decomposition, multi-turn planning, and result reliability.
• Multimodal Understanding (if applicable): Comprehension and reasoning over inputs like images and videos.

In practice, category weights should be tied to business relevance. If your business focuses primarily on data analysis and process automation, the weights for reasoning and instruction following should be higher than those for creative generation.

2. Response Time: Look Beyond Averages to the Tails

Reasoning-oriented models, designed to “think before answering,” exhibit significantly higher latencies than standard conversational models. Average response times of several hundred seconds per call are not uncommon. During evaluation, it is advisable to record:

• Average Latency: Reflects overall efficiency.
• P95/P99 Latency: Reflects worst-case scenarios and directly dictates timeout and retry strategies.
• Time to First Token (TTFT): Has a greater impact on user experience in streaming output scenarios.

3. Token Consumption: A Dual Signal for Cost and Reasoning Efficiency

Token consumption must be separated into input and output. For reasoning models, the output side contains a large number of “implicit reasoning” tokens. This is the primary driver of cost, but it also serves as an observation window into reasoning path efficiency. If a new version shows a significant drop in token consumption on identical problems while accuracy remains stable or improves, it indicates the model solved the problem using a shorter reasoning chain rather than brute-forcing it with compute-heavy search.

4. Cost: Converted per 1,000 Calls for Horizontal Comparison

Different services have varying pricing for input and output tokens, making direct “single call price” comparisons less meaningful. A more practical approach is to run a fixed batch of test queries, calculate the total cost per 1,000 calls, and further compute the “accuracy per unit cost” (accuracy / cost per 1,000 calls) as a rough measure of cost-effectiveness.

3. Engineering Implementation Key Points

1. Test Set Construction

• Scale: A question set in the tens of thousands significantly reduces sampling fluctuation. Small test sets are prone to percentage-point swings caused by the correctness of just a few questions.
• Coverage: Balance categories and stratify difficulty to avoid concentrating on a single type of question.
• Anti-Leakage: Regularly update questions to prevent artificially inflated scores caused by the model having “memorized the answers” from public datasets.

2. Execution Control

• Fix sampling parameters (e.g., Temperature, Top-P) and document them.
• Run each question multiple times to calculate the pass rate and variance, preventing a single lucky run from skewing conclusions.
• Record reasons for failed requests (timeout, rate limiting, format errors) to distinguish “capability issues” from “engineering issues.”

3. Scoring Methods

• Rule-Based Scoring: Compare final answers for math problems and run unit tests for code questions. This is objective and automatable.
• Model-Based Scoring: Use a strong model to score open-ended responses, being mindful of the scoring model’s inherent biases.
• Manual Spot-Checking: Regularly sample and calibrate the accuracy of machine scoring.

4. Pseudocode: Minimal Statistical Script

function summarize(records):
    n       = count(records)
    acc     = sum(r.correct for r in records) / n
    lat     = sort([r.latency for r in records])
    p95_lat = lat[floor(n * 0.95)]
    avg_tok = mean(r.tokens for r in records)
    cost_1k = 1000 * sum(r.cost for r in records) / n
    return {accuracy: acc, avg_latency: mean(lat),
            p95_latency: p95_lat, avg_tokens: avg_tok,
            cost_per_1k_calls: cost_1k}

Save the raw records of each evaluation to disk. When versions are upgraded in the future, rerun the tests on the same question set to form a longitudinally comparable time series.

4. How to Interpret Evaluation Results: A Sample Case Study

The following uses sample data (not actual test results from any real product) to demonstrate common interpretation perspectives. Assume a reasoning model is upgraded from Version A to Version B on an evaluation set of approximately 15,000 questions:

Metric	Version A	Version B	Change
Total Score (Accuracy)	68.3%	71.3%	+3.0
Reasoning & Math	75.3%	78.0%	+2.7
Language & Instruction Following	63.4%	72.1%	+8.7
Average Latency	—	338s	—
Average Tokens	—	3842	—
Cost per 1,000 Calls	—	77.1	—

Interpretation 1: The Value of a 3-Percentage-Point Increase. An overall increase of 3 percentage points and a notable rise in industry rankings indicate that the overall capability baseline has genuinely improved. However, you should question the source of the improvement. If it primarily comes from a few specific categories, it represents a structural improvement rather than comprehensive superiority.

Interpretation 2: Evaluate Reasoning Capability and Efficiency Together. The reasoning and math categories improved by 2.7 percentage points, with more stable results across multiple tests. Concurrently, for questions that previously required massive token consumption and relied on repeated search to “barely solve,” the new version’s token consumption dropped significantly. An increase in accuracy coupled with a decrease in consumption is a signal of “smarter reasoning paths,” which holds greater value than scores achieved simply by burning compute.

Interpretation 3: Instruction Following is the Lifeline of Agent Scenarios. The nearly 9-percentage-point jump in the instruction following category means the model has shifted from “answering based on its own understanding” to “stably executing explicit instructions.” Direct instructions can be executed reliably, and lightly reasoned instructions no longer derail. For applications relying on structured output, tool calling, and process automation, this metric’s significance may surpass the total score improvement itself.

Interpretation 4: Latency and Cost Acceptability Depends on the Scenario. An average of 338 seconds per call, roughly 3842 tokens, and a specific cost per 1,000 calls is generally acceptable for offline batch processing or deep research tasks. However, for online customer service or real-time assistant scenarios, this clearly exceeds the waiting budget. The conclusion is not simply whether it is “expensive,” but whether it is “worth it for the target scenario.”

Interpretation 5: Stability is an Underestimated Metric. High consistency in results across multiple runs of the same question means online behavior is predictable and retry strategies are simple. A model with high variance, even if its average score is higher, must be discounted during engineering implementation.

5. Common Pitfalls and Implementation Recommendations

Four Common Pitfalls:

1. Focusing solely on leaderboard total scores: Leaderboards are screening tools, not selection criteria.
2. Ignoring statistical significance: Percentage-point differences derived from small test sets and single runs may be completely meaningless.
3. Ignoring data contamination: Public evaluation questions may have entered the training corpus, leading to artificially inflated capabilities.
4. Substituting general test sets for business evaluations: Strong general capabilities do not equate to strong performance on your specific business tasks.

Four Implementation Recommendations:

1. Build a Golden Set of 200 to 500 high-frequency questions with standard answers mined from your business logs as a private evaluation baseline.
2. When upgrading versions, run both the old and new models on the same question batch for canary testing, validating evaluation conclusions against real traffic.
3. Leverage cloud platform capabilities to build an evaluation pipeline. For example, use object storage to store question sets and results, logging services to record API call details, and monitoring tools to track online accuracy and cost drift (cloud providers like Baidu AI Cloud offer such general services).
4. Incorporate “accuracy per unit cost” into your selection reports, ensuring that technical decisions and budget conversations operate within the same coordinate system.

6. Conclusion

The value of evaluating new LLM versions lies not in ranking models, but in breaking down the vague notion of “it got stronger” into four quantifiable and reproducible dimensions—accuracy, latency, tokens, and cost—and judging trade-offs in the context of your business scenarios. Establishing your own evaluation system—comprising fixed question sets, controlled execution, transparent scoring, and continuous monitoring—ensures that you consistently make rational, defensible technical decisions amidst the rapid wave of model iteration.