LLM Evaluation in Practice: Designing Metric Frameworks and Implementation Methods

Why LLM Evaluation Must Be Systematic

LLM performance is highly scenario-dependent. The same model may handle casual conversation fluently while struggling with complex instruction following, multi-step reasoning, or specialized domain questions. Making selection decisions based on a few demo sessions introduces enormous variance. For developers and enterprise users, evaluation is not about subjective impressions—it is a serious engineering decision. Its value lies in three areas:

1. Supporting version migration decisions. Model services iterate frequently. Deciding whether a new version is worth migrating to requires quantitative comparison under identical conditions, not a vague sense that it “feels stronger.”
2. Quantifying cost and efficiency. Production traffic means token consumption and unit pricing directly determine your bill. Response latency directly affects user experience and system throughput.
3. Identifying capability gaps. Dimension-level data beyond the overall score pinpoints where a model needs improvement through prompt engineering, retrieval-augmented generation, or fine-tuning.

Four Core Metrics: Quality, Speed, Efficiency, and Cost

A mature evaluation framework typically covers at least four metrics, each answering a distinct question:

1. Accuracy (quality): How many questions did the model answer correctly? Objective questions can use exact matching or unit-test-style assertions for automatic grading. Subjective questions require scoring rubrics plus a judge model, with periodic human sampling to calibrate consistency.
2. Average latency (speed): The end-to-end delay from sending a request to receiving the complete response. It is affected by inference speed, generation length, service load, and other factors, and directly determines user experience.
3. Average token consumption (efficiency): The total input and output tokens per call, reflecting the model’s “expression density.” Longer reasoning chains and verbose explanations drive consumption higher.
4. Cost per thousand calls (cost): The aggregate bill calculated from average token counts and input/output unit prices—the hardest constraint for commercial deployment.

These four metrics are coupled. Accuracy gains often come from longer reasoning chains, which increase token consumption, latency, and cost. Conversely, if tokens remain roughly flat while latency drops significantly, it usually signals real improvements in underlying inference efficiency—such as inference frameworks, operator optimization, or scheduling strategies. Cross-validating metrics often yields more insight than any single number alone.

Test Set Construction: Stratification, Scale, and Grading Rules

The credibility of evaluation results depends heavily on the test set itself. In practice, three points matter most:

1. Domain and difficulty stratification. Taking Chinese text and logic evaluation as an example, coverage typically includes: language understanding and instruction following, reasoning and mathematical computation, and vertical domains such as education, finance, legal and administrative affairs, and healthcare and mental health. Each domain should be further stratified by difficulty to avoid concentrating questions in the model’s comfort zone.
2. Sufficient scale. Small test sets—tens or hundreds of questions—have sampling errors large enough to mask real differences between models. At a scale of tens of thousands of questions, differences around one percentage point begin to carry meaning. Larger question counts produce more trustworthy rankings.
3. Explicit grading rules and leakage prevention. Grading criteria should be written down before seeing answers, not reverse-engineered after inspecting outputs. Test questions should rotate periodically to prevent them from entering training distributions through public dissemination and inflating scores.

It is worth emphasizing that even when evaluation focuses purely on text and logic, it is not a narrow exercise. Demanding language tasks serve as a stress test for a model’s linguistic core and reasoning capabilities. Image, audio, and video capabilities require separately built evaluation sets—one set of text questions cannot cover all capability dimensions.

Evaluation Execution: A Reusable Code Framework

From an engineering perspective, the core of an evaluation script is: concurrent invocation, complete logging, and resumable execution. Below is a vendor-agnostic framework:

import asyncio, time, statistics
async def eval_one(client, q, semaphore):
    async with semaphore:
        start = time.perf_counter()
        try:
            resp = await client.chat(
                model="target-model",      # Model identifier under test
                messages=[{"role": "user", "content": q["text"]}],
                temperature=0,             # Reduce randomness
                timeout=120,
            )
            return {
                "id": q["id"],
                "correct": judge(q["answer"], resp.text),  # Grading function
                "latency": time.perf_counter() - start,
                "tokens": resp.usage.total_tokens,
            }
        except Exception:
            return {"id": q["id"], "correct": False, "latency": None, "error": True}
def summarize(results):
    ok = [r for r in results if not r.get("error")]
    return {
        "accuracy": sum(r["correct"] for r in ok) / len(ok),
        "avg_latency_s": statistics.mean(r["latency"] for r in ok),
        "avg_tokens": statistics.mean(r["tokens"] for r in ok),
    }

Key engineering notes:

• Keep concurrency moderate: Excessive concurrency triggers rate limiting and queueing, which contaminates latency measurements. Run a small stress test first to determine a reasonable concurrency window.
• Set temperature to zero or keep it fixed: This ensures reproducible results and reduces variance from decoding randomness.
• Log the usage field completely: Record input and output tokens separately for accurate cost calculation.
• Cost formula: Cost per thousand calls = (average input tokens × input unit price + average output tokens × output unit price) × 1000. Note that input and output pricing differs for most model services; mixing them introduces significant bias.

Interpreting Results: Version Comparison and Cross-Model Comparison

Version Comparison: Control Variables First

When comparing old and new versions of the same model, you must use the same evaluation set, the same grading criteria, and the same time window. The following table shows illustrative data (fictional, for demonstrating interpretation methods only; it does not represent any specific model’s actual results):

Metric	Before	After	Change
Overall score (accuracy)	72.5%	74.8%	+2.3pp
Average latency	64s	53s	-11s
Average tokens	3119	3157	+38
Overall ranking	6th	2nd	↑4 positions

Four layers of insight can be drawn from this data:

1. Overall improvement: A 2.3 percentage point gain and a four-position ranking jump represent substantive progress, not statistical noise.
2. Structural view by dimension: Suppose language understanding and instruction following improved by 4.9 percentage points to 72.4%—particularly valuable for agents and automated workflows that depend heavily on instruction adherence. Reasoning and math gained another 3.3 points on top of an already high 85.1%, indicating continued strengthening of core capabilities. Education rose from 64.8% to 68.6%, showing clear gains in a vertical domain. Healthcare held steady at a high 88.7%.
3. Efficiency signal: Tokens increased marginally by 38 (about 1.2%) while latency dropped by 11 seconds (about 17%). This is a classic signature of underlying inference efficiency optimization—the model “thinks more but responds faster,” a significant advantage for latency-sensitive scenarios.
4. Cost accounting: The slight token increase has limited cost impact. If unit pricing remains flat, cost per unit of quality actually decreases.

Cross-Model Comparison: Three Principles of Fairness

When ranking candidate models on the same leaderboard, three bottom lines must hold:

• Same questions, same grading: All models use the same question set and the same grading logic. Never let each model be evaluated on its own preferred questions.
• Consistent pricing basis: Cost comparisons should use publicly available, uniform unit pricing and note the statistical time window.
• Beware leaderboard fixation: Different evaluation sets have different difficulty distributions and domain weights, which can produce opposite rankings for the same pair of models. The right approach is to use public leaderboards for initial screening, then run your own business-specific evaluation set for the final decision.

A practical combined perspective is “cost-effectiveness”: accuracy per dollar, or accuracy per second of latency. It compresses quality, cost, and speed into a single decision-making view.

Common Pitfalls Checklist

1. Looking only at overall scores: Two models with similar totals may differ dramatically in vertical domains.
2. Ignoring variance and significance: Fluctuations in a single evaluation run may come from sampling and decoding randomness. Key conclusions should be verified through repeated runs.
3. Test set disconnected from business distribution: A high leaderboard score does not mean high performance in your scenario. Always validate against real business traffic.
4. Measuring quality without cost and latency: The accuracy champion in the lab may be unusable in production due to cost and delay.
5. Treating demos as evaluation: Hand-picked showcase examples have no statistical meaning.
6. Using one question set for all capabilities: Text, multimodal, code, and long-context evaluations should each be designed independently.

Conclusion

LLM evaluation is a systematic engineering effort that requires continuous iteration: test sets need regular rotation, grading rules need ongoing calibration, and cost and latency baselines must be updated as market conditions change. Whether you use cloud model services such as Baidu AI Cloud or private deployment, the same methodology applies. For teams, the right approach is to turn evaluation into a repeatable pipeline that runs automatically with every version release and every model upgrade—so that technical decisions always rest on evidence rather than intuition.