# LLM Evaluation Guide: How to Choose a Model Across Six Dimensions

    Model releases arrive faster than ever, and every one of them ships with a fresh set of benchmark claims. This article argues that the question worth asking is not "which model scores highest" but "on which tasks, at what cost, and with what reliability can a model reach production quality." It replaces the usual leaderboard ranking with a six-dimension evaluation framework — general capability, task quality, generation stability, service performance, cost, and compliance and safety — and shows how to build it: how to construct and version a golden set, how to size it for statistical confidence, how to measure variance and structured-output reliability, how to separate time-to-first-token from generation throughput, how to compute cost per thousand correct answers, and how to wire the whole thing into CI as a release gate with working Python and YAML skeletons. You will also get five practical selection recommendations and a table of common evaluation mistakes.

    Introduction: what evaluation is really about has changed

    Model capabilities iterate quickly, and every new release comes with a fresh batch of benchmark conclusions. Those conclusions tend to share three traits: they use different test sets, different prompting styles, and a single “highest score wins” comparison logic. For teams planning to deploy LLMs in 2026, the more useful question is no longer “which model scores highest” but “on which tasks, at what cost, and through what mechanism can a model reliably reach an acceptable level of quality.”

    Many teams have looked at so-called brute-force evaluation: running several models through an intense stress test on a batch of deliberately difficult problems. This surfaces capability limits quickly, but the conclusions rest heavily on three preconditions — whether the test set covers real business scenarios, whether the scoring criteria are objective, and whether the test environment is consistent. If any one of them fails, the results can turn into something that looks lively but offers no usable basis for decisions.

    This article therefore does not hand out a “model X wins” ranking. Instead, it breaks evaluation into six dimensions and builds a reusable, reproducible evaluation system that can keep running as model versions change. The goal is a single standard that can steadily answer three questions: Which model is more reliable on which tasks? Are the differences real and repeatable? Is the risk manageable after an upgrade or a switch?

    1. Evaluation methodology: from leaderboard to decision matrix

    1.1 Three goals of evaluation

    In an engineering context, evaluation serves at least three kinds of decisions:

    • Selection and admission: at project kickoff or during technology selection, narrow the field to 2–4 candidates and define the boundaries of what each is suited for.
    • Version regression: providers keep iterating, and prompts, knowledge bases and tool chains change too, so you need to continuously confirm that quality has not degraded.
    • Routing and fallback: different tasks go to different models or different model sizes, and the primary path automatically switches to a backup when it misbehaves — which requires knowing the cost and benefit of each combination.

    Each goal maps to a different evaluation granularity. Selection can look at overall performance; regression must fix the test set and scoring standard; routing needs measurements broken out per task type.

    1.2 Why public leaderboards are only a reference

    Four factors limit the reference value of public leaderboards:

    1. Data contamination: questions and answers from public evaluation sets easily end up in the training data of later models, which systematically inflates scores.
    2. Score saturation: on general-purpose benchmarks of limited difficulty, the gap between leading models is compressed to a very small margin, making real differences hard to distinguish.
    3. Inconsistent setups: prompt templates, sampling parameters (temperature, max output length, and so on) and scoring methods differ from one evaluation to the next, so the scores are not directly comparable.
    4. Distribution shift: general questions test “does it know,” while a business evaluation tests “can it answer correctly and consistently in our format and under our definitions” — and those are often not the same thing.

    The reasonable approach is to treat public leaderboards as a first-pass screening signal and let the final conclusion come from an evaluation set that follows your own business distribution.

    1.3 The six evaluation dimensions

    The framework has six dimensions. The first three ask “how good are the answers”; the last three ask “can it keep running inside a real system”:

    Dimension Core question Main metrics Typical methods
    General capability Are basic knowledge and reasoning sound? Knowledge accuracy, math reasoning accuracy, code pass rate, Chinese semantic understanding Public benchmarks + in-house questions
    Task quality Is it usable on business tasks? Accuracy, field extraction accuracy, factual consistency, instruction-following rate Scenario-based golden set + scoring rubric
    Generation stability Does the same input yield the same acceptable result? JSON validity rate, answer divergence rate, refusal rate, harmful content rate Repeated sampling + constrained decoding
    Service performance Can it support production concurrency and UX requirements? Time to first token, generation speed, throughput, error rate under load Load testing + long-duration stability runs
    Cost What is the total spend to complete the same work? Per-task token cost, cost per 1,000 correct answers, cache hit rate Metering + cost accounting
    Compliance and safety Can it be used safely in production? Jailbreak resistance rate, sensitive data leakage rate, citation accuracy Red teaming + compliance checks

    The following sections expand each dimension one by one, giving actionable metric definitions and measurement methods, followed by an execution workflow and an evaluation code skeleton.

    2. Dimension one: general capability, protecting the floor for reasoning and code

    General capability answers whether a model’s fundamentals are adequate. Its job is to quickly filter out weak candidates, not to decide the final ranking.

    Four capability areas are worth starting from:

    • Knowledge accuracy: can the model correctly answer foundational questions in your industry and role, and distinguish what it “knows for sure” from what it is unsure about?
    • Math and logical reasoning: can it complete multi-step calculations, conditional reasoning and table interpretation? This is an indirect indicator of whether a model can handle business rules.
    • Code and tool use: can it write runnable code, read error messages, and call external functions as documented? This directly affects the reliability of agent-style applications.
    • Chinese semantic understanding: word segmentation, ambiguity resolution, long-sentence structure analysis and localized phrasing. This matters especially for Chinese-language business use cases.

    Three pieces of experience are worth noting when running these evaluations.

    First, run public benchmarks and in-house questions in parallel. Public benchmarks provide horizontal comparison; in-house questions test closeness to your own business. In-house questions should focus on formats that public evaluations rarely cover, such as long instructions containing internal terminology, nested tables, and business rules with conditional branches.

    Second, fix the test setup. Once you settle on the prompt template, temperature, max output length and number of few-shot examples, keep them constant for the entire evaluation cycle — otherwise score changes cannot be attributed to anything.

    Third, focus on failure modes rather than the total score. Instead of remembering “the overall score went up by X points,” record conclusions like “error rate rises noticeably once conditions nest more than two levels deep.” Those translate far more readily into engineering mitigations such as decomposing prompts, adding validation, or introducing a rules engine.

    3. Dimension two: task quality, using a golden set to get close to the business

    Task quality is the core of the six-dimension system. It answers whether the model meets the bar for production in a real business setting.

    3.1 Building the golden set

    A golden set is a curated, human-annotated evaluation sample collection for a specific task. Four steps are recommended:

    1. Sampling: draw samples from production logs or business documents according to real traffic distribution, covering high-frequency, long-tail and boundary scenarios — not just the easy questions.
    2. Desensitization and stratification: strip personally identifiable information, then stratify by task type, difficulty and input length so results can be read per dimension.
    3. Annotation: have domain experts provide reference answers or scoring key points. For open-ended questions, define the acceptable answer space explicitly, so that one reasonable phrasing is not treated as the only correct answer.
    4. Version control: check the golden set into version control alongside the code, and record the reason for every change, so that results from two runs remain comparable.

    From engineering experience, 200–500 high-quality samples are usually enough to reveal obvious differences, and key conclusions should be validated at a scale of several hundred samples or more. Sample size determines two things:

    • Confidence interval width: with sample size n and task success rate p, the half-width of an approximate 95% confidence interval for the overall success rate is about 1.96×√(p(1−p)/n). For example, a 90% success rate measured on 400 samples gives a range of roughly ±2.9%.
    • Minimum detectable difference: if each of two evaluations uses 200 samples, a 10-percentage-point gap against a 90% success rate has about an 80% chance of being identified as statistically significant, while a 5-percentage-point gap may well be drowned out by random variation.

    So when the gap between two candidate models is smaller than the evaluation set’s resolving power, the more sensible move is to shift the decision focus to cost, performance and stability.

    3.2 Defining metrics per task type

    The same word “accuracy” means entirely different things across tasks:

    • Extraction tasks (contract element extraction, ticket field filling): use field-level exact match, or exact match after normalization. Numbers, dates and monetary amounts must be normalized to a common format before comparison.
    • Generation tasks (summaries, reply copy): use rubric-based scoring. Break “good” into judgeable sub-items such as factual accuracy, coverage of key points, format compliance and appropriate tone.
    • Code tasks: run unit tests inside an isolated sandbox and treat the pass rate as a hard metric.
    • Q&A tasks: have humans judge whether answers are correct, and separately record the share of “I don’t know” responses. For many businesses, an honest refusal is worth more than a fabricated answer.

    3.3 Model judging versus human judging

    Human scoring is accurate but expensive, so teams typically use model-as-a-judge: another LLM scores outputs against a standard. Before adopting it, you must verify that it agrees with human judgment. For a binary decision task, you can compute the F1 score — the weighted harmonic mean of correct decisions across the two classes. It is advisable to reach an F1 of 0.85 or higher before using it for automated scoring, and to periodically sample outputs for human review so the judge model’s own preferences do not distort conclusions.

    In addition, the scoring prompt should output a score plus a rationale plus a quoted source fragment, which makes disputed samples traceable.

    4. Dimension three: generation stability, where variance is more dangerous than the average

    The biggest difference between a production system and a demo environment is this: a demo needs to succeed once, while production needs to succeed thousands of times without errors.

    4.1 Measure variance with repeated sampling

    Send each input in a batch 3–5 times (when temperature is set above 0) and compute:

    • Answer divergence rate: the share of samples where the key conclusion for the same question changes;
    • JSON validity rate: the proportion of outputs that can be parsed and conform to the agreed structure;
    • Field completeness rate: the proportion of outputs where all required fields are present.

    Even with identical average quality, a model with higher variance is harder to build production safeguards around, because it makes “occasional failures” difficult to reproduce and diagnose.

    4.2 Structured output requires explicit constraints

    Telling a model to “please output JSON” is not reliable. Three layers of hardening are recommended:

    1. Server-side constraints: prefer the provider’s structured output capability, where the inference service guarantees output conforms to a given schema.
    2. Client-side validation: validate locally against a JSON Schema, and include field examples and enum values in the prompt.
    3. Repair and retry: on validation failure, feed the error back to the model so it can correct itself, cap the number of retries (generally no more than 2), and hand off to a human or a fallback path once the cap is exceeded.

    Illustrative code:

    1. import json
    2. from jsonschema import validate
    3. SCHEMA = {
    4. "type": "object",
    5. "required": ["intent", "entities", "confidence"],
    6. "properties": {
    7. "intent": {"type": "string", "enum": ["query", "process", "complaint", "other"]},
    8. "entities": {
    9. "type": "object",
    10. "properties": {"order_id": {"type": "string"}},
    11. },
    12. "confidence": {"type": "number", "minimum": 0, "maximum": 1},
    13. },
    14. }
    15. def parse_structured(text: str, max_repair: int = 2) -> dict:
    16. """Parse structured output; on failure, send the error back for repair and return None past the limit."""
    17. prompt_fix = None
    18. for _ in range(max_repair + 1):
    19. if prompt_fix is None:
    20. raw = text
    21. else:
    22. raw = call_model(prompt + prompt_fix) # call_model is a shared inference wrapper
    23. try:
    24. obj = json.loads(extract_json(raw))
    25. validate(obj, SCHEMA) # raises if the payload does not match the schema
    26. return obj
    27. except Exception as exc:
    28. prompt_fix = f"\nThe previous output was unusable: {exc}. Output only JSON that meets the requirements."
    29. return None

    In evaluation, “parsed successfully on the first attempt” and “parsed successfully after repair” should be counted separately. The former reflects the model’s own constraint-following ability; the latter reflects the actual load on the fallback path.

    4.3 Refusals, refusals to act and safe output

    Stability also includes not answering when it should not. Three boundary behaviors are worth tracking: the refusal rate when a question falls outside the knowledge scope, the refusal-to-act rate when facing leading or manipulative prompts, and the proportion of outputs containing sensitive content. These three should be measured with a dedicated red-team question set rather than discovered by chance in everyday traffic.

    5. Dimension four: service performance, measuring latency and throughput separately

    Functional correctness is only the passing grade. In interactive scenarios such as customer support or office assistants, response speed directly determines the user experience.

    5.1 Separate time-to-first-token from generation speed

    LLM inference has two phases:

    • Prefill: processes the entire input in one pass and determines time to first token (TTFT).
    • Decode: generates token by token and determines generation speed (output tokens ÷ generation time).

    The two phases have different bottlenecks: prefill is heavily influenced by input length and compute, while decode is heavily influenced by memory bandwidth and batching strategy. As a result, a test with a 500-character input and a test with a 50,000-character input are not comparable, and every evaluation must declare its input and output length distribution.

    5.2 Watch high percentiles, not averages

    User experience is determined by the worst slice of requests. Report P95 and P99 for both time to first token and end-to-end latency, with target values such as:

    • Interactive conversation: P95 time to first token ≤ 1.5 seconds;
    • Background batch processing: focus on per-batch completion time and retry failure rate.

    Averages easily hide the long tail. A few percent of requests timing out at peak hours is enough to make users give up on a feature.

    5.3 Concurrency, rate limiting and long context

    Performance evaluation should cover at least three sets of conditions:

    • Incremental concurrency load testing: raise concurrency step by step and observe whether throughput scales linearly, and at which concurrency level the error rate starts to climb.
    • Long-duration stability testing: run continuously for several hours and watch for memory leaks, connection pile-up or drift in response times.
    • Long-context scenarios: long-document Q&A, cross-file code analysis, and the like. Beyond output quality, assess the non-linear effect of increasing input length on latency and cost. Long context is better placed in the step that produces key conclusions than enabled by default across the whole pipeline.

    6. Dimension five: cost, accounting for the total rather than the unit price

    The model with the lowest unit price is not necessarily the cheapest. Cost evaluation should compute the full expense of completing the same body of work:

    1. Per-task cost = (input tokens × input price + output tokens × output price)
    2. + retry and repair overhead
    3. + rework caused by unusable output
    4. Cost per 1,000 correct answers = total call cost / number of correct answers × 1000

    Three things are easy to overlook here:

    • Input and output are priced separately: in scenarios with long documents or codebase context, input often accounts for far more tokens than output, which amplifies the impact of the input-side price.
    • Failure cost must be counted: a request that needs two retries to succeed can cost more than twice a single request.
    • Cost per 1,000 correct answers is the real business metric: it puts quality and cost over the same denominator. A model with a lower unit price but noticeably worse success rate may actually be more expensive on this measure.

    On top of that, three general optimization levers apply:

    1. Cache reuse: prefix caching suits stable, unchanged prefix content such as system prompts and knowledge base fragments, and can significantly cut the cost of repeated input; semantic caching suits high-frequency repeated questions.
    2. Tiered routing: send simple tasks such as intent classification and format conversion to lightweight models, and complex reasoning and long-document understanding to larger models.
    3. Batching and asynchronization: merge offline tasks into batches and submit them together to take advantage of the typically lower unit cost of batch processing, while online tasks must preserve interactive latency.

    Illustrative routing logic:

    1. SIMPLE_TASKS = {"intent_classify", "format_convert", "field_extract"}
    2. def select_model(task_type: str, input_len: int) -> str:
    3. """Pick a model tier by task type and input length; each branch must be validated during evaluation."""
    4. if task_type in SIMPLE_TASKS and input_len <= 8000:
    5. return "light" # lightweight model
    6. if task_type == "long_doc_qa":
    7. return "long_context" # long-context model
    8. return "flagship" # flagship model

    One point deserves emphasis: routing is itself an object under test. The accuracy loss of a lightweight model on simple tasks must be quantified, or the “money saved” may come back in the form of user complaints.

    7. Dimension six: compliance and safety, a red line that cannot be skipped

    The first five dimensions decide whether a model can be used; this one decides whether you dare to use it.

    • Jailbreaks and prompt injection: construct attacks such as role-play, encoded obfuscation and instructions embedded in documents, and check whether the model can be induced to break the constraints of its system prompt. In agent scenarios, also verify that instructions hidden in external documents are correctly ignored.
    • Sensitive data protection: test whether the model repeats personally identifiable information from the input, and whether it leaks information across sessions. It is advisable to filter both inputs and outputs at the application layer rather than relying on the model’s self-discipline alone.
    • Content and value risks: build a harmful-content test set that covers your business red lines, and run it as a regular regression.
    • Traceability and auditability: log an input summary, output summary, model version, latency and cost for every call, so that when something goes wrong you can pinpoint the specific version and prompt template.

    In the evaluation report, compliance and safety items should be a veto rather than a weighted factor. When an item crosses a red line, the candidate is out no matter how high the other dimensions score.

    8. Execution: workflow and code skeleton for a six-dimension evaluation

    8.1 The five-step loop

    1. Define scenarios and metrics: settle the task list, business metrics (such as first-contact resolution rate and human handoff rate) and technical metrics (quality, performance, cost).
    2. Build the evaluation set: sample according to business distribution, complete desensitization and annotation, put it under version control, and freeze a “baseline snapshot.”
    3. Run the evaluation in batch: unify prompt templates, sampling parameters and the call wrapper, collect data for all six dimensions, and write results in a single format.
    4. Analyze and decide: score per dimension with weights, use confidence intervals to judge whether differences are real, and produce a selection or regression conclusion.
    5. Keep regressing: hook the evaluation into the CI pipeline so the golden set runs automatically whenever a model or prompt changes, and block the release when a key metric drops beyond a threshold.

    8.2 Evaluation script skeleton

    1. import hashlib, json, os, statistics, time
    2. import requests
    3. from concurrent.futures import ThreadPoolExecutor
    4. ENDPOINT = os.environ["MODEL_ENDPOINT"] # shared inference endpoint
    5. API_KEY = os.environ["MODEL_API_KEY"]
    6. def call_model(messages, **params):
    7. """Shared call wrapper: switching models changes config only, never the evaluation code."""
    8. payload = {
    9. "model": params.get("model", "default"),
    10. "messages": messages,
    11. "temperature": params.get("temperature", 0.0),
    12. "response_format": params.get("response_format"),
    13. "stream": False,
    14. }
    15. t0 = time.perf_counter()
    16. resp = requests.post(
    17. ENDPOINT, json=payload,
    18. headers={"Authorization": f"Bearer {API_KEY}"},
    19. timeout=60,
    20. )
    21. resp.raise_for_status()
    22. data = resp.json()
    23. latency = time.perf_counter() - t0
    24. usage = data.get("usage", {})
    25. return {
    26. "text": data["choices"][0]["message"]["content"],
    27. "latency": latency,
    28. "input_tokens": usage.get("prompt_tokens", 0),
    29. "output_tokens": usage.get("completion_tokens", 0),
    30. }
    31. def cache_key(case):
    32. raw = json.dumps(case, ensure_ascii=False, sort_keys=True).encode()
    33. return hashlib.sha256(raw).hexdigest()
    34. def run_case(case):
    35. """Run one case and collect four signals: quality, stability, performance and cost."""
    36. result = call_model([
    37. {"role": "system", "content": case.get("system", "You are a rigorous business assistant.")},
    38. {"role": "user", "content": case["prompt"]},
    39. ])
    40. out = result["text"]
    41. # Quality: the case supplies its own scorer, e.g. exact match, unit tests or a judge model
    42. quality = case["score_fn"](out, case["reference"])
    43. # Stability: whether the structured output parses on the first attempt
    44. try:
    45. json.loads(extract_json(out))
    46. json_ok = True
    47. except Exception:
    48. json_ok = False
    49. # Cost: estimated from output volume; input/output unit prices come from config
    50. cost = (result["input_tokens"] * PRICE_IN
    51. + result["output_tokens"] * PRICE_OUT)
    52. return {"id": case["id"], "quality": quality, "json_ok": json_ok,
    53. "latency": result["latency"], "cost": cost}
    54. def evaluate(cases, workers: int = 8):
    55. """Run the evaluation set concurrently and cache results for repeatable comparison."""
    56. results, todo = {}, []
    57. for c in cases:
    58. k = cache_key(c)
    59. results[k] = None
    60. todo.append((k, c))
    61. def worker(item):
    62. k, c = item
    63. return k, run_case(c)
    64. with ThreadPoolExecutor(max_workers=workers) as pool:
    65. for k, r in pool.map(worker, todo):
    66. results[k] = r
    67. return report(results.values())
    68. def report(rows):
    69. """Emit the core metrics of the six-dimension report."""
    70. ok = [r for r in rows if r is not None]
    71. n = max(len(ok), 1)
    72. q = statistics.mean(r["quality"] for r in ok)
    73. p95 = sorted(r["latency"] for r in ok)[int(n * 0.95) - 1] if ok else 0
    74. return {
    75. "cases": len(rows),
    76. "quality": round(q, 4),
    77. "json_pass_rate": round(sum(r["json_ok"] for r in ok) / n, 4),
    78. "p95_latency_s": round(p95, 3),
    79. "cost_per_1k_correct": round(
    80. sum(r["cost"] for r in ok) / max(q * len(ok), 1e-9) * 1000, 4),
    81. }

    The skeleton follows three principles:

    • Unified interface: the evaluation code is decoupled from any specific model, so switching models means changing configuration only.
    • Cacheable results: identical inputs are not called twice, which saves money and keeps repeated runs consistent.
    • Centralized metrics: one run yields quality, stability, performance and cost together, making side-by-side comparison easy.

    8.3 Release gate example

    1. # ci/eval-gate.yaml: block the release when the evaluation does not meet the bar
    2. thresholds:
    3. quality: 0.90 # overall golden set quality
    4. json_pass_rate: 0.98 # first-attempt structured output parse success rate
    5. p95_latency_s: 2.0 # end-to-end P95 latency
    6. cost_per_1k_correct: 12 # cost per 1,000 correct answers
    7. safety_block_rate: 0 # red-line samples must be zero
    8. policy:
    9. compare_with: baseline # compare against the baseline snapshot to catch silent regressions
    10. max_regression: 0.02 # allow at most a 2-percentage-point drift on any single metric

    9. Choosing in 2026: five practical recommendations

    Back to the opening question: under a framework like this, what kind of approach is most likely to hold up in real projects in 2026? The answer is usually not a single model, but a model usage strategy grounded in evaluation data:

    1. Evaluate first, conclude later: do not commit to a technology binding before you have real evaluation data on the candidates. Keep business logic decoupled from any specific model, and reach different model services through a unified gateway or abstraction layer.
    2. The golden set is the core asset: it holds more long-term value than any single evaluation report. It can be reused across version iterations and becomes the team’s shared definition of what a good answer looks like.
    3. Compose per scenario rather than standardizing globally: high-frequency simple tasks such as structured extraction and intent recognition go to a lightweight model with strict validation; complex reasoning and long-document understanding go to a stronger model; critical paths get a backup model and a fallback strategy.
    4. Make regression a habit: models iterate far faster than business systems, and only by wiring evaluation into the release process can you avoid “it quietly got worse after the upgrade.”
    5. Account for cost per business unit: deciding on cost per 1,000 correct answers rather than unit price is the only way to reflect both quality differences and resource consumption.

    Common pitfalls at a glance

    Common practice Potential problem Suggested improvement
    Picking a model by overall ranking alone The ranking has nothing to do with your task distribution Re-test with a scenario-based golden set
    Judging capability from public benchmarks only Data contamination inflates scores Combine public benchmarks with in-house questions
    Evaluating once and never updating Quality shifts silently after a model upgrade Hook evaluation into CI and regress on a schedule
    Comparing scores obtained under different setups Different prompts and parameters make differences unattributable Fix the template, temperature and output length
    Measuring only average scores and average latency Long-tail failures stay hidden Track variance plus P95 and P99
    Looking at unit price only Retry overhead and low success rate costs are ignored Decide on cost per 1,000 correct answers

    Conclusion

    The essence of model evaluation is building a trustworthy basis for decisions in an uncertain technical environment. A six-dimension framework does not chase the “strongest model”; it makes every selection and every upgrade defensible. When evaluation data can reliably answer whether quality meets the bar, whether differences are real, whether cost is controllable and whether the risk is bearable, a team can stay composed no matter how fast models iterate.