Evaluating a New Large Language Model: Key Capability Dimensions and Practical Methods
A model's "feels stronger" reputation is not enough to base a production decision on. This article frames LLM evaluation as an engineering problem: building a reproducible, quantifiable benchmark aligned with your business goals. It walks through eight dimensions worth measuring separately — parameter scale, long-context usability, coding ability, structured creative output, domain accuracy, long-horizon agent performance, and inference speed — and explains how to test each one, followed by a practical five-step process for building an evaluation pipeline, from assembling a test set from real traffic to continuous regression before every model or prompt change.
Introduction: Benchmark Scores Alone Cannot Tell You If a New Model Fits
Every new generation of large language models triggers a wave of subjective “feels more capable” opinions from the community. For developers and enterprise users, however, subjective impressions do not translate directly into production decisions. Whether a model is genuinely usable in your business depends on how it performs under your own data distribution, task mix, and latency and cost constraints.
“Evaluating a new model” is therefore fundamentally an engineering problem: how to design a reproducible, quantifiable evaluation system that is aligned with business objectives. This article breaks down the key capability dimensions and gives practical methods for putting them to work.
1. Parameter Scale and Capability: Not a Linear Relationship
Headline sizes such as trillion-parameter scale are often cited as intuitive proof that a model is “the most capable,” but in practice there is a clear distinction to keep in mind: parameter scale and downstream capability are not in a simple linear relationship.
- Returns from scale diminish at the margin; beyond a certain size, the gain on the same task narrows.
- Training data quality, data mixture, and post-training strategies (instruction tuning, alignment with human feedback) can influence final capability as much as adding parameters.
- Under architectures such as mixture-of-experts (MoE), “total parameter count” has to be understood alongside the compute actually activated at inference time; it is not equivalent to effective compute.
Evaluation tip: Do not judge capability by parameter count alone. Measure task completion, accuracy, and output stability directly on your own evaluation set.
2. Long Context: From Fitting the Input to Actually Using It
Supporting million-token context is a common selling point of new-generation models, but long-context capability has two distinct levels:
- Effective length: whether the model can accurately locate and use key information in long input, rather than performing well only at the beginning or the end;
- Positional robustness: whether recall drops noticeably when key information sits in the middle of the context, the phenomenon often called “lost in the middle.”
A practical test is needle-in-a-haystack: insert one key fact at different relative positions in a long text and measure recall accuracy. Example code:
import randomdef build_context(fact: str, total_len: int, pos_ratio: float) -> str:"""Build a long context with the key fact inserted at a given relative position."""filler = ("This is filler text used to pad the context. "* max(1, total_len // 20))chars = list(filler)insert_at = int(len(chars) * pos_ratio)chars[insert_at:insert_at] = list(fact)return "".join(chars)def evaluate_long_context(model_call, fact, answer, positions=(0.1, 0.3, 0.5, 0.7, 0.9)):"""Insert the fact at several relative positions and measure recall accuracy."""results = {}for p in positions:ctx = build_context(fact, total_len=32000, pos_ratio=p)prompt = f"Answer the question based on the material.\nMaterial: {ctx}\nQuestion: {fact}"output = model_call(prompt)results[p] = answer.lower() in output.lower()return results
If accuracy around position 0.5 is clearly lower than at the two ends, the model’s long context is more about input capacity than genuine usability.
3. Coding Ability: Test It Close to Real Engineering Work
Evaluating code generation easily falls into the trap of “passing algorithm problems.” A more production-relevant approach is layered testing:
- Snippet level: function completion, code explanation, locating common bugs;
- File level: cross-function refactoring, unit test generation, understanding dependencies;
- Repository level: completing an end-to-end feature given a repository structure, which tests whether the model can retrieve and call the relevant files from its context.
Build a private evaluation set that matches your own tech stack, and focus on compilable rate, test pass rate, and the gap between pass@1 and pass@k. The latter is a better signal of how stable the model is in a real development workflow.
4. Creative and Multimodal Tasks: Accuracy of Structured Generation
In creative scenarios such as 3D authoring and image understanding, the point of evaluation is not subjective impressions like “does it look good” but the accuracy of structured output: whether the generated scene descriptions, coordinate parameters, and hierarchy relations are valid, parseable, and reusable. A sensible combination is rule-based validation plus manual spot checks: automatic checks for syntactic validity, and small-sample human scoring for semantic plausibility.
5. Domain Expertise: Let Task Accuracy Speak
In vertical domains such as financial research, evaluation should focus on verifiable factual metrics: whether data references are accurate, whether the reasoning chain is self-consistent, and whether conclusions can be traced back to their sources. Hallucination rate deserves particular attention here, meaning the share of outputs that look plausible but are factually wrong. Domain evaluation must include adversarial samples that specifically test whether the model “makes things up” when information is insufficient.
6. Long-Horizon Agent Capability: Completion Rate on Multi-Step Tasks
Long-horizon agent capability measures how a model performs on multi-step tasks that require tool calling and state maintenance, such as a “retrieve material, analyze, generate report” flow. Evaluation dimensions include:
- Task completion rate: whether the stated goal was ultimately achieved;
- Step efficiency: whether intermediate decisions are efficient, with no redundancy or dead loops;
- Error recovery: whether the agent can self-correct after a mid-task failure.
These evaluations usually need replayable trace logs, so that failures can be traced to the planning, execution, or summary stage.
7. Inference Speed vs. Intelligence: The Value of Fast Variants
Beyond peak capability, engineering teams also care about response latency. A common industry practice is to ship a high-speed inference variant at the same level of intelligence, using techniques such as speculative decoding, quantization, and operator optimization to increase output speed several times over. Evaluation should separate these coordinates clearly: time to first token (TTFT), output tokens per second (TPS), and throughput under high concurrency. For real-time interactive scenarios, speed is often as important as capability.
8. Building an Evaluation System You Can Actually Use
Across all the dimensions above, a practical evaluation workflow can be summarized as five steps:
- Define the scenario: list the high-frequency tasks in your target business along with hard constraints (latency, cost, compliance);
- Build the evaluation set: sample from real traffic, covering typical, edge, and adversarial cases;
- Automate scoring: use rules where possible (accuracy, pass rates) and model-based review with manual spot checks where things are hard to quantify;
- Canary validation: run an A/B comparison on real traffic and calibrate offline conclusions against online metrics such as conversion rate, adoption rate, and human edit rate;
- Continuous regression: freeze the evaluation set as a regression baseline and rerun it whenever the model or prompts change.
On the platform side, this kind of evaluation and deployment can rely on general-purpose capabilities such as model serving, monitoring and alerting, and logging. Baidu AI Cloud, for instance, offers general capability categories such as model services, logging, and monitoring that teams can use to connect offline evaluation with online observation, so that selection decisions rest on evidence.
Conclusion
Evaluating a new-generation model comes down to translating “feels stronger” into “measurably better in my scenario.” Parameter scale, context length, coding and creative ability, domain reasoning, long-horizon agent capability, and inference speed are separate axes, each of which needs to be measured on its own. Only by building your own evaluation set and sticking to canary validation and continuous regression can you make rational model choices that genuinely serve the business, even when surrounded by marketing claims.
