Comprehensive LLM Evaluation: Methods and Engineering Practices
This article presents a practical methodology for evaluating large language models beyond parameter counts and public leaderboard rankings. It breaks business-relevant capabilities into repeatable, auditable test sets and examines four high-frequency areas: web retrieval, cross-domain agents, deep research, and text-to-executable 3D code. It covers task definitions, execution protocols, scoring rubrics, failure taxonomies, environment reproducibility, safety boundaries, leaderboard interpretation, and a sustainable evaluation pipeline.
Evaluating large language models should not rely only on parameter scale or rankings on public leaderboards. Parameter count indicates a capacity ceiling; it does not show whether a model can reliably call tools in real tasks, follow constraints, or maintain factual consistency across long chains. A more valuable approach is to break the capabilities that matter to the business into repeatable, auditable test sets and compare different candidate models and system configurations under the same execution protocol.
This article focuses on four high-frequency capability areas: web retrieval, cross-domain agents, deep research, and text-to-executable 3D-related code. It discusses evaluation design and engineering implementation without binding the discussion to any specific model or vendor product.
Define What to Measure First
Comprehensive evaluation can easily turn into “the more questions, the better.” If task definitions are vague, adding more questions makes scores harder to explain. It is better to specify four things first.
First, the task goal. A retrieval task, for example, should require “a verifiable answer with sources,” not “a seemingly complete literature review.” An agent task should require “a state change completed within limited tools and steps,” not “fluent conversation.”
Second, allowed actions. Whether the system can access the web, execute code, modify files, the maximum number of steps, timeout, and whether it can retry after failure must all be fixed in advance. Otherwise, the same model under a different agent framework can produce large score fluctuations, and comparison loses meaning.
Third, scoring granularity. Outcome scores check whether the final answer is correct; process scores check whether the system selected the right tools, read key evidence, and avoided unauthorized operations. Looking only at outcomes hides lucky guesses; looking only at process lets through cases with plausible steps but wrong conclusions.
Fourth, failure types. Errors should be classified into retrieval miss, citation that does not support the conclusion, tool parameter error, looping calls, context truncation, non-runnable code, unmet geometric constraints, and similar categories. Classification is more useful for system improvement than a total score.
Web Retrieval: Test the Evidence Chain, Not the Prose
Web retrieval tasks usually include open questions, multi-hop facts, and time-sensitive information. A common mistake is to give a high score directly to a long model-generated answer. A more reliable protocol is:
- The question must have verifiable criteria: at least one of the key entities, values, time, or relationships should be checkable automatically or semi-automatically.
- The system must return evidence pointers: title, publication time, paragraph location, or an equivalent traceable marker. A correct sentence without a source should only be recorded as “unverified.”
- Distinguish “retrieval failure” from “reading failure.” The former means the answer is absent from candidate documents; the latter means the answer exists but summarization or reasoning changes it incorrectly.
- Control the retrieval budget: query count, number of returned results, and maximum webpage body length must be standardized. Otherwise, scores mainly reflect the retriever and crawling strategy rather than the model itself.
- Add adversarial samples: outdated pages, clickbait, same-name entities, and contradictory sources. The model should flag conflicts rather than force a single definite answer.
Scoring can be split into hit rate, citation support rate, conflict-handling correctness, and redundant query rate. Citation support is especially important: every key assertion in the answer should have direct support in the cited snippet. This reduces speculative behavior such as writing a conclusion first and then attaching links.
In engineering, it is advisable to split retrieval, reranking, reading, and answering into observable stages and log each query and hit document identifier. During regression, if the score drops, teams can locate whether query rewriting got worse or the reading stage began ignoring constraints.
Cross-Domain Agents: Environment Matters More Than Prompts
Cross-domain agent evaluation simulates real workflows: looking up information, calling APIs, changing state, and replanning based on errors. The difficulty is not single-turn question answering but multi-step decision-making.
The environment must be reproducible. If tools are real external services, results drift with network, permissions, and data changes. A more reliable approach is a controlled sandbox: fixed initial state, fixed tool schema, fixed clock and random seed, and external dependencies replaced with recorded stubs or read-only snapshots.
Tasks should cover multiple domains, but each task should test only a limited combination of skills, such as “query—calculate—write” or “read logs—locate anomaly—give repair suggestion.” Domain breadth prevents the model from memorizing a single vertical template; it should not encourage unbounded open-ended operations.
Process metrics should include:
- Planning efficiency: whether the first step points to a tool that can accomplish the goal rather than unrelated chat.
- Parameter validity: whether required fields, types, and enum values match the schema.
- State progress rate: whether the distance to the goal state decreases after each step. This can be measured with checkpoints or diffs.
- Recovery ability: after a tool returns an explicit error, whether the system corrects parameters or changes path instead of retrying unchanged.
- Termination discipline: whether the system stops when the goal is reached or confirmed impossible, avoiding empty loops that consume steps.
Safety boundaries should be scored separately. Even if the correct answer is produced, the item should fail if the model attempts to access unauthorized resources, perform destructive operations, or leak key placeholders in the environment. If agent scores reward only “getting things done,” they systematically encourage overreach.
When comparing different models, the agent policy, retry count, temperature, and maximum steps must be locked. Otherwise, you are comparing the whole orchestration rather than the model. If the business goal is to compare orchestration approaches, then fix the model instead, change only the planner and memory strategy, and state this clearly in the report.
Deep Research: Traceable Synthesis, Not Length
Deep research tasks require a model to collect multi-source materials around a question, compare viewpoints, and provide limited conclusions. The difference from ordinary question answering is the larger information volume, conditional conclusions, and more hidden errors.
The rubric can be split by dimension instead of giving a single subjective impression score:
- Coverage: whether all required points pre-listed in the task appear. The points come from task design, not from the model’s own outline.
- Traceability: whether each point is bound to specific material and whether the material matches the statement.
- Synthesis: whether facts, inferences, and uncertainties are distinguished, rather than writing inferences as facts.
- Structure: whether the answer is organized around the question rather than stacked by source material.
- Restraint: whether the system clearly says it does not know when evidence is insufficient, rather than filling in details.
Human review is easily influenced by prose quality. Automated checks can first handle format, citation existence, and point coverage, and then disagreement samples can go to double review. During review, hide model identity and provide a uniform rubric to reduce the bias that “longer is better.”
Research tasks also need to prevent data leakage. If questions and reference answers come from public long-form text, the model may have seen similar text during training. Mix in newly constructed scenarios, rewritten values, and details that exist only in the evaluation corpus, and rotate questions regularly. When reporting scores, state whether the questions were “visible” to the model, to avoid treating memory as research ability.
Text to 3D Code: Executability Matters More Than Looking Correct
Generating 3D-related code from natural language tests specification understanding, geometric constraints, and engineering executability. A good-looking render cannot replace testing.
Three layers of acceptance are recommended.
Specification layer: whether the dimensions, counts, relative positions, materials, or naming given in the input are respected. Structured assertions can check the scene graph or intermediate representation rather than only screenshots.
Execution layer: whether the code can complete parsing, building, and export in a fixed runtime. Syntax errors, missing dependencies, timeouts, and runtime exceptions should be captured. When sampling multiple times for the same task, report the pass rate rather than only the best run.
Geometry layer: whether bounding boxes, volume, intersections, clearance, orientation, watertightness, and other constraints are satisfied. For subjective items that cannot be fully determined automatically, such as style similarity, reduce their weight and separately note their subjectivity.
An example check can be simple: after exporting the generated result as a mesh or scene description, read the vertex ranges and determine whether the target object falls within the allowed error. Error thresholds, coordinate systems, and units must be specified in the task; otherwise, different implementations may be misjudged due to unit conversion.
Assertion examples (semantic level):1. The program exits successfully within the time limit.2. The scene contains the specified subject by name.3. The subject's dimensions along each axis are within the allowed error of the requirements.4. The subject's relative relationship with the ground or reference object matches the task.5. No prohibited intersection or penetration exists.
Do not treat a specific high score from one run as proof of capability. Sampling temperature, post-processing repair scripts, and hidden retries all mix “the model wrote it correctly in one pass” with “the system repaired it until it ran.” Reports should separately give the zero-repair pass rate and the pass rate after allowed automatic repair.
How to Read Leaderboards Without Being Led by Them
A summary table across many models and tasks is easy to misread. A more reliable reading approach is:
- Compare only scores produced under the same protocol: same tools, steps, temperature, scoring scripts, and data version.
- Look at mean, variance, and worst percentile at the same time. Agent and code tasks often have high variance; reporting only averages hides instability.
- View results by capability cluster instead of combining retrieval, research, and code into one total score. Composite weights are essentially business choices, not objective truth.
- Publish the proportions of incomplete, refused, and timed-out cases. Treating all refusals as zero penalizes cautious models; treating all refusals as correct rewards inaction.
- Keep raw traces for spot checks. A high score without traces cannot be reviewed or improved.
For launch gates, use “item thresholds plus a regression set” rather than a single champion threshold. For example, retrieval citation support should not fall below a defined line, the agent unauthorized-operation rate should be zero, and the code zero-repair pass rate should not fall below a defined line. Thresholds should be determined by misjudgment cost and human review capacity, not copied from external leaderboards.
Building a Sustainable Evaluation Pipeline
One-off scoring quickly becomes outdated. Evaluation should be treated as continuous engineering.
Data layer: version questions, reference points, environment snapshots, and scoring scripts. Any question change should increment the version, and old scores should not be compared directly with new ones.
Execution layer: fix hardware or at least fix timeout and concurrency; record model identifiers, decoding parameters, random seeds, and dependency versions. For non-deterministic APIs, repeat runs to estimate confidence intervals.
Observation layer: save queries, tool parameters, error codes, token or equivalent length statistics, latency, and cost basis. If billing rules are unclear, record only call counts and elapsed time rather than estimating monetary amounts.
Analysis layer: produce reports by error type and manually sample cases that are “high score but citation unsupported” and “low score but direction correct.” The former prevents metrics from being gamed; the latter prevents overly strict metrics from missing usable systems.
Governance layer: isolate evaluation data from production user data; apply least privilege to sandboxes by default; redact logs. Evaluation itself should not become a channel for leaking internal materials or attacking external systems.
Trade-offs in Deployment
No single score can represent all product forms at once. Retrieval assistants should weight citation support and conflict handling more heavily; automation workflows should weight state progress, recovery, and permissions; research briefs should weight coverage and uncertainty expression; 3D generation should weight executability and geometric constraints.
Parameter scale, context length, and training data volume can serve as background information, but they cannot replace the measurements above. System prompts, retrievers, memory, and retry strategies often affect results no less than swapping the model. Therefore, an evaluation report should include at least: task definitions, environment constraints, scoring rubric, sample size, repetition count, and failure distribution. Without these, rankings have little explanatory power.
Turning these four task types into a small regression-ready set is more useful than piling up hundreds of open questions that cannot be reviewed. First make the correctness of each question clear, then expand coverage; only then do scores become engineering signals rather than marketing material.
