How to Evaluate a New Large Language Model: Capability and Cost
When a general-purpose large language model ships a new version, developers want to know how it performs in practice—but "experience" is subjective. Turning it into comparable, reproducible conclusions means answering three questions: what the model can do, how consistently it does it, and how much a completed task actually costs. This article proposes a "scope first, then task set, then total cost" evaluation workflow and shows how to build a versioned, reproducible evaluation record—covering scope definition, task-matrix design, per-dimension scoring, statistical stability testing, and total cost accounting—so teams can keep reusing it across fast-moving model releases.
Every time a general-purpose large language model ships a new version—especially a general-availability (GA) release—developers first ask, “How does it actually perform in practice?” But “experience” is a subjective impression. To turn it into comparable, reproducible conclusions, an evaluation must answer three questions: what the model can do, how consistently it does it, and how much completing a task actually costs.
The takeaway up front: The sense of a “leap” from preview to GA usually shows up in two areas—the breadth of task generalization and the quality of individual task completion. On engineering metrics such as stability and cost, however, clear gaps often remain. This article therefore advocates an evaluation workflow of “define the scope first, build the task set next, and tally the total cost last.” Its output is not a single “strong” or “weak,” but a versioned evaluation record with a defined statistical basis that teams can reuse in an environment of fast-moving releases.
1. Establish the premises first: what you are evaluating and where the boundaries lie
1.1 Recognize the “preview then GA” release cadence
Most mainstream general-purpose models today follow a two-stage release pattern: preview first, GA later.
- Preview stage: mainly used to collect feedback; whether the direction is right becomes clear quickly.
- GA stage: focused polishing during continued training over the following weeks to months.
Many teams adapt and validate their pipeline during the preview stage, then rerun the same task set and criteria when GA ships. This directly answers “which issues were resolved, and which remain?” Comparisons across versions and generations then share a common baseline.
1.2 Define the test scope
LLM evaluation typically spans logical reasoning, knowledge Q&A, code generation, agent tasks, and more. But different capabilities demand different interaction patterns:
- Reasoning questions usually require a complete answer in a single pass.
- Agent tasks are a multi-turn process of “understand the goal → break down steps → call tools → produce artifacts,” where the usability of the output matters more than intermediate reasoning.
Many teams prioritize agent tasks because they most closely mirror the real workflow of “hand the model a development task and see whether it can deliver usable artifacts.” But a model limited in single-response length may fail to complete a complex reasoning question in one reply, distorting scores in that category. To avoid misreading the conclusions, the evaluation report should state the coverage and the reason explicitly—for example, “This round tests agent tasks only, excluding reasoning items.”
1.3 Lock down the evaluation environment
The same model can score noticeably differently under different execution frameworks (harnesses—the outer layer that assembles prompts, schedules tools, and manages context), for three reasons:
- The framework determines which tools are available, the context-management strategy, and the retry behavior after failures.
- Frameworks differ widely in how well they support long tasks and multi-file edits.
- The instruction-format fit between the model and the framework also affects real-world performance.
In practice, a “one primary harness + one comparison harness” approach works well: the primary harness handles vertical comparison, and the comparison harness provides cross-validation, separating “a problem with the model itself” from “a problem with the framework integration.”
Record the following metadata for every experiment. It is the basis for traceable, reproducible results:
| Metadata item | Purpose |
|---|---|
| Model version and call time | Distinguish preview from GA, attribute issues correctly |
| Framework name and version | Rule out framework-driven differences |
| Task set version and scoring-rubric version | Keep vertical comparisons on a consistent basis |
| Hardware and environment configuration | Ensure artifacts can be reproducibly built |
2. Task set design: making “generalization” measurable
2.1 From “prefers certain tasks” to “broadly usable”
In the preview stage, models often show obvious specialization. Take front-end page generation: some models do well on UI fidelity and interactions, but the moment you switch to back-end services, data processing, or ops scripts, completion quality drops sharply.
For GA, the key measure of progress should become: whether all standard tasks can be autonomously pushed past the “usable line.” “Usable” means it can be integrated into a project without major rework; “pushed” means it does not require step-by-step human prompting. This shift marks the model’s evolution from a “specialist” toward a “general-purpose executor.”
2.2 Structure of the standard task set
Build the task set as a two-dimensional matrix of “scenario dimension × difficulty level”:
| Scenario dimension | Typical task examples | Core focus |
|---|---|---|
| Front-end and UI | Component development, page reproduction, interactive effects | Visual fidelity, component breakdown, accessibility |
| Back-end and services | API implementation, authentication, concurrency and error handling | Edge-case handling, testability |
| Data and scripts | Data cleaning, statistics scripts, report generation | Data correctness, idempotency |
| Engineering and ops | Build scripts, container configuration, log integration | Reproducibility, configuration safety |
| Algorithms and problem solving | Common algorithm implementation and performance optimization | Correctness, complexity |
| Maintenance and refactoring | Legacy system rework, multi-file consistency changes | Global consistency, regression risk |
Suggested difficulty levels:
- L1: single file, clear requirements.
- L2: multiple files, with clear acceptance criteria.
- L3: vague requirements that require the model to search and clarify on its own.
- L4: cross-module refactoring where one change affects everything downstream.
2.3 The value of non-standard tasks
Beyond a fixed task set, design a large number of “non-standard tasks,” such as:
- Uncommon tech stacks.
- Business rules that require substantial external search.
- Legacy system rework with no documentation.
These tasks specifically test whether the model will “do a sloppy job” in unfamiliar situations. The ideal behavior: the model first searches for background, then progresses step by step, proactively filling in missed logic in later prompts.
2.4 Acceptance: defining the “usable line”
“Usable” must be defined by acceptance conditions, not by feel. Common practice:
- Predefine an automatically runnable check script for each task—for example, does the build pass, do all unit tests pass, and does the interface contract match expectations.
- Artifacts must pass preset lint and type checks.
- Automatable parts are scored by script; subjective parts (for example, UI aesthetics) are scored independently by multiple people, then take the median.
3. Scoring system: per-dimension judgment and vertical calibration
3.1 Score by dimension, not by overall impression
A single overall score easily lets “highlights” mask “hard defects.” A more robust approach judges each dimension separately. Common dimensions:
- Runnability: whether artifacts build, start, and pass tests.
- Requirement coverage: whether explicit requirements and reasonable implicit ones are all implemented.
- Instruction following: whether specified tech stack, directory structure, and code style are respected.
- Global consistency: whether naming, error handling, and configuration management are consistent across the whole project.
- Engineering quality: readability, testability, and security.
3.2 Make deductions explicit
Each dimension should list concrete deduction points, for example:
- Runs, but fails edge cases.
- Excellent locally, but the overall architecture is fragmented.
- Introduces heavy dependencies without confirmation.
- Hard-codes secrets, tokens, or other sensitive data in source code.
The more specific the deduction points, the smaller the divergence between different evaluators.
3.3 Watch for rubric “drift”
Evaluation has an easily overlooked problem: as stronger models appear, the ceiling of completion quality keeps rising. Artifacts that were excellent six months ago may only count as “barely usable” by today’s standard.
Suppose an evaluator raises the bar at some stage—for example, downgrading a certain interaction-refinement task from “excellent” to “medium.” Historical scores can then no longer be directly compared with new scores. The right handling:
- Version the rubric: record each change and the reason for it.
- Recalibrate the baseline: in each round, rescore an older model against the new rubric to get a comparable baseline.
- Report accordingly: present results as “this round’s score / comparable baseline score.”
3.4 Example: a unified scoring-record structure
from dataclasses import dataclass, field@dataclassclass ScoreRecord:task_id: str # Task identifierrunnable: bool = False # Whether it builds and startstests_passed: float = 0.0 # Automated test pass ratecoverage: float = 0.0 # Requirement coverage rateinstruction_ok: bool = True # Whether specified constraints are followedconsistency_penalty: int = 0 # Global consistency deductionquality_penalty: int = 0 # Engineering quality deductiondeducations: list = field(default_factory=list) # Concrete deduction pointsdef usable(self) -> bool:# Example criterion: runnable and test pass rate >= 80% counts as usablereturn self.runnable and self.tests_passed >= 0.8
4. Generalization: from “can do it” to “can push every task forward”
4.1 A typical gap: rework from unfamiliar knowledge
For models whose task generalization has improved markedly, the common gap is no longer “cannot understand the goal,” but insufficient familiarity with unfamiliar domains:
- Requires substantial searching.
- Easily misses domain-specific details.
- Low completion from a single direct output.
Whether this gap is acceptable hinges on whether the model can fill it autonomously through follow-up dialogue. For example, after being prompted that “login-state validation is missing,” if the model proactively locates the relevant module and completes the logic in one go, it has a usable “self-correction” ability—it simply started with insufficient information.
4.2 Evaluation method: controlled experiments
Split standard tasks into two groups:
- Group A: provide ample domain background and an acceptance checklist.
- Group B: give only the goal, with no extra background.
Compare the “direct-output usable rate” and the “usable rate after supplemental prompts” of the two groups, quantifying it as:
Self-correction index = (usable rate after supplemental prompts − direct-output usable rate) / direct-output usable rate
The higher this index, the better the model is at converging to the correct result through interaction when information is lacking.
4.3 From evaluation to deployment
For teams that frequently switch business domains, a “slightly lower direct-output usable rate” is not fatal. What truly needs guarding against is “failing to understand the task goal and doing a sloppy job.” In engineering terms, break the prompt flow into four stages—“explore → plan → implement → self-check”: first have the model search and read the relevant code, then produce an implementation plan, confirm it, then code, and finally run a self-check script. This structured flow often improves completion in unfamiliar scenarios more than simply stacking prompts.
5. Stability: quantifying “high ceiling, low floor” with statistics
5.1 Why repeat runs
LLM output is stochastic, so a single test can be misled by extreme results. Some models produce a highly usable result on the first try for a given task, with UI refinement and animation even better than some peers in the same tier, at modest cost; but with a different random seed, they may exhibit “floor problems”—ignoring instructions and attending to local details while disregarding global consistency.
To evaluate such “high ceiling, low floor” models, you must run repeated trials and report the mean, minimum, and standard deviation together.
5.2 Repeated-trial scheme
import statistics as stdef evaluate_task(task, run, repeats=5):records = [run(task, seed=s) for s in range(repeats)]scores = [r.usable() for r in records] # 1 = usable, 0 = not usablereturn {"task": task.task_id,"pass_rate": st.mean(scores), # usable rate, reflects the average level"min_score": min(scores), # worst case, reflects the floor"std": st.pstdev(scores), # volatility"cost": st.mean([r.cost for r in records]), # average cost per run"worst_case": min(records, key=lambda r: r.usable()), # keep the worst sample}
- Number of repeats: 3 to 5 per task recommended; increase to 10 for critical tasks.
- Worst-case sample: must be kept in full for failure attribution.
5.3 Attributing the two classes of floor problems
| Problem type | Typical behavior | Likely cause | Mitigation |
|---|---|---|---|
| Instruction not followed | Specified tech stack or directory structure ignored | Constraints diluted in long context | Restate key constraints in a structured way and auto-check artifacts with a script |
| Local-first | Each file looks reasonable alone, but style, config, and naming are inconsistent overall | Lack of a global view | Generate a project-conventions file first, base every later step on it, then validate consistently at the end |
5.4 Stability determines whether unattended operation is possible
A high ceiling means “room for delight”; a low floor means “cost of a safety net.” If a task must run unattended, use the strict criterion “usable in all worst-case runs too.” For tasks with poor reproducibility, you must set up human review or an automatic fallback rather than rely on average performance.
6. Cost: from “unit price” to “task cost”
6.1 Common pitfall: looking only at unit price
Many selection decisions stop at comparing the price per million input/output tokens, but what truly drives the bill is “the total cost of completing a task,” estimable as:
Task cost ≈ call cost × average attempts + tool execution overhead + failed-rerun cost + human intervention cost
Teams that trial models across tiers often share one impression: some models still have a clear gap from the “cost-performance benchmark” in task cost, usually for three reasons:
- Agent tasks include a large amount of code context on the input side, so input consumption is often far higher than output.
- Insufficient stability causes multiple retries.
- Floor problems require human rework.
6.2 Three controllable cost types
| Cost type | Main sources | Optimization ideas |
|---|---|---|
| Call cost | Repeatedly sending context, long outputs | Trim context on demand, load relevant files in layers |
| Retry cost | Repeated regeneration due to unusable artifacts | Task splitting, step-by-step validation, fail-fast |
| Human cost | Need for human review and rework | Increase automated acceptance coverage, define clear human-takeover conditions |
6.3 The judgment from “expensive” to “worth it”
A cost disadvantage is not necessarily a disadvantage. If a model is clearly better in UI refinement, interactive animation, and the like, then “reaching higher completion with fewer prompt rounds” itself offsets some of the extra spend. So a more reasonable comparison basis is:
Cost per usable artifact = total task cost / number of artifacts meeting the usable standard
Only when completion quality is comparable but cost is significantly higher does it truly constitute “insufficient cost-performance.”
7. Engineering responses under output-length constraints
7.1 Effects of the constraint
Some general-purpose models currently cap single-response output length on their public interfaces (for example, around 64K tokens). For agent tasks, this means:
- Large codebase refactors may not be completable in a single reply.
- Reasoning questions that require a complete derivation in one shot may lose points due to truncation.
This is also why many evaluation reports explicitly “don’t test reasoning items.” Rather than avoid it, it is better to address this constraint head-on in the evaluation.
7.2 Task splitting
Break large tasks into several independently verifiable subtasks. For example, split a backend system into four steps—“data model definition → API implementation → front-end pages → integration and testing”—and validate each step’s artifacts separately.
7.3 Chunked generation and self-checking
Illustrative code (to explain the idea, not an actual API call):
def build_module(spec, max_chunk=8000):"""Generate code in chunks per the spec doc, with step-by-step self-checking."""pieces = []offset = 0while offset < len(spec):chunk = spec[offset:offset + max_chunk]# Leave headroom for a single output to reduce truncation risktext = model.generate(prompt=f"Please implement the code for the following segment; output code only:\n{chunk}",max_output_tokens=max_chunk,)if is_truncated(text):# When output is truncated, halve the chunk size and retrymax_chunk //= 2continuepieces.append(text)offset += max_chunkreturn "\n\n".join(pieces)
7.4 Step-by-step self-checking
Run build, static checks, and unit tests immediately after each step. This prevents errors from propagating downstream and avoids the waste of “getting halfway then hitting the output cap, forcing a restart from scratch.”
8. Putting evaluation into practice: keeping the process running
8.1 An automated evaluation pipeline
Models iterate fast, so evaluation should be automated too. Recommended flow:
- Pull: batch-pull models to be evaluated by version list.
- Execute: start isolated environments on a container platform and run the task matrix in batch.
- Collect: write call logs, artifacts, elapsed time, and consumption to object storage in a unified way.
- Accept: run test scripts automatically and generate per-dimension scores.
- Aggregate: write results into a long-term tracking dashboard, and sync anomalous samples to log services and monitoring alerts.
8.2 Visualization dashboard
| Metric | Meaning | Focus |
|---|---|---|
| Usable rate | Share of tasks meeting the usable standard | Whether it keeps rising across versions |
| Worst case | Lowest score in repeated trials | Whether it touches the floor |
| Volatility | Standard deviation of scores for the same task | Whether it suits unattended operation |
| Cost per artifact | Average cost per usable artifact | Whether cost-performance improves |
Conclusion
Back to the original question: how do you judge how a newly released general-purpose LLM “actually performs”? The answer lies not in any single benchmark, but in whether you have built an evaluation system matched to your own business scenarios. Such a system should include at least:
- A clear scope statement.
- Reproducible environment and version records.
- A task matrix with complete structure, covering both standard and non-standard scenarios.
- Per-dimension scoring and repeated trials.
- A cost basis that folds in calls, retries, and human intervention.
Version iteration will not stop, but the evaluation method can be accumulated as a team asset. With it, every new release can be quickly and accurately folded into your existing technology-selection framework, leading to more rational decisions.
