LLM Evaluation: How to Measure Capability and Cost-Effectiveness of New Models

1. From Launch Hype to Business Value

When a new generation of large models is released, official leaderboards often show broad improvements across metrics. In real business use, however, developers usually care about two questions:

1. Actual capability: Can the model reliably produce directly usable results on tasks like mine? Is the improvement large enough to cross the production deployment threshold compared with the current solution?
2. Cost-effectiveness: How much does the cost per useful output increase? Can the total cost be covered by the business benefit?

A release can introduce two kinds of change at the same time:

• Capability improvements: stronger reasoning depth, longer context windows, better instruction following, and improved code generation and tool calling.
• Pricing and quota changes: tiered pricing, cache-read discounts, batch discounts, subscription quota limits, and call limits such as requests per minute or tokens per minute.

Focusing on only one side can lead to the wrong conclusion. If you focus only on capability, you may pay a premium for capabilities that are not needed. If you focus only on unit price, you may ignore hidden costs caused by more calls, retry expansion, and manual rework.

The neutral goal of this article is to define a reusable method for assessing the real capability and overall cost-effectiveness of new large models in enterprise scenarios. The following sections cover terminology, evaluation dimensions, evaluation-set design, cost modeling, quota and channel strategy, gradual rollout, and long-term governance.

2. Terminology and Basic Assumptions

• Input tokens and output tokens: These are the billing units for model text processing. Different models use different tokenizers, so the same text may produce different token counts. When comparing unit prices across models, you must normalize by each model’s tokenization result.
• Reasoning tokens / chain-of-thought tokens: Some models that support deep reasoning generate intermediate reasoning content before producing the final answer. These tokens are usually counted as output tokens, billed at the output price, and not visible to users.
• Cache hit: For repeated prefixes, such as system prompts or long knowledge contexts, some model service platforms support caching. Once a cache hit occurs, the request may be billed at a discounted rate. Cache hits usually require a minimum length and are limited by cache lifetime.
• Batch processing / asynchronous batch inference: Requests are grouped into batches and submitted asynchronously. This is suitable for offline tasks that are not sensitive to latency. Common industry platforms often provide a clear discount for batch inference.
• Cost per task: The total cost of all model calls needed to complete one business task. A single task may involve multiple dialogue turns, multiple tool calls, and several retries.
• Unit cost per qualified output: Cost per task divided by the first-pass success rate. This is the most important metric in cost-effectiveness evaluation.

One point should be clarified: reasoning enhancement does not mean that “more computation is always better.” For simple tasks, overly long reasoning only increases output tokens and latency, which raises cost. Therefore, evaluation must distinguish between task difficulty levels.

3. Measuring Real Capability: From “It Feels Stronger” to Quantifiable Metrics

3.1 Six Core Evaluation Dimensions

Dimension	Evaluation focus	Recommended metrics
Reasoning and planning	Multi-step logic, numerical calculation, solution decomposition	Accuracy, step correctness
Code generation	Runnability, defect rate, complex refactoring	Unit test pass rate, number of static-check issues, review rework rate
Long context	Long-document understanding, cross-section association, retrieval and localization	Recall of key information, answer correctness
Instruction following	Format constraints, role settings, boundary conditions	Format compliance rate, violation rate
Tool calling	Correct parameter parsing, multi-step task orchestration, exception recovery	Tool-call success rate, average number of tool calls
Stability and safety	Consistency for repeated inputs, privilege-escalation and injection risks	Consistency rate, high-risk output blocking rate

Stability is often overlooked. If the same question produces highly different outputs across repeated attempts, it is difficult to guarantee user experience in production, and automated workflows may also break because parsing fails. It is recommended to sample the same input multiple times and observe the consistency of the output distribution.

3.2 Build a Private Evaluation Set: More Reliable Than Public Benchmarks

Public benchmarks may suffer from data contamination and scenario bias. A more reliable approach is to build an enterprise-owned evaluation set:

1. Sampling: Stratify and sample 200 to 500 examples from real business logs over the last 3 to 6 months, covering high-frequency, long-tail, and boundary scenarios.
2. Labeling: Have domain experts write reference answers, scoring rules, or executable acceptance scripts for each example, such as unit tests or validation rules.
3. Freeze: Keep the evaluation set decoupled from the baseline model. After each new model release or version update, run the evaluation set again for regression comparison.

Scoring can be divided into two types:

• Objective tasks: Judged automatically by scripts, such as whether code passes tests, whether JSON passes schema validation, or whether numerical results match the reference answer.
• Subjective tasks: First scored by a model according to a rubric, also known as “model review,” then 10% to 20% are checked manually to calibrate the consistency between the scoring model and human judgment.

3.3 Evaluate by Difficulty Tier

It is recommended to divide tasks by complexity into three tiers:

• L1: Single-step, template-based tasks.
• L2: Multi-step tasks that require some reasoning.
• L3: Complex planning, cross-file collaboration, or cross-system collaboration.

Track pass@1 for each tier separately. A new model is worth adopting for more complex workflows only if it shows significant improvement at L2 or L3. If the improvement is concentrated at L1, a cheaper model with careful prompt engineering may already be sufficient.

3.4 Example Evaluation and Statistics Framework

from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
import json
@dataclass
class EvalCase:
    case_id: str
    level: str                    # Task difficulty tier: L1 / L2 / L3
    task_type: str                # Task type: code / reasoning / extraction ...
    prompt: str
    reference: str = ""           # Reference answer; may be empty for subjective tasks
    rule: str = ""                # Automated acceptance rule or validator identifier
    rubric: dict = field(default_factory=dict)   # Manual review rubric
def run_eval(model_client, cases: list[EvalCase], workers: int = 8,
             repeats: int = 3) -> list[dict]:
    """Sample each case `repeats` times. pass@1 is the average across repeated results.
    model_client must return an object containing output, prompt_tokens, and output_tokens."""
    def evaluate(case: EvalCase) -> dict:
        results = []
        for _ in range(repeats):
            try:
                resp = model_client.invoke(
                    model=MODEL_ID,
                    messages=[{"role": "user", "content": case.prompt}],
                    max_output_tokens=2048,
                )
            except Exception as exc:               # Record failures so a single error does not stop the full evaluation
                results.append({"passed": False, "error": repr(exc), "usage": None})
                continue
            passed = judge(case, resp.output)      # Run rules for objective cases; use the rubric for subjective cases
            results.append({"passed": passed, "usage": resp.usage})
        return {"case": case, "results": results}
    with ThreadPoolExecutor(max_workers=workers) as pool:
        return list(pool.map(evaluate, cases))
def summarize(records: list[dict]) -> dict:
    agg = defaultdict(lambda: {"n": 0, "passes": 0, "prompt_tokens": 0,
                               "output_tokens": 0, "errors": 0})
    for rec in records:
        case = rec["case"]
        key = (case.level, case.task_type)
        agg[key]["n"] += 1
        for r in rec["results"]:
            if r.get("error"):
                agg[key]["errors"] += 1
                continue
            agg[key]["passes"] += int(r["passed"])
            usage = r.get("usage") or {}
            agg[key]["prompt_tokens"] += usage.get("prompt_tokens", 0)
            agg[key]["output_tokens"] += usage.get("output_tokens", 0)
    report = {}
    for (level, task_type), s in agg.items():
        calls = s["n"] * 3                     # Assumes the default repeats=3 setting
        report[f"{level}/{task_type}"] = {
            "pass_at_1": round(s["passes"] / calls, 4) if calls else 0.0,
            "error_rate": round(s["errors"] / calls, 4) if calls else 0.0,
            "avg_prompt_tokens": round(s["prompt_tokens"] / calls, 1) if calls else 0,
            "avg_output_tokens": round(s["output_tokens"] / calls, 1) if calls else 0,
        }
    return report
if __name__ == "__main__":
    cases = load_cases("eval_set.jsonl")   # Load the private evaluation set from a file
    records = run_eval(model_client, cases)
    print(json.dumps(summarize(records), ensure_ascii=False, indent=2))
    # It is also recommended to save raw outputs for manual inspection and future regression comparison

This framework has three design points:

• Group statistics by “difficulty tier + task type” to avoid overall averages hiding weaknesses.
• Record both failure exceptions and token consumption to provide a basis for cost calculation.
• Preserve raw outputs to support manual sampling checks and version regression comparison.

4. Cost-Effectiveness: From “Price per Million Tokens” to “Unit Cost per Qualified Output”

4.1 Single-Call Cost Formula

Let P_in be the input price, P_out be the output price, both expressed as price per million tokens. Let T_in be the number of input tokens and T_out be the number of output tokens. Then:

C_single = (T_in × P_in + T_out × P_out) / 1,000,000

There are three common mistakes in applying this formula:

1. Treating input and output prices as the same: Output prices are usually much higher than input prices. In reasoning-enhanced mode, output tokens can increase substantially, which may significantly amplify cost.
2. Ignoring system prompts and tool definitions: These are repeatedly sent in every call. In high-frequency agent scenarios, they often account for a major part of token consumption.
3. Ignoring failed calls: Timeouts, rate-limit errors, and retries caused by invalid output formats also consume tokens.

4.2 Task-Level and Business-Level Cost

C_task  = C_single × (1 + r_retry) × n_turns × (1 + p_tool)
C_unit  = C_task / pass@1
C_total = C_unit × Q_monthly + C_engineering + C_ops

Where:

• r_retry: Retry rate.
• n_turns: Average number of dialogue turns needed to complete a task.
• p_tool: Context growth factor caused by feeding tool results back into the context.
• Q_monthly: Monthly volume of qualified tasks.
• C_engineering: One-time engineering costs such as prompt engineering and evaluation-set maintenance, amortized monthly.
• C_ops: Operations costs such as monitoring, logging, and storage.

When comparing a cheaper model with a higher retry rate against a more expensive model with a lower retry rate, looking only at unit price can lead to the wrong conclusion. For example, suppose Model A has a unit price that is 60% of Model B’s, but its first-pass success rate is only 75%, compared with 92% for Model B. The ratio of unit cost per qualified output is:

(0.6 / 0.75) ÷ (1 / 0.92) ≈ 0.74

Model A’s real cost advantage is only about 26%, which is much smaller than the 40% difference in unit price.

4.3 Example Cost Calculation

from dataclasses import dataclass
@dataclass
class PriceBook:
    """Model price list. Prices are expressed in currency units per million tokens."""
    input_per_mtok: float
    output_per_mtok: float
    cached_input_per_mtok: float = 0.0      # Cached input price; 0 means caching is not supported
    batch_discount: float = 1.0             # Batch discount factor; 1.0 means no discount
@dataclass
class TaskProfile:
    """Business task profile derived from evaluation statistics and production logs."""
    pass_at_1: float            # First-pass success rate on the evaluation set
    avg_prompt_tokens: int      # Average input tokens, including system prompt and tool definitions
    avg_output_tokens: int     # Average output tokens, including reasoning tokens
    avg_turns: float = 1.0      # Average number of dialogue or call turns
    retry_rate: float = 0.0     # Failure retry rate
    tool_context_growth: float = 0.0  # Context growth caused by returning tool results
    cache_hit_rate: float = 0.0 # Input cache hit rate
    cache_min_tokens: int = 1024    # Minimum length required for a cache hit; follow platform documentation
def task_cost(price: PriceBook, p: TaskProfile, use_batch: bool = False) -> dict:
    cacheable = p.avg_prompt_tokens >= p.cache_min_tokens
    hit = p.cache_hit_rate if (cacheable and price.cached_input_per_mtok > 0) else 0.0
    c_in = (hit * price.cached_input_per_mtok
            + (1 - hit) * price.input_per_mtok) / 1e6
    c_out = price.output_per_mtok / 1e6
    single = (p.avg_prompt_tokens * c_in + p.avg_output_tokens * c_out)
    single *= (1 + p.tool_context_growth)
    single *= price.batch_discount if use_batch else 1.0
    task = single * (1 + p.retry_rate) * p.avg_turns
    unit = task / max(p.pass_at_1, 1e-6)
    return {"single": single, "task": task, "per_pass": unit}
if __name__ == "__main__":
    incumbent = PriceBook(input_per_mtok=3.0, output_per_mtok=15.0,
                          cached_input_per_mtok=0.3)
    candidate = PriceBook(input_per_mtok=4.5, output_per_mtok=22.5,
                          cached_input_per_mtok=0.45,
                          batch_discount=0.5)
    profile = TaskProfile(pass_at_1=0.78, avg_prompt_tokens=6000,
                          avg_output_tokens=900, avg_turns=1.4,
                          retry_rate=0.12, tool_context_growth=0.35,
                          cache_hit_rate=0.6)
    for name, price in [("Current model", incumbent), ("Candidate new model", candidate)]:
        print(name, task_cost(price, profile))
        print(name, "Batch mode", task_cost(price, profile, use_batch=True))

4.4 Hidden Costs That Must Be Included

• Manual rework cost: If model outputs require human editing, the cost should be converted into engineering hours.
• Downstream processing cost: Longer outputs increase storage, retrieval, logging, and other downstream expenses.
• Migration cost: Rewriting prompts, expanding the evaluation set, and modifying pipelines require one-time investment.
• Latency cost: In real-time interaction scenarios, longer response times directly reduce user satisfaction. This loss is difficult to measure using token price alone.

5. Quota and Channel Strategy: Apply Cost Estimates to Real Constraints

5.1 Distinguish Quota Types

• Quota: The total amount available within a billing period, such as monthly or weekly call quotas for subscription plans, as well as separate quotas for different access channels.
• Rate limit: Requests per minute, tokens per minute, concurrent requests, and similar limits that constrain instantaneous throughput.
• Feature limits: Usage caps for specific features or high-intensity reasoning modes, such as consuming only part of a weekly quota or having a limited number of quota resets. These rules change frequently and should be based on the latest official documentation.

In high-concurrency scenarios, the limiting factor is often the rate limit rather than total usage. During load testing, record the proportion of rate-limit errors and the actual success rate after retries.

5.2 Subscription or Pay-as-You-Go

The key difference between the two billing modes is that subscription plans have fixed costs but are subject to quota caps and rate limits, while pay-as-you-go is more elastic but linearly correlated with usage. A rough estimate can be made as follows:

Monthly call volume = daily active users × average tasks per user per day × average calls per task

If monthly call volume is consistently below the subscription quota and the rate limits can handle peak demand, a subscription plan is usually more economical. If usage fluctuates significantly or has clear peaks, a hybrid model of “subscription for baseline usage + pay-as-you-go for overflow traffic” is often more stable.

5.3 Channel Differences: Quota Attribution and Observability

The same model may be accessed through multiple channels:

• The API of a model service platform.
• A coding tool or IDE plugin.
• A hosted application on a cloud platform.

Quotas across different channels are usually calculated separately and are not interchangeable. The granularity of call details and billing transparency also differs. When evaluating cost-effectiveness, calculate quotas and billing separately according to the channel actually used in business. Do not convert the quota of one channel into an equivalent quota for another channel.

5.4 Multi-Model Routing: Optimize Total Cost Through Tiered Dispatch

Routing tasks by difficulty to models at different price points is an effective way to reduce cost without sacrificing quality:

import re
def route(prompt: str, user_tier: str) -> str:
    """Route by task complexity. Rules should be driven by evaluation data, not subjective assumptions."""
    hard_signals = ("refactor", "root cause", "migration", "architecture", "performance tuning", "security audit")
    if len(prompt) > 8000 or any(s in prompt.lower() for s in hard_signals):
        return "reasoning_model"      # Complex task: use a reasoning-enhanced model
    if re.search(r"```|def |class |SELECT ", prompt):
        return "code_model"           # Medium-difficulty code task
    if user_tier == "free":
        return "light_model"          # Simple request from a free-tier user: use a lightweight model
    return "standard_model"

Before routing rules are deployed, verify on the evaluation set whether the quality of each branch meets requirements. Also set a fallback: when a high-priced model hits a rate limit or insufficient quota, automatically downgrade to an available channel or queue for retry.

6. Engineering Practices: Four Common Ways to Reduce Unit Cost

1. Prompt caching: Place stable prefixes such as system prompts, tool definitions, and long documents at the beginning of the request to improve cache hit rates. For multi-turn dialogue and agent scenarios, this often provides the largest benefit.
2. Batch processing: Use asynchronous batch inference for offline tasks such as daily report generation, document summarization, log classification, and data labeling, trading latency for lower unit cost.
3. Context trimming: Use retrieval-augmented context injection to include only relevant fragments, summarize or compress tool results, and use rolling summaries for historical dialogue to prevent context from growing indefinitely.
4. Output constraints: Use structured outputs, such as JSON mode, and explicit length limits to control output tokens while also reducing downstream parsing failures and retry rates.

Example: Organize messages with “stable prefix + dynamic suffix” to improve cache hit probability.

def build_messages(system_prompt: str, tools_schema: str,
                   history: list[dict], query: str) -> list[dict]:
    """Place stable content first and dynamic content later to improve prefix-cache matching.
    Actual caching rules, such as minimum length and lifetime, should follow each platform's documentation."""
    messages = [
        {"role": "system",
         "content": f"{system_prompt}\n\nAvailable tool definitions:\n{tools_schema}"},
    ]
    messages.extend(history[-6:])      # Keep only the most recent few turns to control context length
    messages.append({"role": "user", "content": query})
    return messages

7. Canary Rollout: Validate Conclusions With Production Data

An evaluation set reflects sample performance, while production traffic distribution may differ. It is recommended to roll out in stages:

1. Shadow traffic: The new model and the current model process the same requests in parallel. Only results are recorded, without affecting users. This is used to compare quality differences and anomaly rates.
2. Small-percentage canary: Increase traffic gradually from 1% to 5%, then 20%, then 50%. Observe at least one complete business cycle at each stage.
3. Monitoring metrics: First-pass success rate, manual rework rate, average response latency, such as P50, P95, and P99, unit cost per qualified output, rate-limit errors, and error rate.
4. Rollback plan: Keep the ability to switch back to the original model with one action, and set automatic degradation thresholds. For example, if the error rate or latency exceeds a preset upper limit, automatically roll back.

These metrics can be collected through centralized logging and monitoring, and alert thresholds can be configured to form a closed loop of “evaluation, launch, monitoring, and review.”

8. Conclusion and Action Checklist

When facing the release of a new generation of large models, a rational evaluation path can be summarized in five steps:

1. Check capability: Use a private evaluation set to measure pass@1 by task type and difficulty tier, instead of relying only on public leaderboards.
2. Calculate cost: Use “unit cost per qualified output” as the core metric, including retries, turns, tool context, and manual rework.
3. Verify constraints: Check subscription quotas and rate limits separately, and confirm that business peak usage stays within limits to avoid frequent throttling after launch.
4. Choose channels: Calculate quotas and billing transparency separately for the channels actually used, and use a hybrid “subscription + pay-as-you-go” model to balance cost and elasticity.
5. Control risk: Optimize cost through tiered routing, caching, and batch processing, and control rollout risk through canary releases and rollback mechanisms.

What truly determines cost-effectiveness is not the model unit price itself, but the fit among model capability, engineering pipeline, and business value. Only by iterating continuously based on data can every unit of model investment correspond to measurable business output.