Evaluating LLM Engineering Tasks: Measuring Capability and Cost-Effectiveness

1. Why Evaluate Capability and Cost-Effectiveness Separately

Many teams rely solely on an aggregate score or unit price during model selection, which often leads to misjudgments. Capability answers whether a model can complete a specific type of task, while cost-effectiveness addresses the cost incurred to complete it. These two answers can contradict each other: a model might have a higher pass rate on difficult tasks but cost several times more per invocation. If the task itself is not inherently difficult, the former model may not be the better choice.

Therefore, the evaluation should be split into two steps. First, measure quality metrics on a fixed task set to obtain pass rates, completion rates, and stability. Second, merge the quality data with cost data to calculate the average cost per successful task. Only when both steps are recorded can the conclusions be reviewed and verified.

2. Establishing Evaluation Dimensions

Evaluating engineering tasks should include at least the following five categories of metrics:

1. Correctness: Whether the output meets acceptance criteria. Code-based tasks are typically judged by the passing of unit or integration tests, rather than subjective human impressions.
2. Completeness: The percentage of steps completed when a task fails midway. This is particularly important for multi-step tasks, as “80% complete” and “complete failure” have different implications in engineering.
3. Stability: The variance of results when the same task is run repeatedly. Running the same input five to ten times and analyzing the distribution of pass counts is much more reliable than a single execution.
4. Latency: The time from submitting a request to receiving a usable result, including model generation time and tool invocation time.
5. Cost: Token consumption for inputs and outputs, plus additional overheads such as tool execution and sandbox resources.

There are trade-offs between these metrics. For example, increasing the sampling temperature might improve diversity but reduce stability. The evaluation report should present all metrics simultaneously rather than just providing a composite score.

3. Tiered Task Design

The evaluation set should cover the difficulty gradient of real-world work. It is recommended to organize tasks into the following tiers:

• Understanding: Reading a piece of code or log and explaining its behavior and flaws. These tasks test context comprehension.
• Generation: Generating functions or modules based on clear specifications and passing preset tests.
• Fixing: Locating and fixing issues given failed tests and a code repository. These tasks closely resemble daily development work.
• Optimization: Improving performance metrics (e.g., reducing memory usage or shortening computation time) while ensuring correctness. Operator development and model inference optimization fall into this category. Acceptance must rely on benchmark tests under fixed hardware environments.
• End-to-End: Completing a full requirement across multiple files and steps and passing integration tests. These tasks demand the highest planning and state management capabilities.

A portion of the questions in each tier should remain private to prevent the evaluation set from losing relevance after targeted optimization. Questions should be updated regularly and assigned version numbers.

4. Basic Structure of the Evaluation Framework

A reproducible evaluation framework typically contains the following components:

• Task Set: Each problem includes input materials, acceptance scripts, and metadata (difficulty, category, version).
• Sandbox Environment: Each run occurs in an isolated container or virtual environment to ensure consistent file systems and dependency versions.
• Executor: Responsible for handing the task to the model and recording the input, output, tool calls, and duration for each round.
• Scorer: Runs the acceptance scripts and outputs structured results without relying on manual grading.
• Logger: Saves complete logs for post-mortem analysis of failures.

Below is language-agnostic pseudocode illustrating the data structure of a single run’s result and the calculation of cost per success:

record TaskResult:
    task_id
    passed            # Whether acceptance criteria passed
    latency_seconds
    input_tokens
    output_tokens
    tool_cost         # Additional cost for sandbox or tool execution
function cost_per_success(results, price_in_per_m, price_out_per_m):
    total = 0
    successes = 0
    for r in results:
        total += r.input_tokens / 1_000_000 * price_in_per_m
        total += r.output_tokens / 1_000_000 * price_out_per_m
        total += r.tool_cost
        if r.passed:
            successes += 1
    if successes == 0:
        return infinity
    return total / successes

This metric represents the average cost incurred for each successfully completed task. It places quality and cost on the same scale, making comparisons between different models more intuitive. Note that the cost of failed tasks must also be included in the total cost; otherwise, the cost of models with low pass rates will be underestimated.

5. Comprehensive Assessment of Cost-Effectiveness

Cost per success is only one dimension. Actual decision-making should also consider the following factors:

• Task Value Differences: Fixing a live production incident carries significantly higher value than generating boilerplate code. For high-value tasks, a higher unit cost may still be a reasonable choice.
• Manual Review Cost: If model outputs require extensive human verification, this review time should be converted into monetary value and added to the total cost.
• Latency Constraints: Interactive scenarios are sensitive to response time, while batch processing scenarios prioritize total throughput.
• Data and Compliance Requirements: Certain code or data cannot be sent to external services, a constraint that may directly eliminate some options.

Weights can be assigned to each metric to form a weighted score. These weights should be determined before the evaluation begins and documented to avoid adjusting them post-hoc based on results, which would undermine the credibility of the conclusions.

6. Common Pitfalls

1. Running tests only once: Single run results are heavily influenced by randomness; you should run them repeatedly and report the distribution.
2. Overlap between test questions and training data: If questions are sourced from public repositories, the model may have already seen the answers, leading to inflated conclusions.
3. Ignoring failure costs: Failed calls also consume resources and must be factored into the total cost.
4. Conflating tool capability with model capability: The same model can perform very differently under various execution frameworks; reports should specify the framework version and configuration.
5. Over-reliance on composite scores: Composite scores can mask specific shortcomings; you should also review the detailed breakdown by sub-task.

7. Implementation Recommendations

1. First identify the three most common task types in your business, prioritize building evaluation sets for them, and then expand to other scenarios.
2. Fix the model version, parameter configuration, runtime environment, and question version for each evaluation, and save the raw logs.
3. Re-run the evaluation quarterly or after every model upgrade to compare metric changes rather than just looking at absolute values.
4. Include scoring scripts in version control, and record the reasons for any changes to acceptance rules.
5. For optimization tasks, use benchmark test results alongside correctness tests as passing conditions; both are indispensable.

The purpose of evaluation is to support decision-making. The conclusions should be reproducible by a third party following the same steps, which is more important than producing a flattering ranking.