Evaluating LLM Performance and Cost Efficiency for Real Business Tasks

Large language models are being updated at a faster pace. In the past, a model upgrade often meant a local improvement in question-answering fluency or text quality. Today, however, changes in complex reasoning, code generation, long-task execution, and multi-step collaboration can significantly shift both the “capability boundary” and the “cost of use” at the same time. As a result, when a new model generation is released, developers and enterprises should not judge it only by version number, leaderboard score, or per-token price. A more practical evaluation should cover task effectiveness, reasoning intensity, token consumption, end-to-end cost, and engineering maintainability.

1. Do Not Evaluate a New Model Only by Version Number or Leaderboard Score

When evaluating a new model, many teams first check public benchmark leaderboards, such as general intelligence indices, coding leaderboards, and mathematical reasoning leaderboards. These leaderboards do have reference value because they can quickly show how a model performs on standardized tasks. However, they do not fully represent real business scenarios.

There are three main reasons for this.

1. Task distribution differs
   Leaderboards usually cover selected standard question sets. Real business workloads often contain many long-tail tasks, such as domain-specific knowledge question answering, internal document extraction, complex ticket classification, and repository-level code modification.

2. Input and output structures differ
   Many benchmark tasks use short inputs and concise outputs. Production tasks may involve long contexts, multi-turn tool calls, structured output, exception retries, and human review.

3. Cost constraints differ
   Leaderboards usually focus on accuracy or pass rates. They do not fully account for task latency, token consumption, retry rates, human review costs, or downstream operational costs.

A more reliable approach is therefore to build an internal business evaluation set. The set does not need to be very large, but it should cover core scenarios and reflect real usage patterns.

A practical evaluation framework can include the following dimensions:

• Effectiveness metrics: accuracy, pass rate, human adoption rate, and repair success rate.
• Efficiency metrics: first-response latency, full-response latency, concurrent throughput, and retry rate.
• Cost metrics: token consumption per task, cost per task, and human review cost.
• Stability metrics: format error rate, tool-call failure rate, timeout rate, and abnormal output rate.
• Safety metrics: risk of sensitive information leakage, risk of unauthorized operations, and rate of uncontrollable content generation.

Only by viewing these metrics together can teams avoid the misjudgment that “the model looks strong on the leaderboard but is not practical in business.”

2. Reasoning Intensity Is a Key Controller Between Capability and Cost

New-generation large models often provide different reasoning intensity levels. Higher reasoning intensity usually means the model can allocate more computation to analyzing a problem, decomposing steps, and generating a complete answer. At the same time, output tokens, reasoning time, and actual cost may also increase significantly.

For complex tasks, higher reasoning intensity often produces better results. For example, code repair, system architecture design, complex data analysis, and multi-step agent tasks often require more “thinking process” before the model can produce a stable result. For simple question answering, text summarization, or format conversion, however, higher reasoning intensity may not bring a noticeable improvement and may waste budget.

Therefore, the focus of model selection should not simply be “whether to use the highest setting.” It should be “whether a reasonable reasoning-intensity routing mechanism has been established.”

A common tiered strategy is as follows:

• Low reasoning intensity: suitable for simple question answering, text rewriting, field extraction, and batch classification.
• Medium reasoning intensity: suitable for ordinary code completion, document summarization, and routine customer-service responses.
• High reasoning intensity: suitable for complex code generation, cross-file modification, long-context analysis, and fault diagnosis.
• Very high reasoning intensity: suitable for high-risk, high-difficulty, and high-value tasks, such as core module refactoring or complex system design.

The following code provides an illustrative example:

def choose_reasoning_level(task):
    if task.risk_level == "high" or task.expected_difficulty >= 0.8:
        return "high"
    elif task.type in ["code_review", "complex_planning", "multi_step_analysis"]:
        return "medium_high"
    else:
        return "medium"

The core idea behind this strategy is: let high-capability models serve high-value tasks, rather than sending every request through the most expensive path.

For enterprises, it is also useful to introduce task priority and budget limits. For example, core production tasks can use higher reasoning intensity, while offline batch-processing tasks can be limited by maximum output length and call frequency.

3. Lower Token Unit Prices Do Not Mean Lower Total Task Costs

When a new model is released, providers often announce input and output token prices. Lower prices are usually positive, but focusing only on unit prices can easily lead to a wrong cost judgment.

In large-model applications, total cost can often be abstracted as follows:

Total task cost = input token cost + output token cost + tool-call cost + retry cost + human review cost

Among these, output tokens are often the most underestimated part. Complex reasoning models may generate longer explanations, more complete step breakdowns, more detailed code descriptions, or repeated verification of results. This can significantly increase output token consumption per task.

When evaluating cost efficiency, therefore, teams should not only ask “how much does one million tokens cost?” They should ask “how much does it cost to complete the same business task in total?”

A simple cost estimation model is as follows:

def estimate_task_cost(input_tokens, output_tokens, input_price, output_price):
    input_cost = input_tokens / 1_000_000 * input_price
    output_cost = output_tokens / 1_000_000 * output_price
    return input_cost + output_cost

If a stronger model causes a significant increase in output tokens, total cost may still rise even when the unit price drops. Common reasons include:

1. Long reasoning processes: The model may generate longer intermediate steps and final answers for complex tasks.
2. Multi-turn tool calls: Each tool call may require reconstructing the context again.
3. Retry mechanisms: Format errors, timeouts, or unacceptable results may trigger additional calls.
4. Full-context input: Unfiltered code repositories, documents, or conversation history can cause input token expansion.
5. Post-generation validation: Additional model calls may be needed to determine whether the result is acceptable.

To control cost, teams can apply the following engineering optimizations:

• Context compression: Retain only the documents, code, and conversation fragments most relevant to the current task.
• Summary caching: Generate summaries of long documents first and use them as inputs for subsequent calls.
• Structured output constraints: Require the model to output in a fixed format to reduce ineffective explanations.
• Maximum output length limits: Prevent unnecessary generation of overly long content.
• Early stopping: Stop further reasoning or validation once a result meets a defined threshold.
• Batch processing: Combine offline tasks into batches to improve resource utilization.
• Result caching: Reuse historical results for identical or similar requests.
• Tiered routing: Use low-cost models or low reasoning intensity for simple tasks, and upgrade only when needed for complex tasks.

The following code provides an illustrative budget-control mechanism:

class BudgetController:
    def __init__(self, max_tokens_per_task):
        self.max_tokens_per_task = max_tokens_per_task
    def should_continue(self, used_tokens, current_output_tokens):
        if used_tokens + current_output_tokens > self.max_tokens_per_task:
            return False
        return True

The purpose of this mechanism is clear: no matter how capable a model is, it should not operate outside budget boundaries. Enterprises usually do not need “the maximum possible capability at unlimited cost.” They need “the best task completion within an acceptable cost range.”

4. Code Capability Should Be Evaluated as an Engineering Closed Loop

For new-generation models, coding capability is often one of the dimensions developers care about most. However, code capability should not be understood simply as “whether the model can generate a piece of correct code.” Real development tasks usually include:

• Generating modules based on requirements.
• Fixing defects.
• Writing unit tests.
• Explaining legacy code.
• Performing code review.
• Modifying multiple files.
• Continuing iteration after test failures.
• Maintaining consistency during long task cycles.

When evaluating coding capability, it is better to use “engineering closed-loop metrics” rather than relying only on a single generation result.

The following metrics are especially useful:

1. First-pass success rate
   Can code generated or modified by the model on the first attempt pass basic compilation, execution, and tests?

2. Repair success rate
   After receiving error information, can the model complete the repair within a limited number of rounds?

3. Test coverage capability
   Can the model not only write implementation code but also add key test cases?

4. Cross-file understanding
   In multi-module projects, can the model understand dependency relationships and avoid destructive modifications?

5. Long-task stability
   In continuous multi-round tasks, can the model maintain alignment with the original goal instead of gradually drifting away from the requirements?

6. Human takeover rate
   What proportion of tasks still ultimately require human completion?

For more complex long-horizon tasks, such as using multiple agents to collaborate on a larger software module, system-level governance is also required.

Such tasks usually involve the following stages:

Requirement decomposition -> module planning -> code generation -> unit testing -> failure repair -> integration validation -> human review

In this process, the model is not only a “code writer.” It may also act as a “planner,” “tester,” or “reviewer.” Therefore, complete execution boundaries are required:

• Run code in isolated sandboxes.
• Restrict file system and network access.
• Keep human approval for critical operations.
• Record logs for each operation.
• Perform security scans on generated content.
• Provide rollback mechanisms for high-risk modifications.
• Set stage acceptance checkpoints for long tasks.

The following code provides an illustrative process-control example:

def run_code_task(task):
    plan = model.generate_plan(task)
    modules = decompose(plan)
    for module in modules:
        result = model.generate_code(module)
        test_result = run_tests(result)
        if not test_result.passed:
            fix_prompt = build_fix_prompt(result, test_result)
            result = model.repair(fix_prompt)
        if not safe(result):
            raise SafetyException("Code failed security check")
        commit(result)

This shows that coding capability cannot be evaluated outside the engineering system. Whether a model is truly usable for code tasks depends on whether it can fit into version control, testing, logging, monitoring, and audit systems.

5. Cost-Efficiency Evaluation Needs a Scenario-Based Matrix

Different business scenarios have very different requirements for model capability. Using the same standard to evaluate all models can easily lead to incorrect conclusions. A more reasonable approach is to build separate evaluation matrices for different scenarios.

Scenarios can be divided into three categories based on task complexity.

1. High-frequency lightweight tasks
   Examples include text classification, information extraction, summarization, and frequently asked question responses.
   The focus should be on low latency, low cost, and high stability.

2. Medium-complexity tasks
   Examples include customer-service assistance, document question answering, code completion, and report generation.
   The focus should be on balancing effectiveness, format stability, and acceptable cost.

3. High-value complex tasks
   Examples include architecture design, complex code refactoring, difficult fault analysis, and long-horizon agent tasks.
   The focus should be on capability ceiling, task completion, and controllability.

Different scenarios can have different objective functions:

High-frequency lightweight tasks: optimize for cost
Medium-complexity tasks: balance effectiveness and cost
High-value complex tasks: prioritize effectiveness while keeping cost controllable

For implementation, a closed loop of “offline evaluation + online canary + continuous monitoring” is recommended.

1. Offline Evaluation

First, use a fixed sample set to evaluate model performance on core tasks. Samples should cover normal inputs, boundary inputs, and exception inputs. Evaluation results should include not only accuracy but also average response time, average token consumption, and exception rates.

2. Online Canary

When connecting a new model to real traffic, do not switch the entire system immediately. Canary releases can be based on users, tenants, task types, or request ratios. During the canary stage, teams should focus on observing:

• Whether failure rates increase.
• Whether latency increases.
• Whether costs exceed the budget.
• Whether user feedback deteriorates.
• Whether there are format errors or security risks.

3. Continuous Monitoring

After launch, a long-term monitoring system should be established. Monitoring targets include request volume, response latency, error rate, retry rate, token consumption, cost trends, and abnormal model output rates.

If the deployment runs on Baidu AI Cloud, general-purpose cloud components can support related engineering capabilities. For example:

• Use container platforms to host inference services and task executors.
• Use message queues to handle asynchronous long-running tasks.
• Use object storage to store samples, logs, and generated results.
• Use logging services to record request traces and call results.
• Use monitoring and alerting to track latency, error rates, and cost anomalies.

These general capabilities do not change the model evaluation logic itself, but they make model capability easier to observe, govern, and optimize continuously.

6. Enterprise Model Selection Should Shift From “Strongest Model” to “Suitable Model”

After new-generation models improve in capability, many teams develop a tendency to assume that if a model is stronger, it should be used at the highest possible setting. From an engineering perspective, this is neither economical nor always necessary.

A more reasonable judgment method is as follows:

• If the task is simple, repetitive, and low-risk, prioritize low-cost and stable solutions.
• If the task is of medium complexity, prioritize configurations that balance effectiveness and cost.
• If the task is high-value and the cost of errors is high, consider using stronger reasoning capability and more complete context.
• If the task requires long-duration execution, focus on stability, controllability, and human takeover mechanisms.
• If the task involves code, system configuration, or production environment operations, strengthen sandboxing, permissions, auditing, and rollback mechanisms.

In other words, model evaluation is not only a technical issue. It is also a combination of business judgment, engineering design, and cost governance.

Ultimately, teams should not only ask: “Is this model the strongest?”

They should answer the following questions:

1. Can it reliably complete real business tasks on real samples?
2. Is its cost lower than the value of the task?
3. Can its errors be detected and fixed in a timely manner?
4. Can it be integrated into existing R&D, testing, and operations workflows?
5. Can it improve efficiency without significantly increasing risk?

7. Summary

The release of new-generation large models often brings visible changes in leaderboard scores, reasoning speed, output quality, and pricing strategies. For developers and enterprises, however, the most important factor is not a single metric. It is a complete evaluation framework.

A robust evaluation framework should include:

• Validating real effectiveness with business evaluation sets.
• Balancing capability and cost through reasoning-intensity routing.
• Evaluating real cost efficiency using token consumption and total task cost.
• Measuring the usability of code and complex tasks through engineering closed-loop metrics.
• Ensuring safe launch through monitoring, alerting, canary releases, and auditing.

Only when model capability, task cost, engineering governance, and business value form a closed loop can model selection move beyond “looking strong” and become a sustainable productivity tool.