Evaluating the Practical Value of Large Language Models: From Task Success Rates to Delivery Costs
A practical evaluation of large language models should go beyond response quality, per-call pricing, and usage limits. This article explains how to measure end-to-end delivery capability, requirement understanding, constraint adherence, verification quality, effective task cost, quota efficiency, retry waste, and operational governance through reproducible, engineering-focused evaluations.
1. Ask the Right Question When Evaluating a Large Language Model
Discussions about whether a large language model is “good” often focus on three intuitive metrics: how fluent its answers are, how cheap each call is, and how much usage capacity is available. These metrics are useful, but they do not directly answer the question developers actually care about:
Can the model reliably complete real tasks within a defined budget and time limit?
In software development, a model’s job usually involves much more than generating a piece of code. It must understand the requirement, read the surrounding context, locate the problem, design a modification plan, run verification, and adjust its approach based on feedback. If the early steps go wrong, later outputs may still look professional while continuing in the wrong direction.
Evaluation should therefore shift from single-response quality to end-to-end delivery capability. A model that generates answers quickly and costs less per call may not be more economical if repeated changes still fail acceptance. A more expensive model may deliver better value if it identifies critical constraints earlier and reaches an acceptable result with fewer failed attempts.
This is also a more meaningful way to discuss the competitiveness of Chinese-developed large language models. Their value should not be demonstrated through a single impressive demo, but through stable performance, cost efficiency, and engineering usability on reproducible tasks.
2. Turning Ambiguous Requirements into Executable Plans
In real-world use, users rarely provide a complete specification. They may simply ask the model to “fix this error,” “make the query faster,” or “add an approval workflow.” These requests express business intent, but they do not contain all the information needed for implementation.
A capable model should not immediately guess at the solution. It should first clarify the requirements.
For example, optimizing query performance requires answers to at least the following questions:
- What are the current and target response times?
- Under which conditions, data ranges, and concurrency levels do slow requests occur?
- Is the bottleneck caused by data access, computation, network transfer, or resource contention?
- Can the data structure, caching strategy, or update process be changed?
- Could the optimization affect consistency, access control, or result accuracy?
Without these constraints, a model may propose a solution that appears reasonable but is unusable in practice. For example, it may reduce query latency through caching while ignoring data update frequency and cache invalidation. The performance problem would then become a correctness problem.
Requirement understanding should therefore include three layers:
- Identifying known conditions;
- Exposing missing information;
- Distinguishing facts from assumptions.
A model that can explicitly state that “the available evidence is insufficient to determine the root cause” may provide more engineering value than one that immediately produces a long list of code changes.
3. Parameter Count Does Not Equal Effective Delivery Capability
A larger parameter count may provide stronger knowledge representation, complex pattern recognition, and reasoning potential. However, it cannot be used to predict performance on every task. Training methods, context organization, tool reliability, and verification workflows all affect the final result.
For development tasks, at least the following capabilities should be evaluated separately.
Constraint retention.
Can the model continue to respect requirements across multiple turns, such as version compatibility, unchanged interfaces, or restrictions on modifying specific modules? Or does it solve one problem while introducing another?
Evidence-based diagnosis.
Can the model form conclusions from logs, call chains, and relevant code instead of matching a single error keyword to a familiar answer?
Executable planning.
Does the proposed plan specify the modification scope, prerequisites, verification method, and what to do if the approach fails?
Feedback-driven correction.
When a test fails, can the model update its diagnosis based on new evidence rather than repeatedly restating the same idea in different words?
Result verification.
Can the model distinguish between “code has been generated” and “the code has been shown to satisfy the requirement”?
These capabilities can differ substantially. A model that explains concepts well may not be effective at cross-module changes. A model that can generate a local feature may not reliably maintain state during a long task. Overall capability should therefore not be summarized through a single impression.
4. Measuring Cost Effectiveness Through Effective Task Cost
Comparing only the cost of individual calls underestimates the expense of failed retries and human intervention. A more useful metric is:
Average cost per accepted task = Total investment during the evaluation period ÷ Number of accepted tasks
Total investment may include:
- Model invocation costs;
- Computing and tool-execution costs;
- Human time spent organizing and reviewing work;
- Rework costs;
- Risk-handling and incident-response costs.
If human time is converted into monetary value, the conversion method should be defined in advance and applied consistently across all options.
Two details are especially important.
First, the investment in failed tasks must also be included in the total cost. Counting only the model spending associated with successful tasks hides the cost of many ineffective attempts.
Second, “complete” must be defined by acceptance criteria. A model’s statement that a problem has been fixed cannot replace test results, code review, or business confirmation.
Individual developers can begin by recording model spending, time spent on human intervention, and final outcomes. Enterprise teams should also consider compliance review, incident impact, and maintenance burden. Not every factor needs to be monetized immediately, but it should at least be tracked separately rather than ignored simply because it is difficult to price.
5. Quota Efficiency Depends on Workload and Execution Method
Having sufficient usage quota does not necessarily lead to higher effective output. Resource consumption depends on how much context the model reads, how many tools it calls, and how many times it repeats an execution cycle.
Quota evaluation should separate the following dimensions:
- Available capacity: How many calls or how much resource consumption is allowed during a defined period.
- Throughput limits: Whether frequency, concurrency, or queueing restrictions apply.
- Task capacity: How many tasks can actually be completed and accepted.
- Usage stability: Whether key workflows are interrupted by changing limits or insufficient resources.
Different task structures can also produce very different experiences. Short text editing consumes relatively few resources, while debugging across multiple files may require repeated context loading, test execution, and plan revision. Comparing these workloads together usually produces conclusions that do not transfer well to other situations.
Budget evaluation should therefore use a representative combination of the organization’s or developer’s own tasks rather than abstract measures such as how long the model can sustain a conversation or how many rounds it can execute.
6. Detecting Ineffective Retries and Preventing Failure Loops
One of the most expensive situations in development is not an obviously failed response, but a model that appears to be getting closer while continuing in the wrong direction.
Common warning signs include:
- Repeatedly modifying the same location without adding new evidence;
- Running the same checks multiple times without explaining how the results relate to the assumptions;
- Expanding the modification scope simply to eliminate one error;
- Alternating between contradictory explanations for the same problem;
- Claiming that the issue “should now be fixed” without completing verification.
The solution is to establish clear stage gates.
Every attempt should answer four questions:
- What is the current hypothesis?
- What evidence supports it?
- What will change in this attempt?
- What result will determine whether the hypothesis is correct?
If these questions cannot be answered, modification should pause until additional observations are collected.
A retry budget can also be useful, but it should not be applied mechanically with the same limit to every task. Simple problems may require strict limits, while complex problems may need more investigation. The real objective is to constrain repetition without information gain, not evidence-based analysis.
7. The Value of Layered Collaboration
Complex tasks do not necessarily require the same capability configuration from beginning to end. Work can be divided into planning, execution, and verification, with resources allocated according to the difficulty of each stage.
1. Planning: Resolve Uncertainty First
This stage covers requirement clarification, impact analysis, risk identification, and task decomposition. Its output should not be merely a to-do list. It should become an executable task specification.
Each subtask should define at least:
- The objective;
- Required inputs;
- The permitted modification scope;
- Constraints that must remain unchanged;
- Acceptance criteria.
The clearer the boundaries, the less inference is required during execution.
2. Execution: Prioritize Well-Bounded Work
Local code changes, test additions, documentation updates, and formatting adjustments are generally easier to verify when their constraints are clear. They are also more suitable for lower-cost execution configurations.
However, excessive decomposition can create interface inconsistencies and repeated context-understanding costs when subtasks are strongly dependent on one another. Layered collaboration does not mean splitting work into as many tasks as possible. It means giving each task a relatively independent acceptance boundary.
3. Verification: Return to the Overall Objective
Passing local tests does not prove that the overall requirement has been satisfied. The final review must still consider module integration, permission boundaries, exception handling, compatibility, and regression impact.
A review performed by another model should not be treated as deterministic verification. Model-based review can provide an additional perspective, but critical conclusions should still be supported by tests, runtime evidence, and human judgment.
8. Building a Reproducible Evaluation Process
A reliable evaluation can begin with the following process.
Step 1: Select real tasks.
Include both frequent everyday tasks and high-risk tasks, such as defect localization, localized refactoring, test generation, performance analysis, and feature implementation. Tasks should come from a controlled environment and should not contain sensitive data.
Step 2: Define acceptance criteria in advance.
Before the evaluation starts, specify correctness requirements, permitted modification scope, time budget, and rules for human intervention. This prevents the scoring criteria from changing after the results are known.
Step 3: Control inputs and the environment.
Use the same initial code, requirement description, tool permissions, and resource conditions. If the model asks clarification questions, apply consistent principles when answering them.
Step 4: Preserve the complete process.
Record the initial plan, modification history, verification results, resource consumption, and reasons for human takeover. Keeping only the final answer discards the most important cost information.
Step 5: Repeat the evaluation and classify tasks.
Model outputs vary between runs. A single successful attempt does not demonstrate stability. Observe success rates, time distributions, and failure types across multiple executions instead of showcasing only the best case.
The final evaluation report should answer three questions:
- Which tasks are suitable for automated execution?
- Which tasks require human confirmation?
- Which tasks should not yet be delegated to the model?
9. Operational Boundaries for Enterprise Adoption
When a model becomes part of a team workflow, capability evaluation must be complemented by data and permission governance.
On the input side, organizations should follow the principle of least privilege and identify and handle sensitive information appropriately. On the execution side, access should be restricted and high-risk operations should require approval. On the output side, review and rollback mechanisms should be retained.
Operations involving data deletion, permission changes, or production configuration updates should not be executed solely on the model’s judgment.
Observability is equally important. Teams should be able to trace:
- Which inputs were used for a task;
- Which operations were performed;
- Where the process failed;
- Who confirmed the final result.
Without these records, a workflow may appear to save time in the short term but will be difficult to turn into a sustainable engineering system.
10. Conclusion: Competitiveness Must Be Demonstrated Through Stable Delivery
The practical value of a large language model should not be judged by whether its answers sound more like those of an expert or whether its usage quota appears larger. More reliable criteria include whether the model can accurately understand the objective, plan under constraints, complete tasks with limited ineffective trial and error, and provide sufficient evidence for verification.
For Chinese-developed large language models, meaningful progress should likewise be demonstrated through success rates, stability, effective task cost, and governability under real workloads.
For users, the best strategy is not to search for the model that is abstractly “the strongest.” It is to establish clear task boundaries, acceptance rules, and layered execution workflows so that each unit of budget is converted as effectively as possible into a verifiable and maintainable delivery outcome.
