How Chinese Foundation Models Can Break Through: From Task Success Rates to Delivery Cost

    Evaluating foundation models requires more than comparing answer quality, token prices, or usage limits. This article presents a practical framework for measuring whether models can reliably complete real-world tasks within defined budgets, including intent clarification, multi-step consistency, evidence-based recovery, unit cost per successful task, quota efficiency, multi-model workflows, evaluation design, and enterprise governance. The central question is not whether a model can produce an impressive answer, but whether it can deliver predictable, verifiable, and economically sustainable results.

    1. Evaluate Whether the Task Can Be Delivered, Not Just Whether the Answer Looks Good

    Discussions about Chinese foundation models often swing between two extremes. One focuses on whether a model can produce surprisingly good answers to complex questions. The other compares prices, usage limits, and subscription quotas. The first can overlook reliability, while the second can mistake a low per-call price for overall cost efficiency.

    For developers, the more important question is this: given a specific requirement, clear acceptance criteria, and a limited budget, can the model consistently move the task toward completion rather than generate plausible-looking content that cannot be verified?

    A model may perform smoothly in short conversations but repeatedly reintroduce the same defect during an engineering task. Another model may have a higher per-call cost but substantially reduce rework by understanding requirements accurately and breaking the work into appropriate steps. Model value should therefore be assessed across the complete delivery process, not by output length or call count alone.

    A breakthrough for Chinese foundation models should not be defined by whether a single demonstration is impressive. A more meaningful standard is whether the model can provide stable, verifiable, and economically sustainable delivery across a set of real tasks.

    2. Break “Intent Understanding” into Four Testable Capabilities

    Users often want to describe a goal and have the model complete the implementation path on its own. Although this experience may appear to depend on how “smart” the model is, it actually involves several distinct capabilities.

    2.1 Identify Uncertainty in the Requirement

    Real requirements are often incomplete. For example, “Add a bulk import feature” does not specify the data volume, field rules, duplicate-handling behavior, failure-recovery strategy, or permission boundaries.

    A reliable model should not immediately invent a solution it considers reasonable. It should distinguish between:

    • Requirements that are already clear and can be executed directly;
    • Assumptions that may use default values but must be stated;
    • Questions that affect architecture or business behavior and must be confirmed.

    The goal is not to ask as many questions as possible, but to ask questions that affect decisions. Repeatedly asking about irrelevant details slows the work, while overlooking critical constraints causes the implementation to rely on incorrect assumptions.

    2.2 Convert Ambiguous Goals into Acceptance Criteria

    “Improve performance” and “increase stability” are not sufficient execution instructions. The model needs to clarify what will be optimized, how it will be measured, which workloads apply, and which existing behaviors must not be changed.

    For a bulk import feature, acceptance criteria might include:

    • Valid records are written correctly;
    • Invalid records can be traced to specific entries;
    • Repeated submissions do not create duplicate records;
    • Partial failures produce a clear status report.

    When response-time or throughput targets are involved, they should be defined by business requirements or baseline testing rather than promised arbitrarily by the model.

    The earlier a model establishes verifiable goals, the less likely the workflow is to reach a state where many changes have been made but nobody can determine whether the task is actually complete.

    2.3 Maintain Consistency Across Steps

    Engineering work is not a series of isolated question-and-answer exchanges. Field constraints confirmed during design should appear in the implementation, tests, and documentation. The failure-recovery approach selected in one step should also remain consistent with state handling in later steps.

    If the model changes its assumptions as the conversation progresses, developers must take on additional coordination work. Evaluation should therefore check whether key decisions are followed throughout the process, rather than judging only whether the final answer appears complete.

    2.4 Converge After Failure Based on Evidence

    A failed execution does not automatically mean that a model is unusable. The real problem is when the model fails to analyze new evidence and instead repeats similar changes, expands the scope unnecessarily, or presents an unverified guess as a conclusion.

    An effective recovery process should:

    1. Extract information from the failure;
    2. Form a limited set of hypotheses;
    3. Select a validation action;
    4. Narrow the problem based on the result.

    Knowing when to stop ineffective attempts and explain the blocking issue is also part of reliability.

    3. Replace Per-Call Pricing with Unit Cost per Successful Task

    A cheap individual call does not necessarily make a task inexpensive. A typical task consumes model calls, context processing, external execution resources, and human review time.

    A useful calculation is:

    Unit cost per successful task = Total task cost during the evaluation period ÷ Number of tasks that passed acceptance

    Total task cost should include all attempts, including tasks that ultimately failed. Counting only successful cases can hide large amounts of wasted effort.

    Total cost can be divided into the following categories:

    • Model usage cost: Input, output, repeated calls, and related fees;
    • Execution resource cost: Test environments, computing, storage, and other required resources;
    • Human cost: Requirement clarification, result review, issue diagnosis, and rework;
    • Risk-handling cost: Rollbacks, data repair, and additional checks caused by incorrect changes.

    For individual developers, human time can initially be tracked in minutes without forcing it into a monetary value. Enterprises can use a standardized cost-allocation method, but should avoid double counting.

    Two additional metrics are also useful.

    First-pass delivery rate: The percentage of tasks that pass acceptance after the model’s first submission without additional correction.

    Success rate within limits: The percentage of tasks completed within predefined limits for cost, time, or interaction rounds.

    These metrics help explain a common situation: a model may eventually solve a problem, but only after extensive human prompting and repeated execution. Its ultimate capability may not be low, while its practical delivery efficiency is still poor.

    4. Quota Efficiency Depends on the Shape of the Task

    Subscription quotas, call limits, and practical usability do not have a simple one-to-one relationship.

    Short-text editing, long-document analysis, continuous coding, and complex planning consume resources in different ways. The same quota can produce very different experiences in different workflows.

    4.1 A Long Context Is Not Necessarily an Efficient Context

    A model’s ability to accept a large amount of content does not mean that the entire conversation history should be submitted every time. Repeating obsolete logs, unrelated files, and discarded approaches can increase cost and interfere with the model’s judgment.

    A better approach is to maintain structured task state:

    • The current goal and acceptance criteria;
    • Confirmed constraints;
    • Completed steps;
    • Unresolved issues;
    • Evidence directly related to the current problem.

    When compressing context, preserve key constraints and the reasons behind important decisions. Keeping only the conclusions can cause the model to repeat paths that have already been ruled out.

    4.2 Retries Should Produce New Information

    A valuable retry adds evidence or tests a new hypothesis. An unproductive retry simply generates another answer that resembles the previous one.

    Workflows should therefore define stopping conditions. If consecutive attempts produce no new evidence, the process should move to human review or redefine the problem instead of continuing indefinitely. The appropriate number of attempts depends on task risk and budget; there is no universal threshold.

    4.3 Quota Rules Must Match the Work Rhythm

    Beyond the available quota, teams should examine rate limits, concurrency limits, context limits, and reset cycles. Evaluations should record the actual published rules rather than treating a short-term experience as a long-term guarantee.

    For teams that process large numbers of tasks at concentrated times, stable availability during peak periods may matter more than the average per-call price.

    5. The Value of Multi-Model Collaboration Lies in Role Separation

    A common approach is to assign planning to a more capable model and repetitive execution to a lower-cost model. This can be reasonable, but it does not automatically reduce total cost.

    5.1 Planning Must Produce a Handoff-Ready Task Package

    An effective task breakdown should specify at least:

    • Input materials and expected output;
    • What may be changed and what must not be changed;
    • Behavioral constraints that must be preserved;
    • An executable acceptance method;
    • Evidence that must be returned in case of failure.

    “Complete the backend logic” is not a sufficient task description. It leaves too many implicit decisions to the execution stage and can easily cause the work to drift.

    5.2 Execution Results Must Be Independently Verified

    A reasonable planning process does not prove that the execution result is correct. Code that runs may still violate business semantics, edge-case requirements, or security constraints.

    Checks that can be automated should be automated, including formatting, type checks, unit tests, and integration tests. High-risk areas involving permissions, data consistency, or irreversible operations still require human review.

    Similarly, a model’s claim that “the tests passed” cannot replace actual execution records.

    5.3 Collaboration Has Its Own Cost

    Multi-model workflows introduce context transfer, task routing, result integration, and conflict resolution. Collaboration is worthwhile only when its benefits exceed these additional costs.

    Teams should therefore test both a single-model baseline and the collaborative design, comparing the success rate and total cost of the complete workflow rather than comparing the price of only one execution step.

    6. Build a Reusable Evaluation Method Based on Real Tasks

    The most effective way to determine whether a model fits a specific business is not to repeatedly try a few arbitrary prompts, but to create a fixed and traceable task set.

    Step 1: Collect Tasks from Real Work

    The task set should cover frequent work as well as failure-prone edge cases, such as:

    • Creating an implementation plan from an incomplete description;
    • Locating a software defect supported by clear evidence;
    • Adding a feature while preserving existing constraints;
    • Generating traceable, verifiable conclusions from multiple documents;
    • Accurately explaining the missing conditions when a task cannot be completed.

    Tasks should be properly authorized, and sensitive information should be removed or protected.

    Step 2: Freeze Acceptance Rules in Advance

    Before testing, define what counts as success, partial completion, and a serious error. Do not change the criteria after seeing the results.

    For coding tasks, test results and human review can be used together. For analytical tasks, evaluate factual support, constraint coverage, and traceability rather than language fluency alone.

    Step 3: Control Experimental Conditions

    When comparing models, keep task inputs, permissions, available materials, and budgets as consistent as possible. If workflows differ, document those differences clearly to avoid mistaking a better prompt or process design for an inherent model advantage.

    Repeat the same task an appropriate number of times to observe output variance. A small number of examples may provide an initial signal, but cannot support broad performance claims.

    Step 4: Record Process Metrics

    In addition to final success rate, record interaction rounds, human interventions, elapsed time, actual cost, and failure causes. In high-risk scenarios, serious errors should be recorded separately so that they are not hidden by an average score.

    Failure categories may include:

    • Requirement misunderstanding;
    • Factual error;
    • Execution error;
    • Missing context;
    • Insufficient verification;
    • Resource exhaustion.

    Optimization is possible only when the point of failure is known.

    7. Enterprise Deployment Requires Security and Governance

    Task completion is only one admission requirement. In an enterprise environment, teams must also address data permissions, operation auditing, and error recovery.

    Models should follow the principle of least privilege when accessing data. Explicit approval or confirmation should be required before writing data, changing permissions, or performing irreversible operations. Test and production environments should also be isolated so that exploratory attempts are not applied directly to live business systems.

    Logs should support troubleshooting while avoiding unnecessary retention of sensitive content. When external materials or user-provided content are involved, organizations must also prevent embedded instructions from being mistaken for system authorization and causing unauthorized actions.

    These measures do not directly improve a model’s reasoning ability, but they strongly influence whether the overall system can be operated safely. Enterprises are not buying one impressive answer; they are adopting a production workflow with controllable risks.

    8. Where a Real Breakthrough for Chinese Foundation Models Should Appear

    Evaluating Chinese foundation models does not require choosing between emotional praise and blanket criticism. Three areas of sustained improvement matter more.

    First, can the model understand incomplete requirements more accurately and convert them into verifiable tasks?

    Second, can it maintain consistent constraints across multi-step execution and correct itself using real evidence rather than relying on repeated generation?

    Third, can it continuously reduce the unit cost of successful tasks within a defined budget and risk boundary?

    Parameter count, response length, and available quota may all affect the user experience, but they are not substitutes for delivery value. A genuinely competitive model should reduce the time developers spend explaining requirements and reworking ineffective output, while enabling enterprises to predict costs, inspect results, and control risk.

    When these capabilities hold consistently across real tasks, progress in Chinese foundation models becomes more than a temporary discussion topic. It becomes an engineering capability that organizations can adopt and rely on.