How to Evaluate a Production LLM: Capability and Cost Through Real-World Agent Testing

    A production LLM should be evaluated not only by benchmark scores or single-turn response quality, but also by its ability to complete complex tasks reliably and at a reasonable cost. This article compares production and preview-stage models across reasoning, task generalization, agent workflows, frontend development, retrieval, stability, output limits, and total cost per accepted task. It also presents a practical evaluation framework for engineering teams.

    1. Why a Production Model Should Not Be Judged by a Single Score

    Evaluating a production LLM cannot be reduced to whether it is simply “stronger” or “weaker.” Model capability includes at least four dimensions: basic comprehension and reasoning, the ability to advance complex tasks autonomously, output stability, and the cost of completing each task.

    This is especially important in agent-based workflows. An AI agent does not merely answer one question. It must understand the objective, break down the task, call tools, read feedback, revise its approach, and deliver a usable result. A mistake at any stage can make the final output unusable. Comparing response quality in a single round of conversation therefore often fails to reflect the model’s actual engineering value.

    Based on public testing and practical use, this production version shows a clear stage-level improvement over the preview version. It relies less on familiar task templates and demonstrates stronger cross-domain transfer. Across many standardized tasks, it can generally advance toward a usable result with limited intervention. However, this progress does not mean that stability and cost problems have been fully solved.

    2. Key Changes from the Preview Version

    2.1 A Significant Improvement in Overall Capability

    The preview version performed relatively well on familiar frontend development tasks. However, its quality dropped quickly when a task involved an unfamiliar domain, complex constraints, or multi-step reasoning. It could sometimes generate code that appeared complete without truly understanding the objective, leaving the result merely “formally complete.”

    The production version performs better in this respect. After continued training and optimization, it is more accurate at identifying task objectives and can establish a relatively clear execution path before starting. Even when facing an unfamiliar technical topic, it is less likely to provide an immediate, careless answer. Instead, it may retrieve documentation, inspect the project structure, analyze existing implementations, and proceed step by step.

    The value of this change is reflected not only in the final answer, but also in process control. A mature agent model should be able to:

    • Confirm the task scope before modifying code;
    • Identify key dependencies and constraints in the existing project;
    • Break a large task into smaller, verifiable steps;
    • Adjust its approach when a tool returns an error;
    • Run basic tests and review the result after completion.

    The production version shows better continuity across these stages, making it more suitable for engineering tasks that require multiple rounds of interaction.

    2.2 Stronger Generalization Across Tasks

    The production version is also less specialized in a single task category. The preview version generally had an advantage in interface construction and styling, but its performance varied significantly in data processing, configuration changes, documentation work, and business-logic implementation.

    The production version is more balanced across these areas. Even when it cannot complete every detail in one attempt, it can often build a basically usable solution first and then fill in missing pieces through follow-up prompts. This suggests a shift from generating results based primarily on familiar patterns toward understanding objectives and executing tasks.

    Greater generalization does not mean that the model has unlimited knowledge coverage. For niche technologies, less common libraries, or business rules without sufficient context, it still needs substantial retrieval. If the retrieved information is incomplete, the model may choose an approximate solution or mix concepts from different technology stacks.

    For enterprise use, teams should therefore provide project documentation, API constraints, coding standards, and acceptance criteria.

    3. Practical Performance in Agent Tasks

    3.1 Complex Tasks Can Reach a Usable Standard

    Beyond standardized benchmarks, the production version has also improved at handling tasks that do not follow a fixed template. For example, a task may require the model to:

    1. Read an existing project;
    2. Design a new interaction flow;
    3. Modify multiple modules;
    4. Preserve existing functionality;
    5. Run tests and fix errors;
    6. Prepare a summary of the changes.

    The difficulty lies not in writing an isolated piece of code, but in maintaining consistency across the entire project. The production version can usually handle the main path first and then make local corrections based on test results. Even when the initial approach is imperfect, it is less likely to deviate completely from the objective.

    For developers, this means the model can take on more work similar to that of a junior engineer or development assistant, such as generating basic pages, adding routine APIs, organizing configuration files, writing test examples, and locating obvious errors.

    3.2 Frontend Work Has a Relatively High Ceiling

    When the model performs well, its frontend output can be quite complete. In addition to basic layouts, it can handle component hierarchy, visual hierarchy, interaction feedback, and simple animations. Some outputs achieve a reasonable level of detail and motion design rather than merely meeting the minimum requirement of being runnable.

    However, frontend quality is highly sensitive to how the task is described. If a requirement specifies only the functionality without explaining the target users, page hierarchy, visual style, or interaction principles, the model may fill in the design independently and produce a result that differs from expectations.

    Frontend tasks should clearly define:

    • The target users of the page;
    • The primary user flow;
    • Page elements that must be preserved;
    • Responsive layout requirements;
    • Colors, spacing, and visual style;
    • Error, empty, and loading states;
    • Interaction details that will receive particular attention during acceptance testing.

    3.3 Retrieval Can Compensate for Knowledge Gaps, but Also Creates Risks

    When facing an unfamiliar problem, the production version often retrieves relevant information proactively. This can significantly improve the probability of success, but retrieval does not necessarily mean understanding. The model may find documentation for multiple versions, confuse different implementation approaches, or generate code based on outdated information.

    Retrieved information must therefore be validated. A more reliable workflow is to:

    • Prefer official documentation or existing project documentation;
    • Confirm versions and parameters for critical interfaces;
    • Compare retrieved conclusions with the current code structure;
    • Use a minimal experiment to verify behavior;
    • Avoid copying documentation examples directly into production code.

    If the model does not validate its sources on its own, developers should ask it to state its evidence, version assumptions, and verification results explicitly.

    4. Stability Remains the Main Weakness

    The most important issue with the production version is not its capability ceiling, but the relatively wide variation in output quality.

    Under ideal conditions, the model can deliver a result that is close to usable. Its steps are concise, the code structure is reasonable, and the scope of changes is relatively controlled. Under less favorable conditions, however, it may still exhibit several problems.

    4.1 Incomplete Instruction Following

    The model may complete the local function that appears most important while overlooking constraints such as “do not modify these files” or “preserve backward-compatible behavior.” As a task contains more requirements, the probability of omissions often increases.

    A practical solution is to divide requirements into three categories:

    • Hard constraints that must not be violated;
    • Functional objectives that must be completed;
    • Implementation details that the model may decide independently.

    The model should also be asked to restate the key constraints before execution and verify them one by one at the end.

    4.2 Local Optimization Can Break Global Consistency

    The model may focus on fixing one component while introducing style conflicts, state-management issues, or interface inconsistencies elsewhere. In a large project, a locally correct modification is not necessarily correct for the system as a whole.

    To reduce this risk, ask the model to analyze dependencies before making changes and limit the scope of each modification. Afterward, at least verify:

    • Whether other pages are affected;
    • Whether shared components are broken;
    • Whether an API contract has changed;
    • Whether duplicate logic has been introduced;
    • Whether tests and documentation need to be updated.

    4.3 Insufficient Coverage of Exceptional Cases

    Models generally find it easier to implement the main flow than to handle empty data, network failures, insufficient permissions, duplicate submissions, and timeouts. These issues may not be obvious in a demonstration task, but in a real business system, exception handling often determines reliability.

    Acceptance criteria should therefore cover normal, exceptional, and boundary scenarios rather than simply stating that “the feature works.”

    5. Output Limits Affect Complex Reasoning Tasks

    If the public calling method imposes a limit on the length of a single response, the model may be significantly constrained in long-chain reasoning, complex proofs, multi-file refactoring, and large-scale code generation.

    Insufficient output length may result in:

    • Truncated reasoning;
    • Partially generated code;
    • Conclusions without sufficient justification;
    • Multi-file changes that cannot be completed in one response;
    • Premature termination before the task is finished.

    This does not necessarily indicate insufficient model capability. It may instead reflect a limitation of the calling interface. Evaluation should therefore distinguish between model capability problems and service constraints.

    From an engineering perspective, teams can execute tasks in stages: ask the model to produce a plan first, process files or modules individually, and then run a unified review. This approach not only avoids output-length constraints, but also makes errors easier to locate.

    6. Cost Should Not Be Measured by Per-Call Pricing Alone

    Although the production version offers clear capability improvements, it may still have a considerable distance to go before becoming highly cost-effective. Task cost should be calculated based on the total resources required to complete an accepted task, not merely the price of one request.

    Total cost should include at least:

    • Input and output consumption;
    • The number of tool calls;
    • Retrieval and context-preparation overhead;
    • Failed retries;
    • Human review time;
    • Subsequent repair and maintenance costs.

    A model with a low per-call price but frequent rework may ultimately cost more than a model with a higher per-call price and a higher first-pass completion rate. A useful metric is:

    [
    \text{Effective Cost per Task}=\frac{\text{Model Invocation Cost}+\text{Human Repair Cost}}{\text{Number of Tasks That Pass Acceptance}}
    ]

    Teams can also track first-pass completion rate, average number of repair rounds, average number of tool calls, and the rate of severe errors. Only by considering these metrics together can an organization determine whether a model is suitable for large-scale use.

    To reduce the impact of randomness, teams should establish a fixed test set and run each task multiple times. The test set should cover:

    1. Routine code generation;
    2. Multi-file project modifications;
    3. Frontend page and interaction implementation;
    4. Documentation and configuration work;
    5. Retrieval in unfamiliar domains;
    6. Error diagnosis and repair;
    7. Incremental development that must preserve existing functionality.

    Each task should have clear acceptance criteria. Evaluation should distinguish among functional correctness, code quality, visual completeness, instruction following, and exception-handling capability.

    Comparisons between the production and preview versions must use the same evaluation criteria. If the criteria change midway, historical scores cannot be compared directly. A more rigorous approach is to preserve the original scores, add a new unified standard, and explain the reasons for any score changes in the report.

    8. Overall Conclusion

    Compared with the preview version, the production version has made its most important progress in three areas: stronger task generalization, better autonomous execution of complex agent workflows, and a higher capability ceiling for frontend and engineering tasks.

    However, it still exhibits significant randomness, with a wide gap between its best and worst performance. The model can reach a high level of completion on complex tasks, but it cannot guarantee the same result every time. In production scenarios that require high reliability, organizations should not rely on complete automation. A workflow that combines model execution, automated testing, and human review is more appropriate.

    The most accurate description is that the model has moved from preview-level capability suitable mainly for experimentation into a production stage where it can take on real development tasks. It still requires task decomposition, context management, acceptance checklists, and cost monitoring to deliver its full value.

    For developers, the key question is not the model’s ranking on a leaderboard. It is whether the model can reliably complete tasks in the target project and deliver maintainable results at an acceptable cost.