LLM Real-World Capability and Cost-Effectiveness Evaluation Guide

When a new generation of large language models is released, developers and enterprise users usually care about three questions:

1. Can it reliably complete real tasks?
2. Are its knowledge boundaries, factual consistency, and hallucination control acceptable?
3. Is its overall cost reasonable under the same business objective?

These three questions correspond to capability, reliability, and cost-effectiveness. Many evaluations only look at benchmark scores or demo cases, which can overlook complex factors in real systems, such as tool invocation, long-running tasks, permission restrictions, retry costs, manual verification costs, and operational overhead. For scenarios such as intelligent question answering, code assistance, document understanding, workflow automation, and agent execution, model output is not the endpoint; closing the business loop is the endpoint.

Therefore, evaluating a new model should not only ask whether it can answer a question. It should ask whether it can reliably produce usable results under controlled conditions, clear constraints, and auditable processes.

Building a Unified Evaluation Framework: Capability, Stability, Cost, and Risk

Before formally comparing models, it is advisable to establish a four-dimensional evaluation framework.

1. Capability Dimension

The capability dimension focuses on whether the model can understand tasks, generate correct results, and follow constraints. Common evaluation points include:

• Whether it can understand complex instructions.
• Whether it can distinguish facts from guesses.
• Whether it can maintain contextual consistency across multiple turns.
• Whether it can generate structured results.
• Whether it can provide executable content in scenarios such as code, tables, and process descriptions.
• Whether it can correctly invoke tools or explain what additional information is needed.

2. Stability Dimension

Stability determines whether a model can be deployed in production systems. Good performance in a single run does not mean the model is ready for launch. The following aspects should be observed:

• Whether repeated runs with the same input produce consistent results.
• Whether it answers boundary cases inappropriately.
• Whether it clearly states uncertainty when information is insufficient.
• Whether it can degrade reasonably when tools fail.
• Whether it loses the original objective during long tasks.
• Whether it frequently outputs meaningless repeated content.

3. Cost Dimension

Cost is not only the price per invocation. In large-scale applications, factors that truly affect total cost include:

• Input and output token consumption.
• Average response latency.
• Failure rate and retry rate.
• Number of tool calls.
• Manual review time.
• Logging, monitoring, and operational overhead.
• Business rework costs caused by incorrect outputs.

4. Risk Dimension

Enterprise scenarios require special attention to risk control, including:

• Whether it may leak sensitive information.
• Whether it may generate arbitrary conclusions that are difficult to audit.
• Whether it may confuse real events with fictional content.
• Whether it may fabricate sources when evidence is lacking.
• Whether it follows format, permission, and access-scope restrictions.
• Whether it can record the complete invocation chain and decision basis.

Test Environment Setup: Offline, Controlled, and Reproducible

If the evaluation objective is to observe the model’s own knowledge, reasoning, and output habits rather than its external retrieval capability, a controlled runtime environment should be used. For example:

• Disable external network access.
• Do not access external links.
• Do not invoke undeclared external services.
• Log all inputs and outputs.
• Record the duration of each task.
• Run repeated tests on the same task.
• Keep system prompts and tool permissions fixed.
• Preserve raw model outputs to avoid distortion caused by secondary rewriting.

This kind of environment helps answer a key question: given a specific context, can the model provide stable, self-consistent, and explainable answers, rather than relying on external real-time information to fill gaps?

It should be noted that offline testing is suitable for evaluating basic cognition, factual memory, hallucination control, and instruction following. It is not suitable for scenarios that require real-time data, the latest policies, or online business systems. Such scenarios should be evaluated separately with retrieval, tools, and permission boundaries included.

Designing Basic Test Sets: Six Task Categories

For intelligent question answering and agent scenarios, six types of basic test questions can be designed. These tests are not tied to any specific model. Instead, they abstract a general evaluation methodology.

Question 0: Identity, Version, and Knowledge-Boundary Testing

Objective: Observe whether the model can accurately describe its own positioning, version information, and knowledge cutoff range.

Example questions may include:

• What type of model are you?
• What is your version or identifier?
• What is your knowledge cutoff date?
• Do you know what type of environment you are currently running in?
• Does this information come from system prompts, training corpora, or the model’s own inference?

The evaluation focus is not whether the answer is polished, but whether the model has clear boundary awareness:

• Does it acknowledge that some information comes from the runtime environment?
• Does it fabricate training details that do not exist?
• Can it distinguish between information provided by the system and the model’s own knowledge?
• Does it proactively state uncertainty when it is uncertain?

An ideal answer usually contains three layers:

1. Clear identity and runtime environment.
2. Knowledge cutoff date or information source.
3. Acknowledgment that the model cannot fully introspect training details.

This type of test can effectively expose whether a model is overconfident, fabricates parameters arbitrarily, or confuses context.

Question 1: Recent Public Events and Timeliness Testing

Objective: Observe the model’s memory of relatively recent events.

Under offline conditions, the model can be asked to recall several recent public events, such as major activities, disaster incidents, international situations, or technological developments. During evaluation, focus on:

• Whether the event timing is accurate.
• Whether the event subjects are clear.
• Whether events from different years are confused.
• Whether it fabricates events that did not occur.
• Whether it acknowledges sparse memory because of the knowledge cutoff date.
• Whether it clearly states “I may not know” or “This information may be outside my knowledge range.”

The value of this type of test is that many models show fragmented memory near their knowledge cutoff date. They may understand the general framework but have unreliable details. If a model proactively reduces certainty, it may be more reliable than one that forcibly provides complete details.

Question 2: Historical Facts and Common-Sense Consistency Testing

Objective: Observe the model’s memory of stable historical facts.

Test questions can be designed around facts with long time spans and abundant public materials, such as historical elections, major engineering projects, scientific events, or policy changes. Evaluation points include:

• Whether key figures are accurate.
• Whether the timeline is coherent.
• Whether relationships between people are clear.
• Whether similar events are confused.
• Whether it continues to insist on an incorrect conclusion when details are wrong.

This type of test is suitable for evaluating the model’s basic factual stability. Compared with timeliness questions, historical fact questions better reflect the quality of the model’s long-term memory.

Question 3: Controversial Events and Uncertainty Handling Testing

Objective: Observe the model’s performance when information conflicts, casualty figures are disputed, or statistical standards are inconsistent.

Many public events have different statements from multiple parties. It is critical whether the model behaves reasonably in the following aspects:

• Whether it distinguishes between “one party claims” and “independently verified by a third party.”
• Whether it lists different information sources.
• Whether it avoids presenting a single statement as established fact.
• Whether it can point out insufficient evidence.
• Whether it fabricates details in pursuit of a complete answer.

This type of test is suitable for enterprise risk control, public-opinion analysis, knowledge question answering, and content moderation. The model should not only “know,” but also “know its boundaries.”

Question 4: Fictional Entities and Hallucination Recognition Testing

Objective: Observe whether the model can identify non-existent entities and avoid treating fictional content as fact.

Test cases can include several fictional names of people, organizations, or projects, mixed with real entities. For example:

• Have you heard of a certain expert publishing a report in a given year?
• Did a certain organization release a standard in a given year?
• Did a certain project win an award?

The evaluation focus is whether the model:

• Directly admits that it has not heard of the entity.
• States that it cannot confirm the information.
• Requests more background information.
• Falsely fabricates resumes, dates, locations, or achievements.

This type of test is especially important for knowledge-base question answering, customer-service question answering, and enterprise knowledge assistants. In production systems, the most dangerous situation is not that a model cannot answer, but that a model answers non-existent content as if it were real.

Question 5: Disaster, Safety Incident, and Public Information Testing

Objective: Observe whether the model expresses itself prudently in sensitive public events.

The model can be asked about its understanding of certain public events, such as natural disasters, factory accidents, food safety incidents, or anomalies in urban infrastructure. Evaluation points include:

• Whether it accurately describes the event type.
• Whether it avoids unconfirmed casualty figures.
• Whether it can distinguish official reports from media speculation.
• Whether it gives conservative statements when information is insufficient.
• Whether it reminds users to rely on authoritative channels.

This type of test examines not only knowledge, but also the model’s risk-awareness expression and compliance awareness.

Key Metrics for Agent Scenarios: Not Just Answering, But Execution

If a model is used for agents or automation workflows, the evaluation focus changes. The key is not the quality of single-turn question answering, but the completion of multi-step tasks.

1. Task Planning Capability

Can the model break a complex objective into executable steps? For example:

• A user asks the model to organize materials and produce a summary.
• A user asks the model to review code and generate modification suggestions.
• A user asks the model to generate analytical conclusions from tabular data.
• A user asks the model to call tools for queries before answering.

During evaluation, observe whether the model can:

• Clarify the task objective.
• Identify missing information.
• Arrange steps reasonably.
• Avoid irrelevant expansion.
• Propose alternatives when failure occurs.

2. Tool-Calling Capability

Agents usually need to invoke search, databases, file systems, logging systems, ticketing systems, or code execution environments. Evaluation should focus on:

• Whether it selects the correct tool.
• Whether it generates reasonable parameters.
• Whether it stops erroneous tool calls when a tool fails.
• Whether it repeatedly calls the same tool.
• Whether it can continue to the next step based on tool results.
• Whether it clearly states limitations when permissions are insufficient.

A sample evaluation logic is as follows:

def run_agent_task(model, task, tools, max_steps=6):
    history = []
    for step in range(max_steps):
        decision = model.plan(task, history)
        if decision.type == "answer":
            return decision.content
        if decision.type == "tool_call":
            result = tools.execute(decision.name, decision.args)
            history.append((decision, result))
            continue
        return "cannot determine next step"
    return "maximum step limit reached"

This logic can be used to observe whether the model falls into an infinite loop, misuses tools, or fabricates information when required information is missing.

3. Long-Chain Stability

Agent tasks are often not completed in one step. During evaluation, record:

• Average number of steps per task.
• Number of failed retries.
• Context loss situations.
• Objective drift situations.
• Completeness of final output.
• Frequency of human takeover.

If a model performs well in the first two steps but begins deviating from the objective by the fifth step, it is not suitable for long-chain automation scenarios.

Cost-Effectiveness: From Unit Price to Total Cost of Ownership

Many teams evaluate model cost-effectiveness only by the price per million tokens. This approach is too coarse. A more reasonable approach is to adopt a total cost of ownership perspective.

A simplified evaluation formula is as follows:

Total cost = model invocation cost + retry cost + tool call cost + manual review cost + operations cost

Specifically:

• Model invocation cost is related to input and output token usage.
• Retry cost is related to failure rate and timeout rate.
• Tool call cost is related to external interfaces and query frequency.
• Manual review cost is related to error rate and risk level.
• Operations cost is related to logging, monitoring, rate limiting, caching, gateways, and related infrastructure.

Looking only at invocation unit price can lead to misjudgment. For example, a model with a lower unit price but a high error rate may cause many retries and manual corrections, ultimately resulting in a higher total cost.

1. Quality-Weighted Cost

A quality-weighted approach can be used:

Effective cost = total cost / number of successfully completed tasks

If a task runs five times and only two pass acceptance, the effective cost is not the single-run cost, but:

Effective cost = total cost of five runs / 2

This approach is more suitable for evaluating production systems.

2. Cost Weights Vary by Scenario

Different businesses have different cost sensitivities:

• High-concurrency customer-service Q&A: low latency and per-call cost are more important.
• Code assistance: accuracy and reduced rework are more important.
• Agent workflows: task success rate and tool-call stability are more important.
• Risk-control and compliance scenarios: explainability, traceability, and security are more important.
• Content production: generation speed, style consistency, and manual editing cost are more important.

Therefore, cost-effectiveness is not a single metric, but a scenario-dependent function.

Engineering Readiness: Observable, Regression-Testable, and Traceable Evaluation

If model evaluation relies only on manually asking a few questions, it is easily affected by subjective judgment. To form a long-term reusable capability, engineering practices are needed.

1. Test-Sample Management

Test samples should be organized by type:

• Identity and version questions.
• Timeliness fact questions.
• Historical fact questions.
• Controversial event questions.
• Fictional entity questions.
• Public event questions.
• Tool-calling questions.
• Code generation questions.
• Multi-turn dialogue questions.
• Permission and unauthorized-access test questions.

Samples should include input, expected behavior, scoring rules, and notes. Expected behavior is not a single correct answer, but reasonable patterns the model should exhibit, such as “acknowledge uncertainty,” “request additional information,” or “refuse to fabricate.”

2. Batch Evaluation and Repeated Sampling

For models with high stability requirements, each task should be repeated multiple times. By measuring pass rate, variance, and failure types, one can determine whether the model is stable.

A sample evaluation process is as follows:

def batch_evaluate(model, tasks, repeat=5):
    records = []
    for task in tasks:
        for i in range(repeat):
            result = run_task(model, task)
            records.append({
                "task_id": task.id,
                "round": i + 1,
                "output": result.output,
                "latency": result.latency,
                "tokens": result.tokens,
                "tool_calls": result.tool_calls,
                "score": task.score(result.output)
            })
    return summarize(records)

Summary metrics may include:

• Average pass rate.
• Worst pass rate.
• Average latency.
• Average token consumption.
• Average number of tool calls.
• Common failure reasons.
• Whether high-risk outputs exist.

3. Logging and Monitoring

In general cloud environments, such as Baidu AI Cloud, evaluation samples can be stored in object storage, batch tasks can be scheduled through message queues, model-calling services can run on container platforms, and logs and monitoring can be used to observe metrics. The emphasis is on general engineering capabilities, not proprietary features of a specific vendor.

Key information to record includes:

• Request time.
• Input summary.
• System prompt version.
• Model identifier.
• Output summary.
• Tool-call records.
• Latency.
• Token consumption.
• Scoring results.
• Manual review conclusions.
• Exception stacks or failure reasons.

4. Permissions and Isolation

The evaluation environment should be isolated from the production environment. This is especially important when enterprise knowledge, user data, code repositories, or internal documents are involved. Clarify:

• Which data can enter testing.
• Which fields need to be masked.
• Whether external calls are allowed.
• Whether outputs can be saved.
• Whether audit records are required.
• Whether manual approval is required before launch.

For highly sensitive scenarios, private deployment or dedicated resource pools can be used to avoid data exfiltration risks.

How to Interpret Evaluation Results: Higher Is Not Always Better

After evaluation results are generated, do not look only at the overall score. The results need to be analyzed in more detail.

1. High Average Score but High Variance

If a model has a high average score but large variance, it lacks stability. In production systems, occasional serious errors may be more dangerous than an average high score.

2. Low Score but Explainable

If a model often admits “I do not know” but does not fabricate answers, the risk is relatively low. For enterprise question answering, a conservative model may be more suitable for high-risk scenarios.

3. High Accuracy but High Cost

If a model has high accuracy but high latency, large token consumption, and frequent tool calls, its value should be judged together with business benefits. For real-time interaction scenarios, responses that are too slow may offset the accuracy advantage.

4. Strong Tool Use but Weak Planning

Some models call tools accurately but are not good at complex planning and may execute mechanically. Such models are suitable for simple workflow automation but not for open-ended agent tasks.

5. Outdated Knowledge but Low Hallucination

If a model has an earlier knowledge cutoff date but can clearly explain its boundaries, its gaps can be filled through external retrieval, knowledge-base augmentation, or tools. Conversely, a model that appears knowledgeable but frequently fabricates content carries higher risk.

Selection Recommendations by Business Scenario

1. Intelligent Q&A

Key focus areas:

• Factual accuracy.
• Hallucination control.
• Uncertainty expression.
• Multi-turn context consistency.
• Support for structured answers.
• Ease of integration with knowledge-base retrieval.

It is advisable to prioritize models that are willing to say “I do not know” when uncertain, and to use retrieval augmentation to reduce the risk of outdated knowledge.

2. Code Assistance

Key focus areas:

• Code correctness.
• Instruction following.
• Control of modification scope.
• Whether dependencies are introduced without authorization.
• Whether the reason for modification can be explained.
• Whether the generated code can pass test cases.

During evaluation, do not only check whether the generated code runs. Also check whether it breaks existing logic, introduces security risks, or violates team standards.

3. Agent Workflows

Key focus areas:

• Task decomposition.
• Tool selection.
• Parameter generation.
• Failure recovery.
• Maximum step control.
• Permission boundaries.
• Traceable logs.

It is advisable to start with a small-scale sandbox pilot and gradually expand permissions.

4. Enterprise Knowledge Assistants

Key focus areas:

• Data isolation.
• Permission control.
• Source citation.
• Sensitive information recognition.
• Answer consistency.
• Manual fallback mechanisms.

This type of scenario should not rely entirely on the model’s internal knowledge. It should establish a closed loop involving knowledge bases, retrieval, citations, and auditing.

5. Content Generation

Key focus areas:

• Style stability.
• Long-form structure capability.
• Factual consistency.
• Repetition rate.
• Manual editing cost.
• Potential compliance risks.

Cost-effectiveness evaluation should include later review and editing time, not only generation speed.

Pre-Launch Checklist

Before a model is officially integrated into a business system, the following checks can be performed:

1. Have identity, version, and knowledge-boundary tests been completed?
2. Have conservative expressions for recent events and controversial events been verified?
3. Have fictional entity recognition capabilities been tested?
4. Have tool calls and failure degradation been verified?
5. Have multi-turn long-task tests been completed?
6. Have latency, token consumption, and retry rates been recorded?
7. Has a manual review mechanism been established?
8. Have logging, monitoring, and alerting been configured?
9. Have permission isolation and data masking been completed?
10. Has a rollback plan been defined?
11. Has a small-traffic canary release been performed?
12. Has a continuous regression evaluation mechanism been established?

Summary: Cost-Effectiveness Is Reliability, Controllability, and Reusability

After a new generation of large models is released, what truly matters is not short-term attention, but whether the model can enter stable, auditable, and maintainable business systems. When evaluating actual capability, focus on key indicators such as identity consistency, knowledge boundaries, factual recall, hallucination suppression, tool invocation, and long-chain stability. When evaluating cost-effectiveness, start from total cost of ownership and consider invocation fees, failed retries, manual review, operational costs, and business losses together.

For developers and enterprise users, the most reasonable path is as follows: first build a unified evaluation framework, then validate it repeatedly with real tasks; first examine stability, then examine cost; first control risk, then expand scale. Only by combining model capability, engineering capability, and business capability can one determine whether a model is truly suitable for launch, rather than stopping at a stage where it merely appears powerful in demonstrations.