Evaluating Large Language Models for Complex Task Execution: Metrics, Failure Modes, and Engineering Practices

The evaluation focus is moving from generation quality to task completion

Previously, evaluating a large language model often meant checking single-turn Q&A fluency, knowledge coverage, and code snippet accuracy. Once a model is connected to real business pipelines—fixing bugs, editing documents, reading screenshots, organizing information, or running demos—the evaluation criteria must change.

Single-turn generation quality measures local optimality: a sentence sounds good, a function is correct. Real work cares about global optimality: whether the task closes the loop, whether intermediate steps drift away from instructions, whether the output can be delivered directly, and whether the process can be repeated without errors.

This introduces three new evaluation dimensions:

• Instruction-following stability: The more constraints there are—format, wording, prohibited actions—the more clearly model compliance can be measured.
• Context resistance: After tens of minutes and large token consumption, does the model still remember the original rules?
• Self-verification ability: Before delivery, does the model proactively check its own work instead of carrying errors to the end?

These three dimensions form the core of long-chain tasks. The industry is gradually converging on the view that agentic engineering capability and visual understanding are the two hardest problems for assessing a model’s practical value.

Three typical failure modes in long-chain tasks

Before designing an evaluation, it is necessary to define what failure looks like. Otherwise, the metrics will be severely distorted.

1. Instruction drift
As conversations become longer, the weight of early constraints is diluted. A typical example: step 3 requires “only modify configuration, do not touch business code,” but by step 20 the model starts refactoring an entire module. This is especially common in multi-turn tool use because the model’s attention is easily captured by the most recent observations.

2. Context decay and information gaps
In long-document processing, the model may compress middle content into a single summary and later replace the original text with that summary. Once the task requires a precise clause or a specific value, the error rate increases sharply. The correct engineering response is not simply “expand the context window further,” but “reduce dependence on full context.”

3. Error cascades
Tool-call errors are not isolated events. A wrong file path leads to a read failure, which leads to a fabricated explanation based on empty results, which then leads to a patch based on the wrong explanation. The longer the chain, the more important self-repair capability becomes.

How to evaluate complex task completion

End-to-end engineering tasks: use test pass rates instead of human impression

A widely adopted public evaluation paradigm works as follows: given a problem description from a real code repository, ask the model to produce a patch, then use the repository’s own test suite to determine whether the issue is fixed. Two design choices are especially worth borrowing:

• Environment isolation: Each task prepares dependencies in an isolated container, ensuring reproducibility and parallelism.
• Objective results: Pass/fail tests and static-check results are the only judges, rather than asking another model to decide whether the output “feels right.”

When building an internal evaluation set, it is advisable to cover multiple languages and group tasks by type—bug fixes, performance optimization, dependency upgrades, and documentation generation—otherwise the overall score can be dominated by a single task type and lose its guidance value.

Multimodal visual understanding: structured scoring instead of “looks accurate enough”

Visual capability is invisible infrastructure for long tasks. Document parsing, UI screenshot reconstruction, chart reading, and invoice extraction all depend on it. When evaluating, avoid asking only “Does the description look similar?” Instead, reduce the evaluation to verifiable field-level dimensions:

Evaluation dimension	Verification method
Field extraction accuracy	Compare against ground truth field by field and calculate precision/recall.
Layout structure reconstruction	Compare table rows/columns and heading hierarchy with the original document.
Numeric reading	Check chart values with tolerance; units and magnitude must match.
Multi-page consistency	Detect conflicts in values and attributes when the same entity appears across pages.
Instruction compliance	Check whether output strictly follows the specified schema and contains no extra fields.

Example code: validate the contract and self-consistency of visual extraction results

# Use a JSON Schema or type definitions to enforce the output contract.
# Extra or out-of-range fields should fail validation.
INVOICE_SCHEMA = {
    "type": "object",
    "required": ["invoice_no", "total_amount", "currency", "items"],
    "properties": {
        "invoice_no": {"type": "string", "pattern": r"^[A-Z0-9-]{6,}$"},
        "total_amount": {"type": "number"},
        "currency": {"type": "string", "enum": ["CNY", "USD", "EUR"]},
        "items": {
            "type": "array",
            "items": {
                "type": "object",
                "required": ["name", "qty", "unit_price"],
                "properties": {
                    "name": {"type": "string"},
                    "qty": {"type": "number"},
                    "unit_price": {"type": "number"},
                },
            },
        },
    },
}
def verify(result: dict) -> list[str]:
    """Return a list of validation errors; an empty list means validation passes."""
    errors = []
    # 1) Structural and type validation can be handled by a JSON Schema validator.
    # 2) Business self-consistency: the sum of item amounts should match the total.
    if result.get("items"):
        subtotal = sum(i["qty"] * i["unit_price"] for i in result["items"])
        if abs(subtotal - result["total_amount"]) > 0.01:
            errors.append(f"Subtotal {subtotal:.2f} does not match total {result['total_amount']}")
    return errors

The value of such self-consistency checks is that most hallucinations can be intercepted before delivery without human review of images. They can also be fixed as regression tests.

Long-chain stability: record process metrics, not only final results

If the same task is run multiple times and only some runs succeed, an “average score” is almost meaningless. At least four metrics should be recorded:

• First-pass success rate: The proportion of outputs that meet requirements on the first round. This reflects real usability.
• Intervention rate: The proportion of tasks that require human correction. This is a direct source of labor cost.
• Tool-call efficiency: Average number of steps, invalid-call ratio, and retry count.
• Latency distribution: End-to-end P50/P95, used for capacity and quota planning.

Turning evaluation results into engineering practice

Build a self-verification loop

Do not expect the model to get it right in one shot. Instead, give it an executable source of truth: tests, compilation, static checks, and consistency validation. Treat model output as a “hypothesis to be verified.”

def agentic_fix(task, model, verify_fn, max_rounds=5):
    history = [system_prompt(), task]
    for round_idx in range(max_rounds):
        action = model.step(history)        # Produce a patch or a tool call
        observation = execute(action)       # Execute for real: tests / compilation / static checks
        if verify_fn(observation):
            return action.patch, round_idx + 1
        history += [action, observation]
    raise RuntimeError("Maximum retry rounds exceeded; escalate to manual handling")

Key constraint: the round limit and token budget must be hard-coded. A self-verification loop without an upper bound degrades into a resource-consuming loop. This is the most common cause of cost overrun in long-running tasks.

Context engineering: actively discard, rather than passively accumulate

• Segmentation and summarization: Chunk long documents first, generate structured summaries, and read back original fragments only when details are needed.
• Retrieval-based citation: Put original document slices into vector retrieval or a full-text index so the model can “retrieve” rather than “memorize.”
• External memory: Write key constraints, glossaries, and interface contracts into persistent storage outside the task. Re-inject them each round instead of relying on conversation history.
• Structured rewriting: When history becomes too long, rewrite it once to keep only decisions, completed items, pending items, and constraint lists, discarding intermediate noise.

Cost governance: a stronger model does not automatically mean a better total bill

Models that follow instructions better and produce output closer to finished products often also consume more tokens per request—they are more likely to expand reasoning chains, initiate more tool calls, and generate longer intermediate artifacts. Sending every request to such models without differentiation can increase total cost several times over. A practical approach is tiering:

1. Coarse screening layer: Use a low-cost model for classification, extraction, formatting, and rewriting.
2. Deep reasoning layer: Upgrade to a higher-capability model only for complex reasoning, cross-file refactoring, or difficult visual judgments.
3. Cache layer: Use prefix caching for system instructions and retrieval results, and idempotent caching for identical inputs.
4. Quota layer: Set limits on rounds, tokens, and latency for each task; automatically downgrade or escalate to human handling when limits are exceeded.

Cloud engineering support: turning model capability into a stable service

Beyond evaluation and governance, deployment requires supporting infrastructure. Common capability categories include:

• Object storage: Store raw documents, screenshots, evaluation sets, and outputs. Archive them by task ID versioning to ensure reproducibility.
• Container platform: Launch an isolated sandbox for each task and carry runtime dependencies. This solves environment consistency at the root.
• Message queue: Smooth traffic peaks, separate long tasks from synchronous requests, and support retries and dead-letter queues.
• Logging service and monitoring alerts: Record input summaries, tool calls, token consumption, and latency for each round. Aggregate dashboards by task and alert on drops in pass rate, cost anomalies, and rising timeout rates.
• Data platform: Automatically collect online failure samples into evaluation sets, forming a loop of “discover problems online, verify them offline.”

For example, Baidu AI Cloud can be used to assemble the above capabilities through standard product categories such as object storage, container services, message queues, and logging/monitoring. The evaluation framework and governance logic are decoupled from the specific model, allowing smooth migration during model iterations without rewriting the pipeline.

Production checklist

• Is the evaluation set grouped by task type and equipped with an objective judge, such as tests or validators?
• Does the evaluation distinguish first-pass success rate from post-intervention success rate?
• Are token consumption, tool-call steps, and P95 latency recorded?
• Are visual tasks evaluated at field level, structure level, and numeric level?
• Does the self-verification loop have round and budget limits?
• Are key constraints stored externally for repeatable injection rather than relying on conversation history?
• Are tiered routing and result caching implemented?
• Do online failure samples automatically flow back into the evaluation set?

Conclusion

The shift from “does the answer look good?” to “is the task actually completed?” is not a gradual improvement in capability; it is a reconstruction of the evaluation system. A single impressive generation is increasingly insufficient to support production decisions. What truly determines ROI is the set of three numbers: task closure rate, human intervention rate, and cost per delivery. By turning evaluation into engineering, verification into a loop, and cost into observable metrics, each capability upgrade can be converted into deliverable production capacity as large models continue to iterate quickly.