Evaluating LLMs for Complex Tasks and Improving Efficiency

    This article explains why single-turn answer quality is insufficient for evaluating large language models on complex tasks and presents an engineering framework for evaluation and efficiency improvement. It covers task stratification, metric design, baseline comparison, long-context stability, retrieval boundary control, review-oriented output, workflows for mathematical reasoning, quota and cost control, model version regression testing, and prompt design principles.

    Evaluating LLMs for Complex Tasks and Improving Efficiency

    1. Why complex tasks cannot be judged by single-turn answer quality

    In real-world usage, large language models are often applied to tasks such as summarization, question answering, code generation, information organization, fact-checking, solution review, mathematical reasoning, and report writing. Many people evaluate model capability with one or two simple questions, such as asking a common-sense question, requesting a short introduction, or summarizing a short passage. This approach can reveal basic language capability, but it is not enough to evaluate how a model performs on complex tasks.

    Complex tasks usually have several characteristics:

    1. The goal is overall usability, not isolated correctness
      An answer may be partially correct, but if the conclusion is unclear, the structure is disordered, or the evidence is insufficient, the final result may still be unusable.

    2. Process consumption is significantly higher than in single-turn Q&A
      A model may perform multiple rounds of reasoning, retrieval, comparison, and rewriting to verify a single detail. Without constraints, this can easily lead to large amounts of ineffective consumption.

    3. Long context amplifies stability issues
      In short conversations, a model can often produce a clear conclusion quickly. However, as the conversation becomes longer, problems such as style drift, repetitive elaboration, scattered conclusions, and excessive explanation become more common.

    4. Different tasks require different model capabilities
      Fact-checking emphasizes traceable evidence, review tasks emphasize risk identification, mathematical proofs emphasize logical completeness, and project evaluation emphasizes overall feasibility. Applying the same standard to all scenarios can easily lead to misjudgment.

    Therefore, evaluating whether a large language model is suitable for a certain type of work should not focus only on whether it can “speak well.” It should evaluate whether the model can consistently produce usable results under given tasks, given budgets, and given quality constraints.

    2. Building an evaluation framework for complex tasks

    A relatively reliable LLM evaluation framework should include at least four parts: task definition, evaluation metrics, baseline comparison, and cost accounting.

    1. Task stratification

    Tasks can be divided into several levels, with different evaluation focuses for each level.

    Task type Typical scenarios Key metrics
    Lightweight Q&A Concept explanation, simple summarization Accuracy, clarity, response speed
    Information organization Long-document summarization, key-point extraction Completeness, structural stability, coverage
    Fact-checking Data verification, viewpoint validation Evidence chain, source credibility, error rate
    Review and evaluation Paper review, solution review, code review Risk identification, overall judgment, actionable suggestions
    Complex reasoning Mathematical proof, logical deduction, architecture design Step completeness, consistency, verifiability

    The core of complex-task evaluation is not making the model “give an answer,” but making the model “give an answer that can be checked.”

    2. Metric design

    Evaluation metrics can be constructed from the following dimensions:

    • Correctness: Whether key conclusions hold and whether there are obvious factual errors.
    • Completeness: Whether the answer covers the core conditions, constraints, and output formats required by the task.
    • Traceability: Whether the model explains evidence, steps, assumptions, or sources.
    • Clarity of conclusion: Whether the conclusion can be identified quickly instead of being buried in excessive background information.
    • Cost efficiency: The number of calls, input and output length, reasoning time, and quota pressure required to complete the task.
    • Stability: Whether repeated execution of similar tasks maintains acceptable quality.
    • Repairability: Whether the cost of correction is low if the first output is unsatisfactory.

    Cost efficiency is often underestimated. Even if a model produces good answers, its practical engineering value decreases if every task requires multiple retries, long reasoning time, or large amounts of meaningless retrieval.

    3. Baseline comparison

    When evaluating model version changes, conclusions should not be based only on subjective impressions. It is recommended to establish a fixed baseline, including:

    • A fixed task set;
    • Fixed prompt templates;
    • Fixed output length limits;
    • Fixed retrieval settings;
    • Fixed budget limits;
    • Fixed human scoring rules.

    Only in this way can one determine whether changes come from model capability, task design, or prompt differences when comparing different versions, parameters, or calling strategies.

    3. Stability governance in long-conversation scenarios

    Long conversations are a difficult part of LLM capability evaluation. In short conversations, models can often place conclusions first. However, as conversations become longer, common problems include:

    • Forgetting earlier constraints;
    • Repeating information that has already been confirmed;
    • Introducing unnecessary metaphors or generalized expressions;
    • Giving vague or compromising answers to complex questions;
    • Producing inconsistent viewpoints across turns;
    • Losing control of output length, causing key conclusions to be diluted.

    To address these problems, engineering practices can introduce a “state card” and a “task checklist.”

    For example, the core constraints of the current task can be maintained as structured state:

    1. Task goal: Review the technical feasibility of a proposal
    2. Input scope: Only materials already provided in the current conversation
    3. Output requirement: State the conclusion first, then list risks, then provide recommendations
    4. Prohibited actions: Do not proactively expand into areas not requested
    5. Current progress: Material summary completed; key risks still to be checked

    Each request can include the state card as part of the context. This reduces the probability that the model drifts away from the task goal during long conversations.

    In addition, a segmented output strategy can be used:

    1. First output a conclusion summary;
    2. Then output key evidence;
    3. Then list uncertain items;
    4. Finally provide recommended next steps.

    This structured output improves readability and also makes subsequent automated validation easier.

    4. Controlling retrieval boundaries for fact-checking tasks

    In practical tasks, models sometimes exhibit “over-retrieval” or “over-reasoning” behavior: the user does not require verification of all details, but the model automatically expands into large amounts of background material, leading to longer task time, redundant output, and unclear focus.

    To solve this problem, evidence boundaries must be clarified in the prompt.

    A usable template is as follows:

    1. Please perform fact-checking based on the following task:
    2. Task: [input task]
    3. Required verification items: [list points that must be verified]
    4. Allowed retrieval scope: [specify document types or scope]
    5. Prohibited actions: Do not expand into verifications unrelated to the task
    6. Output requirements:
    7. 1. Provide an overall conclusion first;
    8. 2. Mark each required verification item as: supported / not supported / insufficient evidence;
    9. 3. If evidence is insufficient, state what is missing instead of continuing to expand;
    10. 4. Keep the total length within the specified limit.

    The key to this design is distinguishing among three types of issues:

    • Facts that must be verified: Those directly related to the task conclusion;
    • Auxiliary information that can be verified: Helpful for explanation but not affecting the main conclusion;
    • Expanded content that does not need verification: Content unrelated to the task goal.

    Without this stratification, models can easily fall into “endless rigor for the sake of rigor,” eventually leading to high cost, slow speed, and delayed conclusions.

    5. Review tasks should shift from “detail truth” to “overall judgment”

    In tasks such as paper review, project credibility review, code review, and solution evaluation, a common model problem is becoming trapped in local details, such as repeatedly verifying a number, a citation, or a term, without providing an effective judgment on overall quality.

    Review tasks are more suitable for a “general-specific-general” structure:

    1. Overall judgment
      First provide an overall conclusion, such as whether the proposal is feasible, whether major risks exist, or whether it has reached a reviewable state.

    2. Key issue list
      List issues by severity rather than by the order in which they are discovered.

    3. Evidence and basis
      Each key issue should include a brief basis explaining why it is considered risky.

    4. Revision suggestions
      Provide actionable suggestions for important issues rather than only pointing out that “something is wrong.”

    A risk classification mechanism can be used:

    1. High priority: Affects the overall conclusion, validity of the method, or key results
    2. Medium priority: Affects clarity of expression, structural completeness, or reproducibility
    3. Low priority: Formatting, wording, minor redundancy, or non-critical optimizations

    This allows model output to shift from “scattered error spotting” to “effective review.” For complex materials, the model can also be required to first output a review framework and then fill in each item, avoiding becoming trapped in details from the beginning.

    6. Engineering workflows for mathematical proofs and complex reasoning tasks

    Tasks such as mathematical proofs, logical deduction, and complex argumentation place high demands on models. Such tasks should not rely only on the model generating seemingly reasonable text. Instead, reliability should be improved through task decomposition, structural constraints, and result validation.

    An effective processing workflow is as follows:

    1. Clarify the proposition goal
      Ask the model to restate the task first and confirm inputs, outputs, conditions, and limitations.

    2. Split the proof structure
      Require the model to organize content according to “goal - key steps - auxiliary lemmas - final conclusion.”

    3. Mark uncertain items
      If a step depends on unprovided conditions, the model should explicitly state this instead of forcing a proof.

    4. Perform step-by-step validation
      Check each step locally, for example: whether conditions are used, whether conclusions exceed premises, and whether circular reasoning exists.

    5. Generate the final structured text
      After step-by-step checks pass, organize the content into a complete text.

    A representative workflow is as follows:

    1. Input: proposition goal, known conditions, output format
    2. Step 1: List the proof goal
    3. Step 2: List the required premises
    4. Step 3: Construct key intermediate conclusions
    5. Step 4: Check whether each step depends on the premises
    6. Step 5: Output the complete proof
    7. Step 6: Output a self-check list

    For these tasks, the value of the model is not only to “write something,” but to help users quickly locate proof ideas, check logical gaps, and organize material structure. Therefore, prompts should emphasize “checkability” rather than “length.”

    7. Quota, rate limiting, and cost control

    Complex tasks often consume more resources. Without budget control, the following problems may occur:

    • Too many calls for a single task;
    • Repeated input of long context, causing resource waste;
    • Retrying without a strategy after failure, further increasing consumption;
    • High-value tasks and low-value tasks being executed together;
    • Short-term and long-term quotas being exhausted quickly.

    Therefore, budget control mechanisms should be established in engineering practice.

    1. Task priority classification

    Tasks can be classified by value and complexity:

    • High priority: Key conclusions, formal reports, review results, proof outputs;
    • Medium priority: Auxiliary organization, draft generation, material classification;
    • Low priority: Simple Q&A, batch summarization, format conversion.

    High-priority tasks can be allowed a larger budget, while low-priority tasks should have strict limits.

    2. Call budget control

    Before starting a task, budget parameters can be set first:

    1. Maximum number of calls: 10
    2. Maximum input length: specified limit
    3. Maximum output length: specified limit
    4. Allowed retries: 1
    5. When the budget is exceeded: output interim results and request human intervention

    This prevents a single task from expanding indefinitely.

    3. Result caching and reuse

    For tasks with high repetition rates, a caching mechanism can be established. For example:

    • Summaries of the same material can be reused;
    • Rewritten results for the same question can be reused;
    • The same review rules can be reused;
    • Intermediate conclusions can be cached to avoid repeated computation.

    Caching does not mean fully trusting old results. It means allowing later tasks to continue optimization based on old results, reducing repeated processing.

    4. Logging and monitoring

    In cloud service or platform environments, general logging services, monitoring alerts, and call statistics can be used to observe model usage, including:

    • Call frequency;
    • Average latency;
    • Failure rate;
    • Input and output length distribution;
    • Number of retries;
    • Quota consumption trends.

    These data can help teams identify abnormal task patterns, such as a certain type of task frequently triggering retries or a certain type of prompt causing overly long output.

    8. Evaluating model version changes

    When a model version is upgraded or switched, conclusions should not be based on a single experience. A more reliable approach is to establish version regression evaluation.

    The basic workflow is as follows:

    1. Collect real task samples
      Select representative cases from historical tasks, covering simple, medium, and complex difficulty levels.

    2. Establish expected outputs
      Define acceptable answer ranges for key tasks rather than pursuing a single standard answer.

    3. Run comparative tests
      Run different versions under the same prompts, same constraints, and same budget.

    4. Collect automatic metrics
      Collect success rate, failure reasons, average output length, retry rate, and budget consumption.

    5. Perform manual sampling review
      Conduct manual checks for high-risk tasks, especially review, fact-checking, and proof tasks.

    6. Form version conclusions
      The final conclusion should state on which tasks the new version performs better, on which tasks it performs worse, and whether it is suitable for the current business scenario.

    This evaluation approach is more reliable than simply comparing whether answers “sound smart,” and it is also closer to real production environments.

    9. Core principles of prompt design

    In complex tasks, prompt quality often directly affects model performance. The following principles are recommended:

    1. Clarify the task goal

    Instead of only saying “help me look at this material,” specify the goal:

    1. Goal: Determine whether the proposal is suitable as a basis for project approval
    2. Output: feasibility conclusion, key risks, missing information, recommended next steps

    2. Clarify the output structure

    Complex tasks require structured output, for example:

    1. 1. Overall conclusion
    2. 2. Main evidence
    3. 3. Risk points
    4. 4. Uncertain items
    5. 5. Recommendations

    3. Clarify boundaries

    Tell the model what content does not need to be expanded and what content cannot be omitted:

    1. Analyze only the materials currently provided; do not proactively add external background.
    2. If the materials are insufficient, mark the item as insufficient evidence and do not guess.

    4. Clarify priorities

    Ask the model to output key conclusions first rather than expanding all information evenly:

    1. Please provide the final judgment first, then explain the reasons.
    2. If multiple risks exist, order them by severity.

    5. Clarify failure handling

    When the model cannot complete a task, ask it to explain the gap instead of continuing to generate irrelevant content:

    1. If necessary information is missing, list the key missing information.
    2. If the task cannot be completed, explain the reason and possible ways to supplement it.

    10. Conclusion

    Evaluating whether a large language model is suitable for complex tasks should not rely only on whether a single answer is fluent. It should examine the model’s overall performance in scenarios such as long conversations, fact-checking, review, complex reasoning, and cost control. A valuable evaluation should establish a clear metric system around task goals and improve stability through structured prompts, retrieval boundaries, budget control, and regression testing.

    For teams and individual users, a more reasonable usage approach includes:

    • Pursuing efficiency for simple tasks;
    • Emphasizing structure and evidence for complex tasks;
    • Emphasizing overall judgment for review tasks;
    • Emphasizing step-by-step validation for reasoning tasks;
    • Setting budget limits for high-consumption tasks;
    • Continuously performing regression evaluation when model versions change.

    Through these methods, one can judge model capability more objectively without blindly relying on a single experience, reduce ineffective consumption in real business scenarios, and improve the delivery quality of complex tasks.