Evaluating and Using Advanced General LLMs in Engineering Practices

    Evaluating advanced general large language models (LLMs) requires looking beyond benchmark scores to assess their ability to handle complex tasks, maintain stability, and manage resources effectively. This article explores practical methodologies for assessing LLM capabilities across math reasoning and code engineering. It also covers strategies for balancing speed and depth, implementing multi-model collaboration, managing long-running automated tasks, and building an objective evaluation system to integrate LLMs successfully into production workflows.

    In practical applications, evaluating an advanced general Large Language Model (LLM) should not rely solely on a single correct answer or marketing benchmark scores. For developers, researchers, and technical managers, the more critical questions are: Can the model consistently handle complex tasks? Can it reduce manual rework? Can it maintain stability over long running periods? And does its speed and cost align with the team’s workflow?

    From this perspective, evaluating an advanced model should cover multiple dimensions, including mathematical reasoning, code engineering, long-text processing, tool calling, task planning, and resource management.

    I. Model Capabilities Cannot Be Summarized by “Smartness”

    Users often describe models as “capable,” “accurate,” or “deep in reasoning,” but these evaluations are too broad. Excellent performance in simple Q&A does not mean a model can solve open-ended math problems, execute complex system refactoring, or handle long-term code maintenance.

    A more reasonable evaluation approach breaks down capabilities into several interrelated dimensions.

    1. Mathematical Reasoning Capability

    Mathematical tasks can generally be divided into three categories:

    • Explanation and application of known theorems;
    • Problems with clear conclusions but complex proofs;
    • Open-ended problems lacking ready-made standard answers.

    The first category mainly tests the model’s grasp of definitions, theorems, and derivation processes. The second requires the model to organize rigorous proofs and verify that each key step holds. The third is closer to research work, requiring not only reasoning but also proposing conjectures, finding counterexamples, revising proof paths, and distinguishing between what “looks plausible” and what is “actually valid.”

    When using a model to generate mathematical lecture notes, you should not just ask it to “provide a proof.” Instead, explicitly require it to:

    1. List the definitions and prerequisites used first;
    2. Annotate the conditions each derivation step relies on;
    3. Distinguish between known facts, model conjectures, and conclusions to be verified;
    4. Discuss boundary cases;
    5. Provide an independent proof check at the end.

    Especially when a proposition lacks an existing proof, the model might generate content that is formally complete but logically flawed. Therefore, mathematical outputs must undergo manual review and, if necessary, be cross-checked using formal verification tools or independent models.

    2. Code Engineering Capability

    Code capability should not be judged solely on whether it can “write code.” What truly impacts development efficiency is whether the model can understand an existing project and implement modifications without breaking current functionality.

    Engineering tasks usually involve the following steps:

    • Reading the project structure;
    • Understanding dependencies between modules;
    • Locating abnormal behavior;
    • Designing a modification plan;
    • Writing or adjusting code;
    • Adding tests;
    • Analyzing potential regression issues;
    • Continuing to fix based on execution results.

    The last three steps are often more important than the initial coding. Many models can quickly generate a seemingly reasonable implementation but fail to identify incomplete exception handling, unsafe concurrency conditions, or broken interface compatibility.

    Therefore, model evaluation should include complete tasks rather than isolated coding problems. For example, you could provide a medium-sized codebase, ask the model to fix a bug and add a small feature, and observe whether it can:

    • Accurately restate the problem before making changes;
    • Clearly identify the scope of impact;
    • Explain why a specific implementation was chosen;
    • Generate targeted tests;
    • Locate the root cause after a test failure;
    • Avoid unnecessary, large-scale refactoring.

    This type of evaluation is much closer to a real production environment than simply comparing code length.

    II. The Trade-off Between Speed, Depth, and Stability

    Advanced models typically offer different inference tiers or running modes. Faster modes are suitable for simple Q&A, code formatting, log summarization, and routine documentation. Deeper modes are better for complex debugging, system design, and mathematical derivation.

    Speed and quality do not have a simple linear relationship. If a task requires multiple rounds of thinking, repeated checking, and tool calling, the model’s total time consumption may increase significantly. In such cases, single-response speed does not represent actual efficiency.

    Tasks can be divided into three tiers:

    1. Fast Tasks

    Include:

    • Code comment generation;
    • Simple interface encapsulation;
    • Routine format conversion;
    • Log summarization;
    • Direct fixes for known errors.

    These tasks should prioritize faster response modes to reduce wait time and resource consumption.

    2. Medium-Complexity Tasks

    Include:

    • Module-level refactoring;
    • Multi-file coordinated modifications;
    • Complex exception troubleshooting;
    • Test case completion;
    • Technical documentation writing.

    These tasks are suitable for modes with a certain depth of reasoning, and the model should be required to output a plan before execution.

    3. High-Complexity Tasks

    Include:

    • Open-ended mathematical problems;
    • Large-scale system architecture design;
    • Long-term code maintenance;
    • Cross-module troubleshooting of difficult faults;
    • Research tasks requiring repeated experimentation and validation.

    These tasks should not simply pursue speed. A more reasonable approach is to allow the model a longer analysis time while setting periodic checkpoints to prevent it from continuously consuming resources in the wrong direction.

    III. Multi-Model Collaboration is More Reliable Than Single-Model Monopoly

    In actual workflows, different models or running modes can be assigned specific roles rather than having one model handle all tasks.

    A practical collaboration method is:

    1. A high-capability model breaks down the problem and formulates a plan;
    2. A fast model handles data organization, code scanning, and initial implementation;
    3. Another independent model reviews the results;
    4. Humans confirm key conclusions and high-risk modifications.

    For example, in code development, you can first let the main model analyze the cause of a fault, then have a lightweight model scan relevant files and perform mechanical modifications, and finally have a review model check interface compatibility, exception handling, and test coverage.

    The advantage of this method is reducing the risk of single-point errors. The most common problem with models is not that they completely fail to do a task, but that they develop a self-consistent yet incorrect reasoning based on a flawed assumption. Introducing an independent review step significantly increases the probability of catching such issues.

    A similar process can be adopted for math tasks:

    • Model A proposes a proof;
    • Model B checks the proof line by line;
    • Model C attempts to find counterexamples;
    • Humans confirm key lemmas.

    Note that multi-model collaboration does not mean “the more models, the better.” If all models reason based on the same incorrect premise, repeated outputs only increase a false sense of certainty. Therefore, the review model should use different prompts and be explicitly required to actively look for flaws, rather than simply evaluating whether the answer looks elegant.

    IV. Resource Management for Long-Running Tasks

    Some agentic coding environments allow models to run continuously for hours, automatically reading files, executing tests, modifying code, and re-verifying. This capability is highly beneficial for large tasks but introduces issues regarding quotas, time, and concurrency.

    Common issues users encounter include:

    • A single task running for too long;
    • One task consuming a large amount of cycle quota;
    • The model continuously consuming resources in repeated attempts;
    • The task running continuously but making little actual progress;
    • Subsequent work being interrupted after hitting short-term rate limits.

    The key to solving these problems is not making the model “try harder,” but improving task controllability.

    1. Break Large Tasks into Phases

    Instead of directly asking the model to “complete the entire system refactoring,” break it down into:

    • Phase 1: Read the code and output an impact analysis;
    • Phase 2: Propose a refactoring plan without modifying code;
    • Phase 3: Implement minimal changes;
    • Phase 4: Run tests;
    • Phase 5: Handle failing test cases;
    • Phase 6: Generate a change summary.

    Each phase should have clear completion criteria. This facilitates human intervention and prevents the model from repeatedly exploring when the goal is unclear.

    2. Set Stop Conditions

    The prompt can explicitly specify:

    • Pause and summarize if two consecutive attempts yield no improvement;
    • Ask questions first if the requirements are ambiguous;
    • Do not modify files unrelated to the current issue;
    • Analyze the cause first when a test fails, do not blindly retry;
    • Output current progress and remaining risks upon reaching the time budget.

    Stop conditions can effectively reduce invalid runs, which is particularly suitable for automated programming scenarios.

    3. Record Intermediate States

    Long tasks must retain change records, including:

    • Completed items;
    • Current assumptions;
    • Modified files;
    • Executed tests;
    • Unresolved issues;
    • Potential risks.

    If a task is interrupted due to rate limits, the next session can quickly resume based on the records without needing to re-scan the entire context.

    V. How to Build a More Objective Evaluation System

    Personal experience is valuable but easily influenced by task type, prompt quality, and usage habits. To reach more stable conclusions, you can build your own evaluation set.

    The evaluation set does not need to be large, but it should cover real work:

    • A set of mathematical proof tasks;
    • A set of code bug fixing tasks;
    • A set of long document generation tasks;
    • A set of multi-file refactoring tasks;
    • A set of comprehensive tasks requiring tool calling.

    Each task can be scored on the following metrics:

    1. Correctness: Is the final conclusion or code correct?
    2. Completeness: Does it cover all requirements?
    3. Maintainability: Is the code and documentation easy to modify subsequently?
    4. Stability: Are the results consistent across repeated executions?
    5. Efficiency: Time and number of interactions required to complete the task;
    6. Reviewability: Does the model clearly explain its basis and risks?
    7. Resource Consumption: Is the task likely to trigger time or quota limits?

    If a model is stronger on complex tasks but slower, it can be used for critical steps; if another model is fast but shallower in reasoning, it can handle scanning, organizing, and initial implementation. The final selection should not aim for “first place in all scenarios,” but rather for alignment with your workflow.

    VI. Conclusion: Treat Models as Engineering Systems, Not Chatbots

    The value of an advanced general LLM lies not only in the quality of a single response, but in its ability to integrate into a complete R&D process. For mathematical research, focus on proof rigor, counterexample awareness, and independent review capabilities; for software engineering, focus on code understanding, minimal modifications, test validation, and regression control; for long-term automated tasks, it is essential to focus on time budgets, quota management, and phased outputs.

    Overall, mature usage practices typically include the following principles:

    • Prioritize speed for simple tasks;
    • Prioritize depth for complex tasks;
    • Key conclusions must be independently verified;
    • Large tasks must be split with stop conditions set;
    • Different models or running modes should handle different responsibilities;
    • Use real work samples to build long-term evaluation records;
    • Do not treat a model’s fluent expression as proof of correctness.

    Only by combining model capabilities, task types, verification mechanisms, and resource limitations can you truly determine whether an advanced model fits your work. The model itself is merely a productivity tool; the final outcome depends on the engineering system composed of task design, process management, and human review.