A Practical Guide to LLM Selection: Evaluation Methods from Retrieval to Code

    This guide explains how to evaluate large language models for retrieval, text rewriting, code assistance, and office agent integration. It shows how to split business needs into testable tasks and gives the evaluation dimensions, test methods, and pass criteria for each scenario. It does not recommend specific products.

    Large language models differ widely in capability, so model selection should start by splitting business needs into tasks that can be tested. This guide does not recommend specific products. It describes evaluation methods only. Each scenario includes evaluation dimensions, test methods, and pass criteria that you can apply directly to your own projects.

    1. Evaluating Information Retrieval

    In retrieval scenarios, users usually need a clear answer to “what is it” or “how do I do it,” not a long analysis. For this reason, evaluate retrieval ability on the following five dimensions:

    1. Source quality: Whether the sources the model cites come from authoritative, accessible places.
    2. Citation traceability: Whether the links provided can be opened, and whether the content actually supports the conclusion.
    3. Timeliness: For information that changes over time, such as policies, prices, and version numbers, whether the answer reflects the latest status.
    4. Hallucination rate: The number of incorrect statements divided by the total number of statements.
    5. Cross-domain coverage: When a question involves several disciplines, whether the answer covers all of them.

    Build a test set of 50 questions: 30 for common knowledge and factual queries, 10 for time-sensitive questions, and 10 for cross-domain questions. For each question, record whether the answer is correct, whether the citations are traceable, and whether any claims lack support.

    Reasoning ability needs its own test. Prepare science and math calculation problems and multi-step reasoning problems. Check whether the final answer is correct and whether the intermediate steps contain logical leaps. Also measure how often the model makes simple questions complicated, meaning it gives long, redundant explanations for questions that could be answered in one sentence.

    Common Question: How Do I Evaluate Whether a Retrieval Model Suits an Enterprise Knowledge Base?

    Extract 100 real questions from internal documents, and measure the answer hit rate and the accuracy of citation localization. Also write a set of questions that the internal documents cannot answer, and check whether the model clearly declines to answer. If the model gives a definite answer when there is no supporting evidence, the model is not suitable for direct use in knowledge base Q&A.

    2. Text Rewriting and Multi-Style Generation

    Models differ noticeably in their language expression. You can therefore split content production into two pipelines. First, a model with strong retrieval ability organizes the source material. Then a model with strong rewriting ability converts the style. The two stages need an agreed data format. Without one, it is hard to tell which stage caused a problem.

    Stage Input Output fields or requirements Checks
    Retrieval and organization Topic and keywords Source list, key facts, conclusions, uncertainties Sources are accessible, and facts have references
    Rewriting and generation Retrieval results, target genre Title, summary, and body within the required word count Facts are consistent, style matches the genre, and there are no repeated paragraphs
    Manual review Rewritten draft Revision comments Numbers, names, citations, and medical or legal statements

    The input template for the rewriting model should include the target genre, the audience, the word count range, banned words, and a list of facts. The fact list helps prevent shifted numbers and substituted names during rewriting.

    You can apply the same approach to multi-modal generation. A reasoning model proposes creative plans, and a multi-modal model turns those plans into images or videos.人工 review is needed between the two outputs to confirm that the visuals match the text descriptions.

    3. Evaluating Code Assistance

    Code assistance covers three types of tasks, and each has different evaluation criteria:

    1. Code completion: Continuing a function or statement based on the surrounding context.
    2. Code refactoring: Changing the structure of code without changing its behavior.
    3. Defect repair: Locating and fixing a bug based on an error message or failing test results.

    Prepare at least 20 cases for each task type. For every test, use the same prompt and the same sampling parameters, and record the model version and the test date.

    The main metrics are:

    • Unit test pass rate: The share of generated code that passes the defined tests.
    • Revision rounds: The number of interactions needed to go from the first generation to passing the tests.
    • Regression failure rate: The share of cases where existing functionality breaks after a change.
    • Manual review items: Naming conventions, exception handling, logging, and readability.

    Academic Q&A scenarios call for a different focus. Give the model an error message and measure how often it correctly identifies the root cause. For difficult logic problems, verify the answers step by step by hand. UI generation and color matching usually need manual correction. Record these issues separately in the evaluation report.

    Code Evaluation Case

    The following is a simple HTTP request function that serves as the test baseline:

    1. import requests
    2. def fetch_data(url):
    3. response = requests.get(url)
    4. return response.json()

    The test instruction asks the model to add these features to the function:

    1. Set the timeout to 5 seconds.
    2. Add exponential backoff retries, with up to 3 retries.
    3. Catch network exceptions and write them to a log.

    “Up to 3 retries” can be read as either 3 or 4 total requests. Before acceptance testing, state the requirement clearly in the prompt: one initial request plus up to 3 retries, for a maximum of 4 requests in total, with retry intervals of 1, 2, and 4 seconds.

    The acceptance scoring table is below:

    Check Pass criteria
    Timeout setting The requests call includes timeout=5
    Retry count After the first request, no more than 3 retries, and no more than 4 total requests
    Backoff strategy Retry intervals grow exponentially, not fixed
    Exception types Catches requests.RequestException and does not use a bare except
    HTTP status Calls raise_for_status or handles error status codes equivalently
    JSON parsing Catches parsing exceptions so the program does not crash
    Logging Uses the logging module, and logs do not contain full tokens or sensitive query parameters
    Runtime environment Records the actual Python and requests versions

    If all checks pass, the result is a pass. If the “Timeout setting,” “Exception types,” or “Logging” check fails, the result is a fail.

    Before going live, also address these security points. Validate external URLs, and allow only https and allowlisted domains. Set upper limits for timeouts and retry counts. Mask sensitive parameters in logs. Enable retries only for idempotent requests, so write operations are not submitted twice.

    4. Office Collaboration and Agent Integration

    Personal assistant agents can connect to enterprise collaboration platforms and handle tasks such as scheduling, approvals, and document organization. Whether a team should adopt one depends on how the team actually uses it, not on the feature list.

    Before paying for a subscription, run a verification period of 1 to 3 months on a monthly plan. Check each item on the following list:

    1. Target users: Identify which roles will use the tool and how many people they include.
    2. Usage frequency: Set a threshold, for example, at least 5 uses per person per week.
    3. Data security: Confirm whether user data is used for model training, whether data can stay in a specific region or be deployed privately, and how long data is retained.
    4. Integration cost: Estimate the working hours and technical staff time needed to connect to the platform.
    5. ROI calculation: Multiply the working hours saved by the hourly cost of staff, then subtract the subscription fee and integration cost. The result is the monthly net benefit.
    6. Exit conditions: For example, if ROI stays negative for two consecutive months, or if key features cannot be used reliably, stop renewing.

    Consider an annual subscription only if the trial period meets the requirements for usage frequency, ROI, and data security. This approach keeps sunk costs within an acceptable range.

    Conclusion

    No single large language model performs best in every scenario. Good selection starts by turning business needs into testable tasks. Compare models with the same metrics, then combine their strengths based on the results.