A Practical Guide to LLM Selection: Evaluation Methods from Retrieval to Code
This guide explains how to evaluate large language models for retrieval, text rewriting, code assistance, and office agent integration. It shows how to split business needs into testable tasks and gives the evaluation dimensions, test methods, and pass criteria for each scenario. It does not recommend specific products.
Large language models differ widely in capability, so model selection should start by splitting business needs into tasks that can be tested. This guide does not recommend specific products. It describes evaluation methods only. Each scenario includes evaluation dimensions, test methods, and pass criteria that you can apply directly to your own projects.
1. Evaluating Information Retrieval
In retrieval scenarios, users usually need a clear answer to “what is it” or “how do I do it,” not a long analysis. For this reason, evaluate retrieval ability on the following five dimensions:
- Source quality: Whether the sources the model cites come from authoritative, accessible places.
- Citation traceability: Whether the links provided can be opened, and whether the content actually supports the conclusion.
- Timeliness: For information that changes over time, such as policies, prices, and version numbers, whether the answer reflects the latest status.
- Hallucination rate: The number of incorrect statements divided by the total number of statements.
- Cross-domain coverage: When a question involves several disciplines, whether the answer covers all of them.
Build a test set of 50 questions: 30 for common knowledge and factual queries, 10 for time-sensitive questions, and 10 for cross-domain questions. For each question, record whether the answer is correct, whether the citations are traceable, and whether any claims lack support.
Reasoning ability needs its own test. Prepare science and math calculation problems and multi-step reasoning problems. Check whether the final answer is correct and whether the intermediate steps contain logical leaps. Also measure how often the model makes simple questions complicated, meaning it gives long, redundant explanations for questions that could be answered in one sentence.
Common Question: How Do I Evaluate Whether a Retrieval Model Suits an Enterprise Knowledge Base?
Extract 100 real questions from internal documents, and measure the answer hit rate and the accuracy of citation localization. Also write a set of questions that the internal documents cannot answer, and check whether the model clearly declines to answer. If the model gives a definite answer when there is no supporting evidence, the model is not suitable for direct use in knowledge base Q&A.
2. Text Rewriting and Multi-Style Generation
Models differ noticeably in their language expression. You can therefore split content production into two pipelines. First, a model with strong retrieval ability organizes the source material. Then a model with strong rewriting ability converts the style. The two stages need an agreed data format. Without one, it is hard to tell which stage caused a problem.
| Stage | Input | Output fields or requirements | Checks |
|---|---|---|---|
| Retrieval and organization | Topic and keywords | Source list, key facts, conclusions, uncertainties | Sources are accessible, and facts have references |
| Rewriting and generation | Retrieval results, target genre | Title, summary, and body within the required word count | Facts are consistent, style matches the genre, and there are no repeated paragraphs |
| Manual review | Rewritten draft | Revision comments | Numbers, names, citations, and medical or legal statements |
The input template for the rewriting model should include the target genre, the audience, the word count range, banned words, and a list of facts. The fact list helps prevent shifted numbers and substituted names during rewriting.
You can apply the same approach to multi-modal generation. A reasoning model proposes creative plans, and a multi-modal model turns those plans into images or videos.人工 review is needed between the two outputs to confirm that the visuals match the text descriptions.
3. Evaluating Code Assistance
Code assistance covers three types of tasks, and each has different evaluation criteria:
- Code completion: Continuing a function or statement based on the surrounding context.
- Code refactoring: Changing the structure of code without changing its behavior.
- Defect repair: Locating and fixing a bug based on an error message or failing test results.
Prepare at least 20 cases for each task type. For every test, use the same prompt and the same sampling parameters, and record the model version and the test date.
The main metrics are:
- Unit test pass rate: The share of generated code that passes the defined tests.
- Revision rounds: The number of interactions needed to go from the first generation to passing the tests.
- Regression failure rate: The share of cases where existing functionality breaks after a change.
- Manual review items: Naming conventions, exception handling, logging, and readability.
Academic Q&A scenarios call for a different focus. Give the model an error message and measure how often it correctly identifies the root cause. For difficult logic problems, verify the answers step by step by hand. UI generation and color matching usually need manual correction. Record these issues separately in the evaluation report.
Code Evaluation Case
The following is a simple HTTP request function that serves as the test baseline:
import requestsdef fetch_data(url):response = requests.get(url)return response.json()
The test instruction asks the model to add these features to the function:
- Set the timeout to 5 seconds.
- Add exponential backoff retries, with up to 3 retries.
- Catch network exceptions and write them to a log.
“Up to 3 retries” can be read as either 3 or 4 total requests. Before acceptance testing, state the requirement clearly in the prompt: one initial request plus up to 3 retries, for a maximum of 4 requests in total, with retry intervals of 1, 2, and 4 seconds.
The acceptance scoring table is below:
| Check | Pass criteria |
|---|---|
| Timeout setting | The requests call includes timeout=5 |
| Retry count | After the first request, no more than 3 retries, and no more than 4 total requests |
| Backoff strategy | Retry intervals grow exponentially, not fixed |
| Exception types | Catches requests.RequestException and does not use a bare except |
| HTTP status | Calls raise_for_status or handles error status codes equivalently |
| JSON parsing | Catches parsing exceptions so the program does not crash |
| Logging | Uses the logging module, and logs do not contain full tokens or sensitive query parameters |
| Runtime environment | Records the actual Python and requests versions |
If all checks pass, the result is a pass. If the “Timeout setting,” “Exception types,” or “Logging” check fails, the result is a fail.
Before going live, also address these security points. Validate external URLs, and allow only https and allowlisted domains. Set upper limits for timeouts and retry counts. Mask sensitive parameters in logs. Enable retries only for idempotent requests, so write operations are not submitted twice.
4. Office Collaboration and Agent Integration
Personal assistant agents can connect to enterprise collaboration platforms and handle tasks such as scheduling, approvals, and document organization. Whether a team should adopt one depends on how the team actually uses it, not on the feature list.
Before paying for a subscription, run a verification period of 1 to 3 months on a monthly plan. Check each item on the following list:
- Target users: Identify which roles will use the tool and how many people they include.
- Usage frequency: Set a threshold, for example, at least 5 uses per person per week.
- Data security: Confirm whether user data is used for model training, whether data can stay in a specific region or be deployed privately, and how long data is retained.
- Integration cost: Estimate the working hours and technical staff time needed to connect to the platform.
- ROI calculation: Multiply the working hours saved by the hourly cost of staff, then subtract the subscription fee and integration cost. The result is the monthly net benefit.
- Exit conditions: For example, if ROI stays negative for two consecutive months, or if key features cannot be used reliably, stop renewing.
Consider an annual subscription only if the trial period meets the requirements for usage frequency, ROI, and data security. This approach keeps sunk costs within an acceptable range.
Conclusion
No single large language model performs best in every scenario. Good selection starts by turning business needs into testable tasks. Compare models with the same metrics, then combine their strengths based on the results.
