Selecting a Domestic Large Language Model: A Practical Evaluation Framework from Requirements to Deployment

    This article provides a systematic approach for choosing a large language model (LLM) for production use, moving beyond public benchmark rankings. It outlines a step‑by‑step methodology that starts with clarifying business scenarios and constraints, then defines key evaluation dimensions including task capability, context length, cost, latency, stability, compliance, and integration effort. The article also explains how to build a custom evaluation set from real business data and includes a minimal Python benchmarking script for comparing candidate models. The goal is to help engineering teams make data‑driven decisions that align with their specific operational and quality requirements.

    Why Selection Requires More Caution Than Ranking

    Public leaderboards reflect model performance on standardized test sets. Real business data often differs noticeably from those benchmarks. Leaderboard tasks are mostly well‑formed, while production inputs frequently contain industry jargon, noisy text, mixed formats, and implicit context. A model that ranks high on a leaderboard may not be the best for customer‑service ticket classification, contract summarization, or code assistance.

    Therefore, leaderboards are useful for creating a shortlist, but final decisions should be based on your own data. First use benchmarks to narrow candidates, then validate with business samples—this order is more reliable.

    Clarify Scenarios and Constraints First

    Before evaluating models, answer three questions.

    1. What is the task type? Common types include text generation, information extraction, classification, question answering, code assistance, multi‑turn dialogue, and tool calling. Different tasks place very different demands on the model.
    2. What level of quality is required? An internal assistant that can tolerate occasional errors has different accuracy and consistency requirements than a user‑facing system that must reliably output structured data.
    3. What are the hard constraints? Consider whether data can leave your environment, whether private deployment is required, the maximum acceptable cost per request, the allowed response time, and the team’s existing technology stack.

    Write these constraints into a table and score each candidate model against the same criteria. This is far more reliable than relying on impressions.

    Core Evaluation Dimensions

    1. Task Capability

    Test the model’s accuracy, format adherence, and reasonableness of refusals using business samples. Focus not only on whether the answer is correct, but also on whether it can produce fixed‑structure data as required—for example, complete JSON fields and valid enum values—and whether it remains consistent when instructions are long or contain many constraints.

    2. Context Length

    The context window determines how much input a model can process at once. Long‑document summarization, multi‑file code analysis, and long conversations are especially sensitive to this parameter. Note that a large window does not guarantee that information in the middle of the text is effectively utilized. Design tests that place key facts at different positions in a long text and measure the recall rate.

    3. Inference Cost

    Most platforms charge per input and output token, with output often priced higher. Estimate costs using real average input and output lengths, not assumed values. For example, a system handling 100,000 requests per day with an average input of 2,000 characters will have costs dominated by the input side; in that case, optimizing prompt compression or reusing fixed prefixes may yield more savings than switching models.

    A rough monthly cost estimate:

    Monthly cost ≈ daily requests × 30 × (average input tokens × input price + average output tokens × output price)

    Use the actual prices published by each platform. Base calculations on real quotes and real traffic.

    4. Latency and Throughput

    Latency has two parts: time to first token and total completion time. Interactive products are sensitive to time‑to‑first‑token; the user’s experience of waiting for the first character matters immediately. Batch processing cares more about throughput—how many requests can be completed per unit of time. Queue behavior during concurrent peaks must be measured separately; a fast single request does not guarantee stability at peak load.

    5. Stability and Availability

    Check the platform’s rate‑limiting rules, error code meanings, and retry policies. Success rates for the same type of request across different time periods often affect the online experience more than average latency. Run continuous tests over a period and record timeout rates and failure types to produce comparable stability data.

    6. Compliance and Data Security

    Clarify the following: where data is stored, whether it is used for model training, how long logs are retained, and whether it meets regulatory requirements for your industry. Scenarios involving personal information or trade secrets should be reviewed by legal and security teams during the project definition phase—not just before launch.

    7. Ecosystem and Integration Effort

    Consider whether the API is compatible with your existing calling code, whether it supports function calling, streaming output, and structured output, and whether documentation and developer tools are mature. Integration cost is often underestimated. A model that requires significant rework to integrate may have lower overall value even if its quality is slightly better.

    Build Your Own Evaluation Set

    An evaluation set is the most important asset for model selection. Follow these steps.

    1. Sample from real business logs, covering high‑frequency scenarios, long‑tail scenarios, and error‑prone cases.
    2. Define a standard answer or scoring rule for each sample. Use rule‑based checks wherever possible—for example, whether a field exists, whether a numeric value is correct, or whether the format is valid. For content that is difficult to automate, use human scoring with at least two independent reviewers and reconcile disagreements.
    3. Aim for at least one to two hundred samples, balanced across categories, to avoid one task type dominating and hiding problems.
    4. Mask sensitive information before testing, especially when using external services.

    The evaluation set needs regular updates. Business changes and shifts in user phrasing will gradually make older samples less representative.

    Example Batch Comparison Script

    The script below demonstrates a minimal evaluation approach. It calls multiple models with the same set of samples, records average latency, and calculates a pass rate based on rule‑based checks. The API base URL and key are read from environment variables, not hard‑coded.

    1. import os
    2. import time
    3. import requests
    4. BASE_URL = os.environ["MODEL_BASE_URL"]
    5. API_KEY = os.environ["MODEL_API_KEY"]
    6. HEADERS = {
    7. "Authorization": f"Bearer {API_KEY}",
    8. "Content-Type": "application/json",
    9. }
    10. def call_model(model_name: str, prompt: str):
    11. payload = {
    12. "model": model_name,
    13. "messages": [{"role": "user", "content": prompt}],
    14. "temperature": 0,
    15. }
    16. start = time.time()
    17. resp = requests.post(
    18. f"{BASE_URL}/chat/completions",
    19. headers=HEADERS,
    20. json=payload,
    21. timeout=60,
    22. )
    23. resp.raise_for_status()
    24. latency = time.time() - start
    25. text = resp.json()["choices"][0]["message"]["content"]
    26. return text, latency
    27. def run_benchmark(models, cases):
    28. results = []
    29. for model in models:
    30. passed, total_latency, errors = 0, 0.0, 0
    31. for case in cases:
    32. try:
    33. text, latency = call_model(model, case["prompt"])
    34. total_latency += latency
    35. if case["check"](text):
    36. passed += 1
    37. except Exception:
    38. errors += 1
    39. n = len(cases)
    40. success = max(n - errors, 1)
    41. results.append({
    42. "model": model,
    43. "pass_rate": round(passed / n, 3),
    44. "avg_latency": round(total_latency / success, 2),
    45. "error_count": errors,
    46. })
    47. return results
    48. cases = [
    49. {
    50. "prompt": "Please extract the order number from the text below and output only the order number itself: The order number is A20391, please process it ASAP.",
    51. "check": lambda text: text.strip() == "A20391",
    52. },
    53. ]
    54. if __name__ == "__main__":
    55. candidate_models = ["Candidate Model A", "Candidate Model B"]
    56. for row in run_benchmark(candidate_models, cases):
    57. print(row)

    Note the following when using this script. First, the scoring function should be strict—do not treat an answer that contains the correct result but also extra text as a pass, unless the business explicitly allows such output. Second, besides aggregated results, save the input and output of each failed sample for manual review. Third, replace the model names with the actual identifiers provided by your chosen platform. Fourth, keep the temperature parameter fixed during evaluation to reduce randomness in the results.