How to Evaluate Chinese LLMs: A Practical Selection Guide for Developers and Everyday Users

    Public leaderboards alone are a poor basis for choosing a large language model. This guide breaks LLM evaluation into two perspectives—developer-facing engineering capabilities such as API stability, function calling, and latency, and everyday-user capabilities such as long-context recall, multi-turn instruction following, and safety policy behavior. It provides a weighted scoring framework, a runnable function-calling code example, needle-in-a-haystack testing methods, a pre-integration checklist, common mistakes, and answers to frequently asked questions. Use measurable metrics and your own business data rather than leaderboard rankings to make a defensible long-term decision.

    Why Leaderboards Alone Are Not Enough

    Public evaluation benchmarks reflect how a model performs on specific test sets, but those test sets often differ greatly from real business scenarios. A model that scores highly on math reasoning may still fail to consistently return valid JSON, or to locate key clauses in a 20,000-word contract. In this article, “top tier” does not refer to any specific ranking; it means models that perform consistently well on the measurable indicators described below.

    This guide splits evaluation into two sets of dimensions for two types of readers. The first is the developer perspective on engineering capabilities: API stability, function-calling accuracy, and latency. The second is the everyday-user perspective on interaction quality: long-context information recall, instruction following across multi-turn conversations, and the reasonableness of safety policies.

    1. An Evaluation Framework with Suggested Weights

    The table below offers a scoring framework you can apply directly. The weights are a suggested starting point—adjust them based on your business priorities.

    Dimension Core Metrics Suggested Weight How to Measure
    API stability Request success rate, timeout rate, rate-limit frequency 15% Continuous load testing for 24+ hours
    First-token latency TTFT (time to first token) P50 and P95 15% Fixed input length, sampled across time slots
    Generation speed Output tokens per second 10% Fixed output length, record the average
    Function calling Tool selection accuracy, parameter validity rate, false-call rate 20% Build 200+ labeled test samples
    Long-context recall Recall rate for information at different positions 15% See the test method in Section 3
    Instruction following Pass rate on multiple constraints 10% Build a test set with checkable constraints
    Safety policy Harmful-request interception rate, false-rejection rate on normal requests 15% Track both error types simultaneously

    The safety policy dimension is easy to overlook. Measuring only the interception rate encourages models to over-reject, so the false-rejection rate must always be evaluated alongside it.

    2. Developer Perspective: API Integration and Function Calling

    2.1 Always Document Test Conditions When Measuring Latency

    Latency data is meaningless without test conditions. When recording TTFT and generation speed, state at least the input token count, output limit, concurrency level, request region, and time of day. The same model can perform very differently during peak evening hours versus early morning. For each model, sample in three different time slots, with no fewer than 100 requests per slot, and report P50 and P95 rather than averages. Averages hide tail latency—and tail latency is exactly what users complain about.

    2.2 A Function-Calling Example

    The following example uses the OpenAI-compatible chat/completions format. Field names may vary slightly across providers, so check the official documentation before using it in production. The example does three things: reads credentials from environment variables, sets up timeouts with exponential-backoff retries, and reads tool_calls from choices[0].message.

    1. import os
    2. import json
    3. import time
    4. import requests
    5. API_URL = os.environ["LLM_API_URL"]
    6. API_KEY = os.environ["LLM_API_KEY"]
    7. MODEL = os.environ.get("LLM_MODEL", "your-model-name")
    8. HEADERS = {
    9. "Content-Type": "application/json",
    10. "Authorization": f"Bearer {API_KEY}",
    11. }
    12. TOOLS = [
    13. {
    14. "type": "function",
    15. "function": {
    16. "name": "get_weather",
    17. "description": "Get weather information for a specified city",
    18. "parameters": {
    19. "type": "object",
    20. "properties": {
    21. "city": {"type": "string", "description": "City name, e.g., Beijing"}
    22. },
    23. "required": ["city"],
    24. },
    25. },
    26. }
    27. ]
    28. def call_model(messages, retries=3, timeout=30):
    29. payload = {
    30. "model": MODEL,
    31. "messages": messages,
    32. "tools": TOOLS,
    33. "tool_choice": "auto",
    34. }
    35. for attempt in range(retries):
    36. try:
    37. resp = requests.post(API_URL, headers=HEADERS, json=payload, timeout=timeout)
    38. if resp.status_code == 429 or resp.status_code >= 500:
    39. raise RuntimeError(f"Retryable error: HTTP {resp.status_code}")
    40. resp.raise_for_status()
    41. return resp.json()
    42. except (requests.Timeout, requests.ConnectionError, RuntimeError):
    43. if attempt == retries - 1:
    44. raise
    45. time.sleep(2 ** attempt)
    46. def main():
    47. data = call_model([{"role": "user", "content": "What's the weather in Beijing today?"}])
    48. message = data["choices"][0]["message"]
    49. tool_calls = message.get("tool_calls") or []
    50. if not tool_calls:
    51. print("Direct model reply:", message.get("content"))
    52. return
    53. call = tool_calls[0]
    54. name = call["function"]["name"]
    55. try:
    56. args = json.loads(call["function"]["arguments"])
    57. except json.JSONDecodeError:
    58. print("Arguments are not valid JSON; retry or refuse to execute")
    59. return
    60. print(f"Model requested function call: {name}")
    61. print(f"Extracted arguments: {args}")
    62. if __name__ == "__main__":
    63. main()

    This code only demonstrates the first round of the call. A real tool-calling loop also needs to execute the function, append the result to the conversation as a tool-role message, and request the model again to generate the final answer. Also, the arguments the model returns are strings and should never be trusted directly—validate them against a JSON Schema before they enter your business system.

    2.3 How to Measure Function-Calling Quality

    Function-calling quality needs data. Build a test set of at least 200 samples, each labeled with the expected function and expected arguments. Track three numbers: whether the correct tool was selected, whether the parameters passed Schema validation and matched the labels, and whether the model made unnecessary calls on questions that required no tool at all. The false-call rate matters especially in customer service and order-management scenarios, where a single erroneous call can trigger a real refund or modification.

    3. Everyday Experience: Long Context and Multi-Turn Conversations

    3.1 How to Test Long-Context Recall

    Advertised context windows vary widely across providers, but nominal length is not the same as effective length. The most practical approach is a needle-in-a-haystack test: insert a fact unrelated to the document—say, a fictional meeting ID—at the 10%, 50%, and 90% positions of a long document, then ask about it and record whether the model answers correctly. Test at document lengths of roughly 8K, 32K, and 128K tokens, repeating each configuration 20 times.

    Report results by both length and position. Many models perform well at the beginning and end of a document but show a noticeable drop in recall in the middle. If your business requires extracting clauses from long contracts, run the same test on real contracts rather than relying only on synthetic data.

    3.2 Multi-Turn Conversations and Complex Instructions

    In everyday conversation, users often drop the subject of a sentence or refer back to a name mentioned five turns earlier. Design dialogues with such references and check whether the model resolves them correctly. For instruction-following tests, use constraints with checkable items—for example, “write something humorous, no more than three sentences, with no numbers at all.” Every constraint must be verifiable programmatically or by a human reviewer item by item; otherwise, results are hard to reproduce.

    3.3 Evaluate Safety Policies in Both Directions

    The goal of a safety policy is to block harmful requests without rejecting legitimate ones. Your test set should include both categories: harmful requests that should be refused, and superficially similar but entirely legitimate requests—for instance, questions about historical events or security knowledge. If you measure only the interception rate, a model can score highly through blanket refusal, and real users will keep hitting unreasonable rejections.

    4. A Pre-Integration Checklist

    Before going live, work through this list:

    1. Have you run at least one round of testing with your own business data, rather than relying solely on public leaderboards?
    2. Have you recorded P50 and P95 for TTFT and generation speed, with test conditions documented?
    3. Does your function-calling test set include questions that should not trigger any tool call?
    4. Did your long-context tests use real documents, covering the beginning, middle, and end?
    5. Is the pricing structure clear, with input, output, and cache-hit rates accounted for separately?
    6. Do rate limits and quota ceilings cover peak traffic, and is there a channel to request higher limits?
    7. Have you verified data retention policies, training usage, and compliance requirements—especially for scenarios involving personal information?
    8. Do you have a backup model and a fallback plan, with failover drills scheduled regularly?

    5. Common Mistakes

    The first common mistake is treating leaderboard scores as the selection criterion. Leaderboard questions differ greatly from business needs—use them only as a preliminary screen.

    The second is ignoring output-token costs. In many scenarios, output length far exceeds expectations, particularly for code generation and long reports, and output tokens are usually priced higher.

    The third is skipping timeouts and retries. Occasional slow responses from the model service will, without timeouts, stall the entire call chain.

    The fourth is assuming the advertised context limit is the effective limit. Use the method in Section 3 to measure the length that actually works reliably, and leave headroom in your application.

    The fifth is executing model-returned arguments directly. No matter how well the model performs, always apply Schema validation and business-rule validation; high-risk operations like deletions or transfers also require human confirmation.

    6. Frequently Asked Questions

    How should I choose a Chinese LLM?
    First clarify whether your scenario is developer integration or everyday interaction, then score each dimension using the weighted framework in Section 1. Testing with your own data is more reliable than comparing leaderboards. If your team needs both function calling and long-document processing, evaluate them separately—don’t assume one model is best at both.

    What should I watch out for in API integration?
    Focus on timeouts and retries, handling disconnections in streaming responses, Schema validation of parameters, and billing and rate-limit rules. When a streaming output is interrupted, the client should be able to detect the incomplete result and decide whether to retry or notify the user.

    Why do function calls fail?
    Common causes include unclear tool descriptions, ambiguous parameter names, the model invoking tools for questions that don’t need them, and invalid returned JSON. Improving tool descriptions is usually cheaper than switching models—try that first.

    What comes after the selection is done?
    Set up continuous monitoring. Log latency, error codes, and token consumption for every call, and run periodic human spot checks on output quality. When a model version is updated, rerun your test suite—capability changes often aren’t fully listed in release notes.

    There is no one-and-done answer to choosing an LLM. Measurable metrics, real business data, and a clear evaluation checklist will support long-term technical decisions far better than chasing a single ranking.