How to Evaluate Open-Source LLMs: Lessons from China’s Emerging Models
China’s emerging open-source large language models show why parameter counts and leaderboard scores are not enough to guide model selection. This article explains how to evaluate language understanding, multi-step reasoning, coding, inference speed, and deployment costs using realistic tasks and reproducible tests. It also covers the reliability and operational checks enterprises need before moving a model into production.
Competition among large language models (LLMs) developed in China has shifted from whether they can hold a conversation to whether they can reliably complete complex tasks. As more open-source models emerge, some with smaller parameter counts are gaining attention for their engineering efficiency, fast responses, and balanced performance across tasks.
No single benchmark or vendor claim can establish whether a model is ready for a particular workload. Developers and enterprises need an evaluation process that reflects how the model will actually be used, from understanding instructions and generating code to serving requests within latency and cost constraints.
1. From Parameter Count to Task Performance
Parameter count gives a rough indication of a model’s capacity, but it does not directly predict how well the model will perform in production. A practical evaluation should consider five dimensions:
- Task completion quality: Does the model understand the user’s intent and deliver a complete, actionable answer?
- Complex task handling: Can it maintain context across several steps without skipping important parts of the problem?
- Code generation: Is the generated code syntactically correct, runnable, and maintainable?
- Response efficiency: Does the model return results quickly enough for the intended application, especially in interactive settings?
- Deployment feasibility: Do inference resource requirements, concurrency, service stability, and cost meet business needs?
An emerging open-source model does not have to lead in every category to merit a place on the shortlist. Production workloads rarely depend on just one metric, so a model that performs consistently across several dimensions may be the better fit.
2. What a Comprehensive LLM Evaluation Should Cover
A handful of general knowledge questions or simple prompts cannot show how a model will handle real work. The test set should reflect the structure and constraints of the tasks the model will face.
Language Understanding and Knowledge-Based Q&A
Language understanding includes following instructions, connecting information across a conversation, extracting facts, and summarizing text. A test might ask the model to extract key conclusions from a long technical document or produce a plan in a prescribed format while satisfying several constraints.
Do not score an answer solely on whether it sounds plausible. Check whether the model:
- Follows all explicit constraints.
- Avoids adding information that is not supported by the prompt or source material.
- Distinguishes facts from assumptions and recommendations.
- Remains consistent across multiple turns.
- Produces output in the required format.
Format compliance matters when another system consumes the output. Even a factually correct answer can increase post-processing costs if the model does not reliably return the expected structure.
Logical Reasoning and Multi-Step Tasks
Use tasks that involve analyzing conditions, comparing options, locating errors, planning a workflow, or verifying a result. The final answer alone does not show where a model went wrong. Common failures include overlooking implicit constraints, confusing necessary and sufficient conditions, changing the objective midway, reaching a conclusion that conflicts with the preceding work, and expressing too much confidence when information is incomplete.
Break the evaluation into four stages: problem understanding, planning, execution, and verification. Where the task provides observable intermediate outputs, score them separately. This helps distinguish a misunderstood requirement from an execution error or a correct result that was communicated poorly.
3. Evaluate Coding Beyond Short Snippets
Code generation is now a core part of LLM evaluation for software development. A short coding exercise can test a specific skill, but it cannot show whether a model can take a requirement through implementation and verification.
Code Generation and Modification
A representative coding test set can ask the model to:
- Implement a function from a natural-language requirement.
- Complete existing code.
- Fix syntax and logic errors.
- Add a feature to an existing codebase.
- Write API calls, data processing logic, and error handling.
- Refactor code into clearer modules.
Evaluate correctness, readability, and safety. Running successfully is the minimum requirement. Check how the code handles edge cases, exceptions, resource cleanup, and input validation.
Front-End and Visualization Tasks
Interactive web pages, data visualizations, and 3D scenes test whether a model can coordinate interface structure, interaction logic, visual presentation, and runtime requirements. A 3D scene task, for example, might require the model to:
- Organize objects into a scene hierarchy.
- Configure cameras, lighting, and materials.
- Implement interaction controls.
- Handle animations and state changes.
- Ensure the page loads in the target environment.
- Check the display at different screen sizes.
The challenge is getting the components to work together. A model may generate plausible snippets for each part without producing a working application. Application-level tests reveal integration problems that isolated coding questions can miss.
4. Measure Inference Speed Beyond Peak Throughput
Some models report peak generation rates of hundreds to more than a thousand tokens per second in high-speed inference modes. Those figures are only meaningful in the context of the test setup. Hardware type and quantity, quantization, input and output lengths, reasoning mode, concurrent requests, server scheduling, network transmission, and first-token latency can all affect reported performance or the user experience.
Distinguish three metrics:
- Time to first token (TTFT): The elapsed time between sending a request and receiving the first output token. This strongly influences perceived responsiveness in Q&A, code completion, and real-time assistants.
- Output throughput: The rate at which the model generates tokens once output begins, typically measured in tokens per second. It is useful for long-form generation, batch summarization, and automated processing.
- End-to-end response time: The total time required to complete a request, including queueing, preprocessing, model computation, output generation, and network transmission. This is the most direct measure of how long a user or downstream system waits for the complete result.
A model can generate tokens quickly and still feel slow if users wait too long for the first one. For interactive products, a short time to first token and steady output can matter more than a high peak generation rate.
5. Build a Fair, Reproducible Evaluation Process
Step 1: Define the Task Set
Include general Q&A, long-document processing, logical reasoning, code generation, code repair, structured output, and multi-turn dialogue. For each task, specify the input, expected result, and scoring criteria before testing.
Step 2: Control the Test Conditions
Standardize the following wherever possible:
- Input prompts and context lengths.
- Maximum output length.
- Sampling settings, such as temperature.
- Whether an extended reasoning mode is enabled.
- Inference hardware and serving configuration.
- Network conditions and concurrent load.
If these conditions differ, the results may reflect the serving environment rather than differences between the models. Record the conditions so that others can interpret and reproduce the tests.
Step 3: Combine Automated Scoring With Human Review
Use executable test cases to score coding tasks and format checks to verify that structured outputs contain the required fields. For speed tests, run multiple requests and report the mean, median, and relevant percentiles.
Open-ended Q&A, solution design, and long-document summaries still benefit from human review. Use a common rubric that scores accuracy, completeness, logical coherence, actionability, and clarity separately.
Step 4: Measure Consistency, Not the Best Run
Repeat tests across multiple runs and record the mean, median, worst result, and failure or anomaly rate. For enterprise workloads, also check:
- How response times change under high concurrency.
- How much performance degrades with longer contexts.
- Whether outputs become less consistent across repeated requests.
- How the service recovers from interruptions or timeouts.
- How easily incorrect answers can be detected and corrected.
6. Check Production Readiness Separately
Strong evaluation results do not automatically make a model production-ready. Enterprises still need to assess data security, access control, audit logging, content compliance, and integration with existing systems.
One option is to decouple model serving from business applications through a shared service layer that handles request routing, rate limiting, retries, caching, and monitoring. Different tasks can use different models or inference configurations so that routine requests do not always run in the most expensive mode.
Build validation into the workflow:
- Validate required fields in structured outputs.
- Test generated code in a sandbox.
- Use retrieval and verify citations for answers that depend on external knowledge.
- Require human confirmation for high-risk operations.
- Retain logs of anomalous responses so they can be traced and investigated.
7. Put Emerging Models to the Test in Real Workflows
The rise of new open-source contenders shows that competitive performance is not limited to the largest systems. A model that balances general capabilities, coding performance, and inference efficiency can earn a place on an enterprise shortlist. But strength in coding does not imply the same level of performance in long-context consistency, complex knowledge questions, or high-concurrency serving.
Use the intended workflow to decide what matters most. For real-time interaction, prioritize time to first token and steady output throughput. For software development, measure how often generated code runs and how often requested changes succeed. For enterprise knowledge Q&A, focus on factual accuracy, citation completeness, and access controls that prevent information from reaching unauthorized users.
Model selection ultimately comes down to the balance between capability, speed, cost, and reliability for a specific workload. A reproducible evaluation process will remain useful long after any single leaderboard ranking changes.
