Domestic LLM Iteration Is Accelerating: How to Evaluate Models Objectively and Choose Rationally

Why Evaluation Methodology Matters More Than Any Single Conclusion

China’s large language model ecosystem has entered an unusually intense release cycle. New versions are appearing one after another, parameter counts are moving toward the trillion scale, context windows have expanded from tens of thousands to hundreds of thousands or even millions of tokens, and multimodal and coding capabilities continue to improve. At the same time, inference prices have fallen sharply, and earlier practices such as waitlists and restricted subscriptions are being replaced by broad availability.

Taken together, these changes point to a clear trend: the overall gap between domestic models and leading international models is narrowing quickly, and domestic models are forming differentiated advantages in areas such as pricing, context length, and ease of access.

However, a narrowing gap does not mean domestic models have already overtaken their international counterparts. Two extreme views are common on social media: one claims domestic models have fully surpassed international models, while the other insists domestic models are still only suitable for simple tasks. This article does not take either side, nor does it attempt another subjective “leaderboard-style” comparison. Instead, it addresses three more practical questions:

1. How can we use public data and independent third-party evaluations to objectively assess the current level of domestic models?
2. Which quantifiable dimensions reveal both remaining gaps and competitive strengths?
3. Why can’t we judge whether a model is “smart” based on one or two conversations, and what should a proper evaluation process look like?

A note on methodology: for capability comparisons, this article relies primarily on official benchmark results published by model providers and reports from independent evaluation organizations. For pricing information, it uses publicly verifiable data available at the time of writing. It does not cite unverified figures, nor does it constitute a recommendation of any specific product.

A Realistic View of the Current Landscape: Neither Extreme Holds Up

The claim that “domestic models can only handle simple tasks” does not stand up to scrutiny. Most judgments of this kind come from individual experience: a user gets an unsatisfactory answer in one conversation and concludes the model is inadequate. This kind of generalization has clear methodological flaws, which are discussed in detail in the next section. Looking at public benchmarks, domestic models have steadily improved on code generation, mathematical reasoning, and Chinese-language understanding, with some individual results now in the first tier. On the engineering side, long context support, low cost, and reliable availability have even become relative advantages.

The claim of “full superiority” is also premature. Leading international labs still have deep reserves in fundamental research, compute and chip availability, data engineering, and product ecosystems. They remain ahead in the upper bounds of complex reasoning and in the stability of long-horizon agent workflows. A more realistic assessment is therefore: domestic models have not yet fully surpassed leading international models, but the gap is narrowing quickly, and in many real-world deployment scenarios, the remaining difference is small enough that it no longer affects business decisions.

A different perspective: model selection is not about “which is strongest” but “which is most suitable.” As with buying a phone, talking about which model “crushes” another without considering the use case has limited value. Users who prioritize photography, battery life, or ecosystem integration will give completely different answers. Model selection works the same way: a business that mainly processes Chinese documents, is cost-sensitive, and must keep data within a specific jurisdiction may have a very different optimal choice than a developer tool centered on complex English-language reasoning. Only what truly fits your needs is what works best.

Why One or Two Conversations Cannot Determine Model Quality

This is a key misconception that deserves clarification. From a technical perspective, at least four factors make conclusions drawn from a single conversation unreliable:

1. Sampling randomness: Generative models sample token by token from a probability distribution. Parameters such as temperature and top_p can cause the same prompt to produce different answers across multiple calls. You may have encountered the best possible output—or the worst.
2. Task distribution bias: An offhand question represents an extremely small sample and may not overlap with the model’s actual capability distribution. A model that is strong at coding but weak at writing will be simultaneously overestimated and underestimated if tested with a single math problem.
3. Prompt sensitivity: The same capability can produce dramatically different results depending on wording, output format, or whether examples are provided. A single test cannot distinguish between “the model cannot do it” and “the prompt was poorly written.”
4. A single sample has no statistical significance: A single success or failure is a high-variance random event and does not provide a valid estimate of the model’s expected performance.

The correct approach is to turn evaluation into a repeatable statistical process: use a fixed evaluation set, sample each task multiple times, standardize decoding parameters, and compare models under identical conditions. Here is an illustrative example:

import statistics
def evaluate(model, eval_set, n_samples=8, temperature=0.7):
    """Repeat sampling for each case and return overall pass rate and volatility."""
    pass_rates = []
    for case in eval_set:                # case contains a prompt and a check function
        outcomes = []
        for _ in range(n_samples):
            answer = model.chat(case.prompt, temperature=temperature)
            outcomes.append(1 if case.check(answer) else 0)
        pass_rates.append(statistics.mean(outcomes))
    overall = statistics.mean(pass_rates)
    volatility = statistics.pstdev(pass_rates)
    return overall, volatility

The core idea is to replace “feels smart” with two comparable numbers: pass rate and variance. In engineering practice, stability often deserves more attention than peak performance. A model that occasionally performs brilliantly but frequently fails is far riskier in production than a model that performs consistently at a moderate level.

A Practical Three-Layer Evaluation Framework

Single-turn conversations are insufficient, and leaderboards alone are not enough. A better approach organizes evaluation into three layers:

Layer	Purpose	Key Actions
Public benchmarks	Initial screening and capability positioning	Cross-reference multiple independent evaluations; understand what each benchmark actually measures
Business evaluation set	Decision-making and scenario fit validation	Extract 100–300 real tasks and annotate expected outputs
Online experiments	Verify real-world benefit	Roll out gradually and observe adoption rate, edit rate, and cost

Two additional points: public benchmarks are susceptible to data contamination—some questions may have appeared in training data, inflating scores. Cross-referencing multiple sources is therefore more reliable than relying on a single leaderboard. Your business evaluation set should cover both high-frequency paths and edge cases. Conclusions from this layer should be the primary basis for model selection.

From Evaluation to Selection: Five Engineering Decision Points

1. Capability dimension matching: Break down requirements into sub-dimensions such as reasoning, coding, long context, multimodality, and Chinese-language capability, and evaluate each separately. Do not chase a single “best at everything” model. For long-context scenarios, also verify effective retrieval performance, since the “lost in the middle” phenomenon means nominal context length does not equal effective usable length.
2. Cost estimation: Pricing is typically separated for input and output tokens, and some providers offer cache discounts and batch asynchronous pricing. Estimates should be based on real token distributions from production logs and should include hidden costs such as retries after failures and context expansion:

# Monthly cost estimate (prices are symbolic variables; use official public pricing)
calls = 500_000
avg_in, avg_out = 1_800, 400
price_in, price_out = p_in, p_out        # per million tokens
monthly = calls * (avg_in * price_in + avg_out * price_out) / 1e6

1. Deployment and compliance: Direct API access is suitable for rapid validation, while dedicated instances are better for data-sensitive scenarios. Businesses operating in specific jurisdictions need to account for data residency and content safety requirements. Cloud platforms such as Baidu AI Cloud typically integrate model inference, fine-tuning, vector retrieval, object storage, and logging and monitoring within the same ecosystem, which can reduce end-to-end integration complexity. Whether to adopt such an approach depends on your team’s existing infrastructure.
2. Stability and availability: Pay attention to rate limiting policies, concurrency limits, and availability history. No matter how low the price, a model that gets throttled during peak traffic and disrupts your business is unacceptable.
3. Multi-model strategy: Production systems should not be locked into a single model. Route tasks by type: send simple tasks to a low-cost model and complex reasoning to a high-capability model, while abstracting the interface so models can be swapped later:

def route(task):
    if task.needs_deep_reasoning or task.needs_long_context:
        return MODELS["high"]
    if task.is_simple_extraction:
        return MODELS["cheap"]
    return MODELS["default"]
answer = call(route(task))
if not quality_check(answer):            # Upgrade and retry if the quality gate fails
    answer = call(MODELS["high"])

Conclusion

Domestic large language models are in their fastest period of improvement: version releases are frequent, prices continue to fall, and context and multimodal capabilities keep expanding. The overall gap with leading international models is narrowing significantly, and domestic models have carved out advantages in several dimensions. But more important than asking “which is stronger” is building your own evaluation process: use public benchmarks for initial screening, business evaluation sets for decision-making, and online experiments for validation. Replace intuition with statistics, measure stability through variance, and compare real costs rather than nominal unit prices. Models will keep changing and leaderboards will keep refreshing, but the engineering methodology for evaluation and selection is the asset that survives version cycles and truly compounds over time. And it comes back to that simple principle: only what truly fits your needs is what works best.