Chinese Open-Source LLMs Are Converging on the Same Architecture—What Should You Evaluate When Choosing One?

    As Chinese large-model development matures, the real competition has shifted from "who can build a usable model" to "who can deliver more stable, more cost-effective capability in real production workloads." This article examines a striking pattern in public technical reports: the base architectures of mainstream open-source models are converging fast, so differences increasingly come from data recipes, long-context effectiveness, alignment and agent capabilities, and inference engineering rather than from the skeleton itself. You'll learn why architecture alone is a poor basis for selection, how to reason about token efficiency, effective context windows, alignment, multimodal support, and serving cost, plus a practical, reproducible evaluation framework and a scenario-based decision matrix to guide model choice.

    In recent years, competition in the Chinese large-model landscape has shifted away from “who can build a usable model” toward “who can keep delivering more stable, more cost-effective capability in real business workloads.”

    Judging by public technical material, one pattern stands out: the base architectures of mainstream open-source large models are converging rapidly. The sparse mixture-of-experts (MoE) models described in many public technical reports are highly similar to the sparse base architecture widely used across the industry in their attention mechanism and expert design; the differences mostly come down to hyperparameter trade-offs such as the number of experts and attention heads. This has made “the architectures look alike—so where do the real advantages actually differ?” a question that developers and technical leads revisit constantly.

    Two claims should be treated with caution. Asserting that “similar architecture means no essential difference in capability,” or conversely that “identical architecture means near-identical capability,” is not rigorous either way. Architecture is only the container that holds the data and the training methodology. What truly separates models is the data recipe, context strategy, alignment training, multimodal capability, and inference engineering efficiency. Using publicly verifiable technical material and hands-on engineering experience, this article systematically breaks down these sources of difference and offers an actionable selection framework.

    1. Architecture Layer: Sparse MoE Becomes the De Facto Standard, and Architectural Differences Compress Fast

    1.1 Dense vs. Sparse MoE: Decoupling Parameter Count from Activated Parameters

    Sparse MoE (Mixture of Experts) splits the feed-forward layers of a Transformer into multiple “experts,” with a router selecting a few experts to participate in computation for each token. Its core value is decoupling “total parameter count” from “activated parameters per token”:

    Dimension Dense Model Sparse MoE Model
    Compute per token Proportional to total parameters Mainly determined by activated parameters
    Weight memory Smaller Substantially larger; all experts must be resident
    Distributed deployment Data parallelism plus tensor parallelism suffices Usually requires expert parallelism
    Training communication overhead Relatively low Heavy all-to-all communication between experts
    Inference load balancing Stable Depends on routing balance; tail imbalance possible
    Suitable scenarios Edge devices, low latency, resource-constrained Large general-purpose base models, complex tasks

    “More parameters, less compute” has become the mainstream approach for large-scale open-source models in recent years, and the room for architectural differentiation has compressed accordingly.

    1.2 Attention Mechanisms: The Focus Shifts from Compute to Memory Access

    Most mainstream models use multi-head, multi-query, or grouped-query attention to shrink the KV cache. Some public technical reports also show that certain models adopt a latent attention mechanism, using low-rank compression to significantly reduce the KV cache while adjusting the number of attention heads accordingly.

    These choices rest on the same judgment: in long-context, high-concurrency inference, the bottleneck has shifted from raw compute to memory bandwidth and cache capacity. Architectural evolution has therefore turned toward “effective information throughput per unit of compute.”

    1.3 Routing and Parallelism: Differences Hide in Implementation Details

    Design choices such as the number of experts, top-k activation count, shared experts, and load-balancing loss directly affect training stability and inference efficiency. Public material indicates that pushing the number of experts up from around 256 to several hundred—while reducing attention heads—has become a typical architectural trade-off for some teams. On the engineering side, expert parallelism, communication optimization, and operator fusion further amplify or offset the benefits that the architecture itself provides.

    1.4 Architectural Information Is Only Partially Available

    Open-source models expose their configurations and weights, but training details—data mixtures, data-synthesis pipelines, reinforcement-learning recipes, parallelism strategies, and cluster-stability handling—often receive only high-level descriptions even in technical reports, and some reports do not directly disclose architectural details at all. This means:

    • From the outside, it is hard to infer real capability from architectural configuration alone; external speculation should be treated only as a reference.
    • Selection should be based on measured results, not on “whether the architectures look alike.”

    2. Source of Real Capability Difference (1): Data Recipe and Token Efficiency

    2.1 High-Quality Text Is Peaking, Making Token Efficiency the Key Variable

    The industry broadly agrees that high-quality human text is approaching its availability ceiling. Against this backdrop, training effectiveness increasingly depends on “how much effective learning each token delivers,” and the emphasis on “token efficiency” in public technical reports reflects exactly this trend. Common practices include:

    1. Rigorous data cleaning, deduplication, and quality tiering;
    2. Curriculum-style data arrangement that moves from general to specialized and from easy to hard;
    3. Synthetic data generated by the model itself and then verified—code with test checks, math and reasoning problems with reference answers;
    4. Targeted supplementation of domain data such as code, math, and multilingual content during continued pre-training.

    2.2 Optimizers and Training Stability

    AdamW and its variants are the mainstream choice. Some public technical reports also disclose that certain teams have adopted matrix-orthogonalization-based Muon-style optimizers to improve token efficiency, along with techniques such as QK-Clip to mitigate the training instability caused by exploding attention scores. In large-scale training, loss spikes are a common risk. A simplified monitoring approach:

    1. import torch
    2. class LossSpikeGuard:
    3. """Detect loss spikes via a sliding median and flag that the training run should
    4. roll back and skip the problematic data segment."""
    5. def __init__(self, window: int = 20, factor: float = 3.0):
    6. self.history: list[float] = []
    7. self.window = window
    8. self.factor = factor
    9. def update(self, step: int, loss: float) -> str:
    10. self.history.append(loss)
    11. if len(self.history) > self.window:
    12. self.history.pop(0)
    13. if len(self.history) < self.window:
    14. return "ok"
    15. baseline = torch.tensor(self.history[:-1]).median().item()
    16. if loss > baseline * self.factor:
    17. # In production, roll back to the most recent checkpoint, mask the
    18. # problematic data segment, and if needed lower the learning rate or
    19. # adjust optimizer hyperparameters.
    20. return f"spike@{step}: {loss:.3f} vs median {baseline:.3f}"
    21. return "ok"
    22. # Example: call this step by step inside the training loop
    23. guard = LossSpikeGuard()
    24. status = guard.update(step=1000, loss=2.35)
    25. if status != "ok":
    26. print(status)

    2.3 How Data Mixes Shape the Capability Profile

    The quality of code and math data directly affects code generation and logical reasoning. The proportion of long documents determines long-context ability, and the multilingual mix affects cross-language performance. Human preference data and verifiable-reward data used during alignment shape answer style and instruction following. Because each vendor makes different trade-offs in these mixes, the result is a distinct “capability profile” with its own emphases.

    3. Source of Real Capability Difference (2): The Effectiveness of Long Context

    3.1 A Nominal Window Is Not an Effective Window

    Extending the context window from a few thousand to hundreds of thousands or even millions of tokens mainly relies on the following techniques:

    • Position-encoding extrapolation: adjusting the base frequency or scaling factors of rotary position embeddings (RoPE) so the model can extrapolate beyond its training length;
    • Long-context continued training: exposing the model to long sequences through concatenated long documents and multi-document combinations;
    • Attention optimization: controlling memory and memory-access overhead via KV-cache compression, paged management, and prefix reuse.

    However, long context introduces two nonlinear problems: attention compute grows quadratically with sequence length, and KV-cache usage grows linearly with length and concurrency. The larger the nominal window, the more you must verify retrieval and reasoning quality at real lengths.

    3.2 Effective-Window Testing: From “Needle in a Haystack” to Multi-Hop Reasoning

    The industry commonly uses a “needle in a haystack” test: bury a key piece of information at different positions in a long document and check whether the model can retrieve it accurately. This measures basic retrieval ability. Real workloads also demand cross-segment aggregation and multi-hop reasoning, so a three-tier validation is recommended:

    1. Single-point retrieval: insert facts at different depths and measure recall accuracy;
    2. Aggregative statistics: sum values across multiple segments;
    3. Multi-hop reasoning: chains of multiple conditions required to reach a conclusion.

    The example below provides a single-point retrieval test harness:

    1. FACT = "The acceptance report states that the concurrency limit was adjusted to 1,200 requests per second."
    2. def build_prompt(fact: str, filler_line: str, total_lines: int, depth: float) -> str:
    3. """Insert the fact at relative depth in the document; fill the other lines
    4. with unrelated text."""
    5. insert_at = int(total_lines * depth)
    6. lines = [filler_line] * total_lines
    7. lines[insert_at] = fact
    8. lines.append("Question: " + fact.replace("adjusted to", "set to"))
    9. return "\n".join(lines)
    10. def probe(model_call, depths=(0.0, 0.25, 0.5, 0.75, 0.99), repeats: int = 5):
    11. results = {}
    12. for d in depths:
    13. hits = 0
    14. for _ in range(repeats):
    15. prompt = build_prompt(FACT, "This is an unrelated filler line of text.", 2000, d)
    16. answer = model_call(prompt) # implement this as a unified model call
    17. hits += int("1,200" in answer)
    18. results[d] = hits / repeats
    19. return results
    20. # Example: plug in any model by replacing model_call for a side-by-side comparison
    21. scores = probe(model_call=lambda p: "The concurrency limit is set to 1,200 requests per second.")
    22. print(scores)

    There is no unified standard for the effective window. A common rule of thumb: if retrieval accuracy stays high (for example, above 95%) at all depth positions, the corresponding length can be considered that model’s effective window.

    3.3 Long Context Is a Double-Edged Sword

    • Benefits: it reduces chunking and retrieval steps, preserves full context, and suits tasks such as contract review, codebase understanding, and meeting-minutes analysis.
    • Costs: time-to-first-token rises noticeably, cost and memory usage increase, and “attention dilution” within long text may cause key details to be missed.

    Long-context capability should therefore be understood as a composite of “nominal length × effective accuracy × latency and cost,” not a single number.

    4. Source of Real Capability Difference (3): Alignment and Agent Capability

    4.1 Instruction Following and Structured Output

    Business systems rely on models producing JSON, function-call arguments, or fixed-format text. Instruction-following ability comes mainly from supervised fine-tuning data and preference-alignment training, and it directly determines integration cost. When evaluating, focus on:

    • Field completeness and format validity of JSON output;
    • Accuracy of function-call arguments and support for parallel calls;
    • The ability to hold constraints across multi-turn dialogue (for example, “no more than 200 words”).

    4.2 Coding Ability: From “Completion” to “Repository-Level Understanding”

    Coding ability depends not only on the proportion of code corpus but also on the training objective:

    • Models trained mainly on text prediction excel at completion and snippet generation;
    • Models that introduce execution feedback, unit-test verification, and reinforcement learning on verifiable tasks are better at complex engineering work such as fixing bugs and refactoring across files.

    4.3 Balancing Reasoning Depth and Cost

    Models that support a reasoning mode (deep thinking) can raise accuracy on complex problems by generating longer chains of thought, but token consumption increases several-fold. Some models support switching between reasoning and non-reasoning modes, making it easier to choose dynamically based on task difficulty.

    4.4 Alignment Strategy Defines the “Product Personality”

    • Preference alignment (such as reinforcement learning from human feedback) shapes answer style;
    • Reinforcement learning with verifiable rewards (RLVR) improves accuracy on objective tasks such as math and code;
    • Safety alignment reduces the risk of sensitive content.

    These differences often get the least attention in technical reports, yet they most directly affect the user experience.

    5. Source of Real Capability Difference (4): Multimodal Capability

    Multimodal capability generally falls into three categories:

    1. Bolt-on:专用components such as optical character recognition and speech recognition first extract text, which is then handed to a text large model for understanding. Integration is simple, but understanding of charts, layouts, and video timing is limited;
    2. Native multimodal: images, audio, and other inputs are encoded into representations isomorphic to text tokens and handled uniformly by a single model, making cross-modal reasoning more natural;
    3. Hybrid: native understanding combined with specialized components, balancing effectiveness and cost.

    When evaluating multimodal capability, don’t just look at “whether it supports image input.” Also look at the maximum image resolution, OCR and small-text recognition accuracy, chart and table understanding, video-timing understanding, and the impact of image input on latency and cost.

    6. Inference Engineering: The Invisible Variable That Determines Deployment Cost and Stability

    6.1 MoE’s “Cheapness” Is Conditional

    MoE achieves the effect of a large model with few activated parameters, but at deployment time all expert weights must be resident in memory. Assume a total of 300 billion parameters using FP8 quantization (about 1 byte per parameter); the weights alone occupy roughly 300 GB, requiring at least four 80 GB GPUs, plus expert parallelism and all-to-all communication. This means:

    • The compute cost per inference is low;
    • The memory and deployment barriers, however, rise substantially.

    For small and mid-sized teams, “can it be deployed and fine-tuned further at low cost” is often a more realistic consideration than a leaderboard score.

    6.2 A Framework for Estimating Inference Cost

    Memory usage can be split into two parts:

    • Weights: total parameter count × bytes per parameter (FP16 is about 2 bytes, FP8 about 1 byte, INT4 about 0.5 bytes);
    • KV cache: 2 × number of layers × number of KV heads × head dimension × sequence length × concurrency × bytes per element.

    From this, you can roughly plan for different scenarios:

    Scenario Weight Usage KV Cache Main Bottleneck
    Short requests, low concurrency Unchanged Very small Memory capacity, expert-parallel communication
    Long context, single concurrency Unchanged Grows linearly with length KV-cache capacity, time-to-first-token
    High concurrency, medium length Unchanged Grows linearly with concurrency Memory bandwidth, throughput scheduling

    6.3 Key Inference Optimization Techniques

    Technique Effect Caveats
    Weight quantization (FP8, INT4, etc.) Reduces memory and bandwidth pressure May cause accuracy loss; regression validation required
    KV-cache quantization Significantly reduces long-context memory Affects long-range information retention; must be measured
    Prefix-cache reuse Reuses identical system prompts or documents Only effective when the prefix matches exactly
    Continuous batching Improves overall throughput May increase per-request latency variance
    Speculative decoding Reduces per-token latency Requires a draft model or extra training; higher implementation complexity
    Prefill/decode disaggregation Optimizes time-to-first-token and throughput separately More complex scheduling and resource planning

    It bears emphasis that these techniques are strongly tied to specific model versions and are not necessarily available out of the box in every deployment environment. During selection, include “whether the deployment environment supports it” and “accuracy changes after optimization” within the validation scope.

    6.4 Observability: From “Working” to “Controllable”

    Metric Definition Focus
    Time-to-first-token (TTFT) Time from request sent to first token received Conversational experience, real-time scenarios
    Per-token generation latency Average time to generate each subsequent token Smoothness of streaming output
    End-to-end latency Time from request sent to complete response Offline batch processing, long-document tasks
    Throughput Number of tokens processed per unit of time Capacity planning and cost control
    Error and degradation rate Proportion of failures, timeouts, and fallbacks System stability

    7. Build Your Own Evaluation: Turn “Gut Feeling” into Reproducible Data

    Public leaderboards are useful for an initial screen, but their questions may already be in the training data, and they differ from your own business distribution. It is worth building a lightweight, reproducible evaluation pipeline:

    1. Scenario sampling: draw no fewer than 200 tasks from real business, covering typical, boundary, and long-tail cases;
    2. Unified input: the same prompts and the same decoding parameters (for example, temperature fixed at 0);
    3. Unified adaptation: wrap different models behind a unified interface to hide API differences;
    4. Automated scoring: format validation, rule matching, unit tests, and custom scoring scripts;
    5. Human blind review: shuffle the sources, then have business experts double-blind score a sample of no fewer than 20% of the total;
    6. Cost linkage: record token consumption and latency alongside each evaluation to form an “effectiveness–cost” matrix.
    1. import json, hashlib, time
    2. from math import comb
    3. from statistics import mean
    4. class ModelClient:
    5. """Unified model-call interface: implement only the chat method to plug in
    6. a different model service."""
    7. def chat(self, messages: list[dict], **params) -> str:
    8. raise NotImplementedError
    9. def evaluate(client: ModelClient, tasks: list[dict], k: int = 1, rounds: int = 3):
    10. """Run rounds passes over the task set and report average latency and pass@k."""
    11. per_task = {t["id"]: [] for t in tasks} # results of repeated runs per task
    12. latencies: list[float] = []
    13. for _ in range(rounds):
    14. for task in tasks:
    15. start = time.perf_counter()
    16. output = client.chat(task["messages"], temperature=0)
    17. latencies.append(time.perf_counter() - start)
    18. per_task[task["id"]].append(int(task["scorer"](output)))
    19. scores = []
    20. for results in per_task.values():
    21. n, c = len(results), sum(results)
    22. # Unbiased pass@k estimate: expected proportion passing at least once in n samples
    23. scores.append(1.0 if n - c < k else 1.0 - comb(n - c, k) / comb(n, k))
    24. return {
    25. "tasks": len(tasks),
    26. "pass_k": round(mean(scores), 4),
    27. "avg_latency_s": round(mean(latencies), 3),
    28. "p95_latency_s": round(sorted(latencies)[int(len(latencies) * 0.95)], 3),
    29. }
    30. # Example: plug in different models through the unified interface for a side-by-side comparison
    31. report = evaluate(client=ModelClient(), tasks=[], k=1, rounds=1)
    32. print(json.dumps(report, ensure_ascii=False))

    An optional scoring-script example (for format illustration only):

    1. import hashlib
    2. # Each task keeps only the scorer field
    3. task = {
    4. "id": "json_extract_001",
    5. "messages": [
    6. {"role": "user", "content": "Extract the order number and amount from the text and return them as JSON"}
    7. ],
    8. "scorer": lambda out: int(is_valid_json(out))
    9. }
    10. def is_valid_json(text: str) -> bool:
    11. import json
    12. try:
    13. json.loads(text)
    14. return True
    15. except json.JSONDecodeError:
    16. return False
    17. def cache_key(messages: list[dict], params: dict) -> str:
    18. raw = json.dumps({"m": messages, "p": params}, sort_keys=True, ensure_ascii=False)
    19. return hashlib.sha256(raw.encode()).hexdigest()

    Evaluation results are best summarized in a table like the one below, updated regularly as model versions iterate:

    Model Version Test Set Size Avg Latency pass@k Avg Tokens per Task Overall Conclusion
    Version A 200+ to be filled to be filled to be filled to be evaluated
    Version B 200+ to be filled to be filled to be filled to be evaluated

    A caveat: when models of the same family act as automatic reviewers, they may favor their own or sibling models’ outputs, so keep human review as a calibration baseline for critical tasks.

    8. A Scenario-Based Selection Decision Framework

    Map business requirements to capability dimensions, then score them against engineering constraints:

    Business Scenario Key Capabilities Secondary Considerations Validation Points
    Intelligent coding and code assistants Code generation, repository-level understanding, function calling Completion latency, IDE integration Cross-file fix tests on your own codebase
    Long-document analysis (contracts, research reports, papers) Effective long-context accuracy, citation localization Long-input cost, structured output Long-document QA and multi-hop reasoning tests
    Enterprise knowledge Q&A and customer service Instruction following, format stability, tool calling Concurrency throughput, security compliance Knowledge-base regression test set
    Multimodal content moderation and understanding Image and video understanding, OCR accuracy Cost per image, latency Sampling tests on business images and documents
    Private deployment and secondary development Open weights, license, fine-tunability Deployment memory threshold, framework support Quantized inference test on target hardware
    Large-scale offline data processing Per-token cost, batch throughput Output stability, deduplication ability Sampling comparison of cost and accuracy

    From this, three principles emerge:

    1. Capability first: identify the one or two capabilities your business relies on most, then compare across models;
    2. Engineering as the backstop: capability advantages cannot be realized if the deployment environment, framework support, observability, or security compliance fall short;
    3. Dynamic evaluation: model iteration cycles are measured in months, so treat the evaluation pipeline itself as a long-term asset to maintain.

    9. Common Selection Pitfalls

    1. Parameter worship: total parameter count does not equal effective capability. For MoE models, pay more attention to activated parameters and training-data quality;
    2. Leaderboard worship: public leaderboards suffer from data contamination and distribution bias, and correlate only weakly with business performance;
    3. Window worship: treating the nominal context length as the effective length, ignoring accuracy decay and latency cost in long text;
    4. Unit-price worship: looking only at unit price while ignoring cache hit rate, routing strategy, and retry rate—actual costs can differ several-fold;
    5. Open-source worship: open source means control and customization, but it also means you must build deployment, operations, and security governance yourself.

    10. Architectural Recommendations for Production: Routing, Caching, and Observability

    Regardless of which model you choose, a production system should follow the principles of “decoupling, replaceability, and observability”:

    1. Unified access layer: hide API differences between model services to make switching easy;
    2. Intelligent routing: split traffic across models by task type, complexity, latency budget, and cost constraints;
    3. Caching layer: serve cache hits directly for identical requests, and enable prefix reuse for common prefixes such as system prompts and knowledge bases;
    4. Asynchronous processing and fallback: handle long-document tasks asynchronously via a message queue; when the primary model times out, automatically fall back to a backup model;
    5. Observability: continuously track latency, error rate, and cost through logging services and monitoring/alerting, with anomaly thresholds in place.
    1. import time
    2. from collections import OrderedDict
    3. class Cache:
    4. """Request-level cache based on LRU."""
    5. def __init__(self, capacity: int = 10000):
    6. self.data: OrderedDict[str, tuple] = OrderedDict()
    7. self.capacity = capacity
    8. def get(self, key: str):
    9. if key in self.data:
    10. self.data.move_to_end(key)
    11. return self.data[key]
    12. return None
    13. def set(self, key: str, value, ttl: int = 3600):
    14. if len(self.data) >= self.capacity:
    15. self.data.popitem(last=False)
    16. self.data[key] = (time.time() + ttl, value)
    17. def route_and_call(client, messages: list[dict], budget_ms: int, cache: Cache):
    18. """Check the cache first, then choose an execution mode by latency budget,
    19. and write the result back to the cache."""
    20. key = cache_key(messages, {})
    21. hit = cache.get(key)
    22. if hit and hit[0] > time.time():
    23. return hit[1], "cache"
    24. if budget_ms < 2000:
    25. return client.chat(messages, stream=True), "fast" # returns a streaming iterator
    26. return client.chat(messages), "standard"

    The cost savings from caching and routing are often more substantial than switching models. In the framework above, capabilities such as object storage, message queues, logging services, monitoring and alerting, and container platforms all belong to general-purpose cloud categories that can be combined as needed and are decoupled from the specific model choice. When using cloud platforms such as Baidu AI Cloud, you can also draw on these general capabilities to build model routing and observability systems.

    Conclusion

    Back to the original question: what are the core advantages of each Chinese open-source large model?

    From a technical standpoint, they share a similar base architecture exemplified by sparse MoE, so the real differences lie not in the “skeleton” but in four areas:

    • Data recipe and token efficiency;
    • The effectiveness of long context;
    • Alignment and agent capability;
    • Inference engineering and cost efficiency.

    For developers, rather than chasing any single model’s short-lived spotlight, it is better to build three long-term capabilities:

    1. Build a scenario-specific evaluation set and let data speak;
    2. Keep architectural flexibility with a unified access layer plus routing and caching;
    3. Make observability part of your go-live standard.

    Only then can you consistently make technical choices that match your business goals amid the wave of rapid model iteration.