Open-Source LLM Architecture Convergence: A Practical Guide to Model Selection
As new-generation open-source large language models converge on a shared architectural skeleton, model selection is shifting away from comparing network designs toward comparing performance on your own data and inference cost per token. This article examines the common technical foundations—ultra-sparse Mixture-of-Experts, low-rank latent attention, load balancing and training stability, token-efficient training, and multi-stage post-training—then breaks down the key trade-offs behind each architectural dimension. It closes with a practical workflow for building evaluation sets, estimating deployment cost, avoiding common pitfalls, and choosing between managed and self-hosted inference.
1. Architecture Convergence: From “Roadmap Disputes” to Convergence on the Best Solution
Over the past two years, several teams have open-sourced a new generation of large language models. Read their technical reports side by side, and you find they are highly similar in architecture:
- Mainstream models: Total parameter counts generally range from tens of billions to the trillions, with only a small fraction activated in a single inference pass.
- Feed-forward layers: A typical configuration of fine-grained experts plus shared experts.
- Attention: Designs that widely adopt low-rank compression of the key-value cache.
- Training: Low-precision mixed-precision training (BF16, FP8, and so on) as the main line, balancing stability and throughput.
- Post-training: After pre-training, models generally go through a multi-stage process of “reasoning-capability fine-tuning + large-scale reinforcement learning.”
This similarity is often read as mutual “borrowing.” However, whether or not the reports state it, after multiple iterations each team’s roadmap converges on the same skeleton. A more accurate description is “convergence”: facing the same hardware constraints and the same publicly available research, the local optima that designers arrive at grow closer and closer.
Two points are worth emphasizing:
- Shared architecture does not mean shared capability. The public structure can be reproduced, but data mixtures, training recipes, and reinforcement learning strategies are hard to fully reconstruct from a report—and these “invisible parts” are often the main source of capability differences.
- Architectural convergence does not mean benchmark convergence. The same architecture performs differently in different teams’ hands, showing that final results depend more on data and post-training than on the network structure itself.
As a result, the focus of model selection is shifting: from comparing “whose network design is newer” to comparing “who performs better on your business data at a lower inference cost per token.”
2. The Common Skeleton: Core Technologies Behind the New Generation of Open-Source Models
2.1 Ultra-Sparse Mixture-of-Experts: Decoupling Parameter Capacity from Inference Compute
In a dense model, every additional unit of parameters costs a computation for every token. Mixture-of-Experts (MoE) splits the feed-forward layer into several “expert” subnetworks, and a gating network decides which experts each token activates. This creates two independent scaling dimensions:
- Total parameters: Determine knowledge capacity, and also determine memory usage and the communication volume during expert parallelism.
- Activated parameters: Determine the actual compute per token, and thus throughput and latency.
The ratio of the two is called sparsity. Higher sparsity means carrying more parameters with less compute. A common observation in public technical reports is that within the range of continuously increasing sparsity, model quality still improves noticeably. This is why many new-generation models choose hundreds of fine-grained experts, activate only a few per token, and set a small number of shared experts to carry general knowledge.
Raising sparsity is not without cost:
- The finer the experts, the more sensitive routing decisions become.
- Load is more prone to imbalance, and the all-to-all communication of expert parallelism grows more frequent.
So sparsity is essentially a balance between “capacity and efficiency.”
2.2 Low-Rank Latent Attention: Compressing the KV Cache to Make Long Contexts More Affordable
Standard multi-head attention must cache all Keys and Values for every token at every layer. The cache overhead can be expressed as:
KV cache ≈ 2 × sequence length × number of layers × number of KV heads × head dimension × bytes per element × concurrency
In long-context, high-concurrency serving, the KV cache is often the main memory consumer, directly limiting the maximum supported context length and concurrency.
The idea behind low-rank latent attention is:
- First compress each token’s hidden state into a low-dimensional latent vector, and cache only that vector.
- When attention must be computed, reconstruct each head’s Key and Value through projection matrices.
- Handle positional encoding separately, so that compression does not break the positional relationships that rotary position encoding (RoPE) relies on.
The example below simplifies details such as normalization and attention masks, and only illustrates the principle:
# Illustration: compression and caching in low-rank latent attention (simplified pseudocode)def compress_kv(h, W_dkv):# h: [batch, seq, d_model], cache only c instead of the full k and vc = h @ W_dkv # [batch, seq, d_c]def attention_from_cache(c, W_uk, W_uv, W_q, H, d_h):k = (c @ W_uk).reshape(..., H, d_h) # reconstruct per-head Key on demandv = (c @ W_uv).reshape(..., H, d_h) # reconstruct per-head Value on demandq = (W_q.reshape(H, d_h, -1)) # Illustration: Query projected from the current statereturn scaled_dot_product(q, k, v) # scaled dot-product attention
When the compression dimension is far smaller than twice “number of KV heads × head dimension,” the KV cache can drop by roughly an order of magnitude, while the matrix sizes inside the module stay essentially unchanged. This is especially critical for long-document processing, multi-turn dialogue, and code-repository-level understanding.
Latent attention is not a “free lunch” either:
- Although the cache is smaller, the reconstruction projections add extra computation.
- Setting the compression dimension too low sacrifices representational power, so you must trade off quality against cost.
- Custom cache layouts place higher demands on inference frameworks, and some open-source inference frameworks still need dedicated adaptation for such structures.
2.3 Load Balancing and Training Stability: The Invisible Infrastructure of Large-Scale Training
When the number of experts reaches the hundreds, two classes of problems are amplified.
First, routing collapse. Early in training, a few experts are repeatedly selected thanks to their initial advantage and grow stronger, while the rest gradually “fall idle.” Two common countermeasures exist:
- Auxiliary-loss balancing: Beyond the language modeling loss, add a penalty term that pushes each expert’s selection frequency toward uniformity.
- Loss-free balancing: Introduce a dynamically adjustable bias term for each expert’s gating score. Lower the bias when an expert is overloaded and raise it when it is underused—achieving balance without disturbing the main task’s gradient. Existing technical reports treat this as one of the key practices.
Second, training instability. This specifically includes divergence caused by attention scores that keep growing numerically, jitter in expert routing under parallel splitting, and numerical overflow during low-precision training. Common countermeasures include:
- Normalizing Query and Key.
- Backpropagating the auxiliary loss’s gradient only, without logging it, so its values do not drown out the real training loss.
- Monitoring the maximum attention score, and once a particular head’s value grows too large, proportionally shrinking that head’s Query and Key projection weights. Illustration below:
# Illustration: weight correction to stabilize attention scores (simplified pseudocode)if max_attention_score > threshold:for head in affected_heads:gamma = (threshold / max_score[head]) ** 0.5W_q[head] *= gamma # proportionally shrink this head's Query projectionW_k[head] *= gamma # shrink this head's Key projection in sync
2.4 Token-Efficient Training: Coordinating Optimizers, Data, and Precision
High-quality human text is nearing its usable ceiling, and “how much effective information can be learned per token” is gradually replacing “how many tokens to pile up” as the core scaling question. The main industry approaches to improving token efficiency include:
- More efficient optimizers: Matrix-based update optimizers are drawing more attention. They apply approximate orthogonalization to weight matrices as a whole, aiming for faster and more stable convergence under the same training budget. Such methods are more sensitive to update values and often need to be paired with the stabilization techniques above.
- Synthetic data: Under strict validation, generate scarce data such as reasoning traces, code, and tool-calling trajectories to compensate for the shortage of high-quality corpora.
- Low-precision training: Mixed precision that is primarily BF16 with FP8 as a supplement, keeping higher precision for sensitive modules (normalization, routing, accumulation) and maintaining high-precision copies of the weights.
Step t total loss = language modeling loss + balancing loss (gradient only) + auxiliary loss (not logged)Routing bias ← routing bias + update rate × (average load - each expert's load)
2.5 Multi-Stage Post-Training: Activating Reasoning and Tool-Use Capabilities
After pre-training, the mainstream practice is staged post-training:
- Long-context continued pre-training / long-text SFT
- Reasoning-capability SFT
- Large-scale reinforcement learning
- General preference alignment
The most typical route in the reinforcement learning stage is Reinforcement Learning with Verifiable Rewards (RLVR): tasks whose answers can be automatically judged by programs, such as math, code, and logic reasoning, can use rules or test cases directly as rewards. This method does not need a separately trained reward model. The basic framework is:
- Sample a group of answers for the same question.
- Normalize with the group’s reward mean and standard deviation to obtain a relative advantage.
- Update the policy from the relative advantage, clip the probability ratio between the new and old policies, and constrain the distance from the reference model.
# Illustration: reinforcement learning based on group-relative advantage (simplified pseudocode)rewards = [verify(question, answer) for answer in sample_group(policy, question)]baseline = mean(rewards)for answer, r in zip(group, rewards):adv = (r - baseline) / (std(rewards) + 1e-8) # group-relative advantageratio = policy.log_prob(answer) - old_policy.log_prob(answer)ratio = exp(ratio) # new/old policy probability ratioclipped = clip(ratio, 1 - eps, 1 + eps)surrogate = min(ratio * adv, clipped * adv)loss += -surrogate + beta * kl(policy, reference) # policy loss + reference constraint
Once large-scale training begins, new engineering bottlenecks surface:
- Sampling dominates. Every sample requires full generation, making sampling the throughput bottleneck, so a high-throughput inference engine and the training framework must be co-designed—for example, with shared weights and asynchronous pipelines.
- Response length keeps growing. In the later stages of reinforcement learning, the model produces longer reasoning traces, raising memory and compute pressure.
- Policies tend to collapse. Techniques such as entropy regularization, diversity constraints, and difficulty-stratified sampling are commonly used to preserve exploration ability.
- Optimizer choice varies. Some work prefers to reuse mature, stable adaptive optimizers, while others experiment with matrix-based optimizers.
These engineering details rarely make it into technical reports, yet they directly affect final capability—further showing that, given the same architecture, the post-training recipe is where models become distinct.
3. Key Trade-offs Under the Same Skeleton: Where Do the Differences Really Lie?
The table below summarizes common architectural dimensions, what raising them can buy, and the cost to pay. Specific values should follow each model’s public technical report.
| Architectural dimension | Typical benefit of raising it | Main cost and risk |
|---|---|---|
| Total parameters | Knowledge capacity, potential for complex reasoning | Memory, expert-parallel communication, inference hardware threshold |
| Activated parameters | Single-token expressiveness | Compute per token, latency, throughput |
| Number and granularity of experts | Carry more parameters under the same activation compute | Routing difficulty, load imbalance, communication frequency |
| Shared experts | Concentrated general knowledge, more stable routing | The shared part is computed for every token, consuming compute |
| Number of attention heads | Expressiveness | Parameter and compute overhead; with compressed attention this can be reduced further to concentrate the compute budget on the expert layers |
| Compressed latent dimension | Stronger representational power after compression | Larger KV cache |
| Optimizer | Faster, more stable training, better token efficiency | Complex implementation, sensitive tuning |
| Post-training mix | Capability emphasis aligned with target business | High data-construction cost, easy to trade one thing off against another |
Teams may make different choices on any single dimension, yet they are highly consistent across four directions: “sparse decoupling, cache compression, training stability, and post-training enhancement.” Architectural convergence is the result of engineering continually approaching shared constraints; the real differences lie in how each dimension is traded off.
4. From Architecture to Selection: How Developers Evaluate and Deploy
4.1 Capability Profiling: Build Your Own Evaluation Set
When architectures converge, the reference value of both parameter counts and public leaderboards declines. A more reliable approach is to build an evaluation set that fits your business:
- Stratified sampling: Cover typical inputs and long-tail scenarios of real business, starting at a few hundred items.
- A clear judging method: Use accuracy for tasks with standard answers, and human or model-review scoring for open-ended tasks.
- Fixed versions: Record the model version and decoding parameters so that before-and-after comparisons are reproducible.
- Two easily confused metrics:
4.2 Deployment Cost Estimation: Do the Full Math
Inference cost is made up of at least four parts:
- Compute per token (determined by activated parameters).
- Weight memory (determined by total parameters and quantization method).
- KV cache memory (determined by sequence length and attention design).
- Expert-parallel communication overhead.
Take two hypothetical models as examples (values only demonstrate the method):
- Model A: about 1T total parameters, about 30B activated, about 500 cached elements per token per layer.
- Model B: about 200B total parameters, about 10B activated, with a cache element count 4× that of Model A.
# Hypothetical Model A KV cache estimate (FP8, 1 byte per element, 64 layers)concurrency 32 × context 128K = about 4.19M tokensabout 4.19M × 500 × 64 × 1B ≈ 134 GB# Hypothetical Model B KV cache estimate (same workload)134 GB × 4 ≈ 536 GB# Compute-per-token comparison (Model B as the baseline)activated parameter ratio 30B : 10B = 3 : 1
In long-context scenarios, Model A’s KV cache is about one quarter of Model B’s, while its compute is about 3×. The final cost depends on the business’s input-to-output length ratio, context length, and concurrency scale. If requests are short and generation-dominated, activated compute dominates and the smaller model has the edge; in long-document and long-dialogue scenarios, the cache advantage from compressed attention gradually shows. So “more parameters means stronger” cannot simply be equated with “higher cost per unit of quality.”
4.3 Common Pitfalls
- Treating the nominal context length as the effective length. Measure with several kinds of “long-text retrieval and multi-hop reasoning” tasks instead.
- Ignoring the input-to-output ratio. Code completion is input-heavy; generation tasks are output-heavy.
- Ignoring tail latency and looking only at average throughput. Measure both time-to-first-token and streaming output latency.
- Looking only at raw precision and ignoring quantization loss. Evaluate quantized versions such as FP8 and INT8 as well.
- Choosing only one model. Add routing and fallback at the gateway layer: send simple requests to smaller models and complex requests to larger ones.
4.4 Deployment Paths: Managed Service or Self-Hosted Inference
In the cloud, deployment usually follows one of two paths:
- Model-as-a-Service: Call models hosted on a cloud platform directly, skipping cluster operations and scaling on demand.
- Self-hosted inference: Build an inference cluster from general cloud products such as elastic compute, container services, object storage, logging, and monitoring—more controllable, but requiring operational investment.
Take Baidu AI Cloud as an example: developers can quickly validate results through the platform’s model services, and can also deploy open-source models themselves on top of general cloud products. The two paths can be combined—first use managed services for result validation and to absorb peak traffic, then optimize cost on self-hosted clusters for stable traffic.
For self-hosted deployment, the following practices are worth referencing:
- Parallelism strategy: Attention and the shared part suit tensor parallelism, while the numerous routed experts suit expert parallelism, often organized along node boundaries.
- Memory optimization: Paged KV cache management, prefix-cache reuse, FP8 weight quantization, multi-copy weight loading.
- Scheduling optimization: Continuous batching, speculative decoding, and unified orchestration of inference and training services.
- Rollout strategy: Blue-green deployment, plus task-based routing and automatic fallback at the gateway layer.
Compliance note: Some models use custom open-source licenses, such as “must be renamed after modification and redistribution,” so review the license terms carefully before commercial use; private deployment in industries such as finance and healthcare may also need to meet requirements like keeping data within a defined boundary.
5. Trend Outlook and Practical Recommendations
Architectural convergence is a sign that open-source large models are maturing. Once the basic skeleton stabilizes, the focus of competition shifts from “network structure” to three things: data and post-training recipes, inference cost per token, and productization for real business. Predictable directions include:
- Native multimodality: Text, image, and audio trained jointly in a unified expert architecture.
- Longer effective context: Achieved through more efficient attention rather than simply stacking positional-encoding length.
- Agent-oriented training: Reinforcement learning environments moving from single-turn Q&A to multi-turn, multi-tool collaboration.
- Training–inference co-design: Optimizers, low-precision training, and inference engines tuned together to keep pushing token cost down.
For developers, the following are the recommended focus areas:
- Use your business evaluation set as the standard, not public leaderboards.
- Use the model from Section 4.2 to compute per-request and per-token cost.
- Check the training-data cutoff, license, and deployment requirements on the model card.
- Reserve model routing and fallback capability at the gateway layer.
Architectures can converge, but business value will not. Whoever binds convergent model capabilities more tightly to their own scenario’s data, workflow, and evaluation system is the one most likely to gain a real efficiency advantage.
