Common Architecture and Training Techniques in Mainstream LLMs

    This article explains why mainstream large language models converge on similar designs and examines the architecture and training choices behind that convergence. It covers MoE routing and sparsity, attention-head and KV-cache trade-offs, token efficiency, optimizer selection, training stability, synthetic data practices, and a reproducible framework for comparing models and planning deployment.

    1. Introduction: Why Architectures Converge

    Large language models released over the past two years show a high degree of similarity in public technical materials. Most mainstream approaches use Mixture of Experts (MoE) structures, attention mechanisms commonly compress memory and bandwidth for inference, and training pipelines increasingly emphasize data efficiency. This convergence has two main causes. First, these designs have been repeatedly validated in large-scale experiments, and the balance between performance and cost is relatively clear. Second, hardware conditions and inference service constraints are similar, so the same bottlenecks naturally lead to similar solutions.

    When comparing different models, therefore, it is not enough to look only at names or marketing claims. Each layer of design should be examined to understand what problem it solves. The following sections analyze this from three perspectives: architecture, training, and engineering.

    2. Core Architecture Design

    2.1 Basic Principles of Mixture of Experts

    In a dense model, all parameters participate in every forward computation. As parameter count grows, training and inference costs increase linearly. A Mixture of Experts structure splits the feed-forward network into multiple expert sub-networks, and a router selects a small number of experts for each token. The total parameter count can therefore be very large, while the number of parameters actually activated per token is only a small fraction.

    The following simplified top-k routing example illustrates the computation flow:

    1. import torch
    2. import torch.nn.functional as F
    3. def moe_forward(x, router, experts, top_k=2):
    4. # x: [tokens, hidden]
    5. logits = router(x) # [tokens, num_experts]
    6. weights, indices = torch.topk(logits, top_k, dim=-1)
    7. weights = F.softmax(weights, dim=-1) # Normalize only over selected experts
    8. output = torch.zeros_like(x)
    9. for k in range(top_k):
    10. for e, expert in enumerate(experts):
    11. mask = indices[:, k] == e
    12. if mask.any():
    13. output[mask] += weights[mask, k:k+1] * expert(x[mask])
    14. return output

    Actual engineering implementations add load-balancing losses, expert capacity limits, and distributed communication scheduling. The code above retains only the core logic.

    Assume a model has 1 trillion total parameters and activates 30 billion parameters per token. The activation ratio is then about 3%. This ratio determines the computation per inference pass, while the total parameter count determines the knowledge capacity the model can hold. The gap between the two is precisely the value of sparse structures.

    2.2 Expert Count and Sparsity

    With activated parameters fixed, increasing the number of experts raises the ratio of total parameters to activated parameters, that is, sparsity. Industry experience suggests that within a certain range, higher sparsity usually reduces loss, but the benefit diminishes with scale. At the same time, routing instability, uneven expert load, and higher communication overhead become increasingly prominent.

    Engineering trade-offs include:

    • As the number of experts increases, the capacity of each expert becomes smaller, requiring more careful load-balancing strategies.
    • Distributing experts across devices amplifies all-to-all communication latency, and network bandwidth may become a bottleneck.
    • Inference memory demand depends on total parameter count. Even if only a few parameters are activated, storage must be reserved for all experts.

    Sparsity should therefore be determined together with training set size, hardware topology, and inference deployment method, rather than pursued as a standalone numerical target.

    2.3 Attention Heads and Inference Cost

    The number of attention heads directly affects KV cache size and attention computation cost. In long-context scenarios, the KV cache is often the main component of memory usage. Reducing the number of attention heads lowers the cache cost per token and reduces attention-layer computation, but too few heads limit the model’s ability to capture multiple relationships simultaneously.

    This trade-off requires empirical testing. A common approach is to compare loss curves on the validation set for different head configurations under the same parameter budget, while recording single-GPU throughput and first-token latency, and then choose the configuration with the best overall performance.

    2.4 KV Cache Compression

    Another common approach is to compress the KV cache itself. The low-rank projection idea maps keys and values into a lower-dimensional latent space, stores only the compressed representation in the cache, and reconstructs it when computing attention. The core benefit is reduced inference memory usage, at the cost of additional projection computation and implementation complexity.

    To judge whether such a scheme is worth adopting, three metrics matter: the memory usage curve under long context, the maximum number of concurrent requests at the same latency, and quality changes on standard evaluation sets. Conclusions are reliable only when all three are evaluated together.

    3. Key Training Techniques

    3.1 Token Efficiency Becomes a Core Constraint

    The growth of high-quality text data has not kept pace with model scaling. In this situation, how well a model can be trained from the same data is more important than simply increasing data volume. Token efficiency refers to the marginal contribution of each token to model capability. There are three main ways to improve it: optimizing training objectives and optimizers, improving data cleaning and mixing, and introducing validated synthetic data.

    3.2 Optimizer Choice

    Most current large-model training still uses adaptive optimizers as a baseline. They adjust step sizes by maintaining first- and second-order momentum for each parameter. Recently, another class of optimizers has attempted to orthogonalize the momentum matrix so that updates in different directions are more balanced, thereby achieving faster convergence with the same number of tokens.

    When choosing an optimizer, consider the following questions:

    1. Can it stably reproduce the baseline convergence curve on smaller models?
    2. How transferable are the hyperparameters? Do learning rate and weight decay need to be re-searched after scaling the model?
    3. In distributed training, are the communication and storage overheads of optimizer states acceptable?

    3.3 Training Stability and Logit Clipping

    During large-scale training, attention logits occasionally explode numerically, causing a sudden loss increase or even training divergence. A common stabilization method is to clip the inner product of queries and keys, scaling it proportionally when it exceeds a threshold.

    Example code:

    1. def clip_attention_scores(scores, max_logit):
    2. # scores: [batch, heads, q_len, k_len]
    3. current_max = scores.amax(dim=(-1, -2)).detach() # Maximum per head
    4. scale = torch.clamp(max_logit / (current_max + 1e-6), max=1.0)
    5. return scores * scale[..., None, None]

    The key is setting the threshold. If it is too small, normal attention distributions are suppressed; if it is too large, it does not stabilize training. In practice, the threshold is usually determined by observing historical logit distributions at each layer in training logs, with monitoring during early training.

    3.4 Principles for Using Synthetic Data

    Synthetic data can supplement samples in scarce domains and can also convert existing high-quality corpora into forms more suitable for training, such as rewriting, reorganization, and reasoning-chain expansion. However, synthetic data can easily amplify the biases of the generating model. Without constraints, it can lead to distribution collapse.

    Reliable practices include controlling the proportion of synthetic data in the total corpus, filtering samples with an independent quality classifier, and retaining real data as an anchor. The real validation set should be used periodically to check whether the model shows capability degradation.

    4. How to Compare Different Models Objectively

    When comparing models, it is advisable to establish a unified evaluation framework to avoid being misled by a single metric or marketing language. The following dimensions can be checked one by one.

    Dimension Focus Common evidence
    Architecture parameters Total parameters, activated parameters, number of experts, attention heads, hidden dimension Official technical reports and model configuration files
    Training recipe Pretraining token volume, optimizer, learning rate schedule, data mix Training chapters in technical reports
    Inference cost Memory usage, single-GPU throughput, first-token latency Own stress tests or independent third-party tests
    Capability performance General, code, math, long-text evaluations Public benchmarks and internal business evaluation sets
    Alignment and safety Instruction following, refusal boundaries, harmful content control Human evaluation and red-team test results

    Note that public benchmark scores are affected by prompt templates, decoding parameters, and data contamination. The same model may produce quite different results under different evaluation frameworks. Cross-team comparisons should therefore fix evaluation scripts and versions.

    At the same time, similar architecture does not mean identical capability. Data quality, post-training pipelines, and inference service optimization all significantly affect the final experience. Two models with close architectures may perform very differently in practice.

    5. Engineering Deployment Recommendations

    5.1 Model Selection Process

    When choosing a large model, enterprises can proceed as follows:

    1. Define the core metrics for the business scenario, such as question-answering accuracy, code pass rate, or long-document summary quality.
    2. Estimate the required memory and compute based on concurrency, latency requirements, and budget.
    3. Build a small evaluation set from real business samples, covering common requests and edge cases.
    4. Stress-test candidate models and record the relationship between quality and cost.
    5. Determine the final solution based on data compliance and deployment method.

    5.2 Basic Structure of an Evaluation Script

    Evaluation should be reproducible. The following is a simple evaluation loop example:

    1. def evaluate(model, dataset, metric_fn, decode_cfg):
    2. results = []
    3. for sample in dataset:
    4. prediction = model.generate(
    5. sample["prompt"],
    6. temperature=decode_cfg["temperature"],
    7. max_tokens=decode_cfg["max_tokens"],
    8. )
    9. results.append(metric_fn(prediction, sample["reference"]))
    10. return sum(results) / len(results)

    Fixing decoding parameters, recording random seeds, and saving raw outputs are basic requirements for traceable results.

    5.3 Common Pitfalls

    • Equating parameter scale with capability. Parameter count is only part of the upper bound; data and training methods are equally important.
    • Looking only at a single leaderboard ranking. Leaderboards may be disconnected from business tasks, so internal data should be the basis.
    • Ignoring inference cost. If training-stage advantages cannot translate into acceptable service latency, deployment is difficult.
    • Ignoring training stability. Divergence or loss spikes waste substantial compute, so monitoring and rollback mechanisms are essential from the start.

    6. Key Takeaways

    The commonality of mainstream large models lies in using sparse structures to balance capacity and cost, using attention compression to reduce inference cost, and using more efficient optimization and data methods to improve token utilization. To judge a model’s value, return to specific architecture parameters, training recipes, and measured data, and combine them with business requirements to form reproducible evaluation conclusions.