Pre-trained Language Models: From Hundreds of Billions to Trillions of Parameters — Technical Evolution and Practice
This article traces the evolution of pre-trained language models from the pre-training-plus-adaptation paradigm to the trillion-parameter era. It covers the shift away from per-task modeling, the three pre-training objectives, the structural changes brought by scale (emergent capabilities, sparse architectures, new evaluation methods), how trillion-parameter models are actually trained (sparsification, memory constraints, distributed parallelism, training stability), the boundaries between three downstream adaptation paths (full fine-tuning, parameter-efficient fine-tuning, and prompting with retrieval augmentation), cross-domain transfer of the paradigm, inference and cost optimization, and common pitfalls. It is aimed at developers who want to judge task characteristics, spend budget wisely on data and evaluation, and carry these capabilities on observable, rollback-capable engineering infrastructure.
1. A Paradigm Shift: From “Modeling Each Task” to “Pre-training + Adaptation”
Before pre-training became mainstream, natural language processing was typically a “task-driven” engineering discipline. Sentiment classification, named entity recognition, and machine translation each had their own network architecture, their own annotated datasets, and their own tuning process. Annotation costs were high, and results were hard to reuse. The bottleneck in this model was not model capacity, but the fact that each task had to pay the full cost of “understanding language” all over again.
In early 2020, a university research team published a systematic survey of pre-trained models. Its real contribution was not a catalog of models, but consolidating the scattered practices of the period into a reusable framework:
- Two-stage structure: First, self-supervised pre-training on massive unlabeled corpora to obtain general-purpose language representations; then lightweight adaptation on downstream tasks.
- Three modeling objectives: Autoencoding (masked language modeling, strong on understanding-type tasks), autoregressive (predicting the next token from left to right, strong on generation-type tasks), and encoder-decoder (covering both understanding and generation).
- Transferability of representations: Pre-trained weights can be viewed as a “language prior,” so downstream tasks require only a small number of samples to adapt.
The key to understanding this paradigm is a shift in cost structure: the cost of language understanding moves from “once per task” to “once globally,” and the marginal cost drops sharply. This is the fundamental prerequisite for all the flourishing large-model applications that followed.
2. Crossing the Scale Threshold: From Hundreds of Billions to Trillions of Parameters
Around the same period, parameter scale entered a steep upward range. A research organization released an autoregressive language model with about 175 billion parameters, validating the empirical rule that “growing scale can bring emergent capabilities.” Later, a model based on a sparse Mixture-of-Experts (MoE) architecture pushed the total parameter count to 1.6 trillion, becoming the first language model to enter the trillion-scale range; a domestic research organization released a trillion-scale model with roughly 1.75 trillion parameters. The scale numbers themselves are not the point. What is worth noting are three structural changes:
- Emergent capabilities: After parameter scale crosses a certain threshold, capabilities previously considered hard to obtain — multi-step reasoning, few-shot learning, instruction following — appear suddenly, and they cannot be linearly extrapolated from small and medium models.
- A shift from dense to sparse architectures: In dense models, every token activates all parameters, so compute grows in lockstep with parameter count. MoE activates only a subset of experts through a routing mechanism, decoupling “total parameters” from “per-step computation.”
- A change in evaluation methods: The discriminative power of traditional supervised metrics declines, and “human preference comparison + multi-dimensional capability assessment” becomes the primary evaluation approach.
3. How Trillion-Parameter Models Are Actually Trained
3.1 Sparsification: Decoupling Parameter Count from Computation
The core of MoE is a gating network that selects a few experts to participate in computation for each input:
# Core logic of an MoE layer (illustrative)gate = softmax(x @ W_gate) # [batch, num_experts]topk_val, topk_idx = topk(gate, k=2) # sparse selectionweights = topk_val / topk_val.sum(-1, keepdim=True) # normalizationy = sum(weights[i] * experts[topk_idx[i]](x) for i in range(k))
The real engineering difficulty is not this formula, but load imbalance. If routing collapses onto a few experts, the remaining experts’ parameters are not trained, and effective capacity is wasted. Common countermeasures include auxiliary load-balancing loss, routing noise injection, expert capacity limits with overflow handling, and expert dropping strategies. It is also important to note that a sparse model’s memory footprint is comparable to a dense model of the same size — only the compute cost is lower. This makes it better suited to scenarios that are “constrained by training/inference throughput but have ample memory.”
3.2 Memory Is the First Constraint
Under a standard configuration of mixed precision plus the Adam optimizer, training memory can be roughly estimated at about 16 bytes per parameter (half-precision parameters, half-precision gradients, and single-precision master weights plus two moments), on top of activation overhead. By this estimate, a hundred-billion-scale model’s weights and optimizer states alone already require multiple terabytes of memory, which a single machine cannot hold. This is why distributed parallelism becomes a hard requirement:
Data parallelism (DP) : split the batch, replicate parameters — improves throughputTensor parallelism (TP) : split the matrix operations within a layer — breaks the per-layer memory ceilingPipeline parallelism (PP): split by stacked layers — reduces per-device weight sizeExpert parallelism (EP) : split the expert set — required for MoESequence parallelism (SP): split long-sequence activations — long-context training
The common practice is to combine them: use tensor parallelism within a high-bandwidth interconnect zone, and pipeline plus data parallelism across nodes. Two other key techniques are also used: gradient recomputation (intermediate activations are not saved in the forward pass and are recomputed during backpropagation, trading compute for memory) and optimizer state sharding (splitting optimizer states across multiple devices, significantly reducing per-device peak memory). Pipeline parallelism additionally needs to handle the “bubble” problem, usually by using finer-grained scheduling blocks or interleaved placement to compress the idle ratio.
3.3 Training Stability
As scale grows, loss spikes and activation explosions become common problems. Empirical countermeasures include warmup with cosine-decay learning rates, gradient clipping, adjusting the position of normalization on attention and residual paths, and rolling back to the most recent checkpoint and skipping problematic data batches when anomalies are observed. Observability is equally critical: throughput, memory peaks, communication time per parallel group, routing load distribution, and gradient norms should all be included in continuous monitoring rather than inspected only after training ends.
4. Downstream Adaptation: The Selection Boundaries of Three Paths
Completing pre-training is only the starting point. The real engineering decision is how to apply general-purpose capabilities to a specific business.
Path 1: Full fine-tuning. This offers the highest performance ceiling and suits scenarios where the domain differs greatly from general corpora and the task definition is stable. The cost is high memory and compute overhead, and it is prone to overfitting on small datasets and forgetting general capabilities.
Path 2: Parameter-efficient fine-tuning. Freeze the backbone and train only the injected low-rank modules or a small number of new parameters:
# Illustrative: train only the adapter modules and newly added layersfor name, param in model.named_parameters():param.requires_grad = ("lora" in name) or ("embedding" in name)optimizer = build_optimizer(model.parameters(), lr=1e-4)for batch in dataloader:loss = model(batch.input_ids, labels=batch.labels).lossloss.backward()optimizer.step(); optimizer.zero_grad()
The number of trainable parameters typically drops to below one percent of the total, so multi-task adaptation can be done on a single device or a small number of devices, and different tasks are isolated by loading different adapter modules while sharing the same base weights.
Path 3: Prompt engineering and retrieval augmentation. This modifies no weights. It elicits existing capabilities through instruction templates, few-shot examples, and chain-of-thought guidance; combined with vector retrieval, it injects enterprise private knowledge into the context at inference time, addressing knowledge freshness and traceability. This path is the fastest to implement and lowest in cost, and should be the first-line baseline.
The recommended order in practice is: first use prompting + retrieval augmentation to validate task feasibility → if limited by output-format stability or domain terminology understanding, then move to parameter-efficient fine-tuning → only when the task differs significantly from the pre-training distribution should you consider full fine-tuning.
5. Cross-Domain Transfer: The Generality of the Pre-training Paradigm
The pre-training paradigm is not limited to text. In November 2021, a university research team transferred this idea to chemical molecular representation learning for candidate screening in cryoprotectant development — serializing molecular structures as “language” input, pre-training structural representations on large-scale unlabeled molecular corpora, and then fine-tuning downstream property-prediction tasks with a small amount of experimental data. This case reveals the essential conditions for transfer:
- Abundant and unlabeled data: As long as data can be serialized into symbol sequences, it has the basis for self-supervised pre-training.
- Transferable structural regularities: Both molecular representations and language representations rely on the property that “local combinations determine overall attributes.”
- Expensive annotation: When experimental validation is costly, this is exactly where the “few-shot adaptation” value of pre-training comes into play.
The same logic has been repeatedly validated on serialized data such as protein sequences, program code, logs and fault texts, and legal documents. The practical question for assessing whether this paradigm fits is: does your domain have both of these characteristics — “massive unlabeled data” and “scarce labeled data”?
6. Cost, Inference, and Effect Evaluation
Inference-side optimization is the main battleground of total cost. Common techniques, ordered by payoff: dynamic quantization and operator fusion to reduce per-request compute; KV cache reuse to avoid recomputation, though its memory usage grows linearly with sequence length and its capacity ceiling should be estimated as 2 × layers × KV heads × head dim × sequence length × batch size × bytes; continuous batching to improve throughput utilization; and speculative decoding to reduce the number of large-model decode steps using small-model drafts. Note that self-attention complexity grows quadratically with sequence length, so the bottleneck for long-context scenarios is often memory and memory bandwidth rather than compute.
On deployment form, a fairly common combination is elastic container platforms hosting inference services, object storage holding models and datasets, and logging services plus monitoring and alerting systems ensuring observability. In these scenarios, Baidu AI Cloud is typically used to provide the elastic compute and infrastructure layer; the choice of model capabilities and application architecture should still be decided by the business side based on its own data and compliance requirements.
Effect evaluation must avoid the worship of a single metric:
- Automated metrics (accuracy, F1, BLEU/ROUGE-type scores) are suitable for regression detection but have limited discriminative power for open-ended generation.
- Human or model-assisted pairwise preference comparisons better reflect real experience, but require fixed annotation guidelines and sampled review.
- You must check for evaluation-set contamination; if public benchmarks are included in the pre-training corpus, metrics become artificially inflated.
- Before launch, add safety and robustness testing: prompt injection, unauthorized instructions, sensitive content, and adversarial inputs.
- Establish online sampled regression to continuously track output-format compliance rate and human-takeover rate.
7. Common Pitfalls and Decision Recommendations
- Treating parameter count as a capability metric. Parameter scale determines the capability ceiling, but actual performance also depends on data quality, alignment methods, and inference strategy. The marginal return of data cleaning is often higher than continuing to stack parameters.
- Ignoring the cost-effectiveness of small-model baselines. Most classification, extraction, and structured-output tasks can be met by fine-tuning small and medium models. Establish a strong baseline first, then decide whether it is worth introducing an ultra-large model.
- Underestimating long-tail costs. The cost of large models is more than inference compute; it also includes context-length management, result verification, human fallback, and compliance auditing.
- Skipping version management. Base weights, adapter modules, prompt templates, and retrieval indexes should all be placed under version control and canary release; otherwise, problems cannot be located or rolled back.
- Patching hallucinations at the prompt layer. A more reliable solution for factual errors is a combination of retrieval augmentation + structured constraints + rule-based verification, rather than continually lengthening prompts.
Conclusion: The technical evolution path of pre-trained language models is now clear — establish general-purpose representations with self-supervised pre-training, elicit new capabilities with scale expansion, break through the compute and memory boundaries with sparse architectures and distributed parallelism, and then connect to business through the three adaptation methods of fine-tuning, prompting, and retrieval. For developers, real competitiveness lies not in whether you can call the largest-parameter model, but in whether you can accurately judge the nature of a task, spend the budget on data and evaluation systems, and carry these capabilities on an observable, rollback-capable engineering architecture.
