Early Architecture Evolution of Large Language Models: From Tens of Billions of Parameters to Global Deployment

    This article reviews the early architecture evolution of large language models across four phases: architecture exploration and small-scale validation around 2020, academic validation and reproducibility around 2022, dialogue model release and engineering deployment around 2023, and global deployment across data, evaluation, and infrastructure. It summarizes the goals, practices, and risks at each stage and provides a reusable checklist for R&D teams and technical decision-makers. The methods cited are available in public papers; teams should revalidate them with their own data and compute.

    The rapid adoption of large language models (LLMs) makes them look like a technology that appeared overnight. In practice, some teams began systematic exploration around 2020 and advanced algorithms, training, evaluation, and product work along the same chain. This article follows that chain and reviews the goals, practices, and risks to check in advance at each stage for R&D and technical decision-makers. The methods mentioned here can be found in public papers; teams need to revalidate them with their own data and compute.

    Around 2020: Architecture Exploration and Small-Scale Validation

    Traditional autoregressive language models generate text from left to right, and each position can see only the preceding content. This works well for continuation and dialogue, but for tasks such as fill-in-the-blank, classification, and extraction that require understanding the whole context at once, information utilization is relatively low. Some teams therefore began trying to combine understanding and generation in the same model.

    There are public reference implementations for this direction. UniLM (NeurIPS 2019) designs different attention masks so that the same Transformer uses bidirectional or unidirectional attention at different positions. GLM (ACL 2022) uses autoregressive blank infilling: it first masks several spans in the input and then restores them segment by segment in autoregressive order. Both show that hybrid attention structures can balance understanding and generation, but their applicable tasks and training costs differ.

    In practice, teams usually start with small-scale validation and then scale up gradually. The following steps can be used:

    1. On corpora ranging from hundreds of millions to billions of tokens, compare the new mask with a baseline model on the same validation set in terms of perplexity and downstream task scores.
    2. Fix the random seed and data order, and compare the impact of different positional encoding schemes (for example, absolute positional encoding, relative positional encoding, and rotary positional encoding) on long text.
    3. Record the loss curve for each change and confirm that there is no persistent oscillation or gradient explosion.

    Training stability is the most common issue at this stage. Loss oscillation often comes from a learning rate that is too high, insufficient warmup steps, or anomalous samples in the data. Common handling methods include extending warmup, using gradient clipping, adopting mixed-precision training, and removing duplicate and overly long samples during data cleaning. Mixed precision reduces numerical precision, and some numerically sensitive layers may need to retain single-precision computation. Memory pressure in long-sequence training can be relieved through gradient checkpointing and sequence parallelism, but both increase computation or communication cost, so the tradeoff must be made according to hardware conditions.

    Before entering the tens-of-billions-of-parameters scale, it is advisable to confirm three things: whether gains on small models can be maintained, whether training can run stably for at least one complete cycle, and whether inference latency can be kept within an acceptable range. If any of these is not met, continuing to scale up usually only amplifies existing problems.

    Around 2022: Academic Validation and Reproducibility

    After an architecture works internally, public papers are an important way to test its value. A submission usually needs to answer three types of questions: how large the improvement is over strong baselines, whether training details can be reproduced by others, and whether the model is balanced on both understanding and generation tasks. Reviewer comments often point out issues overlooked in engineering, such as inconsistent evaluation protocols, hyperparameters that are too sensitive to initialization, or baselines that have not been tuned to the same level.

    The value of academic submission goes beyond acceptance. The process of organizing experimental settings itself turns experience into standards. Teams can document the following items to form a reusable training manual:

    • Data mixture: proportions of corpora by language and domain, as well as deduplication and filtering rules.
    • Learning rate schedule: peak learning rate, warmup steps, decay method, and minimum learning rate.
    • Checkpoint strategy: save interval, retention count, and steps for rolling back from an abnormal state.
    • Evaluation protocol: dataset version, prompt templates, decoding parameters, and random seeds.

    Teams planning public release should list a complete hyperparameter table in the paper and provide access to code or configuration files. When citing, use the official conference or journal name and publication year, and check differences between the preprint and the formal version. Evaluation results should report both mean and variance, rather than showing only the best run.

    An acceptance criterion for this stage can be: whether another engineer can reproduce the main results on comparable hardware using the documentation alone, with error within an acceptable range. If not, the documentation still has gaps.

    Around 2023: Dialogue Model Release and Engineering Deployment

    After the base model is stable, teams usually release a dialogue-oriented version. The core work at this step has three parts: instruction tuning, so the model learns to follow the format of user instructions; human feedback alignment, so the style and quality of answers match preferences; and safety filtering, so harmful requests are refused and inappropriate content is reduced. These three parts affect one another. Excessive safety filtering may cause the model to refuse normal questions, while weak filtering brings compliance risks. Both types of errors need to be continuously monitored with independent test sets.

    There are three engineering directions that need particular attention.

    The first is inference efficiency. Quantization can compress weights from 16-bit to 8-bit or 4-bit and significantly reduce memory usage. However, low-bit quantization loses accuracy on some tasks, so regression testing on target tasks is required before launch. Speculative decoding first has a small model generate candidate tokens and then has the large model verify them in batches, which can reduce latency in some scenarios. Its benefit depends on how close the output distributions of the small and large models are; when they differ greatly, the speedup may be limited.

    The second is multi-turn consistency. In long conversations, early instructions or user settings are easily forgotten. Teams can retain a summary of key information at each turn, fix the position of the system prompt, and design multi-turn test cases. The test cases should cover cross-turn coreference, condition changes in the middle of a conversation, and detail recall in long contexts.

    The third is safety and compliance. The refusal mechanism needs to be designed in tiers, handling clearly illegal content, privacy leakage, and general controversial topics separately. Content moderation rules should have version records, and after each change, historical test sets should be rerun to prevent old issues from reappearing.

    Open-source release is also part of engineering. Weight files, licenses, usage instructions, and known limitations should be provided together. When documentation is incomplete, secondary developers often give up because of configuration errors, and community activity declines accordingly. Before release, an external developer can be asked to deploy from scratch according to the documentation, record blockers, and fix them.

    Global Deployment: Synchronizing Data, Evaluation, and Infrastructure

    Internationalization is not just interface translation. It involves at least three layers: data, evaluation, and deployment.

    At the data layer, corpora covering multiple languages and cultural backgrounds are needed, and the sparsity of samples for low-resource languages must be handled. Common practices include: completing base training mainly with English and Chinese, then extending other languages through continual pretraining or adapters; using backtranslation or manual annotation to supplement samples for low-resource languages; and recording the data source and licensing status for each language.

    At the evaluation layer, scores in a single language are not enough. An independent test set should be established for each target language rather than simply translating existing questions. Evaluation results need to be reported separately by language and task type, so that an average score does not hide an obvious weakness in a particular language.

    At the deployment layer, latency, data storage, and compliance requirements in each region need to be considered. Common metrics include first-token latency, tokens generated per second, and round-trip time for cross-region requests. Some regions require user data to be stored domestically, which affects the design of inference nodes and logging systems. Export controls, content safety, and intellectual property boundaries should also be reviewed early in the project and confirmed by legal before proceeding.

    Reusable Checklist

    The following checklist items can be used in retrospectives at each stage. Each item should have a clear owner and verifiable evidence, such as experiment records, test reports, or audit files.

    Architecture validation:

    • The new structure and the baseline are compared under the same data and compute.
    • Masks and positional encoding have unit tests covering boundary lengths.
    • Gains from small-scale experiments are recorded after scaling up.

    Training stability:

    • Loss curves, gradient norms, and learning rates are monitored with alerts.
    • Training can resume from the latest checkpoint and produce consistent subsequent loss.
    • Data cleaning rules have version records.

    Evaluation reproducibility:

    • Evaluation scripts, prompt templates, and decoding parameters are archived.
    • A third party can reproduce the main results in a similar environment based on the documentation.
    • Metrics include mean, variance, and sample size.

    Internationalization and compliance:

    • Each target language has an independent test set.
    • The mapping between data storage locations and local regulations has been confirmed by legal.
    • Content safety rules are validated separately for each language.

    Conclusion

    The value of early exploration lies in turning uncertain ideas into testable processes. For teams today, directly copying past architectures may not be appropriate, but their rhythm can be borrowed: validate the structure at small scale first, then scale up steadily, include papers, open source, and product in the same plan, and consider internationalization requirements from the beginning of the project.