Large Language Model Improvements: Pre-training, Post-training, and Practical Evaluation
This article explains how large language models improve from preview to generally available releases, and what developers should evaluate. It covers parameter scale, activated parameters, context windows, pre-training and post-training, data flywheels, multimodal capabilities, and a practical evaluation framework for complex instruction following and project-level coding.
1. From Preview to GA: Technical Trade-offs Behind an Iteration
In the large-model industry, major vendors usually release models in two stages: first, a preview version for developers to collect feedback and validate direction; then, weeks or months later, a generally available version. The GA version is often not just a bug-fix release. It often significantly expands pre-training and post-training, adds foundational capabilities such as multimodal understanding, and aims to better fit real productivity scenarios.
For developers, understanding this iteration pattern matters. It affects how you evaluate a model, how you select one for a project, and how you predict its capability boundaries. This article breaks down the core mechanisms behind large-model improvement from four perspectives: parameter scale, training paradigm, data flywheel, and practical evaluation.
2. Parameter Scale and Architecture: Total Parameters, Activated Parameters, and Context Window
Before evaluating a model, it helps to understand three key metrics:
- Total parameters: the capacity of the model weights. It defines the model’s theoretical upper limit and influences training and deployment costs.
- Activated parameters: a metric specific to Mixture-of-Experts (MoE) architectures. MoE models contain many expert subnetworks, but each forward pass activates only a small subset. As a result, a model with hundreds of billions or even trillions of total parameters may require computation per inference call that is closer to a dense model with only a few billion activated parameters.
- Context window: the maximum number of input tokens the model can process in a single request. As windows expand from tens of thousands to millions of tokens, a model can ingest a long document, a large codebase, or a multi-turn long conversation at once. This matters for long-context understanding, repository-level code analysis, and document summarization.
It is important to note that MoE’s value is to provide larger model capacity with controllable inference cost. Total parameters determine the breadth of knowledge, while activated parameters determine the compute cost of a single inference. Decoupling these two makes “large and efficient” possible. Therefore, a combination such as “total parameters in the 700B class with dozens of billions of activated parameters” should not be treated as equivalent in cost to a 700B dense model; it should be evaluated based on the activation ratio.
3. Pre-training and Post-training: Cost, Objectives, and Division of Labor
These are the most important concepts to clarify in this article, and they are key to judging the substance of a model iteration.
Pre-training
Pre-training is the process in which a model performs self-supervised learning on massive unlabeled corpora. Using basic objectives such as next-token prediction, the model learns language patterns, world knowledge, and basic reasoning capabilities from the data. Its characteristics are extremely large data scale, very high compute cost, and long training cycles. It often requires GPU clusters ranging from thousands to tens of thousands of cards running for weeks or longer. Pre-training determines the model’s foundation and ceiling.
Post-training
Post-training builds on an existing base model. Through methods such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), it teaches the model to follow instructions more reliably, complete specific tasks, and align with human preferences. Its characteristics are relatively controllable data, lower cost, and faster iteration.
Because post-training is cheaper and faster, most same-generation model iterations keep the base model unchanged and only optimize post-training. This is an engineering-rational choice, but it also means that the model’s knowledge boundary and foundational capabilities are largely frozen at the previous generation.
4. What It Means When a Same-Generation Model Performs Incremental Pre-training
For this reason, when a model iteration significantly expands pre-training while strengthening post-training, it stands out. It means the vendor is willing to bear high training costs to expand the model’s knowledge boundary and foundational capabilities, rather than merely refining the existing base model.
From a technical standpoint, this combination of incremental pre-training and stronger post-training usually brings three benefits:
- Overall improvement in foundational capabilities: reasoning, understanding, and generation are systematically enhanced rather than optimized only for specific tasks.
- Knowledge and recency updates: expanded pre-training corpora can help the model absorb newer and broader knowledge.
- Stronger generalization: post-training on a stronger base makes instruction following and task generalization easier to achieve at a high level.
The takeaway for developers is that when evaluating model iterations, you should not look only at benchmark scores. You should also examine whether the training paradigm has changed. An iteration that truly expands pre-training often has greater long-term value for adoption.
5. The Data Flywheel: How User Scale Feeds Model Evolution
A leap in model capability is not an isolated technical event. It is often supported by a data flywheel. The data flywheel refers to the following loop: real users generate large amounts of high-quality interaction data while using products such as AI coding assistants and conversational agents. Under compliant, de-identified, and user-authorized conditions, this data is filtered and labeled, then fed back into pre-training and post-training, creating a positive cycle: more users → more real data → stronger models → better experience → more users.
This mechanism explains why leading products can maintain continuous capabilities: they not only have algorithms and compute, but also access to real-world scenario data. When a coding assistant with a large user base is deeply coupled with an in-house large model, both model iteration speed and scenario alignment can benefit significantly.
6. Multimodal Capabilities and Fit for Productivity Scenarios
Another common upgrade from preview to GA is moving from text-only to multimodal: for the first time, the model can process visual information alongside text. Multimodal capability is significant for productivity scenarios:
- It can interpret design mockups, UI screenshots, and error screenshots, assisting frontend and UI work.
- It can understand charts and scanned documents, improving information extraction efficiency.
- It can combine text and images to support more complete task understanding and code generation.
When multimodal capability is combined with long context, a model can handle composite productivity tasks such as reading long documents, viewing charts, and writing code. This is also an important sign of whether a model has engineering usability.
7. How to Evaluate a Large Language Model Scientifically: Complex Instruction Following
Benchmark scores reflect only part of a model’s capabilities. Real engineering selection requires practical evaluation. A high-quality capability test should focus on two dimensions: complex instruction following and project-level coding. The example below shows how to design an evaluation by asking a model to generate a single-file HTML mini-game.
7.1 Evaluation Design
Do not provide a reference prototype. Instead, use a detailed prompt with multiple constraints to test how the model decomposes and implements complex requirements. The main evaluation points include completeness of requirement understanding, correctness of physical logic, state management, exception and boundary handling, and code runnability.
An example of a high-quality evaluation prompt is:
Build a 2D side-scrolling off-road driving mini-game in a single HTML file. Requirements:1. The vehicle must support both acceleration and braking, controlled by keyboard arrow keys or on-screen buttons.2. The UI must display fuel level and vehicle posture in real time.3. Include simple physics: gravity, acceleration, and the effect of terrain slope on vehicle speed.4. Declare game over if fuel runs out or the vehicle overturns and remains stuck, meaning the body angle exceeds a threshold and the vehicle cannot continue.5. Provide a restart option.6. All logic must be inline in HTML/CSS/JS, without external resources, and the file must run by double-clicking it.
7.2 Expected Code Skeleton
A good response should provide a clear and runnable engineering skeleton rather than scattered fragments:
<!DOCTYPE html><html lang="en"><head><meta charset="UTF-8"><title>Side-Scrolling Off-Road Driving Demo</title><style>#stage{background:#87CEEB;display:block;margin:0 auto}</style></head><body><canvas id="stage" width="800" height="400"></canvas><script>const canvas = document.getElementById('stage');const ctx = canvas.getContext('2d');// Vehicle state: position, velocity, angle, fuel, and game-over flagconst car = { x:100, y:300, vx:0, vy:0, angle:0, fuel:100, over:false };const keys = { throttle:false, brake:false };function update(){if(car.over) return;// Throttle acceleration and fuel consumptionif(keys.throttle && car.fuel > 0){ car.vx += 0.3; car.fuel -= 0.05; }// Braking decelerationif(keys.brake){ car.vx *= 0.95; }// Gravity and terrain slope effect (illustrative)car.vy += 0.2;// Posture and overturn checkif(Math.abs(car.angle) > Math.PI/2 && car.vx < 0.1){ car.over = true; }if(car.fuel <= 0){ car.over = true; }}function draw(){ctx.clearRect(0, 0, canvas.width, canvas.height);// Draw terrain, vehicle body, fuel bar, and game-over message}function loop(){ update(); draw(); requestAnimationFrame(loop); }loop();</script></body></html>
7.3 Scoring Dimensions
To avoid vague conclusions such as “it looks good,” score the output quantitatively across the following dimensions:
- Instruction following: whether all constraints are covered, including throttle, braking, fuel, overturn detection, and restart.
- Physical plausibility: whether gravity, slope, and acceleration logic are consistent.
- Engineering completeness: whether the output is a directly runnable single file without missing dependencies.
- Edge cases and exceptions: whether states such as fuel exhaustion, getting stuck, and restarting are handled correctly.
- Code readability: whether naming, comments, and structure are clear and easy to maintain or extend.
This evaluation method can be applied to any model for horizontal comparison or longitudinal tracking of the same model’s iterations.
8. Engineering Practice Recommendations
Based on the analysis above, here are practical recommendations for developers and enterprises:
- Select based on the training paradigm, not only leaderboards. Prioritize whether there is a dual upgrade in both pre-training and post-training, since this often indicates stronger long-term value.
- Replace intuition with practical evaluation. Design multi-constraint tests based on real business scenarios, and quantify model performance in complex instruction following and project-level coding.
- Pay attention to engineering usability. Long context, multimodality, and single-file runnability directly determine whether a model can be embedded into production workflows.
- Use cloud capabilities to support deployment. In real deployments, you can use object storage to host static assets, container platforms to run inference services, and logging services with monitoring and alerting to maintain stability. When using Baidu AI Cloud’s general cloud product capabilities, teams can more efficiently engineer and scale high-quality model capabilities.
Overall, the leap in large-model capability is the result of parameter scale, training paradigm, data flywheel, and multimodal capability working together. By understanding the technical logic behind these changes and applying scientific evaluation methods, developers can make rational and sustainable model selection decisions in a rapidly iterating ecosystem.
