Designing Learning Rate Schedules for Multi-Stage Model Training: Practices and Optimization

    This article explores how learning rates should be designed and adjusted across multi-stage training pipelines for large-scale models. It covers the rationale behind staged training, the distinct role of learning rates at each phase, common pitfalls during stage transitions, and practical strategies such as warmup, parameter grouping, and optimizer state handling. A reproducible tuning workflow and a generic training framework are also provided.

    In large-scale model training, the learning rate is not a static parameter that remains constant from the first step to the last. This is especially true when training is divided into multiple stages, each serving a fundamentally different purpose. One stage may focus on learning general-purpose representations, another on adapting to a specific data distribution, and yet another on strengthening complex reasoning, tool use, or task execution. Applying the same learning rate strategy across all stages often makes it difficult to simultaneously achieve fast convergence, stable training, and strong final generalization.

    The core challenge of multi-stage training is therefore not “what value should the learning rate be?” but rather answering three more fundamental questions:

    1. What is the training objective of the current stage?
    2. How risky is the parameter update at this point?
    3. Do the capabilities acquired in previous stages need to be preserved, transferred, or rebuilt?

    Why Multi-Stage Training Is Necessary

    Single-stage training typically assumes that the data distribution and optimization objective remain relatively stable. In real-world tasks, however, training data often exhibits clear hierarchical structure.

    For example, a model may first be trained on a large, general-purpose corpus to learn language, vision, or cross-modal representations. It may then be fine-tuned on high-quality data for capability calibration, and finally trained with preference data, task feedback, or environment interaction to learn complex decision-making policies. These stages differ in data volume, quality, noise level, difficulty, and objective function.

    Mixing all data together in a single training run can lead to several problems:

    • High-quality data is diluted by large amounts of low-quality samples;
    • Late-stage task objectives fail to effectively change model behavior;
    • Gradient directions from different data sources conflict with each other;
    • A compromise learning rate must be chosen to accommodate all data;
    • The model remains strongly influenced by early data distributions late into training.

    The value of multi-stage training lies in decomposing a complex optimization problem into several more controllable subproblems. Each stage has a well-defined data recipe, loss function, parameter update scope, and learning rate strategy, which improves the interpretability of the overall training process.

    The Role of the Learning Rate at Different Stages

    The learning rate determines how far parameters move along the gradient direction. It affects not only training speed but also whether the model forgets previously acquired capabilities.

    In the early stages of training, the model parameters are far from the optimal region. A relatively large learning rate helps build basic representations quickly. However, gradient noise is also typically high at this point, and an excessively large learning rate can cause severe loss fluctuations or even destroy structure already present in the initialization.

    In the middle stages, the model gradually enters a more stable optimization region. The primary role of the learning rate is to balance exploration and convergence. It is usually necessary to progressively reduce the update magnitude, allowing the model to shift from coarse-grained learning to fine-grained fitting.

    In the later stages, the model already possesses strong foundational capabilities, and training focuses on local refinement. A large learning rate can cause the model to oscillate around the optimum and may also lead to catastrophic forgetting of general abilities. Therefore, later stages typically require a smaller learning rate, stricter data filtering, and clearer stopping criteria.

    A multi-stage learning rate schedule can be abstracted as:

    [
    \eta(t)=\etai \cdot g_i(t), \quad t\in [T{i-1},T_i)
    ]

    where (\eta_i) is the base learning rate for stage (i), (g_i(t)) is the decay function within that stage, and (T_i) marks the stage boundary. Stage boundaries need not be determined solely by training steps; they can also be defined by validation metrics, data epochs, or capability evaluation results.

    Easily Overlooked Issues at Stage Transitions

    1. Does the Learning Rate Need a New Warmup?

    When the data distribution or loss function changes significantly, directly continuing with the very small learning rate from the end of the previous stage may prevent the model from learning anything new. Conversely, jumping back to a large initial learning rate can disrupt capabilities acquired in the previous stage.

    A safer approach is to introduce a short warmup at the stage boundary:

    [
    \eta(t)=\eta{\text{target}}\cdot
    \min\left(1,\frac{t}{T
    {\text{warmup}}}\right)
    ]

    The warmup does not need to be long. Its main purpose is to reduce the risk of parameter updates when gradient statistics change abruptly. If the new stage has a small dataset, the warmup length should be shortened accordingly to avoid wasting computation at low learning rates.

    2. Should Optimizer States Be Inherited?

    Optimizers typically maintain first-order momentum, second-order statistics, and other states. These states reflect the gradient history of the previous stage. But when the data distribution changes, historical statistics may no longer be reliable.

    Common strategies include:

    • Fully inheriting optimizer states: suitable when the stage shift is small;
    • Inheriting only parameters, not momentum: suitable when the data distribution changes significantly;
    • Decaying states before inheritance: a middle ground between continuous training and task switching;
    • Handling parameter groups differently: for example, retaining stronger historical information for base layers while resetting states for newly added modules.

    It is not always true that “inheriting states is more stable” or “resetting states is always better.” Decisions should be based on loss curves, gradient norms, and stage objectives.

    3. Does the Learning Rate Jump Cause a Spike?

    If the learning rate is near zero at the end of the previous stage and jumps to a large value at the start of the next, parameter updates will show a noticeable spike. This can cause a sudden increase in loss, especially in scenarios involving small datasets, high-quality data, or reinforcement feedback.

    When switching stages, the following metrics should be tracked:

    • Current learning rate;
    • Global gradient norm;
    • Gradient norm per parameter group;
    • Training loss and validation loss;
    • Ratio of parameter update magnitude to parameter magnitude;
    • Results of key capability evaluations.

    If both gradient norm and loss spike abnormally after a stage transition, check the learning rate, data batch composition, and optimizer states first—rather than blindly increasing the number of training steps.

    Typical Learning Rate Strategies for Different Stages

    1. Pre-training Stage: Rise Then Decay

    Pre-training stages typically use large amounts of data with a relatively stable objective. A scheme of “short warmup followed by continuous decay” is commonly adopted. The warmup stabilizes the initial phase of training, while the decay helps the model converge gradually.

    Common decay forms include linear decay, cosine decay, and piecewise decay. There is no absolute conclusion about which is best; the key is whether the training budget and validation metrics align.

    It is important to note that a learning rate curve cannot be discussed independently of the effective number of training steps. If the number of training steps changes, the learning rate schedule usually needs to be rescaled. The same final learning rate does not imply the same optimization trajectory.

    2. Adaptation Stage: Smaller Base Learning Rate

    Adaptation stages typically use higher-quality data with a narrower distribution. At this point, the model already possesses foundational capabilities. The goal is not to relearn everything, but to adjust output style, task format, or domain behavior.

    This stage suits a smaller base learning rate, with different update magnitudes for different parameter groups. For example, base layers can use a lower learning rate while task-related layers use a relatively higher one. This reduces the risk of forgetting general capabilities while accelerating adaptation to the new task.

    3. Reinforcement or Feedback Stage: Stability First

    When the training objective depends on model feedback, environment rewards, or preference signals, gradient noise tends to be larger and training can become unstable. The learning rate should not be increased in isolation, as the model may quickly amplify incorrect behaviors.

    A more sensible approach is:

    • Control the magnitude of each update;
    • Clip anomalous rewards or gradients;
    • Use a smaller learning rate with more frequent evaluations;
    • Monitor whether base task capabilities regress;
    • Mix in a proportion of stable supervised data when necessary.

    The goal of this stage is not rapid loss reduction but sustained improvement in behavior quality. Therefore, single loss metrics should not be the only criterion for success.

    Parameter Grouping and Layer-wise Learning Rates

    For models with complex architectures, parameters can be grouped by function:

    1. Base representation layers: smaller learning rate
    2. Intermediate feature layers: medium learning rate
    3. Task adaptation layers: larger learning rate
    4. Newly added modules: independent warmup and separate decay

    The rationale is straightforward: parameters closer to the general capability foundation need more protection, while parameters closer to the current task objective need faster adaptation.

    However, more parameter groups do not necessarily mean better results. Too many groups increase tuning complexity and may lead to inconsistent update speeds across modules. In practice, grouping should prioritize functional roles and training objectives, rather than mechanically grouping by network depth.

    Weight Averaging and Learning Rate Strategy

    In the later stages of training, model parameters may oscillate among multiple near-optimal regions. Weight averaging can reduce this fluctuation and produce a smoother final parameter state.

    That said, weight averaging does not replace a well-designed learning rate schedule. If the learning rate is too large, the parameter trajectory remains far from stable regions, and averaging may still yield poor results. A more reasonable combination is:

    1. Use a normal learning rate early on for rapid learning;
    2. Gradually reduce the learning rate in the middle and later stages;
    3. Apply averaging after parameters enter a relatively stable region;
    4. Select the final weights using a held-out validation set.

    It is important to distinguish between “parameters used during training” and “averaged parameters used for evaluation or export.” Mixing the two can lead to misleading evaluation results.

    Building a Reusable Tuning Workflow

    A reliable multi-stage training workflow should follow these steps.

    First, define the objective of each stage clearly. Do not simply divide stages by data files; specify what capability the model is expected to acquire in each stage.

    Second, estimate the distribution shift between stages. The more different the data, the more carefully warmup, optimizer states, and learning rate upper bounds need to be reconsidered.

    Third, determine the order of magnitude for the learning rate before fine-tuning the shape of the curve. If the magnitude is wrong, debating between cosine and linear decay is usually pointless.

    Fourth, use small-scale experiments to validate stage transition behavior. Focus on loss, gradient, and capability metrics before and after the transition, rather than only on final scores.

    Fifth, set a stopping criterion for every stage. If validation metrics show no improvement over an extended period, continuing training may only increase overfitting and capability forgetting.

    Sixth, maintain complete experiment records, including data version, stage boundaries, learning rate, batch size, optimizer state handling, and evaluation results. Multi-stage training involves many variables, and without full records it is difficult to diagnose issues.

    A Generic Training Framework

    Below is a pseudocode skeleton that is platform-agnostic:

    1. for stage in stages:
    2. configure_dataset(stage.dataset)
    3. configure_objective(stage.loss)
    4. configure_parameter_groups(stage.parameter_groups)
    5. if stage.reset_optimizer:
    6. optimizer = build_optimizer(model.parameters())
    7. else:
    8. optimizer.load_state(previous_state)
    9. scheduler = build_scheduler(
    10. base_lr=stage.learning_rate,
    11. warmup_steps=stage.warmup_steps,
    12. total_steps=stage.total_steps
    13. )
    14. for batch in stage.batches:
    15. loss = forward_and_compute_loss(model, batch)
    16. loss.backward()
    17. clip_gradients(model, stage.max_grad_norm)
    18. optimizer.step()
    19. scheduler.step()
    20. optimizer.zero_grad()
    21. if should_evaluate():
    22. evaluate(model)
    23. previous_state = optimizer.state()

    In actual implementations, safeguard mechanisms should be added, such as detecting non-finite loss values, gradient explosions, learning rate recovery after errors, and avoiding repeated execution of stage boundaries.

    Conclusion

    The essence of multi-stage training is aligning data, objectives, parameter update scopes, and learning rate changes with one another. The learning rate is not an isolated hyperparameter but a core control variable that connects training stages with model behavior.

    A mature training scheme typically exhibits the following characteristics: clear stage objectives, smooth transitions, well-reasoned parameter group division, justified optimizer state handling, rich training metrics, and the ability to stop or roll back promptly based on validation results.

    These principles apply broadly, whether training on an on-premises cluster or in a cloud environment such as Baidu AI Cloud. Effective optimization rarely depends on a single fixed curve. It depends instead on sustained analysis of the relationships among data distribution, model state, and training objectives.