LLM Pretraining Pipeline Explained: Data, Training, and Required Skills

Pretraining quality carries through to alignment, inference, and deployment. The pretraining pipeline can be split into four areas: data construction, architecture and training stability, large-scale parallelism and engineering, and talent and delivery. These areas form a chain, and a gap in any one of them gets amplified in later evaluation. Each section below covers one area and lists checks you can run directly.

Data Cleaning Steps for LLM Pretraining

Run the data pipeline in the following order, and keep a record at each step.

1. Rule-based filtering: Remove text with abnormal length, a high share of garbled characters, or dense clusters of punctuation. Set thresholds by sampling first. For example, manually review 1,000 random samples. If the rules wrongly delete more than 20% of them, loosen the rules.
2. Deduplication: Use hashing for exact deduplication at the document level, and use approximate methods such as MinHash for fuzzy deduplication at the paragraph or sentence level. The deduplication granularity should match the training goal. Coarse granularity deletes valid samples by mistake, while fine granularity leaves many duplicates behind.
3. Quality scoring: Score documents with a small model, and calibrate the scores against human labels. Sample documents, measure how often the model score agrees with human judgment, and then set the retention range by quantiles. Score distributions differ widely across sources, so compute them separately for each source.
4. Domain quotas: Set target proportions for each domain, language, and modality. Downsample sources that exceed their quota. Set minimum floors for long-tail domains so that high-frequency content does not drown them out.
5. Long-tail coverage check: Measure coverage of low-frequency entities, specialized terms, and low-resource languages. Track this as a fixed metric for every data version.

Each data version record should include at least the following fields: data version number, source list and acquisition dates, filtering rule version, deduplication parameters, scoring model version and thresholds, per-domain quotas and actual proportions, number of output samples and tokens, and owner. With these fields, an anomaly in a metric can be traced back to a specific batch of data.

Model Architecture and Training Stability: How to Choose

Mainstream architectures are still built on self-attention. The decisions to make include whether layer normalization goes before or after the residual branch, which activation function to use, and whether to use absolute or rotary position embeddings. Rotary position embeddings generally extrapolate better to long sequences, but measure the effective extrapolation length on a validation set instead of relying on experience.

Mixture-of-experts (MoE) suits settings with very large parameter counts where you want to limit the compute activated per token. It has three costs: higher all-to-all communication across experts, idle experts when load is uneven, and unstable routing training. Before adopting it, estimate communication time on the target cluster, set a load-balancing loss and expert capacity limits, and monitor the share of tokens each expert receives during training.

Training stability depends on several techniques that work together. Gradient clipping usually uses a fixed norm threshold, and you should record the fraction of steps that get clipped. Use learning rate warmup followed by decay, where warmup typically lasts a small fraction of total steps. Mixed-precision training requires loss scaling. If numerical overflow appears, check the gradient norm curve first, then decide whether to adjust the learning rate.

How to Choose a Parallelism Strategy for Large-Scale Training

Measure the bottleneck before choosing a parallelism strategy. In a small-scale configuration, record three metrics: per-GPU memory usage, the share of each step spent on communication, and tokens processed per second. Whichever of the three hits its limit first determines what to adjust first.

The usual selection order is as follows. When the model fits on a single GPU and throughput is the constraint, use data parallelism first. It is simple to implement, and its communication is mainly gradient synchronization. When a single layer’s parameters are too large for memory, or when you need to reduce per-GPU activation pressure, add tensor parallelism. Tensor parallelism is sensitive to bandwidth, so keep it within the high-speed interconnect of a single node. When the model has many layers and cross-node bandwidth is limited, use pipeline parallelism to split the layers. In this case, tune the number of micro-batches to reduce the share of pipeline bubbles.

The best combination depends on the cluster topology: intra-node interconnect bandwidth, inter-node network bandwidth and latency, GPU count, and rack layout. The same strategy can produce very different results on another cluster, so measure again every time you move to a new one.

Several common engineering measures also apply. Activation recomputation trades extra compute for memory. Optimizer state sharding can significantly reduce per-GPU memory usage. CPU offloading is slow, so use it only as a fallback when memory runs short.

Which Metrics Should Pretraining Evaluation Track?

Perplexity is only a training signal. It does not represent model capability. Organize evaluation into evaluation cards, and each card should include at least the following.

• Downstream probe tasks: Choose several tasks relevant to the target scenario. Fix prompt templates and decoding parameters, and use the same version of the task set for every evaluation.
• Human spot checks: Sample a fixed number of examples for each version, at least 200. Have two annotators score them independently. Send disagreements to a third reviewer for a final decision, and record inter-annotator agreement.
• Safety metrics: Measure the rate of harmful content, along with the rates of refusals and false refusals, and set thresholds. Versions that exceed a threshold do not advance to the next stage.
• Regression baseline: Compare against the previous version on the same tasks. Any clear drop in a core task needs an explanation before the version is approved.

Version the evaluation cards too. Record every change to the task set, spot-check samples, and thresholds. Otherwise, results from different versions cannot be compared.

Acceptance Criteria for Small-Scale Validation

Validate on a controllable configuration before scaling up. Controllable dimensions include parameter count, sequence length, training token count, GPU count, and communication topology. The goal of small-scale experiments is to confirm that the data recipe and architecture changes do not cause instability. It is not to chase final quality.

Before scaling up, all of the following metrics must pass.

• The loss curve is smooth, with no sustained rising segments, and the number of spikes stays within the preset range.
• The gradient norm is stable, and the share of clipped steps has not risen noticeably.
• Tokens per second deviate from the estimate by no more than about 20%. If the deviation is larger, first check communication and data loading.
• Checkpoint saving and restoring run end to end, and the loss after restoring continues the original trajectory.

These thresholds are only a starting point. Adjust them based on your team’s historical data.

Fault Tolerance, Checkpoints, and Data Safety

Set the checkpoint frequency by balancing save overhead against rollback loss. Saving too often consumes storage and bandwidth. Saving too rarely means losing many training steps when a node fails. Estimate the save interval from the cluster’s historical failure rate, and keep several past checkpoints.

Practice the recovery process on a schedule instead of trying it for the first time after a failure. The drill should cover restoring from the most recent checkpoint, confirming that the data read position and random seeds match, and verifying that the loss curve is continuous after recovery.

When nodes are preempted, also prevent data inconsistency. For example, if the data loader’s state is not synchronized with the checkpoint, samples will be repeated or skipped after recovery. Define the rollback policy in advance: which checkpoint to roll back to, which data versions must be regenerated, and who confirms the decision.

What Skills Does a Pretraining Role Require?

A pretraining role requires theory, systems, data, and collaboration skills. The table below lists the core content of each area, along with how interviews and projects assess it.

Skill area	Core content	Interview focus	Project evidence
Theory	Optimization theory, generalization analysis, architecture design intuition	Explain the role of learning rate warmup, and analyze possible causes of a loss spike	Designed and validated one architecture or hyperparameter change
Systems	Parallelism strategies, communication protocols, performance tuning	Given a cluster topology, explain the basis for choosing among three parallelism strategies	Completed one throughput optimization with before-and-after data
Data	Corpus construction, quality evaluation, bias control	Design a deduplication and quality filtering plan, and explain how to monitor the risk of wrongful deletion	Maintained a traceable data version system
Collaboration	Coordinating data, compute, and evaluation across teams	Describe a case where inconsistent data versions caused rework, and how it was handled	Led or took part in setting cross-team evaluation standards

Experience in cross-domain areas such as perception systems, education, or generation tasks helps broaden the problem perspective.

How to Iterate in Practice

Adopt an approach of validating at small scale first, then scaling up step by step. Verify the data recipe and architecture changes at a controllable scale, and expand only after they pass acceptance. Record the data version, code version, configuration, and evaluation card ID for every experiment so results can be reproduced. During training, keep watching trends in loss, gradients, and throughput, rather than looking only at final metrics.