Multimodal LLM Pretraining Explained: Data Engineering, Scaling Laws, and Training Strategies

1. Introduction: Pretraining Sets the Capability Ceiling

In the development pipeline of large models, pretraining determines the capability ceiling, while downstream alignment and fine-tuning mostly converge within that ceiling and enforce safety constraints. Several recent public technical reports from major LLM teams signal the same trend: the focus of pretraining competition has shifted from “stacking parameters and data” toward the integrated, fine-grained design of data engineering, scaling laws, and training recipes. Using a typical native multimodal LLM technical report as a reference, this article dissects three stages—pretraining data construction, scaling-law research, and the training recipe—and distills transferable methodology for teams building their own models or selecting a foundation model.

2. Pretraining Data Engineering: Quality Determines the Ceiling

2.1 Four Text Domains and Layered Filtering

The report splits text corpora into four domains: web text, code, math, and knowledge. Domain splitting matters because noise patterns, quality standards, and optimal mixing ratios differ dramatically across domains; a one-size-fits-all filtering strategy systematically damages specific capabilities.

Each domain is filtered through a three-tier combination:

• Rule-based heuristic filtering: quantifiable hard signals such as abnormal length, garbled-character ratio, and duplicate-line ratio remove obviously low-quality documents at minimal cost.
• Classifier-based quality scoring: a lightweight quality classifier scores each document and keeps the high-score band, catching “clean-looking but low-information-density” content that rules miss.
• Deduplication: exact and near duplicates are removed across corpora, preventing overfitting to high-frequency samples and avoiding the illusion of inflated “effective data volume” created by repeated tokens.

An easily overlooked detail: per-domain sampling rates are not set by intuition. They are first ablated on smaller models, observing how different ratios affect each capability dimension, and then extrapolated to large-scale training. This “small model pathfinding, large model deployment” approach approximates the optimal mix at controllable cost—a key step in industrializing data mixing.

2.2 Corpus Rewriting: Balancing Diversity and Faithfulness

For knowledge and math corpora, the report uses a rewriting pipeline to diversify expression, with three control points:

1. Diverse prompts: rewriting prompts cover multiple writing styles and narrative perspectives, so the model does not see only one form of expression.
2. Block-wise autoregressive generation: rewriting proceeds chunk by chunk, balancing local coherence and global structure in long documents.
3. Faithfulness verification: rewritten output must pass a consistency check against the source document, intercepting content drift.

The value: the same knowledge enters the training distribution in multiple expressions, so the model is more likely to learn the essence of knowledge rather than surface wording, while the faithfulness check guards the factual baseline. Any data-processing step involving generative models needs this verification gate.

2.3 Visual Corpora: From Image-Text Pairs to Programmatic Multimodal Data

The visual corpora cover image captioning, interleaved image-text documents, OCR, perception-oriented data, video, and visual programming, built from open-source datasets plus internal filtering, synthesis, and deduplication pipelines.

The most notable trend in the report is the substantial expansion of programmatic multimodal data: code snippets paired with their rendered outputs, spanning SVG, 3D assets, web pages, games, and CAD drawings. The unique advantage of such data is “generative correspondence”—code is a precise structured description, and the rendered image is a deterministic visual result, naturally forming zero-noise cross-modal pairs. It lets the model build a bidirectional mapping between visual structure and formal languages from the start of pretraining, making it an important source of capability for later drawing, generation, and design tasks.

2.4 Dual-Format Coordinate Supervision

For localization-oriented visual supervision, coordinates are provided in both absolute and normalized ([0,1]) formats: absolute coordinates support pixel-level precision, while normalized coordinates are robust to resolution changes and transfer across image sizes. Supervising with both formats in parallel frees the model from choosing between precision and resolution robustness—a subtle but far-reaching design in visual grounding quality.

3. Scaling Laws: Keeping Hyperparameters in Step with Architecture Changes

3.1 Why Retuning Is Necessary

When attention structure, MoE configuration, data recipe, and training method evolve together, the optimal hyperparameter combination of the previous model generation is likely obsolete. The report therefore runs a dedicated scaling-law study, fitting loss curves on held-out out-of-distribution (OOD) validation data and retuning four classes of key hyperparameters:

• Batch size: the balance point between gradient estimation variance and training throughput.
• Learning rate: strongly coupled with model size and batch size.
• Tokens per parameter (TPP): the number of training tokens per parameter, answering the fundamental question of “how much data to feed.”
• Model shape: structural ratios such as depth, width, and number of experts.

3.2 Scaling Efficiency: Quantified Gains from Stacked Improvements

The core output of scaling-law experiments is the compute-versus-validation-loss curve. According to the report, stacking data, architecture, and training improvements yields roughly a 2.5x gain in overall scaling efficiency over the previous-generation baseline—intuitively, the same compute reaches a significantly lower validation loss, or the same level is reached with far less compute. For engineering teams, the implication is that “soft improvements” on the data and training side can be quantified and priced like raw compute, which guides where R&D resources should go.

3.3 Practical Recommendations

• Fit curves at small scale: compute points spanning one to two orders of magnitude suffice to fit the power-law relationship; full-scale training is unnecessary.
• Treat curves as monitoring: if large-scale training loss deviates significantly from the fitted curve, the hyperparameters or the data pipeline are often at fault—the scaling curve can serve as a predictor of training health.
• Control variables strictly: when comparing efficiency gains across model generations, the data and training recipe must be fixed, or the gains cannot be attributed.
• Never skip OOD evaluation: tuning only on in-distribution validation sets yields systematically optimistic conclusions.

4. Training Recipe: Native Multimodality and Stability Design

4.1 Native Multimodal Training: Joint Optimization from Step One

The traditional multimodal route is “pretrain a language model first, then graft on a vision encoder”: pretraining learns text only, after which a vision encoder is attached through adapter layers and aligned via fine-tuning. The report instead adopts native multimodal training: language and vision are jointly optimized from the first step, vision and text tokens share the same backbone, and everything is learned under a single next-token prediction (NTP) objective. The difference between the two routes is fundamental:

• Grafted: the backbone is already fixed when the vision signal enters, so cross-modal fusion is limited by late adaptation, and vision behaves like a bolt-on module.
• Native: cross-modal representations form jointly from the bottom up; the model treats images as first-class modeling objects on par with text, producing a more unified representation space.

The cost is a significant rise in training complexity: modality mixing ratios, sequence-length inflation from visual tokens, and cross-modal interference all require careful recipe design and ablation validation.

4.2 Optimizer, Weight Clipping, and MoE Load Balancing

Training uses an optimizer built on matrix orthogonalization, along with a per-head variant: a classical numerical iteration orthogonalizes the momentum matrix so that the singular-value spectrum of each step’s update is more uniform, and the operation is applied independently to each attention projection head, mitigating the instability where “a few directions dominate updates” in large-scale training. Combined with a weight clipping mechanism that bounds per-step update magnitude, it is a typical “stability for scale” engineering pairing.

For MoE routing, a load-balancing mechanism controls the token distribution across experts, preventing routing collapse (a few experts overheating while the rest idle) from shrinking the effective parameter count. Two common approaches exist in the industry: auxiliary-loss balancing is simple to implement but interferes with the main loss, while auxiliary-loss-free bias adjustment is friendlier to the main objective and has become the recent mainstream choice.

4.3 Scheduling and Context Curriculum

The training configuration in the report can serve as a reference template for similar runs:

• Learning rate: cosine decay schedule with linear warmup over the first 1% of steps.
• Weight decay: fixed at 0.1 throughout.
• Context length: curriculum-style expansion; pretraining starts at 8k tokens and lengthens gradually in later stages.

Illustrative pseudocode:

seq_len = 8_000
for stage in pretrain_stages:
    train(model, data, seq_len, lr=schedule(stage))
    seq_len = extend(seq_len)  # extend context length stage by stage

This “short first, then long” curriculum is not just a compute-saving trick: the short-context stage has high sample density and efficient knowledge absorption, while the long-context stage concentrates on long-range retrieval and multi-document reasoning. The two stages deliver different benefits.

5. Engineering Takeaways

When transferring this methodology to your own development workflow, four points matter:

1. Build data infrastructure first: filtering, rewriting, rendering, and deduplication are high-throughput pipeline jobs. Rely on the general capabilities of a mature cloud platform (for example, Baidu AI Cloud), such as object storage, containerized batch computing, and logging and monitoring, to build reproducible and auditable data and training infrastructure instead of letting each stage operate in isolation.
2. Make small-model ablations routine: decisions such as sampling rates, rewriting strategies, and hyperparameters should be validated in low-cost proxy experiments before scaling up.
3. Monitor health end to end: OOD validation loss, expert load distribution, gradient norms, and update-clipping trigger rate are the four dashboards for judging whether training is “on track.”
4. Keep verification gates as the backstop: every data-processing step involving generative models must have a quantifiable faithfulness check.

6. Conclusion

A close reading of this report condenses into three main threads. First, in data engineering, the goal is information density per token, pursued through domain-based filtering, rewriting with verification, and programmatic multimodal pairs. Second, scaling laws keep hyperparameters in step with architectural evolution and quantify “soft improvements” as comparable efficiency multiples. Third, native multimodality and stabilization design turn vision from a bolt-on into a first-class citizen. For development teams, these threads point to a plain conclusion: every bit of refinement in pretraining pays back multiplicatively in the final model’s capability and training cost.