Deep Dive into Multimodal LLM Pretraining: Data, Scaling, and Training

1. Background: Why Pretraining Requires Systematic Redesign

The competitive gap among large models is increasingly determined by systemic engineering capabilities during the pretraining phase, rather than isolated algorithmic tricks. A recently released technical report on a frontier multimodal model reveals a crucial fact: when model architectures (such as Sparse Mixture of Experts), data composition (coexistence of text and vision), and training objectives undergo generational shifts, the optimal hyperparameters and training curricula validated for previous generations often no longer apply. A holistic redesign around three pillars—data, scaling laws, and training schemes—becomes mandatory. This article follows this主线, outlining the key decisions and reusable engineering methods.

2. Pretraining Data Engineering: Quality, Ratios, and Knowledge Density

2.1 Text Data: Hierarchical Governance Across Four Domains

Pretraining corpora span four main text domains: web text, code, math, and knowledge. Each domain undergoes a three-layer progressive processing pipeline:

1. Rule-based Heuristic Filtering: Deterministic rules are used first to remove low-quality boilerplate content. This is low-cost, interpretable, and serves as the primary gatekeeper.
2. Classifier Quality Scoring: A trained quality discriminative model scores and truncates the corpus, capturing fine-grained quality differences that rules cannot cover.
3. Deduplication: Combines exact and fuzzy deduplication to reduce the memorization risk caused by duplicate samples, thereby improving the information gain per unit of compute.

Regarding data mixture, the sampling rate for each domain is not empirically guessed. Instead, it is determined through ablation experiments on a smaller proxy model and then extrapolated to the large model training. This “small model explores, large model benefits” approach keeps the cost of mixture searching within acceptable limits.

Furthermore, knowledge and math corpora undergo systematic rewrite augmentation, with a pipeline featuring three key points:

• Prompts with diverse styles and perspectives are used to expand a single source into multi-angle expressions, increasing knowledge coverage diversity.
• Chunk-by-chunk autoregressive generation ensures context consistency during the rewriting of long documents.
• Faithfulness verification against the source document is performed on the rewritten results to prevent “distortion during rewriting” and avoid synthetic noise polluting the training signal.

2.2 Visual Data: From Image Captioning to Programmatic Multimodality

Visual corpora follow a clear taxonomy, covering task formats such as image captioning, interleaved image-text documents, OCR, perception, video, and visual programming. This is constructed from a combination of open-source datasets and internal filtering, synthesis, and deduplication pipelines.

Two engineering details are particularly noteworthy:

• Dual-format coordinate supervision: During training, both absolute coordinates and normalized [0,1] coordinates are provided as supervision signals. This makes the model’s localization capability robust to input resolution—eliminating the need for recalibration or fine-tuning when switching inference resolutions.
• Large-scale expansion of programmatic multimodal data: Code snippets are directly coupled with their rendered results in specific domain formats, covering SVGs, 3D assets, web pages, games, and CAD drawings. This type of data inherently carries a “code-vision” alignment signal, serving as a high-quality bridge connecting language reasoning with visual understanding and generation.

3. Scaling Laws: The Navigator for Training Schemes

Any substantial change in architecture, data, or training methods will shift the position of the “optimal training point.” Therefore, this work conducted dedicated scaling law research, re-tuning four key hyperparameters:

• Batch size: Determines the balance between gradient noise levels and throughput.
• Learning rate: Strongly coupled with optimizer characteristics and model scale.
• Tokens-Per-Parameter (TPP): The number of training tokens consumed per parameter, directly dictating “whether compute should be spent on parameters or data.”
• Model shape: The proportional relationship between depth, width, and expert configuration.

For evaluation, the scaling curves are measured on held-out Out-of-Distribution (OOD) validation data rather than in-distribution training data, preventing curves from looking “good but distorted.” Public results show that stacking these improvements yields an overall scaling efficiency improvement of roughly 2.5x compared to the previous generation—meaning the compute required to reach the same loss level is significantly reduced. This validates a key judgment: scaling efficiency is not merely the result of piling up compute, but a function of the co-design of architecture, data, and optimization.

Transferable methodological takeaways include:

1. Fit scaling curves using a family of small models to control single-experiment costs.
2. Fix the evaluation set and protocol to ensure results are comparable across generations.
3. Tune TPP and learning rate first, then batch size and shape, to reduce coupling between search dimensions.

4. Training Schemes: Native Multimodality and Optimization Stability

4.1 Natively Multimodal Joint Training

Unlike the two-stage paradigm of “training a language model first, then grafting a visual encoder via an after-the-fact alignment phase,” this work jointly optimizes visual and text tokens under a unified Next-Token Prediction (NTP) objective from day one of training. A shared backbone network learns unified multimodal representations from scratch. The benefits are threefold: it avoids the degradation of existing capabilities caused by the alignment phase; visual and language gradients mutually shape representations from the early stages; and the training pipeline is cleaner with lower engineering complexity.

4.2 Optimizers and Stability Mechanisms

• Optimizer: Employs a novel matrix optimizer based on orthogonalized momentum updates, processing attention matrix parameters on a Per-Head basis, striking a balance between convergence speed and training stability.
• Weight Clipping: Continues the clipping mechanism from the previous generation to suppress sporadic anomalous updates in sparse architectures, preventing loss spikes.
• MoE Load Balancing: Introduces a queue-based balancing mechanism to alleviate expert load imbalance and routing oscillation, reducing the resulting training instability.
• Scheduling and Regularization: Cosine learning rate scheduling paired with a 1% linear warmup, with weight decay fixed at 0.1 throughout.

4.3 Curriculum-based Context Length Extension

Pretraining starts with an 8k token context length and expands in stages. This “short-first, long-later” curriculum design allows the model to accumulate most of its capabilities on cheaper short sequences before adapting to long contexts at a lower cost, significantly outperforming the compute overhead of training on long sequences throughout.

The core training loop can be illustrated as follows:

# Pretraining main loop (illustrative)
lr = warmup_then_cosine(base_lr, warmup_ratio=0.01)
for step in range(total_steps):
    batch = mixture.sample(text_domains, rates)   # Sample according to domain ratios
    batch += vision_pipeline.sample()             # Mix in visual tasks
    loss = ntp_loss(model(batch))                 # Unified NTP objective
    loss.backward()
    clip_weights(model)                           # Weight clipping for stable sparse training
    optimizer.step()                              # Orthogonalized momentum optimizer (per-head)
    moe_balancer.rebalance(model.experts)         # Queue-driven load balancing
    scheduler.step()

The data pipeline can similarly be abstracted into clean pseudocode:

def build_corpus(raw):
    docs = rule_filter(raw)               # Heuristic rule filtering
    docs = quality_score(docs)            # Classifier scoring and truncation
    docs = dedup(docs)                    # Exact + fuzzy deduplication
    rates = ablation(proxy_model, docs)   # Small model ablation to determine sampling rates
    return rewrite(sample(docs, rates),   # Rewrite augmentation
                   verify_faithfulness=True)

5. Transferable Engineering Insights

1. Data quality is hierarchical: Rule-based fallbacks, model scoring, and deduplication for freshness are all indispensable.
2. Data mixture must be experiment-driven: Small model ablation combined with scaling extrapolation is currently the most cost-effective way to search for optimal ratios.
3. Rewriting enhances knowledge density, but faithfulness verification is the baseline: Synthetic data without verification will inversely dilute the training signal.
4. Scaling laws are hyperparameter infrastructure: Any major architectural or data change should trigger a re-run of the scaling curve experiments.
5. Stability mechanisms must be designed upfront: Clipping, load balancing, and warmup scheduling should be integrated into the scheme before training begins, rather than being applied as after-the-fact remedies.
6. Enterprise implementation path: The aforementioned pipelines can be built leveraging general cloud capabilities—using object storage to manage multimodal corpora, container platforms to orchestrate preprocessing and training tasks, and logging services with monitoring alerts to track training curves and expert loads. For instance, on Baidu AI Cloud, these components fall under general product categories and can be directly mapped into an integrated “data-training-observation” MLOps workflow.

6. Conclusion

From hierarchically governed corpus engineering, to scaling law research anchored by OOD validation, to the unified training objectives of native multimodality, this technical thread sends a clear signal: large model pretraining has entered the era of “systemic co-design.” What truly widens the gap is not any isolated technique, but the capacity to repeatedly calibrate data, architecture, and optimization as a single integrated whole. For teams planning their own large or domain-specific models, building the data pipeline and scaling experiment infrastructure first, before talking about scaling up, is the more prudent path forward.