Large-Scale Vision-Language Model Video Training: Practice and Capability Emergence

1. Why Large-Scale Models Overturn Training Intuitions

Engineers who have long trained small-scale models often undergo a holistic cognitive shift when first participating in large-scale vision-language model (VLM) training. This shift is not a leap in any single metric, but a qualitative change in model behavior patterns.

At the small-model stage, engineers are accustomed to decomposing tasks based on the independent and identically distributed (i.i.d.) assumption: to train capability A, prepare type-A data; to train capability B, prepare type-B data; and evaluate each item separately. This is reasonable when parameters are limited and data volume is small, because model capacity cannot accommodate the stacking of too many capabilities.

However, when model scale reaches a certain level, “synergistic gains” emerge between capabilities (note: this is a concept proposed by the authors of this article, referring to the phenomenon where different foundational capabilities, after sufficient training, can combine to produce composite capabilities beyond simple addition; the applicable boundary is when model parameters reach the tens-of-billions level and foundational capability training quality meets standards). In other words, after training foundational capabilities A and B, the model may ultimately obtain not A+B, but a new capability amplified through the synergy of A and B. This is the core of how scale changes everything.

Take “video-to-code” as an example. The model mastered front-end code generation during text post-training and learned temporal visual understanding during video pre-training. After joint training, the two capabilities naturally gave rise to a composite capability: “generating interactive code from screen recordings.” This is not because the data contained large amounts of “video-code” pairs, but because of synergistic gains.

This phenomenon reminds engineering teams: at the large-model stage, excessively pursuing strict i.i.d. data distribution may instead compress the model’s generalization space.

2. Why Visual Evaluation Can Avoid Frequent Benchmark Watching

According to the team, in the visual capability evaluation of this model, multiple mainstream visual benchmark sets (such as VBench, MMStar, and similar benchmarks) were not monitored at high frequency during training. Typically, a full evaluation was run only once after each joint training merge. This would have been almost unimaginable in the small-model era—at that time, engineers often checked key metrics every few hundred steps to prevent divergence.

The fundamental reason large models can avoid frequent benchmark watching is that foundational capability training quality is sufficiently solid. Once foundational capabilities such as visual perception, text understanding, and code generation are fully consolidated during pre-training, fluctuations in downstream evaluation metrics tend to stabilize. As long as foundational training has not overfitted, evaluation results usually do not fluctuate dramatically.

This places demands on the evaluation system itself: rather than spending effort on repeated tuning for specific benchmark sets, it is better to maintain a small number of high-quality “observational evaluations” to monitor training health, and invest the main resources in continuously improving foundational capabilities. It should be noted that excessive benchmark watching may distort the training process through tuning behavior, and this is especially worth guarding against in large-model training.

3. Video Training Data Construction: Small Proportion, High Quality, and Compliance

This model supports video input for the first time. According to the team, in the pre-training stage with a total training volume of approximately 15T, video data accounted for about 1%. Although the proportion is low, substantial effort was invested in data construction.

The core selection principle is “raw video from natural scenes.” Natural scenes mean the data has not undergone heavy cleaning or staged shooting, preserving real interference such as motion blur, lighting changes, camera shake, occlusion, and complex backgrounds, which is crucial for model robustness. “Raw” means avoiding excessive manual annotation, preventing the model from learning to take shortcuts from annotated text rather than truly understanding visual content.

Video data construction typically goes through four stages, with the following recommended key parameters:

1. Source filtering: Filter low-quality and duplicate content from large-scale public videos, retaining clips with real-scene complexity. Ensure data complies with copyright regulations and anonymize faces and private information.
2. Temporal sampling: Video frame rates vary widely. An adaptive sampling strategy is recommended to preserve key motion information while controlling sequence length to a reasonable range (e.g., 2–4 frames per second).
3. Spatial transformation: Unify resolution to the model input size (e.g., 336×336 or 768×768) while preserving the spatial proportions of the original frames.
4. Weakly supervised text association: Use vision-language alignment to automatically generate text descriptions relevant to video content, replacing frame-by-frame manual annotation.

It should be specially noted that although a small proportion of video data can bring significant improvements, it may also cause uneven capability on certain fine-grained temporal tasks, which can be mitigated through subsequent targeted data supplementation.

4. Video-to-Code: How Three Layers of Foundational Capability Stack and Emerge

Video-to-code capability is a typical case of “synergistic gains,” and its formation can be decomposed into three layers:

Video-to-code capability emerges from the stacking of three layers of foundational capability. The first layer is text-to-code foundational capability: during the pure-text post-training stage, the model learned to transform requirements into HTML structure, styles, and interaction logic. The second layer is video perception capability: video pre-training taught the model to extract spatiotemporal features from consecutive frames and understand layout, component appearance, and interaction state changes. The third layer is cross-modal alignment capability: joint vision-language training established mappings between visual representations and text/code representations.

When all three layers reach a certain quality, the composite capability naturally emerges. After seeing a video of interface operations, the model first parses the screen layout and interactions, then transforms the parsing results into executable code. In actual training, multiple rounds of rejection sampling are also needed to filter low-quality outputs, along with human feedback to correct deviations in interface semantics and interaction details.

5. Does Multimodal Training Degrade Text Style? Engineering Practice

After visual capability was added, some argued that the model’s text style shifted. According to the team, they used only about 256 GPUs for a little over one day of training to produce a version based on the new foundational model that highly restored the old version’s text style characteristics. This shows that the model’s foundational capability structure is quite stable after pre-training is complete, and lightweight post-training can restore a specific text style while maintaining visual capability.

The key lies in the ratio strategy and scheduling of visual and text data, rather than simply “adding visual data” destroying text capability.

6. Engineering Lessons and Limitations

From the overall training process, the following lessons can be distilled:

First, scale changes methodology. Intuitions that hold for small models may not apply to large models. Synergistic gains require redesigning evaluation and training strategies.

Second, data quality takes priority over quantity. A 1% share of high-quality natural video contributes far more than the same proportion of ordinary data. When the data budget is limited, prioritize complexity and naturalness.

Third, evaluation should be observational, not directive. Evaluation is used to monitor health; frequent benchmark watching distorts training.

Fourth, foundational capability is the prerequisite for emergence. Composite capabilities are the result of multiple foundational capabilities meeting standards. Engineering resources should be prioritized for foundational consolidation.

Fifth, data source compliance cannot be ignored. Public videos require copyright review and privacy anonymization, which is a basic prerequisite for large-scale training.

At the same time, limitations should be noted: a 1% video proportion may lead to weaker capability on some fine-grained temporal tasks; the concept of “synergistic gains” currently lacks large-scale quantitative verification, and its applicable boundary still needs further exploration. This article is suitable for teams planning or executing large-scale vision-language model training. What scale brings is not only an increase in parameter count, but a reconstruction of an entire training methodology.