Native Multimodal Models: The Real Bottleneck Is Rich Media Data
A recent wave of open-weight models pairs native multimodality with million-token context, but the real constraint has moved to the data layer. Text corpora are shifting from scarce to abundant, while rich media—video and images—has yet to reach its supply inflection point. This article explains why native multimodal architectures amplify a model's appetite for visual data, quantifies the widening gap between what models can see and what the data layer can deliver, walks through the five barriers that keep rich media expensive (alignment, quality, licensing, privacy, timeliness), lays out a minimal but practical data pipeline, and argues that data-side filtering and curation will decide product differentiation more than parameter scale.
Recent releases in the open-model space have been worth a closer look. Among the newer sparse-activation models, one puts “native multimodal” and “million-token context” on the same spec sheet: hundreds of billions of total parameters with only tens of billions activated per pass, weights released under a permissive license, and per-task pricing pushed down to a few cents. More striking than any single spec was what happened during the anonymous trials the series offered on several public model-aggregation platforms under codenames—cumulative traffic reached trillions of tokens in a matter of days. Once the guessing games ended, those numbers were mostly read as footnotes to a price war.
For developers building on top of these models, though, a different line on the spec sheet deserves attention: this series finally shipped a native multimodal member, and its pretraining corpus runs to the tens of trillions of tokens. Release specifics are a one-off, and this article won’t rehash price cuts. I want to ask the other side of the coin: when even the strongest open models start asking for visual and video data, what happens on the data side?
Here is the conclusion up front: text data is turning from a scarce good into inventory, while rich media—video and images—has yet to cross its supply inflection point. The gap between what models can now “see” and what the data layer can actually deliver is widening fast.
Three Shifts, One Conclusion
Rather than comparing benchmark scores row by row, it’s more useful to see where the new models actually pulled ahead of the previous generation. Those differences look scattered, but they all point back to data.
First, modality has moved from add-on to foundation. The common approach in the previous generation of multimodal models was “text as the body, vision as an attachment”: train a strong text model first, then bolt on a frozen or partially frozen vision encoder, map visual features into the token-embedding space through a projection layer or lightweight adapters, and align them with image-text pairs. The weakness of that attachment design is clear. The vision representation’s optimization objective is decided independently on the vision side and doesn’t serve the language model’s reasoning distribution; the text side’s capabilities are already fixed before vision is plugged in; and for tasks that require repeatedly cross-checking across modalities—reading chart values, document layout, temporal causality—an attachment design often has to compensate for its representational shortfall with longer chains of reasoning.
Native multimodality takes the opposite approach. Vision, video, and even audio are first-class citizens in the pretraining corpus, and a unified or semi-unified modal representation is co-optimized with language objectives from early in training. The cost is a far more complex pretraining recipe: the ratio across modalities, curriculum ordering, deduplication criteria, and temperature sampling all have to be retuned. The payoff is a foundation that can genuinely “see,” rather than a layer of translation. The gap between the two shows up most clearly in exactly those tasks that require the model to visually verify its own output—read an image first, then validate item by item. Back to the original question of how to evaluate this class of models: the first-layer answer is that it changed the source of the capability, not just the list of capabilities.
Second, sparse architecture plus extra-long context changes the default cost shape of video. Hundreds of billions of total parameters with only tens of billions activated per pass means compute scales mainly with activated parameters rather than linearly with total parameters, which significantly lowers unit inference cost. But sparsification saves compute, not memory: the weights for all experts still have to stay resident, and long context turns the attention key-value cache into a major memory cost. Video, once tokenized, is also far larger than text, so the unit cost of “watching video” is no longer simply a function of duration—it’s a function of sampling frame rate, tokens per frame, and time span together. Engineers have to switch their intuition from “bill by the hour” to “model by visual token.”
Third, open weights plus a permissive license pushed multimodal capability out of the API and into private environments. Medical imaging, industrial quality inspection, security surveillance, internal training videos—these are precisely the domains where public corpora are thinnest and unit value is highest, and where data leaving the premises is often a hard veto. Downloadable weights mean these scenarios have, for the first time, the possibility of “the model following the data.” Reports indicate the inference traffic during the anonymous trial ran on domestic AI accelerators. This isn’t something to consume as a nationalistic narrative; its real engineering meaning is that running trillion-token-scale inference workloads on domestic compute is now feasible. The precondition is that the framework and operator ecosystems keep up, especially memory and bandwidth scheduling for long-sequence, visual-token-dense scenarios. For teams doing hardware selection, the sensible move is to run the same set of representative workloads on both domestic and general-purpose hardware and compare throughput, latency, and memory utilization—not to look at single-card peak compute.
All three differences point to the same fact: the model side’s appetite for visual and video data has been structurally amplified.
Quantifying the Gap: Text Is Bottoming Out, Video Is Nearly Infinite
To judge where the bottleneck lies, draw the two supply curves first.
On the text side, the estimate widely cited in the industry is that usable high-quality text on the internet sits at roughly tens of trillions to a quadrillion tokens, and the increment comes mainly from new web pages, code repositories, and digitized publications, growing slowly. Extrapolating from current mainstream training scales, the “inventory” of high-quality text is being consumed quickly—a judgment with fairly broad consensus in the field. Synthetic data can add volume, but the diversity ceiling of synthetic corpora is limited by the model that generates them, and over-reliance introduces distribution narrowing and quality degradation. The core tension on the text side has already shifted from “is there enough” to “how well is it selected.”
Video shows the opposite order of magnitude. Once you pixelate video and cut it into tokens, supply is nearly infinite: on mainstream platforms, uploads run to hundreds of hours per minute, and the vast majority has never been “read” by any model. Here’s a rough calculation of the token scale:
# Order-of-magnitude sketch; per-frame tokens and sampling must be tuned per modelFPS = 30 # frame sampling rateTOKENS_PER_FRAME = 256 # visual tokens per frame, depends on resolution and tilingSECONDS_PER_HOUR = 3600tokens_per_hour = FPS * TOKENS_PER_FRAME * SECONDS_PER_HOUR # ~2.76e7tokens_per_day_upload = 500 * 24 * tokens_per_hour # uploads estimated at 500 hours/minuteprint(f"One hour of video is about {tokens_per_hour/1e7:.2f} ten-million tokens")print(f"Global daily new uploads are about {tokens_per_day_upload/1e12:.1f} trillion tokens")
One hour of video is roughly 27.6 million tokens, and global daily new uploads come out to more than 30 trillion tokens. For comparison, the stock of high-quality text is on the order of “tens of trillions of tokens,” yet daily video increments alone reach the same order of magnitude. Pixels aren’t scarce—this is already consensus. The real bottleneck is one layer down: what models can actually “digest” is the subset of data with aligned labels, filtered for quality and safety, and with clear ownership—and that subset’s daily increment is several orders of magnitude smaller. Order-of-magnitude estimates are only for judging direction; they don’t replace precise measurement. The gap doesn’t come from the silence of the physical world; it comes from the data engineering of filtering and curation.
Compress that gap into one sentence: text is depreciating, because its marginal supply cost approaches zero; rich media is appreciating, because its marginal usable-supply cost stays high—and that cost is becoming the effective tax rate on real gains in model capability.
The Five Barriers to Rich Media
Alignment. The most expensive barrier. Text corpora carry a natural supervision signal—the next token is the label. Video enjoys no such convenience; the usable supervision for a video clip comes from four sources: transcribed speech, on-screen OCR text, human-written or generated descriptions, and annotations with timestamps and coordinates. The problem is the acquisition cost of these signals: transcription needs robust multilingual, multi-accent, high-noise recognition, and speaker diarization and alignment for long videos are especially error-prone; without human calibration, on-screen descriptions tend toward “safe but empty” generic captions that add almost nothing to training or inference.
Quality. Blur, shake, static frames, compression block artifacts. Video’s quality distribution is far longer-tailed than text, and “looks good” is not the same as “carries information.” In practice, teams combine signals like sharpness, exposure, motion magnitude, bitrate, and transcoding artifacts for an initial filter, but the real quality signal comes from downstream: whether a given data slice actually improves results on a validation set. This means data selection is shifting from “rule engineering” to a “measurable experiment workflow.”
Licensing. The ownership chain for images and video is far more complex than for web text. A single clip may involve the shooting party, people on camera, music rights, stock-library licenses, and more—publicly visible does not mean trainable or commercially usable. Treating license information as a first-class field in the data catalog, rather than backfilling it later, is the only way to avoid rework.
Privacy. Frames naturally contain faces, license plates, ID documents, and personal information on screens. Text de-identification already has fairly mature named-entity approaches; visual de-identification needs a full pipeline of detection, tracking, and blurring, plus auditable records proving what processing was applied to which dataset.
Timeliness. Text knowledge goes stale, but the “reality” in video goes stale faster. Product appearances, interface layouts, and streetscapes all keep changing, so models trained on public video lag steadily on these dimensions. What’s needed is a sustainable channel of incremental rich media, not a one-time database build.
Stacked together, the result of these five barriers is that the marginal cost of rich media data is far higher than text, with no order-of-magnitude drop in sight for the near term.
From Judgment to Engineering: A Minimal Viable Pipeline
Turning the above into engineering actions, a multimodal data foundation can be broken into a plain pipeline: register, decode, segment, transcribe, deduplicate, score, isolate, version. What follows is an illustrative minimal implementation—the emphasis is on structure, not specific values.
Step one: registration. Before any sample enters the store, establish a “data passport,” with ownership and source as first-class fields; missing means rejection.
{"sample_id": "v_8f3a1c","modality": "video","source": {"kind": "licensed_corpus", "license_id": "L-2291", "region": "CN"},"duration_sec": 412.7,"fps": 25,"sampled_fps": 1.0,"shot_count": 37,"asr": {"lang": "zh", "cer": 0.06},"ocr_text_density": 0.03,"quality": {"sharpness": 0.71, "motion": 0.18, "nsfw": 0.01},"privacy": {"faces_masked": true, "plates_masked": true},"dedup": {"phash": "9f2c1b", "cluster": "c_12"},"split": "train","data_version": "2026-08-27"}
Step two: the quality gate. Its purpose is to turn “can this data enter the training set” into a versionable, auditable function rather than scattered if statements across scripts.
GATE = {"min_sharpness": 0.45,"min_asr_chars": 200, # lower bound on aligned text sparsity, tune per language"max_nsfw": 0.20,"require_license": True,}def gate(rec, cfg=GATE):fails = []q, a = rec["quality"], rec["asr"]if q["sharpness"] < cfg["min_sharpness"]:fails.append("low_sharpness")if a["chars"] < cfg["min_asr_chars"]:fails.append("weak_alignment")if q["nsfw"] > cfg["max_nsfw"]:fails.append("unsafe")if cfg["require_license"] and not rec["source"].get("license_id"):fails.append("no_license")return (len(fails) == 0, fails)
Step three: deduplication and ratio. MinHash-style near-duplicate detection is fairly mature on the text side; the difficulty on video is that transcoded, cropped, watermarked, and speed-altered versions of the same source material form large clusters of “near duplicates.” A workable approach is a multi-level cascade: file-level hashing, perceptual-hash clustering, frame-level embedding retrieval, combined with modal-ratio sampling so that one modality doesn’t completely overwhelm the others within a batch.
Step four: segmentation and isolation. Timestamped data very easily causes “soft leakage” between training and evaluation—different segments of the same long video can land in both the training and test sets. Splitting by source and cluster ID, rather than randomly splitting by sample, is a rule that must be enforced.
Step five: versioning. Datasets need version numbers, changelogs, and lineage. When a given model version behaves anomalously, being able to quickly trace back to “which rule changed in which batch of data” is more effective than any after-the-fact tuning.
When this pipeline runs on engineering infrastructure, it typically needs a combination of four general-purpose capabilities: object storage to hold raw material and slice outputs, a message queue to decouple and smooth the peaks of decode/transcode jobs, a container platform to run distributed decoding, frame extraction, and scoring jobs, and a logging-and-monitoring service for observing failure rates and backlog. Mainstream cloud platforms such as Baidu AI Cloud offer this category of general capabilities, and teams can choose according to their existing infrastructure conventions; this article doesn’t go into any specific interface or pricing.
Mechanism Level: Why Native Multimodality Makes the Data Gap More Fatal
Attachment-style multimodality has an “interface-friendly” demand for data—as long as there are enough image-text pairs, alignment training can proceed. Native multimodality is different: mixing modalities in the stream amplifies three mechanistic problems.
The first is modality collapse. If text dominates the corpus far more than vision, the model naturally tends to “guess” answers from language priors, diluting the learning signal of the visual channel, which shows up as “looking at the image but not using it.” This kind of degradation can only be mitigated through active intervention at the ratio and curriculum level.
The second is the multimodal version of evaluation contamination. Text leaderboard contamination already has fairly mature detection approaches, but visual contamination is harder to spot: the same image or video may no longer match a hash after cropping or compression, yet its semantics and answer are fully leaked.
The third is “flooding failure.” Average attention cost at million-token context isn’t actually that high; the real problem is that “lost in the middle” is more pronounced in long videos—key information lands in the middle, and the model simply ignores it. The solution isn’t to force the model to push through, but to coordinate on the data and retrieval side by cutting long videos into locatable semantic units so the model can look back on demand.
Taken together, these three points show that the marginal growth in model capability will increasingly be constrained by the organizational quality of the data side, not by parameter scale.
Evaluation: Turning “Can See” into Verifiable Metrics
The most practical piece of advice for developers is: don’t buy the narrative of “can watch video,” buy verifiable metrics. A minimal viable multimodal evaluation should cover at least five task types: numeric extraction from documents and charts, temporal localization in video, cross-frame coreference resolution, counting and spatial relations within frames, and self-verification tasks that require “looking back to cross-check.” Each needs a clear scoring script and a library of failure cases.
Even more important is governance of the evaluation set itself. The holdout set must be isolated from the training corpus at the source and cluster level, rotated regularly, and actively checked for “variant leakage of the same material.” Otherwise, evaluation scores will inflate alongside data expansion while real-world performance stays flat.
EVAL_SUITE = {"doc_chart": {"metric": "exact_match", "n": 500},"temporal_grounding": {"metric": "tIoU@0.5", "n": 300},"cross_frame_coref": {"metric": "accuracy", "n": 300},"count_spatial": {"metric": "accuracy", "n": 300},"self_verify": {"metric": "consistency", "n": 200},}
Conclusion: The Next Bottleneck Is in the Data Layer—Filtering and Curation
Pulling the threads together: native multimodality, sparse architecture, million-token context, and open weights together turned “vision and video” from a demo feature into an infrastructure feature. And once infrastructure takes shape, the focus of competition migrates from model capability to data supply. The text channel is sliding from “scarce” toward “abundant,” while the video channel has only just taken its first step out of “undeveloped”—and what sits between them isn’t algorithms, but the five barriers of alignment, quality, licensing, privacy, and timeliness.
For teams preparing to build on this class of models, three actionable things: first, make the rich-media “data passport” and quality gates mandatory parts of the pipeline rather than after-the-fact audits; second, build a rotatable holdout evaluation set around real business scenarios, turning “can see” into “verifiable”; third, manage modal ratio and curriculum as first-class hyperparameters, and watch out for modality collapse. The arms race on the model side will continue, but what ultimately decides product differentiation is very likely the engineering effort in that data-layer “filtering and curation.”
