Native Multimodal LLMs: Architecture Evolution and a Practical Selection and Benchmarking Guide
This guide explains how native multimodal large models differ architecturally from bolted-on vision-language approaches and how technical teams should evaluate and select them for production. It covers the limits of the bolted-on approach, what makes pretraining natively multimodal, and the role of post-training; a reproducible benchmarking framework built on controlled variables and a tiered evaluation set; a per-task cost model showing how input/output pricing and cache hit rates translate into real savings; and tuning levers such as caching, tiered routing, and output constraints. It includes example Python code for routing and cost estimation, a human-review mechanism for high-risk tasks like 3D reconstruction, and guidance on version lifecycle, data compliance, and release strategy.
Native Multimodal LLMs: Architecture Evolution and a Practical Selection and Benchmarking Guide
Big-model development is shifting from “stacking parameters and data” to “rebuilding architecture and training paradigms.” Especially in the new generation of multimodal models, “natively multimodal” has become a headline selling point: vendors typically emphasize stronger visual understanding, more coherent image-text reasoning, and more stable structured output. But production selection cannot rely on hands-on impressions alone, nor on public leaderboards alone. It needs to place capability, cost, latency, and risk into a single reproducible framework for evaluation.
This article abstracts a typical new-model public-preview scenario into a general engineering problem, organized around four key areas: what is actually “new” in a new version; how to build a reproducible evaluation; how to choose reasoning-effort tiers; and how pricing structure affects per-task cost. At the end, it provides example code for tiered routing and cost estimation, plus a human-review mechanism for high-risk tasks, to help technical teams build an actionable upgrade and selection plan.
1. What’s Actually New: A Technical Breakdown of Native Multimodality
1.1 Limitations of the Bolted-On Approach
“Multimodal” is not the same as “natively multimodal.” A common early approach bolts together a visual encoder, a projection adapter, and a language model:
- The visual encoder splits an image into patches and extracts visual features;
- The projection layer maps the visual features into a vector space the language model can understand;
- The language model receives a concatenated sequence of “visual vectors + text vectors” for inference.
This approach reuses mature components and is fast to implement, but usually has three bottlenecks:
- Information compression loss: The projection layer compresses visual representations, affecting fine-grained recognition such as small text, chart tick marks, and arrow connections;
- Modality fragmentation: Visual understanding and language reasoning belong to different stages, so cross-modal tasks tend to sacrifice one for the other;
- Visual ability “learned later”: Visual ability relies mainly on later continued training and lacks a process of growing in sync with language ability.
1.2 Key Differences of Native Multimodality
Native multimodality typically means that during the pretraining stage, text, images, and possibly audio and video participate in joint modeling as a unified input sequence. Its core is interleaved image-text data and a unified attention mechanism, with three typical characteristics:
- Interleaved image-text pretraining: The corpus is composed of text passages, accompanying images, and mixed image-text pages, letting the model learn “image-text” correspondence from large amounts of raw combinations rather than relying only on manually annotated image-text pairs;
- Unified representation and joint attention: Visual and text tokens interact in a unified vector space, and visual information can be processed in sync with language reasoning at every layer;
- Multimodal data mixture design: The proportion of charts, documents, geometric figures, and code-type data affects the model’s upper bound on structured reconstruction, layout understanding, and reasoning tasks.
1.3 Post-Training: From “Able to Understand” to “Able to Perform”
Post-training usually includes two stages: supervised fine-tuning and preference alignment:
- Supervised fine-tuning uses instruction-response data to help the model follow format constraints, reason step by step, and generate compilable drawing or modeling code;
- Preference alignment uses reinforcement learning from human feedback, direct preference optimization, and similar methods to make outputs better match human preferences, reducing redundant expressions and undesirable outputs. Note that preference alignment should not be understood as eliminating hallucination; high-risk multimodal tasks still require human review.
Therefore, “redoing pretraining + post-training” means capability gains no longer come only from parameter and data scale, but from a change in the training paradigm itself. This usually brings three effects:
- Improved visual capability: More accurate fine-grained entity recognition and understanding of layout and occlusion relationships;
- Changed output style: A greater tendency toward structured, step-by-step expression, and possibly a changed response to prompts;
- Prompt regression: Prompts repeatedly tuned on older versions may no longer be effective and need recalibration.
For adopters, an upgrade decision should not rely only on the vendor’s introduction but should build their own baselines, evaluation sets, and cost models.
2. A Reproducible Evaluation Framework: Turning “Feels Better” into Data
2.1 Controlling Variables: The Prerequisite
The most common mistake in model comparison is changing multiple variables at once, making conclusions impossible to attribute. The following principles are recommended:
| Variable | Recommended practice |
|---|---|
| Model version | Explicitly specify the full version number; avoid automatic drift to “latest default” |
| Prompt | Keep it consistent with production; prepare a separate prompt set tuned for the new version as a control |
| Sampling parameters | Fix parameters such as temperature and maximum output length |
| Reasoning effort | Set low, medium, and high tiers and compare across them |
| Repeated sampling | Run the same input at least 3 times to remove the effect of randomness |
2.2 Layered Evaluation Set: Organized by Task Difficulty
An evaluation set should cover neither only simple Q&A nor only piled-up public leaderboard questions. It is recommended to layer it by difficulty:
| Tier | Task type | Core metrics |
|---|---|---|
| L1 Basic understanding | Image-text Q&A, object recognition, text extraction | Accuracy, response latency |
| L2 Structured reconstruction | Generating drawing code from a diagram; generating page code from a UI screenshot | Compile rate, element completeness, layout fidelity, human rating |
| L3 Complex reasoning and modeling | Geometry calculation, 3D structure reconstruction | Size/proportion error, part completeness, human rating |
Public leaderboards can be a reference, but they have two limitations: first, they may overlap with training data, creating a “leaderboard-gaming effect”; second, they differ significantly from a company’s real business distribution. Therefore, add 100-300 self-built private questions covering the long-tail scenarios of real business. It is recommended that each tier account for roughly one-third, to avoid over-weighting the evaluation set toward any single task type.
Once established, the question set should be version-controlled, recording the private question set, model version, sampling parameters, and evaluation date. This avoids “evaluation-set contamination” from repeated tuning and makes it easy to review historical results.
2.3 Common Problems Quick Reference
Before formal evaluation, you can first locate bottlenecks through these questions:
- If accuracy is unstable, check controlled variables first, see 2.1;
- If you don’t know how to choose reasoning effort, first see whether the task allows higher latency and cost, see Chapter 4;
- If you want to know whether a price cut actually saves money, look at the per-task cost model, see 3.2;
- If request volume is high but content is highly repetitive, prioritize improving cache hit rate, see 4.1.
2.4 Evaluation Dimensions: Quality, Cost, Latency, and Stability
Comparing only “who answers more correctly” is not enough. In production, the real question is: at equal quality, who is cheaper, faster, and more stable. It is recommended to record four types of data per sample:
- Quality: Accuracy, human rating, which can use a 1-5 scale;
- Cost: Input and output token counts, cache hit rate, per-task cost;
- Latency: Time to first token (TTFT), time per output token (TPOT) — TPOT is in milliseconds/token, convertible to tokens/s — and total elapsed time;
- Stability: Result consistency rate across multiple runs of the same input.
2.5 Combining Automation and Human Review
Code tasks should prioritize automatically verifiable metrics, such as “whether it compiles” and “whether it renders.” Diagram reconstruction can be quantified as: whether elements are complete, whether relative positions are correct, whether solid/dashed lines, fills, arrows, and other styles correspond, and the structural similarity between the reconstruction from drawing code and the original image.
For tasks with stronger subjectivity, such as fine-grained recognition, dual-person cross-annotation is recommended to reduce scoring bias.
3. Cost Estimation: From “Lower Unit Price” to “Per-Task Cost”
3.1 Common Pricing Structures
General large-model services are usually billed per token and distinguish different cases:
- Separate input/output pricing: Output requires more computation, so its unit price is usually higher than input;
- Cache-hit vs. cache-miss tiers: Parts hit by the cache, such as prompt prefixes, system instructions, and repeated documents, are billed at a lower unit price;
- Peak/off-peak tiers: Some platforms adjust prices for peak hours, with lower prices during off-peak hours.
When vendors announce a “price cut,” it is often only one or two unit prices that are lowered, and the magnitude varies widely. In the common structure, cache-hit input sees the largest cut, cache-miss input next, and output the smallest.
The following unit prices are illustrative only, used to demonstrate the calculation method, and do not represent any vendor’s real pricing:
| Billing item | Illustrative old price (CNY/million tokens) | Illustrative new price (CNY/million tokens) | Cut |
|---|---|---|---|
| Cache-hit input | 0.05 | 0.02 | 60% |
| Cache-miss input | 1.5 | 1.0 | ~33.3% |
| Output | 4.5 | 4.0 | ~11.1% |
3.2 Per-Task Cost Model
Single-task cost can be estimated as:
Per-task cost = cache-miss input tokens × miss unit price+ cache-hit input tokens × hit unit price+ output tokens × output unit price
For easy comparison, the following is normalized per thousand calls. Assume a task type consumes an average of 4000 input tokens and 1000 output tokens per call:
| Scenario | Cache hit rate | Old price (CNY/1k calls) | New price (CNY/1k calls) | Cut |
|---|---|---|---|---|
| A: No cache | 0% | 10.5 | 8.0 | 23.8% |
| B: Long-document Q&A | 70% | 6.44 | 5.26 | 18.4% |
| C: Reasoning mode | 70%, output increases to 3000 tokens | 15.44 | 13.26 | 14.1% |
| D: Multi-image vision task | 30% | 8.76 | 6.82 | 22.1% |
With no cache, the hit portion is 0: old price 4000×1.5 + 1000×4.5 = 10.5, new price 4000×1.0 + 1000×4.0 = 8.0, a cut of 23.8%, in CNY/1k calls.
Comparing these reveals three patterns:
- The task-level cut is the result of weighting the input and output unit-price cuts by their cost share, usually falling between 11.1% and 33.3%; the higher the output share, the more the overall cut approaches 11.1%.
- The higher the cache hit rate, the more pronounced the benefit from the price cut. In scenario B, a 70% cache hit brings the overall cut to 18.4%.
- The extra thinking tokens in reasoning mode are billed as output and offset part of the price-cut benefit. In scenario C, the cut falls to about 14%.
If a platform bills at a higher factor during peak hours, batch tasks should be scheduled during off-peak hours as much as possible. Compared with simply comparing unit prices, time-based scheduling often brings more substantial savings.
4. Tuning Levers: Caching, Routing, and Output Constraints
4.1 Improving Cache Hit Rate
- Stable prefix: Put fixed content such as system prompts, few-shot examples, and long documents at the very front of the request, keeping it byte-for-byte identical;
- Layered caching: Set different caching policies and expiration times for the fixed system prompt, session-level context, and repeated requests;
- Cache invalidation management: When the knowledge base updates, use version-number isolation to avoid mixing old cache with new content;
- Exact cache first: For fully reusable requests, do exact-match caching first, then consider more complex semantic caching.
4.2 Tiered Routing: Sending Simple Tasks Through a Lightweight Lane
Not all requests need the strongest model. Route by task complexity:
- Keyword extraction, format conversion, short-text summarization → lightweight model, low reasoning effort;
- Multi-step reasoning, code generation, complex image-text analysis → strong model, high reasoning effort.
4.3 Output Constraints and Context Compression
- Use structured output modes to constrain the format and reduce redundant expressions;
- Set a reasonable maximum output length to avoid unbounded “thinking” in reasoning mode;
- Summarize and compress multi-turn conversation history to reduce input tokens;
- Crop or resize images according to business needs first, to avoid irrelevant pixels consuming the token budget.
5. Example Code: Tiered Routing and Cost Estimation
The code below wraps different model-service tiers behind a unified interface, selects a processing path based on the request profile, and estimates the per-call cost. All unit prices in the code are illustrative.
import hashlibimport jsonimport timefrom dataclasses import dataclass, field@dataclassclass Request:"""Request profile for a single model call"""messages: list[dict]images: list[str] = field(default_factory=list)expect_json: bool = Falsetask_type: str = "chat" # chat / structured / vision / reasoning@dataclassclass Profile:"""Service tier corresponding to each task type; tier names are placeholders"""name: strreasoning_effort: str # none / low / medium / highmax_output_tokens: intinput_price: float # CNY/million tokensoutput_price: float@dataclassclass CacheEntry:content: strexpire_at: floatclass Router:def __init__(self):self.profiles = {"lite": Profile("lite-model", "none", 512, 0.8, 2.0),"standard": Profile("standard-model", "low", 2048, 1.0, 4.0),"vision": Profile("vision-model", "low", 2048, 1.0, 4.0),"reasoning": Profile("reasoning-model", "high", 8192, 1.0, 4.0),}self.cache_hit_price = 0.02self.off_peak_factor = 1.0self.peak_factor = 2.0self._cache = {}self.cache_capacity = 128self.cache_ttl = 3600def select(self, req: Request) -> Profile:if req.images:return self.profiles["vision"]if req.task_type == "reasoning":return self.profiles["reasoning"]if req.expect_json:return self.profiles["standard"]return self.profiles["lite"]def cache_key(self, req: Request) -> str:"""Only image-free requests with identical messages can reuse the exact cache"""if req.images:return ""raw = json.dumps(req.messages, ensure_ascii=False, sort_keys=True)return hashlib.sha256(raw.encode("utf-8")).hexdigest()def route(self, req: Request, peak: bool = False) -> dict:key = self.cache_key(req)if key:entry = self._cache.get(key)if entry and entry.expire_at > time.time():return {"source": "exact_cache", "content": entry.content}if entry:self._cache.pop(key, None)profile = self.select(req)return {"source": "model","profile": profile.name,"reasoning_effort": profile.reasoning_effort,"max_output_tokens": profile.max_output_tokens,"cache_key": key,"factor": self.peak_factor if peak else self.off_peak_factor,}def write_cache(self, key: str, content: str, ttl: float = 0):if not key or not content:returnttl = self.cache_ttl if ttl <= 0 else ttlif len(self._cache) >= self.cache_capacity:old_key = next(iter(self._cache))self._cache.pop(old_key, None)self._cache[key] = CacheEntry(content, time.time() + ttl)def estimate_cost(self, profile: Profile, in_tokens: int, out_tokens: int,cache_hit_tokens: int = 0, peak: bool = False) -> float:miss = max(in_tokens - cache_hit_tokens, 0)base = (miss * profile.input_price+ cache_hit_tokens * self.cache_hit_price+ out_tokens * profile.output_price) / 1_000_000return round(base * (self.peak_factor if peak else self.off_peak_factor), 6)if __name__ == "__main__":router = Router()req = Request(messages=[{"role": "user", "content": "Please summarize the key points of this document"}],task_type="structured",expect_json=True,)plan = router.route(req, peak=False)print(plan)# Write the exact cache after the model returnsrouter.write_cache(plan["cache_key"], "Example structured response")print(router.route(req))# Corresponds to scenario B in 3.2: new price 5.26 CNY/1k callsprint(router.estimate_cost(router.profiles["standard"], 4000, 1000, 2800))# Example output: 0.005256
This code illustrates three engineering points:
- Routing decoupled from tiers: Tier configuration is centrally managed; adding models or adjusting unit prices later only requires changing the configuration;
- Cache safety: Requests with images do not participate in exact caching, avoiding mismatches;
- Billing up front: Before launch, historical logs can be replayed to estimate call costs at different time periods.
It should be emphasized that the specific names and available values of “reasoning effort” tiers vary by platform; when actually integrating, defer to the service documentation and make it configurable in code.
6. High-Risk Tasks: 3D Reconstruction and Human Review
Tasks like diagram reconstruction and 3D model reconstruction have errors that are often hard to detect at the text level, such as proportion distortion, missing parts, and illogical internal structure. Taking full-vehicle 3D reconstruction as an example, common problems include:
- Body surface proportions that are distorted;
- Key parts, such as wheels and light assemblies, missing or displaced;
- Internal structure that is invisible but logically inconsistent.
Even with the strongest reasoning effort enabled, the model output may still “look similar but differ in essence.” The following review mechanisms are recommended:
- Multi-dimensional verification: Programmatically check whether dimensions, bounding boxes, and part counts conform to common sense;
- Multi-view rendering: Render the result from different angles and sample-compare;
- Human spot checks: Sample-check by proportion and accumulate them as regression test cases;
- Human-machine collaboration: The model generates a first draft, and a professional corrects it before it enters downstream processes.
7. Limitations, Risks, and Release Strategy
- Version lifecycle: Temporary versions with expiration markers are usually for evaluation only and should not carry production traffic. Before official launch, confirm the stable version and its deprecation plan to avoid “service interruption from an expired version.”
- Evaluation accessibility: General interfaces may not directly expose the reasoning-effort parameter, and some capabilities may require experience through dedicated service channels; the test plan should reserve an adaptation layer.
- Data compliance: Multimodal calls often involve images and documents. Teams should perform necessary data minimization before calling, such as redacting sensitive information like ID numbers, phone numbers, and customer names; clarify whether data is used for training, how long it is retained, and whether it crosses borders; for highly sensitive businesses, use a private deployment or a compliance-reviewed service channel.
- Online monitoring: After an upgrade, continuously track quality, latency, and cost metrics, and set a rollback plan.
- Feedback loop: During public preview, vendors usually collect usage feedback; teams can organize a structured issue list to submit and participate in product iteration.
8. FAQ: Common Selection and Deployment Questions
1. How to choose between native multimodal and bolted-on?
Decide by task first. If the business needs fine-grained recognition, joint image-text reasoning, complex layout, or structured reconstruction, prioritize native multimodal; if the task is simple, latency-sensitive, or cost-sensitive, a lightweight model or bolted-on approach can be used.
2. How to choose reasoning effort?
Conclusion first: tier by quality threshold and cost budget. Ordinary Q&A can use a low tier, structured extraction a medium tier, and complex reasoning, code generation, or 3D modeling a high tier, and validate the marginal benefit with a private evaluation set.
3. Why does a task-level price cut not equal a single-item price cut?
Because overall cost is determined jointly by the input, output, and cache-hit proportions. When the output unit-price cut is small but its share is high, the task-level cut is dragged down; when the cache-hit proportion is high, the cut is amplified.
4. How to prevent self-built evaluation sets from being contaminated?
The core practices are versioning and privatization. Record the question-set version, model version, sampling parameters, and evaluation date; avoid using public original questions directly; periodically rotate long-tail samples; control access permissions to the evaluation set.
5. How to launch high-risk tasks?
Conclusion first: human-machine collaboration and canary release. The model produces a first draft first, then programmatic verification, human spot checks, and multi-view review are applied; after a small-traffic launch, continuously monitor quality, cost, and latency, and retain a rollback path.
9. Conclusion
“Native multimodality + rebuilt pretraining and post-training” represents a new direction for large-model upgrades: capability growth no longer depends purely on scale expansion but comes from structural optimization of the training paradigm. For development teams, rather than chasing leaderboard rankings, it is better to build a closed-loop methodology of “baseline - evaluation - cost - canary”: use a layered evaluation set to validate capability, a cost model to measure returns, caching and routing to control overhead, and human review to backstop high-risk tasks. Only in this way can each iteration of a new version truly translate into stable, quantifiable, and sustainable business value.
