Native Multimodal Sparse LLMs: Technical Breakdown and Low-Cost Practice
A technical walkthrough of a new class of large models that pair 100B+ total parameters with roughly one-tenth of them activated per inference call, plus native text-and-image support. The article explains how MoE sparsity decouples knowledge capacity from compute, how to estimate the real resource footprint (weights, KV cache, compute, and inter-device communication), why sparse and linear attention matter for long context, how to decide between API access and self-hosting, how to build your own evaluation loop, and which cost-control and risk practices actually hold up in production.
1. Introduction: Sparsity, Native Multimodality, and Hybrid Attention
Competition among large models is shifting from “who has more parameters” to “who can deliver sufficient capability at a lower cost per call.” One model shape in particular has drawn attention from developers recently: more than 100 billion parameters in total, with only about one-tenth of them activated on each inference pass, and native support for text, images, and other modalities. Models of this kind typically show up first under an anonymous name on model aggregation platforms and inside coding assistants, accumulate a few days to a few weeks of public testing, and are only then formally released with open weights.
This article discusses no specific product. Instead, it extracts the shared techniques behind this class of models and answers three questions:
- What allows them to be both capable and cheap per call?
- How should developers estimate their resource footprint and inference cost?
- How do you select, evaluate, and control cost for them in real workloads?
2. Total vs. Activated Parameters: The Cost Logic of Sparse MoE
2.1 Two Numbers That Matter
A Mixture of Experts (MoE) model splits the feed-forward network into several parallel “experts,” where each expert is an independent set of weights, and a lightweight router selects a few experts per token. Describing such a model usually comes down to two figures:
- Total parameters: the full size of the model weights. This determines knowledge capacity and also determines GPU memory or host memory usage.
- Activated parameters: the number of parameters that actually participate in computing a single token. This determines the compute per token and, roughly, the inference compute cost.
A label like “320B-A18B” means about 320 billion total parameters and about 18 billion activated parameters. The industry commonly uses the shorthand “A active B” to describe this relationship.
2.2 The Upside: Decoupling Capacity from Compute
In a dense model, doubling the parameters doubles the compute for every token. MoE breaks that binding: growing total parameters only requires adding experts, while compute grows only with the number of experts actually activated.
You can think of it as “wide rather than deep,” with on-demand dispatch:
- The router acts like a front-desk dispatcher, sending each type of request to the experts best suited to handle it.
- Within a single batch, different tokens often route to different experts, which keeps expert utilization reasonably balanced during batched inference.
As a result, the per-call price of a sparse model can be well below that of a same-generation dense flagship. For some newer models, the unit price is only a fraction of the previous generation’s flagship. In practice this shows up as a lower effective price per token, an advantage that is especially visible for high-volume workloads.
2.3 The Costs: Memory, Load Balancing, and Fine-Tuning
Memory cannot be estimated from activated parameters. Regardless of how many experts a request activates, all expert weights must stay resident in memory. With FP16 (2 bytes per parameter), 320 billion total parameters require roughly 640 GB of GPU memory just to hold the weights, before KV cache and runtime overhead. Sparse models are therefore not “small models” — they need multi-device parallelism to run at all.
Load balancing is a systems engineering problem. Under expert parallelism, routing must dispatch tokens across devices, which is communication-heavy, and hot experts create a bucket effect. MoE training depends on routing regularization (such as a load-balancing loss), capacity factors, and token-dropping policies to keep things balanced. As a user you generally cannot adjust these training-time settings, but it helps to know how they affect inference: individual tokens being dropped or routing skewing toward a few experts can make outputs less stable in particular domains.
Fine-tuning behaves differently than with dense models. Updating only a subset of expert parameters is more parameter-efficient, but you still have to keep routing stable, avoid “expert collapse,” and ensure the activated experts can be fully loaded at inference time. For most business use cases, prompt engineering, retrieval-augmented generation (RAG), or lightweight fine-tuning is the better starting point.
3. Native Multimodality: Extending the Capability Boundary
3.1 “Native” vs. “Bolted-On”
The common industry approach to multimodality is bolted-on: a separate vision encoder is attached to a text model, then aligned with a modest amount of image-text data. Native multimodal models typically share one set of parameters and one attention mechanism across modalities from the pretraining stage onward, converting inputs such as images into a token sequence unified with text before modeling them jointly.
The engineering differences are mainly these:
- One unified interface: text and images are passed in through the same chat endpoint, so there is no need to call a vision model and a language model separately.
- More natural cross-modal reasoning: interleaved text-and-image long documents, screenshots, tables, and UI mockups can be questioned together in a single conversation.
- Better fine-grained understanding: because vision and language share representations at the base level, the model tends to be more precise about text inside images, chart values, and layout relationships.
3.2 Engineering Details Developers Should Watch
- Vision tokens consume context budget. An image is split into patches and encoded into multiple tokens. A high-resolution screenshot can take up hundreds or even thousands of tokens, so multimodal long conversations need context length planned in advance.
- Multimodal long context is heavier on memory. Image tokens also need KV cache, so memory grows noticeably when processing images in batches.
- Safety and compliance cannot be skipped. Image inputs may contain faces, identity documents, and other sensitive content, so content filtering and redaction belong in the pipeline.
3.3 Typical Use Cases
- Document understanding: structured extraction from scanned contracts, invoices, and report screenshots, then reconciliation against business system data.
- Development assistance: reading architecture diagrams, flowcharts, and UI screenshots to generate or modify code.
- Content moderation support: identifying key elements in an image first, then applying business rules for the decision.
- Data entry and customer support: turning paper forms and chat screenshots into structured data for downstream processes.
4. Sparse and Linear Attention: The Key to Long Context
4.1 The Quadratic Cost of Attention
Standard self-attention requires every token to interact pairwise with all preceding tokens, so compute grows quadratically with context length while KV cache grows linearly. That makes prefill time and memory usage climb sharply for long-context requests.
The code below estimates the attention cost of a single token during prefill using a generic formulation, and compares it against the MoE feed-forward cost:
def prefill_attention_cost(n_layers, d_model, seq_len):"""Illustrative FLOPs for attention of a single token during prefill (no sparse/linear optimizations counted)."""# Computing attention scores and the weighted sum costs roughly 2 * seq_len * d_model each.# Multiplying by the number of layers gives the per-token attention cost.return n_layers * 4 * seq_len * d_model# Illustrative parameters: layers, model dimension, activated parameters (in billions)N_LAYERS, D_MODEL, N_ACTIVE_B = 60, 6144, 18ffn_flops = 2 * N_ACTIVE_B * 1e9 # about 2 FLOPs per parameter per forward passfor ctx in (4096, 32768, 131072):attn = prefill_attention_cost(N_LAYERS, D_MODEL, ctx)share = attn / (attn + ffn_flops)print(f"Context {ctx:>6}: attention share {share:.1%}")
With these illustrative parameters, as context grows from 4K to 32K and then to 128K, the attention share rises from about 14% to about 57% and then to about 84%. In other words, the longer the context, the more attention optimizations matter for both cost and latency.
4.2 Two Families of Optimization
- Sparse attention: compute interactions for only a subset of token pairs, for example local attention inside a sliding window plus a small number of global tokens, cutting compute substantially while preserving the interactions that matter.
- Linear attention: use kernel functions or recurrent states to rewrite attention in a recursive form, so complexity grows linearly with sequence length and the per-layer state size stays fixed.
4.3 Hybrid Architectures: Balancing Quality and Efficiency
Models that rely purely on linear attention can underperform standard attention on tasks like exact retrieval and verbatim copying. The industry therefore generally adopts a hybrid design: a few layers keep full (or near-full) attention while the rest use linear or sparse attention, preserving long-range modeling ability along with inference efficiency.
For inference engineering, this architecture means:
- A different KV cache composition: linear layers only keep a fixed-size state instead of something that grows with context, which makes memory usage and time-to-first-token for long inputs more predictable.
- Operator and framework support is a prerequisite: a hybrid architecture needs an inference framework that supports sparse, linear, and standard attention operators at the same time. Before any on-premises deployment, confirm how well your hardware and framework support the target architecture.
Worth noting: MoE addresses “too much feed-forward compute,” while hybrid attention addresses “too much attention compute.” They are two complementary cost-reduction paths, and their combination is the technical foundation that lets this class of models offer strong long-context and multimodal capability at a relatively low unit price.
5. Cost and Resource Estimation: From Unit Price to Total Cost of Ownership
5.1 What to Count Beyond Unit Price
The token price is only the surface number. Real cost also depends on the price gap between input and output tokens, the quadratic growth of long context, time-to-first-token, concurrency limits, and, for self-hosted deployments, GPU rental plus operations headcount.
For self-hosting, estimate at least these four items:
- Weight memory: total parameters × bytes per parameter (2 for FP16, 1 for INT8).
- KV cache memory: a function of layer count, number of KV heads, context length, and concurrency; linear attention layers are replaced by a fixed-size state.
- Compute demand: compute per token = 2 × activated parameters, while time-to-first-token also depends on prefill length.
- Communication overhead: inter-device traffic from tensor parallelism across multiple devices and from expert parallelism.
The code below gives a reusable estimation framework; the output is illustrative:
def kv_cache_gb(n_layers, n_kv_heads, d_head, seq_len, batch, bytes_per=2):"""KV cache size in GiB for standard attention layers (one copy each for K and V)."""return 2 * n_layers * n_kv_heads * d_head * seq_len * batch * bytes_per / 1024**3def weights_gb(total_params_b, bytes_per=2):return total_params_b * 1e9 * bytes_per / 1024**3# Illustrative parameters (not a real model configuration)TOTAL_B, LAYERS, KV_HEADS, D_HEAD = 320, 60, 8, 128for ctx in (8192, 65536):cache = kv_cache_gb(LAYERS, KV_HEADS, D_HEAD, ctx, batch=1)print(f"Weights for a {TOTAL_B}B model need about {weights_gb(TOTAL_B):.0f} GiB; "f"KV cache per request at context {ctx} is about {cache:.1f} GiB")
5.2 Choosing Between API Access and Self-Hosting
| Dimension | API access | Self-hosted |
|---|---|---|
| Upfront investment | Low, pay as you go | High, requires multi-GPU servers |
| Operations burden | Low, the provider handles scaling and upgrades | High, includes inference engine, monitoring, and incident handling |
| Data compliance | Depends on the provider’s data processing terms | Data stays inside your network, stronger control |
| Elasticity | Strong, handles traffic spikes quickly | Limited by cluster size |
| Best fit | Fast validation, spiky traffic, long-tail workloads | High-volume steady calls, sensitive data, deep customization |
A simple way to decide is to compare the self-hosted cost per million tokens against the blended API price per million tokens, then compute the break-even point:
# Example: GPU hourly cost, GPU count, and output speed (tokens/sec) are all assumptionsgpu_hourly_cost, gpu_count, output_tps = 8.0, 8, 2000throughput = gpu_count * output_tps # cluster output throughputtokens_per_hour = throughput * 3600self_cost_per_m = gpu_hourly_cost / tokens_per_hour * 1e6api_price_per_m = 4.0 # assumed blended API priceprint(f"Self-hosted: about {self_cost_per_m:.2f} per million tokens; API price {api_price_per_m}")print("Self-hosting only pays off if real utilization is well above the full-load assumption used here.")
Note that the self-hosted figure in the example is a theoretical floor assuming full load and zero idle time. In reality you also pay for batch sizes limited by memory pressure, failures and maintenance windows, and redundancy for elasticity, so real cost is usually higher. Put differently, self-hosting only makes sense when call volume is consistently high and throughput is saturated; with spiky traffic or during a validation phase, an API is the better choice.
5.3 Cost Control Checklist
- Control context length: chunk long documents, summarize before asking questions, and avoid stuffing the entire corpus into every turn.
- Use prefix caching: keep system prompts and knowledge base snippets in fixed positions to improve cache hit rates.
- Compress images before uploading: scale down resolution and crop irrelevant regions as needed to reduce vision tokens.
- Tier by task: use a smaller model or a rules engine for simple tasks and reserve the large model for complex ones.
- Shave the peaks: move non-real-time work into a message queue for asynchronous processing.
- Observability first: log token usage, time-to-first-token, and per-call cost in your logging and alerting stack.
6. Evaluation: An Engineering View Beyond Leaderboards
6.1 Reading Leaderboards and Anonymous Tests Objectively
Launching publicly under an anonymous name and revealing the model later has become a common practice. It gives the vendor real traffic and real feedback — a low-cost stress test and reputation check — and gives developers a chance to try before committing. Keep a clear head when reading the conclusions, though:
- Leaderboards reflect general capability, not your workload. General benchmarks lean toward knowledge and reasoning and may not cover your business scenarios at all.
- Sampling parameters change results. Whether thinking mode is enabled and how sampling temperature is set both alter the output.
- Call volume is not a quality signal. Free and cheap anonymous models naturally attract more traffic; popularity is not quality.
6.2 Build Your Own Evaluation Loop
- Build a business evaluation set: pull 100–500 samples from historical data and real tickets, covering both typical cases and edge cases.
- Define quantifiable metrics: accuracy, field extraction completeness, hallucination rate, time-to-first-token, and cost per thousand calls.
- Fix the evaluation configuration: standardize temperature, maximum output length, and prompt templates so results are comparable across models.
- Start with a small canary: route 5%–10% of traffic to the new model, run an A/B comparison against the model in production, then ramp up.
- Run regression regularly: re-evaluate after model version updates or prompt changes to catch quality regressions.
The code below is a lightweight skeleton for a comparison harness:
from dataclasses import dataclass, field@dataclassclass CaseResult:case_id: strpassed: boolttft_ms: float # time to first tokentotal_ms: float # end-to-end latencyin_tokens: int # input tokensout_tokens: int # output tokensnotes: str = ""@dataclassclass Report:results: list = field(default_factory=list)def add(self, r: CaseResult):self.results.append(r)def summarize(self):n = max(len(self.results), 1)passed = sum(r.passed for r in r_list := self.results)ttft = sum(r.ttft_ms for r in self.results) / ncost = sum(r.in_tokens + r.out_tokens for r in self.results) / nreturn {"samples": len(self.results),"pass_rate": f"{passed / n:.1%}","avg_ttft_ms": round(ttft, 1),"avg_tokens": round(cost, 1),}report = Report()# In practice, loop over the evaluation set and call report.add(...) for each result.# Print report.summarize() to output the comparison report.
The framework is deliberately lightweight: the judging function can be swapped for regex matching, field comparison, rule validation, or manual spot checks. The point is to put quality, latency, and cost side by side in a single report.
7. Deploying on the Cloud: Architecture Recommendations
If you want to adopt this class of models quickly while avoiding the operational burden of running your own cluster, a cloud platform such as Baidu AI Cloud can support a lean deployment path:
- Access layer: call the model through a unified model service endpoint, with per-business-line timeout, retry, and fallback policies.
- Async processing and peak shaving: run batch document processing and offline labeling jobs asynchronously through a message queue.
- Data and storage: keep original files in object storage, and archive processing results and evaluation reports separately for traceability.
- Security and compliance: put content safety checks in front of the model call to screen both inputs and outputs.
- Observability: track token consumption, latency, and error rates with logging and alerting, and configure budget alerts.
- Elastic scaling: use a container platform to scale application services automatically during traffic peaks so the client side never becomes the bottleneck.
8. Risks and Caveats
- Effective utilization of long context: a long advertised context window does not mean the model can reliably retrieve information from any position in a long document. For long-document tasks, locate the key information first, then ask targeted questions.
- Multimodal outputs still need verification: add a validation step for image recognition and number or unit extraction. For critical business flows, do not write model output straight to the database.
- Custom accelerator adaptation takes time: running inference on alternative or in-house accelerators requires the framework, operator libraries, and quantization strategy to be adapted together. Architectures like hybrid attention and MoE place heavier demands on custom operators and communication, so verify accuracy and stability thoroughly before going live.
- Licensing and copyright: open weights usually come with a license agreement; confirm its scope and restrictions before commercial use.
- Avoid single-vendor dependency: decouple prompts, evaluation sets, and business logic from any specific model, and keep a model abstraction layer you can switch behind.
9. Conclusion
Native multimodal sparse large models represent a clear engineering thesis: use MoE sparse activation to cut per-call compute, use native multimodality to widen the capability boundary, and use a sparse-plus-linear hybrid attention design to keep long-context cost under control. For developers, the useful mental model comes down to three lines — total parameters determine capacity, activated parameters determine compute, and attention structure determines long-context cost. Combine that with quantitative estimation for your own workload, small-traffic evaluation, and continuous cost control, and you can find a better balance between capability and cost.
