Native Multimodal LLM Architecture Explained: A Systems Engineering Practice Guide
This guide explains how the evaluation of large models has shifted from benchmark scores to cost per unit of effective output, and walks through the architectural levers behind that shift: sparse MoE activation that decouples parameter capacity from per-token compute, hybrid attention stacks that bring long-sequence memory and latency back under control, practical strategies for turning a nominal long-context window into a usable one, native multimodal training versus bolt-on alignment, agentic loops with independent visual verification, and hardware-portable deployment. It closes with a cost model that makes "an order of magnitude cheaper" accountable, plus a selection and rollout checklist covering architecture, quality, compliance, and operations.
1. The starting point: the engineering yardstick has changed
Models used to be evaluated mainly on general-purpose benchmark scores. Today the question is cost per unit of effective output. Three trends drive that change:
- Contexts are expanding. Million-token windows turned “paste the entire codebase into the prompt” from impossible into routine — but the bill arrives as KV Cache memory and time-to-first-token, both of which climb together.
- Modalities are multiplying. Text, images, and audio now land in the same representation space, so information density on the input side and the token budget per request feed directly into the invoice.
- Interactions are becoming multi-turn. Agent workloads are not single-shot Q&A. A task phrased as “build this feature” can fan out into dozens of model calls, and the cost multiplies.
As a result, architectural optimization is no longer an academic topic. It reduces to three measurable levers: sparsity to cut per-token compute, linearization to cut sequence complexity, and hardware–software co-design to cut the price per unit of compute. The sections below take each in turn.
2. Architecture layer one: how sparse activation decouples “capacity” from “compute”
The core idea of Mixture of Experts (MoE) is to split knowledge capacity from computation. A gating network routes each token to a few experts; total parameter count can be very large while the FLOPs per token scale only with the number of active parameters.
Public industry practice puts designs of this class in a typical range of several hundred billion total parameters with tens of billions up to roughly twenty billion active parameters — a sparsity ratio of more than 100x. The same compute budget can therefore hold far more parametric knowledge, but the engineering complexity moves to three places:
1) Routing load balance. Uneven expert load means some experts saturate while others sit idle, and real throughput falls well short of the theoretical number. The usual countermeasures are an auxiliary balancing loss, headroom reserved through the capacity factor, and either dropping or falling back on tokens that overflow a bucket.
2) Communication becomes the bottleneck. Under expert parallelism, tokens have to be reshuffled across devices (all-to-all), a step that easily dominates the time spent on compute itself. In practice this means overlapping communication with computation, and keeping the expert parallel domain inside the high-bandwidth interconnect.
3) Training and inference see different distributions. Routing distributions inside an inference batch differ from training, and a capacity factor set too tight produces accuracy jitter.
def topk_router(hidden, W_gate, num_experts=64, top_k=8, capacity=1.25):# hidden: [num_tokens, hidden_dim]logits = hidden @ W_gate # [T, E]probs = softmax(logits, dim=-1)topv, topi = probs.topk(top_k, dim=-1)topv = topv / topv.sum(-1, keepdim=True) # renormalize selected expert weights# capacity factor > 1 leaves headroom for routing skew and reduces token droppingcap = math.ceil(capacity * hidden.shape[0] * top_k / num_experts)return topi, topv, cap
Practical rule of thumb: the payoff from a sparse model correlates with batch size. In low-concurrency, short-input scenarios the routing overhead and weight-loading overhead take a larger share, and the benefits of sparsity get diluted considerably. Offline batch processing and medium-to-high concurrency online serving are where it belongs.
3. Architecture layer two: hybrid attention, pushing quadratic complexity back to linear
Standard attention carries two costs: compute grows quadratically with sequence length, and KV Cache memory grows linearly with length while multiplying with concurrency. The second one is usually the real ceiling for long-context serving — not that you cannot compute, but that you cannot fit.
Linear attention and state-space structures bring sequence complexity down to linear, at the price of weaker long-range exact recall. That is why the mainstream approach is a hybrid stack: most layers use linear attention or sliding windows for local modeling, while a few layers keep global attention for precise retrieval and long-range dependencies, inserted at fixed intervals.
Why not go purely linear? Because the trade-off point has to sit between “local coherence” and “global exactness.” Code comprehension and long-document question answering both require the model to point back precisely to a symbol definition far away, and purely recurrent structures degrade noticeably on exactly those tasks.
The engineering gains run in both directions:
- On memory, KV Cache drops sharply, so the same memory budget carries higher concurrency.
- On latency, autoregressive decoding is memory-bound — every step reads the KV Cache again — so a smaller cache converts directly into higher tok/s.
Risks to validate up front: a hybrid structure changes numerical behavior. Accuracy loss after low-bit quantization differs from a pure-attention model, and operator implementations need to be checked one by one on the target hardware. Do not assume “supported by the framework” equals “performance is usable.”
4. Long context: advertised length is not usable length
“Supports a million tokens” is a product parameter. “Completes tasks reliably at a million tokens” is an engineering capability. Three layers sit between them:
1) Positional extrapolation. With rotary position embeddings, the frequency scaling and interpolation strategy determine whether the model can genuinely use the long window; long-text continued training or long-context fine-tuning is usually needed to consolidate it.
2) Effective recall. Recall decay for information buried in the middle of a long context is a repeatedly confirmed phenomenon. The engineering response is to place the most critical information at the beginning or the end, and to chunk and restate long documents rather than simply concatenating head and tail.
3) Memory and latency budget. Viable measures include paged KV Cache management, KV quantization, and prefix cache reuse — system prompts and project context have high hit rates, and the cost reduction from reuse is often more significant than switching models.
The key insight: context is a working set, not a warehouse. In codebase scenarios the right order is “retrieve first, then expand progressively as the task phase requires,” not filling the window in one shot. Mixing long context with vector retrieval is more economical and more controllable than simply piling on length in most scenarios.
For evaluation, split into two classes: needle-in-a-haystack style tests (hiding the needle at different depths in a long document) measure recall, and multi-hop reasoning tests measure information integration. One aggregate score must not mask both.
5. Native multimodality: unified representation and token budget
“Native multimodal” means multimodal corpora were mixed in during pretraining, rather than adapters bolted onto a text model for late-stage alignment. The engineering implications of that distinction are direct:
- Alignment quality cannot be patched later. Cross-modal semantic alignment is determined by the pretraining data mixture. On the engineering side you can only optimize inputs and outputs, not fix deficiencies in the representation.
- Images are token hogs. Tiling and resolution-adaptive strategies decide how many tokens a single image expands into, and therefore both cost and context occupancy. Control information density on the input side: downsample, crop by region, send in segments.
- Audio is usually a continuous or discrete representation after frame-level downsampling, sensitive to sample rate and duration, and it needs its own truncation policy.
- Structured output on the output side is non-negotiable. Requiring the model to emit JSON, a unified patch format, or a scene description file substantially reduces downstream parse failures — the most easily overlooked source of stability in agent scenarios.
6. From “answering” to “delivering”: agentic loops and visual self-checking
The essential difference between single-turn Q&A and an agent is that external state and verifiable signals exist. The general loop is: plan → execute → observe → verify → correct.
Visual self-checking is especially valuable here. For frontend pages and 3D modeling tasks the artifacts are themselves visual, and defects like “is the layout aligned” or “are the model proportions reasonable” are hard to cover with assertions. Feeding the rendered result back in as input — a page screenshot, multi-view previews of a model — produces a verification signal in the visual dimension.
Three hard requirements when designing this loop:
- The verifier must be independent of the generator. If you only ask the model whether its own output is good, it will tend to confirm itself. Bring in external objective signals: test pass rates, render diff ratios, geometry constraint checks.
- Feedback must be localizable. Give diff regions, line numbers, screenshot crops — not a vague “something looks off.” The information density of the feedback determines how fast corrections converge.
- There must be an iteration cap and a fallback path. Visual judgment carries overfitting risk, and the model can oscillate between “fix A, break B.” At that point, converge deliberately to human intervention or shrink the task granularity.
def agentic_loop(task, model, tools, max_iter=8):state = {"goal": task, "history": []}for i in range(max_iter):plan = model.plan(state) # structured planpatch = model.act(plan, tools) # code / scene description filereport = run_checks(patch, tools) # tests, static analysis, rendered screenshotsverdict = model.critique(patch, report) # review including visual evidenceif verdict.passed:return patch, {"iterations": i + 1}state["history"].append({"attempt": i, "patch": patch,"evidence": report.artifacts})raise RuntimeError("Did not converge: fall back to human review or reduce task granularity")
Task granularity is the single biggest determinant of convergence rate. Splitting “build a page” into three independently verifiable subtasks — component structure → styling → interaction states — typically yields a much higher first-pass rate than generating everything at once. The effect on cost is larger than any unit-price discount.
7. Deployment: hardware portability is an architectural constraint, not an afterthought
To run large-scale distributed inference on non-mainstream accelerator hardware, portability has to hold at three abstraction layers at the same time:
1) The operator layer. Do attention, quantized matmul, and normalization custom kernels have equivalent implementations on the target hardware? If not, how much performance is lost falling back to generic implementations? This determines the distance between “it runs” and “it is usable.”
2) The communication layer. The maturity of collective communication libraries and their topology awareness directly determine scaling efficiency. Expert-parallel all-to-all is the most sensitive case, so the high-bandwidth domain inside a node and the network between nodes must be treated differently.
3) The precision layer. Support matrices for low-precision and mixed-precision formats vary widely across hardware. Validate numerical stability during selection, not while debugging accuracy regressions after launch.
Experience-based parallelism configuration: tensor parallelism is bounded by intra-node bandwidth and should stay within a node where possible; expert parallelism needs a high-bandwidth interconnect and should be co-located with its EP domain; pipeline parallelism is latency-sensitive per request and suits throughput-first offline scenarios; then scale concurrency horizontally with data parallel replicas.
Key serving components: continuous batching to raise hardware utilization, prefix caching to amortize fixed overhead, and constrained decoding (schema or grammar constraints) to guarantee parseable output.
When building this service on the cloud, general-purpose capabilities combine into a complete foundation: a container platform for elastic scaling and canary releases, object storage for model weights and artifact files, message queues to decouple long-running tasks asynchronously, and logging plus monitoring and alerting to cover end-to-end SLOs. The general-purpose product categories offered by Baidu AI Cloud can serve as the engineering base for a self-built inference platform.
Change management: any change to a model version, a routing policy, or a prompt template should be treated as a release. That requires versioned offline evaluation sets that support comparison, and a fast rollback path.
8. Cost and performance: turning “an order of magnitude cheaper” into an accountable model
The cost of a single request can be written as:
cost = input tokens × input unit price + output tokens × output unit price + fixed overhead (queuing / retries / cache misses)
def request_cost(in_tok, out_tok, p_in, p_out, retries=1.0, cache_hit=0.0):effective_in = in_tok * (1 - cache_hit) # tokens served from prefix cache are priced lowerreturn (effective_in * p_in + out_tok * p_out) * retries
This formula leads to three counterintuitive conclusions:
- In short-input, low-concurrency scenarios the unit-price advantage of sparsity is diluted by fixed overhead. At that point the bottleneck is weight loading and scheduling, not compute.
- In long-context scenarios cost is capped by memory, which sets the concurrency ceiling. Architecture — attention structure above all — therefore determines unit cost indirectly through concurrency.
- In agent scenarios the invisible majority is failed retries and rework. Moving first-pass convergence from 50% to 80% usually saves far more than switching to a cheaper model. This is why “improve task decomposition and verification quality” takes engineering priority over “cut prices.”
The companion cost-reduction measure is tiered routing: intent classification, format conversion, and field extraction go to small models or a rule engine, while complex reasoning and long-context tasks go to the large model, with verification results used as the escalation condition. Caching splits into three layers — exact cache, prefix cache, semantic cache — and semantic caching should be used cautiously, because the cost of an incorrect hit is usually higher than the amount saved.
9. Selection and rollout checklist
- Architecture assessment: are the sparsity ratio, the attention mixing ratio, and the KV Cache strategy publicly documented? Can you run the benchmark on your own target hardware?
- Quality assessment: do not rely on public leaderboards alone. Build your own task set, and use automatically judgeable metrics: code runnability rate, frontend visual consistency, multi-hop retrieval accuracy, structured-output compliance rate.
- Compliance assessment: are the obligations of the open-source license (attribution, modification notices, patent clauses) compatible with your commercial scenario? Do data masking, retention, and data-transfer policies for inputs meet your industry’s regulatory requirements?
- Operations assessment: are SLOs explicitly defined for time-to-first-token, per-token output latency, success rate, and degradation path? Are cost alert thresholds actionable down to specific business owners?
10. Conclusion
The competitive focus of this generation of models has shifted from parameter scale to cost per unit of effective output, and the latter is determined jointly by architecture and systems engineering. Sparse activation resolves the mismatch between capacity and compute; hybrid attention resolves memory and latency on long sequences; native multimodality resolves information density on the input side; the visual self-checking loop resolves the verifiability of delivered quality; and heterogeneous hardware adaptation resolves supply autonomy. Remove any one of these links and the gains from the others fail to materialize. For developers, the most pragmatic path is to build your own task set and verifiers first, then let the architecture and deployment decisions be tested against them.
