Next-Generation Sparse LLMs: How to Balance Low Cost and Native Multimodality

    This article examines how next-generation sparse large models reconcile low inference cost with native multimodal understanding. It first breaks down where MoE inference costs actually come from, then explains the long-context bottleneck through attention complexity and KV cache. Four coupled design directions are explored in depth: sparse-linear hybrid attention, depth compression with representation reuse, and native multimodality. The article also covers the engineering work needed to turn architectural gains into serving efficiency—kernel support, prefix caching, continuous batching, and context budgeting—alongside a six-dimension evaluation framework and a practical implementation checklist to help developers select, evaluate, and deploy these models with cost in mind.

    Introduction

    Over the past two years, the capability frontier of large models has kept expanding: cross-file understanding across a codebase, long-document question answering, video content analysis, and screenshot-driven UI automation have gradually moved from demos into production. Alongside this progress sits an unavoidable hurdle—inference cost. When inputs grow from a few thousand tokens to hundreds of thousands or even millions, a model must not only “see far” but also “afford to see”—deliver acceptable latency and price.

    At the same time, the shape of the input is changing. End users rarely submit pure text: a screenshot of an error, a PDF technical document, or a screen recording of an operation all need to be understood directly. This pushes models away from “a text model with bolt-on vision” toward “native multimodality.”

    This article discusses the design of next-generation large models across five dimensions:

    • Cost structure;
    • Long-context bottlenecks;
    • Hybrid attention;
    • Depth and representation reuse;
    • Native multimodality.

    The goal is to help developers form a clear framework for selection, evaluation, and engineering deployment.

    Four Design Directions of Next-Generation Models

    If we summarize recent architectural progress, it can be grouped into four directions:

    1. Sparse activation: using a Mixture-of-Experts (MoE) structure so each token activates only a subset of parameters, decoupling “model capacity” from “per-token compute”;
    2. Long-context attention redesign: replacing full attention—costly on long sequences—with linear attention, sparse attention, and their combinations;
    3. Depth and representation-reuse optimization: improving residual pathways so fewer layers carry equal or stronger expressive power;
    4. Native multimodality: jointly modeling text, images, video, and files in the same sequence space.

    These four directions are not independent; they form a coupled whole. Longer context means more visual tokens enter the sequence, and attention cost scales up accordingly; lower activated parameters free up compute budget for native multimodality; fewer layers directly reduce the number of KV cache copies. Understanding this coupling is the foundation for everything discussed below.

    First, Clarify Cost: What Determines MoE Model Overhead

    To discuss a “low-cost model,” we first need to know where cost actually occurs. For a sparse-activation architecture, a single inference request’s cost can be roughly broken into four parts.

    Cost source Main determinant Change with context length
    Weight reading (decode phase) Activated parameter count Essentially unchanged (independent of context length)
    Attention computation Number of layers, attention mechanism, sequence length Full attention grows quadratically with length
    KV cache access and footprint Number of layers, attention-head structure, sequence length Grows linearly and may dominate at long context
    Multimodal encoding (vision, etc.) Size of image/video inputs Depends on multimodal input volume

    From this we can extract three key judgments.

    First, total parameters mainly determine “capacity,” while activated parameters mainly determine “per-call cost.” In a sparse-activation architecture, total parameters determine the breadth of model knowledge; but only a subset actually participates in computation for each token. For scenarios that bill per token and need large-scale online serving, the scale of activated parameters is the direct cost lever.

    Second, Prefill and Decode are two fundamentally different workloads. In the decode phase, generating each token requires reading all activated weights and the entire existing KV cache, while each parameter typically participates in only one operation. Its arithmetic intensity (floating-point operations per byte of data read) is usually very low, so performance is limited by memory bandwidth rather than compute. Therefore:

    • When requests are few and batches are small: reducing activated parameters directly lowers per-token cost; reducing the KV cache directly lowers per-request memory footprint;
    • When concurrency is high and batches are large: computation gradually becomes the bottleneck, and the attention compute of long sequences becomes the key throughput constraint.

    Third, compute efficiency cannot be judged by single-card peak FLOPS alone. Sparse attention often produces irregular memory-access patterns, which standard attention kernels struggle to execute efficiently. Only with specially optimized kernels can a theoretical reduction in computation translate into real cost and throughput improvements.

    Illustrative code (for estimation only, no framework dependency): growth of attention computation at long context

    1. def attention_flops_dense(layers, n, d_model):
    2. """Full attention: QK^T and A·V, two matrix multiplications, ~4 * n^2 * d_model FLOPs per layer"""
    3. return 4 * layers * n * n * d_model
    4. def attention_flops_linear(layers, n, d_model):
    5. """Linear attention: ~O(n * d_model^2) per layer"""
    6. return 4 * layers * n * d_model * d_model
    7. layers, d_model = 45, 4096
    8. for n in [8_192, 32_768, 131_072, 1_048_576]:
    9. dense = attention_flops_dense(layers, n, d_model)
    10. linear = attention_flops_linear(layers, n, d_model)
    11. print(f"Context {n:>8}: dense {dense/1e12:>10.1f} TFLOPs, "
    12. f"linear {linear/1e12:>6.1f} TFLOPs, ratio ~{dense/linear:.0f}x")

    The example output looks like this. In the formula, the ratio of context length to d_model exceeds 200x at the million-token scale. In real models, the actual gap is affected by gating, normalization, kernel efficiency, and other factors, but the order-of-magnitude trend is consistent.

    1. Context 8192: dense 3.7 TFLOPs, linear 2.8 TFLOPs, ratio ~1x
    2. Context 32768: dense 59.2 TFLOPs, linear 11.0 TFLOPs, ratio ~5x
    3. Context 131072: dense 947.6 TFLOPs, linear 44.2 TFLOPs, ratio ~21x
    4. Context 1048576: dense 60626.6 TFLOPs, linear 353.7 TFLOPs, ratio ~171x

    Conclusion: the longer the context, the more pronounced linear attention’s cost advantage; with short inputs of a few thousand tokens, the gap is limited and can even be close due to implementation overhead. This also explains why next-generation architectures generally adopt a “hybrid” rather than “pure linear” approach.

    First Principles of Long Context: Complexity and Memory

    Where Quadratic Complexity Comes From

    The attention mechanism of a standard Transformer requires every token in the sequence to compute relevance with all other tokens. When sequence length is n, the attention matrix is n × n, and both computation and memory grow quadratically with length. This property makes long context one of the most expensive parts of inference serving.

    KV Cache: The Hidden Cost of Long-Context Serving

    During autoregressive generation, to avoid recomputation, the model caches the Key and Value vectors of past tokens—the KV cache. Its memory footprint can be approximated as:

    KV cache bytes ≈ 2 × layers × KV heads × head dim × sequence length × batch × bytes per element

    Take a configuration of “60 layers, 8 KV heads, head dim 128, half precision, batch size 1”:

    • Per-token KV cache is about 2 × 60 × 8 × 128 × 2 B ≈ 240 KB;
    • Processing one 1-million-token request needs about 240 GB of memory for the KV cache alone.

    And this doesn’t count model weights and runtime overhead. The KV cache squeezes out memory available for concurrency and directly limits how many requests a single card can serve at once.

    Conclusion: in long-context serving, reducing attention compute and reducing KV cache scale are equally important. The former determines the compute cost per request; the latter determines the concurrency the system can handle.

    Key Design One: Sparse and Linear Hybrid Attention

    Linear Attention: Replacing History Cache with a Fixed-Size State

    The core of linear attention is compressing historical information into a fixed-size state. Standard attention must store the Key/Value of every past token; linear attention maintains a fixed-dimension “memory matrix” (an association matrix in outer-product form). Each time a new token is read in, the matrix is updated recursively:

    • Compute complexity drops from growing quadratically with sequence length to growing linearly;
    • The state added at each step is constant in size and no longer grows linearly with context.

    It is essentially a “learned compression memory”: using a limited state capacity to summarize all history in exchange for a dual reduction in compute and memory. For tasks dominated by local dependencies—code completion, document summarization—linear attention already performs well. But in scenarios that “recall a specific early instruction accurately across multiple turns of dialogue,” the memory capacity of a fixed state is always an upper bound—like a notebook that keeps being overwritten, details that need to be retrieved precisely may be diluted.

    Sparse Attention: Computing Only Necessary Token Pairs

    Sparse attention abandons full computation and computes attention only between selected tokens, according to a predefined or learned pattern. Common combinations include:

    • Sliding window: each token attends only to nearby local context;
    • Global tokens or global layers: a few positions/layers retain full attention, acting as “information hubs”;
    • Dynamic retrieval-style selection: for each query, dynamically pick the few most relevant historical blocks to participate, with block-level sparsity costing approximately linearly.

    The advantage of sparse attention is that it can precisely access the selected historical content. The cost is that information outside the pattern must propagate indirectly; irregular memory-access patterns also increase the difficulty of kernel implementation.

    Hybrid Architecture: Letting Each Mechanism Do Its Job

    Combining the two by layer, by head, or by budget has been the mainstream direction for long-context models in recent years. Using a publicly disclosed example configuration from a frontier model generation as a reference:

    • The previous generation used full attention: 355B total parameters, 32B activated parameters, 92 layers;
    • The new generation uses sparse-linear hybrid attention: 320B total parameters, 18B activated parameters, 45 layers.

    The two have roughly comparable total parameter counts, activated parameters reduced by about 44%, and layers nearly halved. According to its public description, compared with the previous generation:

    • Attention computation reduced by about 3.01x;
    • KV cache size reduced by about 4.44x;
    • Under the sparse-activation architecture, since fewer parameters participate in computation per token, the model can reach comparable or even better overall performance at lower compute cost.

    This data reveals an important design philosophy: hybrid attention isn’t simply “removing” global attention, but placing the limited global-interaction budget on the most critical layers. Local layers maintain fine-grained modeling at low cost, while a few global layers handle cross-sequence information integration.

    1. Illustration: inter-layer division of labor in hybrid attention (illustrative, not a specific implementation)
    2. Layer 0 1 2 3 ... 40 41 42 43 44
    3. Attn local local local global ... local local global local local
    4. ↑_______ handles local & recent info _______↑ ↑__ handles global aggregation __↑

    Indexer and Index Cache Compression: Optimizing the Cost of the Sparse Mechanism Itself

    Dynamic retrieval-style sparse attention relies on a lightweight “indexer”: it computes a coarse relevance score between each query and historical tokens, then selects the top-scoring subset to participate in attention. This is like having the model “glance at a table of contents” before “flipping to the relevant chapter,” preserving long-range information at far lower cost than full attention.

    But the indexer itself produces a cache and can become a new source of overhead. One publicly disclosed approach uses index pooling:

    • Weighted pooling of multiple cache vectors produced by the indexer, compressed into a single vector;
    • The longer the context, the more significant the benefit of compression, reducing the indexer’s own latency and memory overhead at the million-token scale.

    This design thread is worth noting for anyone building long-context systems: any sparsification mechanism needs cost control of its own, otherwise it just moves the bottleneck from one stage to another.

    Engineering Details to Watch

    • Kernel support: both linear and sparse attention need customized efficient kernels. Before deployment, confirm whether the inference framework and hardware backend support them natively; if you fall back to a generic implementation, theoretical gains may shrink significantly;
    • Inference consistency: if training and inference use different attention implementations (e.g., exact during training, approximate during inference), errors may amplify at long context, requiring dedicated output-consistency verification;
    • Positional encoding and length extrapolation: as sequences get longer, the positional encoding scheme and scaling strategy need to be adjusted accordingly;
    • Quantization compatibility: standard optimizations like KV cache quantization and weight quantization need to be verified for compatibility with the new cache structure;
    • Accuracy boundaries: the performance of linear and sparse components on “needle-in-a-haystack” tasks should be measured with business data, not judged solely by public leaderboards.

    Key Design Two: Depth Compression and Representation Reuse

    Why the Number of Layers Matters

    Every layer of a Transformer contains a full set of parameters and a full KV cache. The more layers:

    • The stronger the model’s expressive power usually is;
    • But forward computation, weight reading, and KV cache all grow linearly.

    In the example above, layers dropped from 92 to 45, meaning the “layer factor” of the KV cache is directly halved. Combined with the sparsification of attention, the overall cache is compressed to about one-fifth of the original, consistent with the publicly disclosed 4.44x figure.

    Residual Connections and Hyper-Connections

    A standard Transformer adds each layer’s output to its input through a residual connection, forming an identity “highway” running through the network. This path ensures gradients flow smoothly in deep networks, but the connections between layers are fixed and static—different layers cannot exchange information on demand.

    Hyper-Connections generalize this fixed path into a learnable connection mechanism:

    • No longer just “previous layer → next layer” single-path addition, but more flexible information pathways between layers;
    • Essentially building a learnable “interchange” in the depth dimension: shallow details can reach deeper layers faster, and deep abstractions can flow back more effectively, enhancing gradient propagation and representation reuse.

    Manifold Constraints: Flexibility Without Instability

    Once connections become highly free, training stability becomes a new problem: signals propagated between layers may be over-amplified or attenuated, and the optimization landscape becomes rugged. Manifold-Constrained Hyper-Connections constrain connections within a suitable manifold space—think of it as setting traffic rules for the “interchange”: retaining the expressive power of flexible connections while keeping the scale and direction of information stable as it propagates through deep networks.

    Its publicly disclosed effect: the model showed no capability regression despite a large reduction in layers, and in fact further amplified the benefits of scaling up model size. This means that, given a fixed compute budget, depth is a “scarce resource” that can be utilized more efficiently. The engineering implications are:

    • Under the same memory and throughput constraints, models with fewer layers are often easier to deploy and shard in parallel;
    • When evaluating a model, don’t just look at total parameter count; also consider “representation efficiency per parameter.”

    Engineering Details to Watch

    • Kernel adaptation: new inter-layer connection patterns may change the computation graph structure. Whether mainstream inference engines and training frameworks support them natively, or require custom operators, affects deployment cost;
    • Training-stability verification: such changes primarily work by influencing the training process; teams that reuse existing weights for incremental fine-tuning or continued pretraining need to verify weight compatibility under the new structure;
    • Distributed sharding: cross-layer connections in the depth dimension change the pipeline-parallel sharding scheme, requiring evaluation of communication overhead.

    Key Design Three: From “Bolted-On” to Native Multimodality

    The Difference Between Two Multimodal Approaches

    Traditional multimodal solutions typically adopt a “bolt-on” structure:

    1. An image first passes through an independent vision encoder to extract features;
    2. A projection layer then maps these into the language model’s feature space;
    3. Finally, they are concatenated into the text sequence.

    This approach is engineering-mature and fast to iterate, but has two inherent limitations:

    • Modality fragmentation: the vision and language sides are pretrained separately, and alignment between their semantic spaces relies on limited image-text data;
    • Information loss: visual features are usually compressed into a small number of “visual tokens,” so details like small fonts, complex charts, and UI layouts are easily lost during encoding.

    Native multimodality jointly models images, video, text, and even file content in the same sequence space during pretraining. Visual encoding is no longer a “bolt-on front stage” but an intrinsic part of the model’s inference process.

    Three Practical Values of Native Multimodality

    First, visual information participates directly in the reasoning chain. Take a coding scenario—a model’s typical workflow might be:

    1. Read a requirements screenshot;
    2. Locate relevant files in the codebase;
    3. Write or modify code;
    4. Run tests and view errors;
    5. Fix code based on error or page screenshots.

    Under a bolted-on architecture, each screenshot may first need to be handed to a vision model to convert into a text description, after which the language model acts on the description—information is continually lost in conversion. Native multimodality lets visual understanding happen directly inside the same model’s reasoning loop, reducing the double loss of “look → describe → re-understand.”

    Second, it complements long-context capability. Once images and video are encoded into tokens, they are likewise part of the sequence; the more visual content, the longer the sequence, and the greater the pressure on attention and KV cache. The long-context cost reduction from hybrid attention is exactly what makes inputs like “long-video understanding” and “multiple high-resolution screenshots” economically feasible; conversely, native multimodality extends the value of long context from “reading far” to “seeing broadly.” This is precisely why next-generation architectures address the two together in the same generation of products.

    Third, a unified architecture brings operational and iteration simplicity. A single model covering multiple modalities means:

    • The server side only needs to maintain one inference stack;
    • Version upgrades, capability rollbacks, and canary releases can be done uniformly;
    • No need to coordinate version compatibility across multiple models.

    A Sense of Scale for Visual Tokens

    Engineering requires a basic awareness: the token consumption of visual input often far exceeds expectations. A single high-resolution image can produce hundreds to thousands of tokens after encoding; a video, sampled by frame, is even more substantial.

    Roughly estimating “1024 visual tokens per frame, 25 fps, 1 minute of video,” the visual portion alone exceeds 1.5 million tokens—equivalent to raising the context length by an order of magnitude. For video understanding and screen-recording analysis applications, the visual-token budget must be included in capacity planning alongside the text budget.

    Common Problems and Responses in Multimodal Deployment

    • Input preprocessing: resolution, slicing strategy, and frame-rate sampling directly determine token count and cost; they should be uniformly configured at the ingestion layer and support on-demand adjustment;
    • Detail fidelity: native multimodality is not omnipotent. Tiny fonts, complex tables, and dense charts still require reasonable high-resolution input or local cropping to ensure fidelity;
    • Evaluation coverage: multimodal capability evaluation should include real business samples, not rely solely on general image-text benchmarks;
    • File inputs: for structured content like documents and tables, clarify the parsing path (input directly as a file, or preprocess into text/image); different paths differ significantly in both cost and effectiveness.

    From Architecture to Engineering: Key Actions for Servicing Deployment

    To turn architectural gains into real serving efficiency, a series of engineering actions is needed.

    Inference Side: Actually Realizing the Benefits of Sparsification

    • Confirm kernel and backend support: whether hybrid attention is efficiently supported by the inference framework is the first question to answer before deployment;
    • Cache reuse: in code-assistant and multi-turn dialogue scenarios, prefix caching can reuse large amounts of repeated context KV, significantly reducing redundant computation;
    • Continuous batching: combined with dynamic batching, achieving higher concurrent throughput on the same hardware;
    • Decoding acceleration: when compatible with the model architecture, general acceleration schemes like speculative decoding can further reduce decode latency;
    • Quantization and parallelism strategy: choose tensor parallelism, expert parallelism, and other sharding schemes based on the activated-parameter scale, avoiding resource waste for small-batch requests.

    Ingestion Side: Controlling Input Cost with Engineering

    • Tiered context budgets: set token quotas separately for system prompts, conversation history, retrieval snippets, and visual inputs; auto-truncate or summarize when exceeded;
    • Retrieval and compression upfront: for long documents, first retrieve relevant snippets, then summarize and compress them, rather than stuffing the entire material into context;
    • Visual input slimming: choose image resolution based on business need, and sample video by key frames rather than defaulting to full-frame input;
    • Streaming plus caching: unified cache management for high-frequency repeated system prompts and knowledge-base snippets.

    Operations Side: Building an Observable Cost View

    By bringing token consumption, time to first token (TTFT), time per output token (TPOT), cache hit rate, and other metrics into monitoring and logging, and breaking down cost by business line, scenario, and user, you can pinpoint the true sources of high consumption and then optimize prompt design, context strategy, and model routing (simple requests to small models, complex requests to large models).

    Cloud Deployment: Elasticity and Observability

    For teams that want to bring inference services online quickly, they can rely on the general capabilities of cloud platforms such as Baidu AI Cloud to complete deployment:

    • Use a container platform for elastic scaling and canary releases of inference services;
    • Use object storage to centrally manage model files and multimodal input assets;
    • Use logging and monitoring/alerting services to establish continuous observation of latency, throughput, and cost.

    Selection and Evaluation: Six Dimensions

    For next-generation architecture models, build an evaluation framework around the following six dimensions:

    Dimension Key question Verification method
    Long-context capability Effective recall and reasoning quality under very long inputs “Needle-in-a-haystack” and long-document reasoning test sets built from business corpora
    Multimodal capability Whether understanding of screenshots, charts, video meets business needs Image-text alignment and detail recognition tests on real business samples
    Cost efficiency Whether activated parameters and KV cache scale can support target concurrency Capacity estimation by target QPS and context length
    Engineering compatibility Whether the inference framework, hardware backend, and quantization scheme are supported Pre-deployment technical validation (PoC) and kernel-availability checks
    Stability and consistency Whether output is stable and reproducible at long context Repeated-sampling consistency tests and abnormal-input tests
    Ecosystem and tooling Whether tool calls and structured output needed for coding scenarios are complete End-to-end run-through tests of real coding tasks

    Among these, cost efficiency is the most easily underestimated. A common mistake is looking only at “price per million tokens” while ignoring the KV cache’s constraint on concurrency at long context. The right approach is:

    1. Clarify the actual distribution of business input lengths;
    2. Estimate single-card concurrency based on this;
    3. Then convert to a combined cost of “per thousand requests” or “per user per month.”

    Limitations and Risks: Viewing the Benefits Objectively

    Every architectural innovation has its applicability boundaries, and next-generation designs are no exception.

    Accuracy boundaries of linear and sparse attention: fixed-state memory may still be limited on tasks requiring precise retrieval of large amounts of historical detail. Hybrid architectures alleviate this by retaining global layers but don’t fully eliminate it. For scenarios with extremely high recall requirements, judge based on actual testing.

    Implementation complexity: designs like hybrid attention and manifold-constrained hyper-connections place higher demands on training and inference infrastructure, including customized kernels, distributed parallelism strategies, and numerical stability. This means teams deploying such models need a certain level of engineering capability, or can lower the barrier by relying on mature inference services on a cloud platform.

    Multimodal ≠ universally applicable: native multimodality improves the integration of visual information with the reasoning chain, but for detail recognition like tiny text and complex charts, the quality of input preprocessing remains critical.

    Evaluation limitations: there is often a gap between public benchmark scores and actual business performance, especially for long-context and multimodal capabilities, which are heavily influenced by how the evaluation set is constructed. Using business data as the standard is always the more reliable choice.

    Implementation Checklist

    1. Clarify scenario priorities: first confirm whether the business is “mostly long text,” “mostly multimodal,” or “mostly coding,” then choose a matching combination of model capabilities;
    2. Run a PoC on a small sample: build a 100–500 item evaluation set from real business samples, covering typical input lengths and modality combinations;
    3. Load-test rather than estimate: run end-to-end load tests at target concurrency and context length, recording TTFT, TPOT, throughput, and memory footprint;
    4. Build a cost model: incorporate token price, cache hit rate, and visual-input ratio into a unified model and review it regularly;
    5. Keep a fallback path: for cost- or latency-sensitive scenarios, design model routing and fallback strategies to avoid a single model carrying all traffic;
    6. Track iterations continuously: sparse architectures and multimodal technology are still evolving rapidly; re-evaluate candidate models quarterly.

    Conclusion

    The design logic of next-generation large models can be summed up in one thread: without sacrificing capability, use the resources of every dimension more efficiently.

    • Sparse activation decouples “capacity” from “compute”;
    • Hybrid attention pulls “long context” back from quadratic complexity to near-linear;
    • Manifold-constrained hyper-connections let “depth,” a scarce resource, carry more representation;
    • Native multimodality turns visual understanding from a bolt-on stage into an intrinsic part of the reasoning chain.

    For developers, understanding the cost logic behind these designs is more valuable than memorizing a model’s parameter specs. Because architectures iterate and specs change, but the engineering essence—“do more with fewer resources”—doesn’t change. In selection and deployment, returning to the business scenario and real load-test data is what makes for architecture decisions that stand up to cost scrutiny.