Ultra-Sparse MoE and Hybrid Attention: How a Low-Cost Multimodal Foundation Model Is Built

    A practical reading of a Flash-class native multimodal design—about 320B total parameters, ~18B active, 45 layers, and a native 1M context—covering where capability comes from, how cost is cut with ultra-sparse MoE, wide residuals, and linear/sparse hybrid attention, how to read the benchmarks, and how to use the model in production.

    1. From “More Parameters” to Intelligence per Token

    For the past two years, the main line of large-model competition was clear: larger scale, longer context, and more tool use. After 2025, that line shifted—from “the strongest model” to “the most useful model per unit of token cost.” One visible signal is that vendors no longer ship only a single flagship. They use the same generation of architecture to offer both a flagship tier and a Flash-class tier, so developers can allocate budget by scenario.

    A recently discussed class of Flash-class native multimodal models (320B total parameters, about 18B active, 45 layers, native 1M context) can be summarized as: ultra-sparse MoE + a shallow Transformer + linear/sparse hybrid attention + wide residual information mixing + native multimodal pretraining.

    Almost none of these components is a new invention. Combining them into a system that actually runs and cuts cost by about 10× is an engineering result. The rest of this article follows four layers: where capability comes from, how cost is reduced, how to read the evaluations, and how to use the model in engineering practice.

    2. Where Capability Comes From: Retraining, Not Compression

    To tell whether a low-cost model is an architectural design or a distillation/pruning artifact, three checks help:

    1. Independent depth, expert count, and attention configuration, rather than layers sliced from the flagship.
    2. A stated independent pretraining scale, for example on the order of 30T tokens, covering text, images, and video.
    3. Whether multimodality is native joint training or late fusion. Native training means image and video tokens share the same vocabulary segmentation and training objective as text tokens from day one of pretraining. Late fusion trains a text base first, then attaches a vision encoder with alignment data.

    On the debated question of whether the Flash model is “just a compression,” the answer sits in the first check: it is a newly trained base model, not a cut-down flagship. Depth, routing, positional encoding, and attention type—variables that are almost impossible to reset together in incremental modification—are rearranged at once. That is the premise of a same-generation, two-tier lineup.

    3. Ultra-Sparse MoE: What 320B Total and 18B Active Means

    Sparsity is about 17:1. Inference FLOPs and memory-bandwidth pressure are close to an 18B dense model, while the parameter count supplies capacity and knowledge density. Compared with earlier sparse models at roughly 100B total and 30B active, this is a clear directional choice: bet that capacity is cheap and compute is expensive.

    Structurally there are 45 layers, and the first three stay dense. The reason is straightforward: early representations are the input to every later layer. Sparse routing at the bottom makes token representations unstable and weakens shared low-level semantics. Keeping the first one to three layers as dense FFNs is common practice.

    Routing uses 288 routed experts and selects 8 per token (top-8). Two tradeoffs matter:

    • Far more experts than active experts (288 vs. 8) means fine expert granularity and more room for specialization, but also a higher risk of load imbalance. In practice this needs a load-balancing loss, a capacity factor, and a token-drop policy. Otherwise a few “hot” experts dominate throughput.
    • Whether a shared expert is used strongly affects general capability. A shared expert handles cross-domain transforms; routed experts focus on the long tail. This difference often disappears on benchmarks and shows up on real business prompts.

    A schematic MoE forward pass (masks and distributed details omitted):

    1. def moe_forward(x, router, experts, shared, topk=8):
    2. # x: [num_tokens, hidden]
    3. logits = router(x) # [T, E]
    4. weights, idx = logits.softmax(-1).topk(topk) # [T, topk]
    5. weights = weights / weights.sum(-1, keepdim=True) # normalize to keep scale
    6. # Group by expert for batched GEMMs instead of a per-token loop
    7. flat_idx = idx.reshape(-1)
    8. token_of = torch.arange(x.size(0)).repeat_interleave(topk)
    9. order = flat_idx.argsort()
    10. token_of, flat_idx = token_of[order], flat_idx[order]
    11. counts = torch.bincount(flat_idx, minlength=router.num_experts)
    12. splits = counts.cumsum(0)[:-1].tolist()
    13. out = shared(x) # shared expert: dense, every token
    14. for eid, (tok, w) in enumerate(
    15. zip(token_of.split(splits), weights.reshape(-1)[order].split(splits))):
    16. out[tok] += w.unsqueeze(-1) * experts[eid](x[tok])
    17. return out

    Two lines dominate online behavior: top-k normalization, which decides whether expert outputs can be added linearly, and grouping by expert, which decides whether the kernel is one large GEMM or hundreds of small ones. The latter is usually the main MoE inference bottleneck, not FLOPs themselves.

    4. The Information-Mixing Problem of a Shallow Stack

    Depth is only 45 layers, shorter than the common 60–80. Fewer layers means each layer must do more. A conventional stack refines representations step by step; with enough depth, the correction per layer can be small. Cutting depth by about a third makes each correction larger and stresses both training stability and expressiveness.

    That is what wide residual mixing—referred to in the source as mHC, manifold-constrained hyper-connections—is meant to address. A standard residual is one stream:

    1. x_{l+1} = x_l + F_l(x_l)

    Wide residuals keep k parallel residual streams. Layers exchange information through a learnable mixing matrix, with constraints such as row normalization, non-negativity, or a spectral-norm bound so training stays stable:

    1. # Conceptual sketch: k residual streams, mixed by a constrained matrix
    2. # H: [B, T, k, d] M: [k, k] learnable mixing matrix (used after row normalization)
    3. H_in = (M @ H.reshape(B, T, k, d).transpose(-1, -2)).transpose(-1, -2) @ W_in
    4. Y = attention_or_moe(flatten_streams(H_in))
    5. H_out = H + W_out(Y).reshape(B, T, k, d) # write back to all k streams, not one

    The cost is direct: activation memory grows linearly with k, and operators plus parallelism (tensor-parallel splits, sequence parallelism) must follow. This is a typical trade of memory for training quality, and it pays off mainly in shallow architectures. With enough depth, the marginal gain of wide residuals is diluted.

    5. Hybrid Attention: 34 Linear Layers + 11 Sparse Layers, Interleaved 3:1

    This is the largest cost cut in the stack. Of 45 layers, about 34 use linear attention and 11 use sparse attention, interleaved as three linear layers plus one sparse layer.

    Why most layers are linear. Linear and gated linear attention (and Mamba-style state-space models) scale linearly with sequence length, and the state size is fixed—it does not grow with context. At 1M context, that matters more than raw FLOP savings: it decides whether one GPU can hold an ultra-long sequence.

    Why not all linear. A fixed-size state is lossy compression. It is good at summarizing and smooth decay, and it degrades on exact retrieval—needle-in-a-haystack, citing an ID across documents, or copying a code span. Periodic sparse attention layers restore addressable, exact global lookup.

    Why 3:1. This is an empirical ratio, not a theoretical optimum. Linear layers handle continuous context cheaply; sparse layers do long-range exact lookup. Fewer sparse layers save KV cache and memory; too few and retrieval accuracy drops.

    How sparse attention is sparse. A typical implementation is two-stage: index, then main path. The query first dots with a set of compressed index keys, selects the top-k blocks or keys, and runs full-precision attention only on the selected part. With a structure such as IndexPool, index keys are compressed by about 4:1. The stated figures versus dense attention are about 3× less attention compute and about 4.4× less KV cache.

    The handoff between linear and sparse layers is a hidden implementation problem. They differ in sensitivity to positional encoding: sparse layers usually need precise relative positions, while linear layers often use a decay-style position bias. Cross-layer initialization scale and normalization also have to line up. Otherwise quality can collapse once length exceeds the training distribution.

    6. A Quantitative KV-Cache Estimate

    The estimate is:

    1. KV_bytes ≈ 2 × N_sparse × L × H_kv × d_head × bytes_per_elem × batch
    2. + N_linear × S_state × bytes_per_elem × batch

    N_sparse is the number of layers with a KV cache (here 11, not 45). N_linear is the number of linear-attention layers. S_state is the fixed per-sequence state size. The linear term does not depend on sequence length, which is the point of long context.

    Illustrative comparison at 45 layers, hidden size 4096, H_kv × d_head = 1024, BF16 (2 bytes). Here 1K = 1024 and M = 1024²:

    Architecture Layers with a KV cache KV at 128K context KV at 1M context KV at 8K
    Full (dense) attention 45 ≈ 22.5 GB ≈ 180 GB ≈ 1.4 GB
    This hybrid (11 layers, index keys compressed 4:1) 11 ≈ 1.3 GB ≈ 10.7 GB ≈ 0.09 GB
    This hybrid (11 layers, uncompressed) 11 ≈ 5.5 GB ≈ 44 GB ≈ 0.34 GB

    The conclusion is counterintuitive but important: the real advantage of the hybrid design is not at 8K. It shows up at long context and high concurrency. Above 128K, full attention’s KV cache can consume an entire GPU, while the hybrid design still has about an order of magnitude of headroom. Within 8K the gap is under 20×, and differences in attention-kernel efficiency can shrink it further.

    7. How to Read the Scores: 10× Cheaper and Near Flagship

    The source reports a set of numbers worth unpacking: the model beats the previous generation across six coding and agent benchmarks, including 63.4 vs. 46.2 on a deep software-engineering task and 48.8 vs. 26.2 on an automated-agent task. A third-party composite intelligence index is about 57, single-task cost is about $0.05, overall quality is close to a major closed flagship, and price is about one-tenth of the previous generation.

    Three reading rules:

    1. Separate capability gains from harness fit. Agent benchmarks are extremely sensitive to scaffolding—tool granularity, retry policy, context compression, and how errors are written back into the prompt. The same model can move by 15 points or more under a different scaffold. A gap such as 48.8 vs. 26.2 should be trusted only after confirming the harness is the same.
    2. A 10× unit-price cut must include retries. Agent tasks are multi-turn. If single-call success is slightly lower, extra tokens from retries can erase most of the price advantage. The right metric is task success rate divided by total task cost, not unit price.
    3. A “gatekeeper” effect explains when a flagship premium is justified. Once a cheap model reliably matches the previous flagship, the remaining reason to pay more is high cost of failure—long-horizon consistency, complex refactors, and strict format constraints.

    A practical mapping:

    Scenario Recommended tier Why
    High concurrency; batch classification, extraction, or summarization Flash-class Cost dominates; failures can be retried
    Long-document QA and cross-document retrieval Flash-class + hybrid attention More KV-cache headroom; long context is the point
    Code completion, unit-test generation, script refactors Flash-class Coding benchmarks already match the flagship
    Multi-file architectural refactors and complex migrations Flagship Long-horizon consistency; failure is expensive
    Strict structured output and compliance checks Flagship + post-checks Format errors break downstream systems

    8. Four Engineering Recommendations

    (1) Route by input features and cost of failure, not by user tier. Useful signals include input length, whether a tool-call chain is required, whether the task writes across multiple files, and whether the session is already long. The router itself should be a gradually rolled-out config, with a path to fall back to the flagship by task type.

    (2) Test long context for usefulness, not for advertised length. A claimed 1M context should be measured with a layered retrieval eval: place the target at 1K / 8K / 32K / 128K / 512K / 1M and plot accuracy. A typical hybrid curve is near-perfect within 32K and a slow decline from 128K, because a fixed linear-attention state does lose detail over very long distances. Treat that as a model property, not a bug, and put segmented summaries and retrieval in front of the model.

    (3) Budget memory for KV cache and batching. Use the formula in Section 6 and include N_sparse × L in per-GPU capacity planning. With multi-GPU tensor parallelism, sparse-layer KV must be split across devices, while linear-layer state is per sequence. The two split strategies differ; mixing them carelessly creates a communication bottleneck.

    (4) Build your own regression set. Official benchmarks are necessary but not sufficient. Freeze 200–500 real prompts from your workload, label expected and forbidden behavior, and run that set before every model change. For agent tasks, pin the scaffold version; otherwise cross-model comparisons are invalid.

    9. Six Design Knobs for the Next Base Model

    Abstracting this Flash design, six adjustable knobs are worth keeping:

    1. Active-to-total parameter ratio. More sparsity saves compute, but routing and expert parallelism get harder.
    2. Depth versus residual width. Shallow layers plus wide residuals are a viable solution; the cost is activation memory.
    3. Attention mix. The linear-to-sparse ratio (for example 3:1) sets the curve between long-context ability and KV-cache cost.
    4. Index compression. A 4:1 compression saves about 4× cache and costs retrieval precision. This knob is worth an A/B test by task.
    5. Native joint multimodality versus late fusion. Native training is stronger on cross-modal reasoning and an order of magnitude more complex to train.
    6. Pretraining data mix and length curriculum. What the short-sequence stage looks like, and how the long-sequence stage is extended, decides whether 1M context is real competence or nominal support.

    In one line: the point of this Flash class is not a leaderboard score. It shows that ultra-sparse MoE, shallow wide residuals, and linear/sparse hybrid attention can sit near flagship quality at about one-tenth the cost. What developers need is not a parameter table. It is the habit of using the KV-cache formula, cost per successful task, and layered evaluation curves to decide when to save—and when not to.