Inside New Open-Source Multimodal LLMs: Architecture and Engineering Guide

    This article examines the architecture and engineering practices behind a new open-source multimodal large model with approximately 320 billion total parameters and about 18 billion activated parameters per inference. It explains how sparse Mixture-of-Experts decouples knowledge capacity from inference cost, how linear attention variants enable million-token context, and how agent-oriented post-training shifts model capability toward task execution. It also covers distributed deployment on non-NVIDIA accelerator clusters, inference optimization, and practical guidance for model selection and integration.

    1. Background: From the Parameter Race to the Efficiency Race

    Over the past two years, the technical evolution of large models has shifted direction. Simply stacking more parameters yields diminishing marginal returns, while inference cost, response latency, and deployment barriers have become core constraints for enterprise adoption. Against this backdrop, a new architectural approach has become increasingly mainstream: use a very large total parameter count to preserve knowledge capacity, while keeping activated parameters extremely low to control inference overhead.

    A multimodal large model that recently attracted broad attention in anonymous blind tests and was subsequently open-sourced is a representative implementation of this approach. With approximately 320 billion total parameters, it uses a sparse mixture mechanism to keep activated parameters per inference at the 18-billion scale. It is released under a fully permissive open-source license and is said to have completed large-scale distributed deployment validation under real production traffic. Setting aside product-specific narratives, this article analyzes the key design choices of this class of models from a general technical perspective and provides an engineering deployment guide.

    2. Core Architecture Analysis

    2.1 Mixture of Experts: The Key to Sparsity

    The core idea of MoE is to replace the dense feed-forward network in a standard Transformer with multiple parallel “expert” sub-networks. A router dynamically selects a small number of experts for each token. Its mathematical form can be simplified as follows:

    1. # Illustrative pseudocode for an MoE layer
    2. def moe_forward(x, experts, router, top_k=2):
    3. logits = router(x) # [batch, seq, num_experts]
    4. topk_idx, topk_w = topk_softmax(logits, top_k)
    5. out = sum(
    6. topk_w[..., i].unsqueeze(-1) * experts[topk_idx[..., i]](x)
    7. for i in range(top_k)
    8. )
    9. return out

    This design introduces a key decoupling: total parameters determine how much the model “knows,” while activated parameters determine how much inference “costs.” The 18 billion activated parameters roughly correspond to the per-inference compute cost of a small or medium dense model, yet the model can use knowledge stored across 320 billion parameters. This is the architectural basis for achieving performance close to flagship models at roughly one-tenth the cost.

    In engineering practice, two inherent MoE challenges require attention:

    1. Load balancing: Routers are prone to expert collapse, so auxiliary load-balancing losses or expert capacity limits are usually needed.
    2. Memory and communication: Although compute is reduced, all expert parameters must reside in memory, and All-to-All communication creates significant overhead in distributed training, placing high demands on cluster interconnect bandwidth.

    2.2 Linear Attention: The Technical Support for Million-Token Context

    Standard Softmax attention has compute and memory costs that grow quadratically with sequence length, making a one-million-token context almost impractical. This model uses a linear attention variant that reduces attention complexity to linear growth with sequence length. Common implementation paths include:

    • Kernelized linear attention: Approximate the Softmax kernel through feature mappings and use associativity to change the computation order, computing Key-Value first and then combining with Query.
    • Hybrid state-space or recurrent mechanisms: Compress long-range dependencies into a fixed-size hidden state and process the sequence segment by segment.
    • Hybrid sliding-window and global-summary layers: Use window attention for local detail and compressed memory for long-range semantics, alternating these across layers.

    Hybrid attention design changes KV-cache memory usage from quadratic growth to near-linear growth. This is what turns “million-token context” from a paper metric into a serviceable engineering capability. For users, this means large code repositories, long contracts, or complete audio and video transcripts can be supplied in a single request, without building complex retrieval-augmented and chunking pipelines.

    2.3 Native Multimodality and Visual Self-Check Feedback

    “Native” multimodality means that text, image, and audio/video inputs are encoded in a unified token space rather than being appended through separate encoders after the fact. This gives the model genuine cross-modal reasoning ability for tasks such as front-end page screenshots and design drafts.

    More notable is the visual self-check feedback loop: after generating front-end code or a 3D model, the model can “visually review” the rendered result, detect layout misalignment or style deviations, and correct them autonomously. This forms an iterative loop of “generate, render, evaluate, repair.” It mirrors the human developer workflow of “write code, run it, inspect the result, revise the code,” and it is an important reason this class of models performs strongly on automated front-end tasks.

    3. Agent-Oriented Post-Training: Why Task Execution Is Stronger

    Evaluation data shows that this new class of models has entered the first tier on coding and automation benchmarks, with multi-step task execution capability nearly doubling compared with the previous generation. However, improvements in difficult pure knowledge reasoning remain relatively limited. This divergence reveals an important trend: the focus of model capability improvement is shifting from “basic intelligence” to “task execution.”

    The training paradigm behind this is commonly called Agent-oriented Post-training. It generally includes:

    1. Multi-turn tool-use trajectory training: Build long-horizon task trajectories involving search, code execution, terminal operations, and API calls, so the model learns the loop of planning, invoking, observing, and adjusting.
    2. Environment-interactive reinforcement learning: Use task completion as the reward signal in sandbox environments to optimize end-to-end execution success rather than the surface quality of a single-step output.
    3. Targeted optimization of error recovery: Deliberately inject errors, timeouts, permission failures, and other abnormal scenarios into training data to train the model to recover from failure. This is the biggest difference between real automation tasks and idealized benchmarks.

    For enterprise applications, this orientation means that if your scenario involves digital employees, pipeline automation, code generation, and repair, this class of models can offer very strong cost-performance. If the scenario involves highly difficult mathematical proofs or frontier research reasoning, careful evaluation is still required.

    4. Systems Engineering Practice: Deployment on Non-NVIDIA Clusters

    The model is said to have handled real global production-level traffic on a large-scale cluster of domestically developed accelerators with more than 10,000 cards. The general technical points behind this engineering achievement deserve closer examination.

    4.1 Combined Distributed Strategies

    Inference for very large MoE models usually requires multiple levels of parallelism:

    • Expert parallelism (EP): Distribute different experts across different nodes and aggregate tokens through All-to-All communication.
    • Tensor parallelism (TP): Split weights within a layer to reduce per-GPU memory pressure.
    • Pipeline parallelism (PP): Split by layer to improve cluster utilization.
    • KV-cache disaggregated serving: Place attention caches in dedicated cache-pool nodes, decoupled from compute nodes, to support long-context and high-concurrency serving.

    4.2 Inference Optimization Stack

    1. request → routing gateway → prefix-cache hit detection
    2. → continuous batching scheduler → hybrid-parallel inference cluster
    3. → streaming output → monitoring/degradation

    Key optimizations include continuous batching to improve throughput; prefix caching to reuse system prompts and common parts of long contexts; speculative decoding to reduce first-token and overall latency; and quantization such as INT8 or FP8 to compress memory and bandwidth when precision loss is acceptable.

    4.3 Heterogeneous Hardware Adaptation

    Deploying on a non-mainstream GPU ecosystem requires operator-level adaptation, including rewriting or compiling core operators for the target chip operator library; replacing the communication library with a vendor-specific collective communication library; and compiler optimization, such as operator fusion and scheduling tuned to chip characteristics. This practice demonstrates that large-model capability is not tightly bound to a specific hardware ecosystem, which is a useful reference for enterprises pursuing supply-chain autonomy and control.

    5. Application Selection and Integration Recommendations

    Based on the analysis above, three practical recommendations emerge:

    1. Start with a task profile: Build a small evaluation set from your own business data and focus on multi-step task completion rate and tool-call accuracy, rather than relying only on general leaderboards.
    2. Use long context to reduce architectural complexity: Million-token context allows “inject the entire repository or full document set” to replace complex RAG pipelines. However, latency and cost must be weighed, and a hybrid approach combining long context with retrieval is often better.
    3. Open-source licensing enables deployment freedom: A permissive license allows private deployment and commercial integration, making this suitable for finance, government, and healthcare scenarios with strict data-egress requirements.

    6. Conclusion

    The real lesson of this new generation of open-source multimodal large models is not a particular benchmark score, but three clear technical directions: use MoE sparsity to decouple capability from cost, use linear attention to unlock ultra-long context, and use agent-oriented post-training to evolve models from “able to answer” to “able to do.” For developers, understanding these underlying principles has more long-term value than chasing specific products, because they are defining how enterprise AI applications will be built over the next three years.