Decoding Domestic LLMs: From Architecture Innovation to Model Selection

I. Background: From Following to Autonomous Evolution

Since the establishment of the self-attention mechanism architecture, the capability boundaries of large language models (LLMs) have been continuously pushed forward. The research and development of domestic LLMs have generally gone through three stages: early architecture replication and technological catch-up, mid-stage refinement of data quality and engineering, and the current autonomous innovation around differentiated technical routes.

Regarding the competitive landscape, different teams have chosen distinct technical focuses based on their resources and target scenarios: some bet on long context and code capabilities, some focus on lightweight models and inference efficiency, some delve into multimodal fusion, while others pursue the optimal solution in compute economics through the Mixture of Experts (MoE) architecture. These routes are not mutually exclusive but represent different branches on the LLM technology tree, collectively shaping the domestic LLM ecosystem.

For developers and enterprise users, understanding the technical logic behind these routes is far more important than remembering a model’s marketing labels. This article will break down these core technical directions one by one.

II. Long-Context Modeling: Breaking the Attention “Memory Bottleneck”

2.1 Technical Challenges

The computational complexity of the standard self-attention mechanism grows quadratically with sequence length. When the context window expands from a few thousand tokens to hundreds of thousands, two bottlenecks emerge:

• Memory Pressure: The memory footprint of intermediate attention matrices and the KV cache expands rapidly with length.
• Extrapolation Failure: The model’s generalization ability drops sharply on position encodings unseen during training, leading to unstable answer quality for long texts.

2.2 Mainstream Solutions

Position Encoding Extension: Based on Rotary Position Embedding (RoPE), methods like linear interpolation and frequency adjustment allow the model to extrapolate to longer sequences with minimal continued training on long texts.

Attention Computation Optimization: Using chunked computation and IO-aware operator design avoids explicitly generating the full intermediate attention matrix. Combined with quantization, compression, and eviction strategies for the KV cache, this significantly reduces the peak memory usage for long-sequence inference.

Sparse and Sliding Window Attention: Local window attention paired with a few global tokens reduces complexity to approximately linear while maintaining access to key information.

Retrieval-Augmented Generation (RAG): For ultra-long document scenarios, semantic chunking and vector retrieval are used to recall relevant fragments, transforming the “infinite context” problem into a “precise retrieval” problem.

Schematic code for long document chunking:

def chunk_document(text, chunk_size=512, overlap=64):
    """Sliding window chunking: overlap design prevents semantics from being split at boundaries"""
    step = chunk_size - overlap
    return [text[i:i + chunk_size] for i in range(0, len(text), step)]

2.3 Applicable Scenarios and Pitfalls

Long-context capability is highly valuable in scenarios like entire repository code understanding, lengthy contract review, and multi-turn deep conversations. However, note that “supported context length” and “effectively utilized context length” are two different things. Information in the middle of a long text is easily ignored by the model. When selecting a model, you should design “needle in a haystack” tests to verify the actual recall effect, rather than just looking at the nominal length on the spec sheet.

III. Mixture of Experts (MoE): The Compute Economics of Sparse Activation

3.1 Core Concept

The key to the Mixture of Experts (MoE) architecture lies in specialization: splitting the feed-forward network into multiple expert subnetworks. A gating network dynamically selects the most relevant few experts for each token to participate in computation, realizing a sparse computation mode of “large total parameters, few activated per inference.”

3.2 Gating Mechanism Schematic

# MoE routing pseudo-code
def moe_route(x, W_gate, top_k=2, temperature=1.0):
    logits = matmul(x, W_gate.T)                    # [tokens, num_experts]
    probs = softmax(logits / temperature, dim=-1)   # Temperature controls routing dispersion
    topk_p, topk_i = top_k_select(probs, top_k)     # Only activate Top-K experts
    topk_p = normalize(topk_p)                      # Weight normalization
    return topk_i, topk_p                           # Weighted aggregation of expert outputs

A higher temperature coefficient results in a more uniform routing distribution, activating more experts. This yields high capacity utilization but increases single-pass computation overhead. A lower temperature leads to a sharper distribution, making inference more focused and cost-effective.

3.3 Engineering Trade-offs

A typical MoE model has a total parameter scale reaching hundreds of billions, while only activating a small fraction during each inference. This brings three benefits and three challenges:

Benefits:

• Under the same inference cost, the effective capacity of the model significantly increases.
• Expert division of labor reduces gradient interference between different tasks.
• Training compute can be more concentrated on high-value parameters.

Challenges:

• Load Balancing: If the gate chronically favors a few experts, it causes compute waste. An auxiliary balancing loss is usually introduced to mitigate this.
• Memory Footprint: All expert parameters must reside in memory, placing high demands on deployment hardware.
• Training Stability: The discreteness of routing easily triggers training oscillations, requiring meticulous hyperparameter tuning.

IV. Lightweighting and Inference Optimization: Running LLMs in Production

4.1 Three Typical Paths

Model Compression: Quantization (reducing weights from FP16 to INT8/INT4), pruning (removing redundant channels), and knowledge distillation (using a large model to guide a small model’s training) are the three pillars. Quantization-aware training can further reduce precision loss.

Small and Refined Dense Models: Abandoning ultra-large parameter scales, high-quality data blending and multi-stage training allow models with under ten billion parameters to approach the performance of much larger models on specific tasks.

Inference Framework Optimization: Technologies like operator fusion, continuous batching, paged management of the KV cache, and speculative decoding can increase throughput several times over without altering the model itself.

4.2 The True Value of Lightweighting

Lightweight does not mean low capability. For high-frequency, low-latency online scenarios (like intelligent customer service or on-device assistants), a carefully distilled and quantized small model is often more cost-effective than directly calling an ultra-large model. It is recommended to measure performance by “task quality per unit cost” rather than simply looking at benchmark scores.

V. Multimodal Fusion: Bridging the Semantic Gap Between Language and Vision

5.1 Technical Routes

The mainstream approach maps image features output by a vision encoder into the language model’s embedding space via a projection layer, enabling text and vision tokens to share the same autoregressive modeling framework. Further directions include:

• Unified Tokenization: Discretizing images and audio into tokens, modeled uniformly with text.
• Cross-Attention Fusion: Inserting visual branches into the intermediate layers of the language model.
• Native Multimodal Pre-training: Mixing multimodal data from the pre-training stage rather than post-hoc stitching.

5.2 Implementation Key Points

Evaluating multimodal capabilities is more complex than text-only. Besides recognition accuracy, attention must be paid to image-text consistency, fine-grained understanding (such as table and chart parsing), and hallucination rates. For enterprise deployment, it is recommended to first validate effectiveness in scenarios with clear boundaries, like document parsing and ticket recognition, before gradually expanding to open-ended visual question answering.

VI. Code Capabilities: From Completion to Agents

Coding capability has become a key dimension for measuring an LLM’s practical value. Its technical composition includes:

• High-Quality Code Corpus: Cleaning, deduplication, and dependency parsing of repository-level code.
• Repository-Level Context Understanding: Perceiving cross-file references and project structures, rather than isolated function completion.
• Fill-in-the-Middle (FIM) Training: Supporting completion modes at the cursor position, fitting the interaction habits of programming tools.
• Tool Calling and Execution Feedback: Allowing the model to run code, read errors, and self-correct, forming an agent loop.

For development teams, evaluating code LLMs should focus on: the compilable rate of generated code, the accuracy of cross-file modifications, and the adaptation cost for internal proprietary frameworks. The latter can be addressed by injecting domain knowledge via retrieval augmentation or bridging gaps using parameter-efficient fine-tuning.

VII. Alignment and Controllability: Two Complementary Technical Paths

Regarding model controllability, the industry has formed two complementary paths:

First, Reinforcement Learning from Human Feedback (RLHF) and its variants (such as Direct Preference Optimization, DPO), which train reward signals based on annotators’ preference rankings of outputs, shaping the model’s style at the behavioral level.

Second, Data and Knowledge Augmentation: Injecting structured knowledge like knowledge graphs into the training or inference process to constrain probabilistic generation with deterministic knowledge, reducing factual errors. Simultaneously, integrating local language habits and cultural contexts into the training data makes value alignment better fit local user expectations.

The former establishes behavioral constraints from within the model, while the latter shapes the output distribution from the data source. The combination of both constitutes the current mainstream paradigm for domestic LLM alignment practices.

VIII. Enterprise Model Selection and Engineering Deployment Framework

8.1 Four-Dimensional Evaluation Model

Dimension	Key Questions
Capability	Real performance on proprietary evaluation sets, rather than just public benchmarks
Cost	Cost per token, time to first token, concurrent throughput
Engineering	API and private deployment options, context length, fine-tuning support
Compliance	Data security requirements, content security moderation capabilities, service stability

8.2 Engineering Practice Recommendations

1. Build Proprietary Evaluation Sets: Extract hundreds of representative samples from real business operations, covering typical and edge cases, and conduct regular regression testing.
2. Tiered Cost Architecture: Route simple tasks to lightweight models and complex tasks to large-parameter models, unified through a gateway layer for scheduling.
3. Disaster Recovery and Degradation: Configure backup models and fallback strategies for critical paths to avoid single points of dependency.
4. Leverage Cloud Platform Capabilities: Under the public cloud route, platforms like Baidu AI Cloud can provide model hosting, container services, monitoring, and logging, allowing rapid deployment of observable inference services so you can focus on business logic.

Schematic code for task routing:

def route(prompt: str) -> str:
    """Route to different model service tiers based on task complexity"""
    if is_simple_task(prompt):        # Lightweight tasks like intent classification, formatting
        return "small-model-endpoint"
    if needs_long_context(prompt):    # Long document analysis
        return "long-context-endpoint"
    return "flagship-model-endpoint"  # Complex reasoning

IX. Conclusion

The competition among domestic LLMs is essentially a contest of technical route selection and engineering execution efficiency. Long context, MoE, lightweighting, multimodality, and code capabilities are not isolated selling points, but rather explorations of optimal solutions within the same underlying tech stack under different constraints. For developers, rather than chasing the “strongest model,” it is more important to understand the principles and trade-offs of each route, build your own evaluation system and cost model, and let model selection return to actual business value.