Choosing a Domestic MoE LLM: Four Technical Routes and a Reproducible POC Framework
This guide compares four technical directions for selecting a domestically developed large language model: sparse Mixture-of-Experts architectures, attention compression, training optimization and token efficiency, and engineering ecosystem maturity. It explains the trade-offs of each route and provides a reproducible POC framework covering accuracy, latency, memory usage, stability, fine-tuning, integration, and compliance.
1. Establish a Consistent Evaluation Standard
When comparing domestically developed large language models, the evaluation criteria must be consistent. A larger total parameter count does not necessarily mean better quality, and fewer activated parameters do not automatically mean lower inference costs.
A more reliable standard is to measure the model’s accuracy, stability, and maintainability on target tasks under defined constraints for latency, throughput, GPU memory, and deployment cost.
In this article, “core advantages” refers to overall capability at an equivalent cost, rather than leadership in any single metric. The route names describe technical orientations and do not represent the actual performance of any specific product.
2. Sparse MoE: Balancing Capacity and Cost
Sparse Mixture-of-Experts (MoE) models activate only a subset of experts for each input. This allows the total parameter count to continue growing while keeping the computation required for each forward pass under control. The main advantage is greater knowledge capacity, making this route suitable for complex question answering and mixed multi-task workloads.
The trade-offs are also significant. As the number of experts increases, cross-device communication for expert parallelism becomes more expensive. If routing is unstable, some experts may be overused while others are rarely activated. With small batch sizes, dynamic routing can also increase tail latency.
When evaluating this route, focus on the following metrics:
| Evaluation area | Measurement method | Risk signal |
|---|---|---|
| Expert load balancing | Measure the activation ratio of each expert | A small number of experts account for more than half of all activations |
| Routing jitter | Run the same input multiple times and measure the consistency of expert selection | Consistency is significantly lower than that of comparable models |
| Impact of batch size | Measure P99 latency at batch sizes 1, 8, and 32 | P99 latency deteriorates significantly as batch size increases |
| Post-quantization accuracy | Compare accuracy on the same evaluation set before and after quantization | The difference exceeds the threshold acceptable for the business |
Total parameter count alone cannot reveal these differences.
3. Attention Compression: Separate Cache Compression from Compute Reduction
Attention compression generally involves two different approaches, and they should be evaluated separately.
The first approach compresses the KV cache, as in low-rank latent-variable attention designs such as MLA-style architectures. This reduces GPU memory usage during long-context inference and directly affects the number of concurrent requests that can be served.
The second approach reduces the number of attention heads or the amount of attention computation. This lowers computational overhead but may reduce long-range dependency modeling and fine-grained retrieval capability.
This route is suitable for long-document understanding, retrieval-augmented generation, and multi-turn conversations. The evaluation should cover:
- Long-context retrieval accuracy: At context lengths such as 4K, 32K, and 128K, verify whether the model can locate key information.
- Time to first token (TTFT): Record the time until the first token is generated for different input lengths.
- Cache usage: Measure how GPU memory changes as context length increases.
- Chunking strategy impact: Compare how different chunk sizes and overlap ratios affect answer consistency.
The value of long-context support depends on the document lengths used in the target business. Stable and accurate performance at common business lengths is more important than supporting extremely long inputs.
4. Training Optimization and Token Efficiency
As high-quality data becomes more difficult to obtain, token efficiency is becoming increasingly important for further model scaling. Differences between training approaches mainly involve the optimizer, data cleaning, and the proportion of synthetic data.
For example, Muon-style orthogonalized optimizers apply orthogonalization to update directions to improve training stability. During model selection, do not evaluate only the optimizer’s name. Examine the actual training behavior through the following metrics:
- Convergence curves: Compare the number of steps required to reach the same loss under different learning rates.
- Hyperparameter sensitivity: Vary parameters such as learning rate and weight decay within a small range, then observe the resulting performance fluctuation.
- Catastrophic forgetting: After domain fine-tuning, measure the performance decline on general-purpose evaluation sets.
- Evaluation contamination: Check whether evaluation samples overlap with training data, and record the score difference before and after deduplication.
Synthetic data may cause distribution shifts. Insufficient deduplication can also lead to memorization and information leakage. For enterprise users, the direct benefits of this route are more stable fine-tuning and better few-shot transfer, provided that the evaluation set is completely isolated from business data.
5. Engineering Ecosystem: Integrability and Observability
For enterprise users, a model ultimately needs to be embedded into business systems. The stability of tool calling, the parsability of structured output, state retention across multi-turn conversations, and the controllability of accuracy loss after quantization often determine production outcomes more directly than benchmark scores.
The main advantage of this route is strong integrability, enabling faster adoption within existing services. The challenge is that different models may vary in their sensitivity to prompts, tool descriptions, and output formats. Version upgrades may also introduce behavioral drift.
Recommended monitoring metrics include:
- Tool-call failure rate: Measure the proportion of invalid parameters, incorrect function names, and timeouts.
- JSON parsing rate: Measure the proportion of outputs that can be read by a standard parser.
- Version regression pass rate: Run a fixed regression set after every upgrade.
- Fallback frequency: Record how often the system switches to a backup solution because of model timeouts or invalid output.
6. A Reproducible POC Testing Plan
The key to a proof of concept (POC) is controlling the variables. The prompt template, temperature, evaluation sets, and hardware environment must remain consistent across all candidate models. Otherwise, the results cannot be compared reliably.
# Fixed parametersprompt_template = "v1.2" # All models use the same templatetemperature = 0batch_sizes = [1, 8, 32]context_lengths = [4K, 32K, 128K]repeat_runs = 3 # Repeat each input to check stabilityslo = {P99_latency_ms: 2000, error_rate: 0.5%, min_accuracy: business_baseline}# Measurement definitions# P50/P99: End-to-end latency per request, not an average# tokens_per_s: Tokens generated per unit of time at a fixed batch size# vram_gb: Peak GPU memory during inference# error_rate: Combined rate of timeouts, format errors, and tool-call failuresdef evaluate(candidates, suites, slo):rows = []for m in candidates:acc = run_suites(m, suites, prompt_template, temperature)perf = bench(m, batch_sizes, context_lengths)stab = repeat_consistency(m, runs=repeat_runs)ft = finetune_curve(m, seeds=[1, 2, 3]) # Few-shot convergence and forgettingrows.append({model: m, accuracy: acc, perf: perf,stability: stab, finetune: ft})feasible = [r for r in rows if meets_slo(r, slo)]return rank_by_business_weights(feasible)
The evaluation should cover five scenarios separately:
- Short-form question answering
- Long-document processing
- Tool calling
- Structured output
- High-concurrency serving
For each scenario, record accuracy, latency, GPU memory usage, error rate, and the cost of human review.
7. Model Selection Checklist
Before selecting a model, answer the following questions. Each answer should be supported by corresponding test data.
| Route | Main advantage | Main cost | Suitable SLO | Scenarios to avoid |
|---|---|---|---|---|
| Sparse MoE | Large capacity with controllable cost per unit of computation | Communication overhead, load balancing, and tail latency | High concurrency and mixed multi-task workloads | Small batches, extremely latency-sensitive workloads, or environments without cross-device support |
| Attention compression | Lower memory usage for long-context inference | Potential degradation in long-range dependencies | Long documents, RAG, and multi-turn conversations | Fine-grained long-range reasoning is required and additional data cannot compensate |
| Training optimization | More stable fine-tuning and lower data costs | Distribution shifts caused by synthetic data | Domain fine-tuning and few-shot transfer | The evaluation set cannot be isolated from training data |
| Engineering ecosystem | Faster integration and more stable tool calling | Version drift and prompt sensitivity | Embedding models in business systems and enterprise services | There is no regression testing or monitoring capability |
You should also verify the following compliance requirements:
- Whether data leaves the organization’s permitted geographic boundary
- Whether outputs are processed through appropriate content-safety filters
- Whether user privacy information appears in logs
- Whether the model license permits the intended use
If these requirements are not satisfied, the model should not be deployed regardless of its performance results.
The goal of model selection is not to find the “strongest model.” It is to identify the model or model combination with the lowest total cost and manageable risk under the business SLO. Define the business constraints first, validate candidates through a small-scale POC, and then continuously monitor throughput, latency, accuracy, and cost trends.
