Choosing a Domestic MoE LLM: Four Technical Routes and a Reproducible POC Framework

    This guide compares four technical directions for selecting a domestically developed large language model: sparse Mixture-of-Experts architectures, attention compression, training optimization and token efficiency, and engineering ecosystem maturity. It explains the trade-offs of each route and provides a reproducible POC framework covering accuracy, latency, memory usage, stability, fine-tuning, integration, and compliance.

    1. Establish a Consistent Evaluation Standard

    When comparing domestically developed large language models, the evaluation criteria must be consistent. A larger total parameter count does not necessarily mean better quality, and fewer activated parameters do not automatically mean lower inference costs.

    A more reliable standard is to measure the model’s accuracy, stability, and maintainability on target tasks under defined constraints for latency, throughput, GPU memory, and deployment cost.

    In this article, “core advantages” refers to overall capability at an equivalent cost, rather than leadership in any single metric. The route names describe technical orientations and do not represent the actual performance of any specific product.

    2. Sparse MoE: Balancing Capacity and Cost

    Sparse Mixture-of-Experts (MoE) models activate only a subset of experts for each input. This allows the total parameter count to continue growing while keeping the computation required for each forward pass under control. The main advantage is greater knowledge capacity, making this route suitable for complex question answering and mixed multi-task workloads.

    The trade-offs are also significant. As the number of experts increases, cross-device communication for expert parallelism becomes more expensive. If routing is unstable, some experts may be overused while others are rarely activated. With small batch sizes, dynamic routing can also increase tail latency.

    When evaluating this route, focus on the following metrics:

    Evaluation area Measurement method Risk signal
    Expert load balancing Measure the activation ratio of each expert A small number of experts account for more than half of all activations
    Routing jitter Run the same input multiple times and measure the consistency of expert selection Consistency is significantly lower than that of comparable models
    Impact of batch size Measure P99 latency at batch sizes 1, 8, and 32 P99 latency deteriorates significantly as batch size increases
    Post-quantization accuracy Compare accuracy on the same evaluation set before and after quantization The difference exceeds the threshold acceptable for the business

    Total parameter count alone cannot reveal these differences.

    3. Attention Compression: Separate Cache Compression from Compute Reduction

    Attention compression generally involves two different approaches, and they should be evaluated separately.

    The first approach compresses the KV cache, as in low-rank latent-variable attention designs such as MLA-style architectures. This reduces GPU memory usage during long-context inference and directly affects the number of concurrent requests that can be served.

    The second approach reduces the number of attention heads or the amount of attention computation. This lowers computational overhead but may reduce long-range dependency modeling and fine-grained retrieval capability.

    This route is suitable for long-document understanding, retrieval-augmented generation, and multi-turn conversations. The evaluation should cover:

    • Long-context retrieval accuracy: At context lengths such as 4K, 32K, and 128K, verify whether the model can locate key information.
    • Time to first token (TTFT): Record the time until the first token is generated for different input lengths.
    • Cache usage: Measure how GPU memory changes as context length increases.
    • Chunking strategy impact: Compare how different chunk sizes and overlap ratios affect answer consistency.

    The value of long-context support depends on the document lengths used in the target business. Stable and accurate performance at common business lengths is more important than supporting extremely long inputs.

    4. Training Optimization and Token Efficiency

    As high-quality data becomes more difficult to obtain, token efficiency is becoming increasingly important for further model scaling. Differences between training approaches mainly involve the optimizer, data cleaning, and the proportion of synthetic data.

    For example, Muon-style orthogonalized optimizers apply orthogonalization to update directions to improve training stability. During model selection, do not evaluate only the optimizer’s name. Examine the actual training behavior through the following metrics:

    • Convergence curves: Compare the number of steps required to reach the same loss under different learning rates.
    • Hyperparameter sensitivity: Vary parameters such as learning rate and weight decay within a small range, then observe the resulting performance fluctuation.
    • Catastrophic forgetting: After domain fine-tuning, measure the performance decline on general-purpose evaluation sets.
    • Evaluation contamination: Check whether evaluation samples overlap with training data, and record the score difference before and after deduplication.

    Synthetic data may cause distribution shifts. Insufficient deduplication can also lead to memorization and information leakage. For enterprise users, the direct benefits of this route are more stable fine-tuning and better few-shot transfer, provided that the evaluation set is completely isolated from business data.

    5. Engineering Ecosystem: Integrability and Observability

    For enterprise users, a model ultimately needs to be embedded into business systems. The stability of tool calling, the parsability of structured output, state retention across multi-turn conversations, and the controllability of accuracy loss after quantization often determine production outcomes more directly than benchmark scores.

    The main advantage of this route is strong integrability, enabling faster adoption within existing services. The challenge is that different models may vary in their sensitivity to prompts, tool descriptions, and output formats. Version upgrades may also introduce behavioral drift.

    Recommended monitoring metrics include:

    • Tool-call failure rate: Measure the proportion of invalid parameters, incorrect function names, and timeouts.
    • JSON parsing rate: Measure the proportion of outputs that can be read by a standard parser.
    • Version regression pass rate: Run a fixed regression set after every upgrade.
    • Fallback frequency: Record how often the system switches to a backup solution because of model timeouts or invalid output.

    6. A Reproducible POC Testing Plan

    The key to a proof of concept (POC) is controlling the variables. The prompt template, temperature, evaluation sets, and hardware environment must remain consistent across all candidate models. Otherwise, the results cannot be compared reliably.

    1. # Fixed parameters
    2. prompt_template = "v1.2" # All models use the same template
    3. temperature = 0
    4. batch_sizes = [1, 8, 32]
    5. context_lengths = [4K, 32K, 128K]
    6. repeat_runs = 3 # Repeat each input to check stability
    7. slo = {P99_latency_ms: 2000, error_rate: 0.5%, min_accuracy: business_baseline}
    8. # Measurement definitions
    9. # P50/P99: End-to-end latency per request, not an average
    10. # tokens_per_s: Tokens generated per unit of time at a fixed batch size
    11. # vram_gb: Peak GPU memory during inference
    12. # error_rate: Combined rate of timeouts, format errors, and tool-call failures
    13. def evaluate(candidates, suites, slo):
    14. rows = []
    15. for m in candidates:
    16. acc = run_suites(m, suites, prompt_template, temperature)
    17. perf = bench(m, batch_sizes, context_lengths)
    18. stab = repeat_consistency(m, runs=repeat_runs)
    19. ft = finetune_curve(m, seeds=[1, 2, 3]) # Few-shot convergence and forgetting
    20. rows.append({model: m, accuracy: acc, perf: perf,
    21. stability: stab, finetune: ft})
    22. feasible = [r for r in rows if meets_slo(r, slo)]
    23. return rank_by_business_weights(feasible)

    The evaluation should cover five scenarios separately:

    1. Short-form question answering
    2. Long-document processing
    3. Tool calling
    4. Structured output
    5. High-concurrency serving

    For each scenario, record accuracy, latency, GPU memory usage, error rate, and the cost of human review.

    7. Model Selection Checklist

    Before selecting a model, answer the following questions. Each answer should be supported by corresponding test data.

    Route Main advantage Main cost Suitable SLO Scenarios to avoid
    Sparse MoE Large capacity with controllable cost per unit of computation Communication overhead, load balancing, and tail latency High concurrency and mixed multi-task workloads Small batches, extremely latency-sensitive workloads, or environments without cross-device support
    Attention compression Lower memory usage for long-context inference Potential degradation in long-range dependencies Long documents, RAG, and multi-turn conversations Fine-grained long-range reasoning is required and additional data cannot compensate
    Training optimization More stable fine-tuning and lower data costs Distribution shifts caused by synthetic data Domain fine-tuning and few-shot transfer The evaluation set cannot be isolated from training data
    Engineering ecosystem Faster integration and more stable tool calling Version drift and prompt sensitivity Embedding models in business systems and enterprise services There is no regression testing or monitoring capability

    You should also verify the following compliance requirements:

    • Whether data leaves the organization’s permitted geographic boundary
    • Whether outputs are processed through appropriate content-safety filters
    • Whether user privacy information appears in logs
    • Whether the model license permits the intended use

    If these requirements are not satisfied, the model should not be deployed regardless of its performance results.

    The goal of model selection is not to find the “strongest model.” It is to identify the model or model combination with the lowest total cost and manageable risk under the business SLO. Define the business constraints first, validate candidates through a small-scale POC, and then continuously monitor throughput, latency, accuracy, and cost trends.