Four Technical Routes in LLM Selection: MoE, Long Context, Inference Engineering, and Multimodality
When people compare Chinese large language models, the discussion usually collapses into "who scores higher" and "who has more parameters." But in real engineering projects, what decides success is rarely a few points on a benchmark. It is four verifiable capability dimensions: the cost curve that architectural sparsification unlocks, the task coverage that context length buys, the per-token price that inference engineering drives down, and the scenario depth that multimodality and tool calling provide. This article dissects each of the four dominant technical routes — sparse Mixture-of-Experts (MoE), long context, lightweight models plus inference engineering, and multimodal/knowledge-enhanced systems — explaining the underlying mechanism, the real engineering costs, and the pitfalls for each. It also includes a scenario-based selection matrix, a four-layer deployment architecture, common mistakes to avoid, and a practical action checklist for choosing and validating a model for production.
1. Before You Compare Benchmarks: Technical Routes Are Where “Core Advantages” Actually Live
When people talk about Chinese large language models, the discussion is too often reduced to “who scores higher” and “who has more parameters.” But in real engineering, what decides whether a project succeeds is usually not a few benchmark points. It is four verifiable capability dimensions: the cost curve that architectural sparsification brings, the task coverage that context length buys, the per-token price that inference engineering drives down, and the scenario depth that multimodality and tool calling provide.
Early on, the routes taken by domestic vendors were highly homogeneous: the mainstream approach was to optimize data mixtures and training techniques around a self-attention decoder architecture, with differences showing up mainly in data quality and fine-tuning strategy. Around 2023, real divergence began to appear, and the dividing line was essentially a question of judgment — does the next increment of intelligence come mainly from continued growth in parameter scale, or from better allocation of information and compute at inference time?
Following that judgment, the routes naturally split into four:
- The sparsification route: use a Mixture of Experts (MoE) to decouple model capacity from per-token compute, betting on the unit-cost advantage of “large capacity + sparse activation”;
- The long-context route: make the effective context window longer, betting that “fitting all the relevant material in one pass” solves long documents, codebases, and long multi-turn dialogues;
- The lightweight and inference-engineering route: shift the focus from pre-training scale to the inference side, betting on the cost-performance inflection point after stacking quantization, distillation, and speculative decoding;
- The multimodal and knowledge-enhancement route: turn the language model into a hub for perception and action, betting on scenario depth rather than a single benchmark score.
These four routes are not mutually exclusive. Mature teams typically combine several of them on the same product line. Below, each is broken down in terms of mechanism, cost, and pitfalls.
2. Route 1: Sparse Mixture-of-Experts (MoE) Architecture
2.1 Core mechanism: decoupling “capacity” from “compute”
A dense model activates all of its parameters on every token during the forward pass. MoE replaces the feed-forward layers with a set of parallel “expert” subnetworks, and a gating network (the router) selects a few experts to participate for each token. The result: total parameters can grow very large, while the actual compute per token stays roughly constant.
This has a direct business implication: the growth of model capacity (which can be understood as the ceiling on how much knowledge it can hold) no longer pushes inference cost up linearly. The core benefit of MoE is unit cost under high training throughput and large-scale online serving — not single-card, low-concurrency scenarios.
2.2 The gate network: the real decider on this route
The gate is usually a linear layer plus softmax, combined with top-K selection. The softmax temperature coefficient is an underappreciated but critical knob:
- Higher temperature flattens the expert distribution, so tokens are handled by more experts; specialization is weaker, but experts are used fully and training is more stable;
- Lower temperature sharpens the distribution, selecting only the highest-scoring few experts; specialization is stronger, but “expert collapse” becomes very likely — most traffic flows to a few experts over the long term, and the rest effectively die.
Engineering a fix for this degradation requires a full set of supporting techniques:
import torch, torch.nn.functional as Fclass TopKGate(torch.nn.Module):def __init__(self, hidden, n_experts, k=2, temp=1.0, noise_std=0.01, cap=1.0):super().__init__()self.w = torch.nn.Linear(hidden, n_experts, bias=False)self.k, self.temp, self.noise_std, self.cap = k, temp, noise_std, capdef forward(self, x): # x: [tokens, hidden]logits = self.w(x)if self.training and self.noise_std > 0:# Noisy gating: breaks routing deadlock so early exploration covers all expertslogits = logits + torch.randn_like(logits) * self.noise_stdprobs = F.softmax(logits / self.temp, dim=-1)topv, topi = probs.topk(self.k, dim=-1)weights = topv / topv.sum(dim=-1, keepdim=True).clamp_min(1e-9)# Load-balancing auxiliary loss: suppresses expert collapsef = torch.zeros_like(probs).scatter_(1, topi, 1.0).mean(0) # actual assignment sharep = probs.mean(0) # average gate probabilityaux = (f * p).sum() * (n_experts := probs.shape[-1])# Expert capacity cap: overflow tokens fall back to the residual path# so tail experts don't slow down the whole batchcap = int(self.cap * x.shape[0] * self.k / max(1, probs.shape[-1]))return weights, topi, aux, cap
There are three key points: the load-balancing auxiliary loss constrains the assignment entropy, a capacity cap plus drop/residual fallback prevents a single expert from becoming a long-tail bottleneck, and routing noise ensures all experts receive gradients early in training.
2.3 Engineering costs you must budget for upfront
MoE is not a free lunch. Several costs belong in the plan from the start:
- Memory is counted across all experts. Sparse activation saves compute, not storage; weights, optimizer states, and the KV cache all have to be sized for total capacity.
- Communication cost rises. Expert parallelism scatters one token’s computation across different devices, and all-to-all communication becomes the bottleneck. Load imbalance shows up directly as the whole batch being held back by the slowest device.
- It is more sensitive to quantization. Each individual expert is already a small-capacity module with a more concentrated weight distribution, so post-training quantization errors are amplified more than in a dense model. This calls for finer calibration sets and grouping granularity.
- The advantage is weak at small batch sizes. When request volume is not enough to reuse experts fully, sparsity actually adds scheduling overhead.
Conclusion: MoE fits scenarios with clearly high concurrency and long-running online serving. When evaluating it, you must compute total memory + communication topology + peak concurrency together, rather than looking only at the number of activated parameters.
3. Route 2: Long Context — the Most Misunderstood Route
3.1 Two cost curves, one ignored multiplier
Long context has two independent cost curves:
- Attention compute grows quadratically with sequence length (without sparse or linear modifications);
- The KV cache grows linearly with sequence length, and it is multiplied by concurrency.
The second curve is often left out of capacity planning:
KV bytes per token per layer = 2 (K and V) × kv_heads × head_dim × bytes_per_valueTotal KV memory = num_layers × bytes_per_token_per_layer × context_length × concurrency
Example: with 60 layers, 8 kv_heads, head_dim 128, and FP16, a single token in a single layer costs about 2 × 8 × 128 × 2 = 4 KB in KV cache. Across the full model that is about 240 KB per token, and a 32K context comes to about 7.7 GB — then multiply by concurrency. This is why “supporting long context” and “cheaply supporting long context at high concurrency” are two different things.
3.2 Position extrapolation and the training data mix
Making the window longer inevitably involves the extrapolation ability of position encodings. The common approaches fall into two types: position interpolation (compressing position indices to fit an unseen range) and base-frequency scaling (adjusting the frequency distribution of rotary position encodings so that remote positions have better resolution). But what really decides extrapolation quality is often not the technique — it is the mix of long-sequence data during training: you need to deliberately construct long-range dependency samples that span paragraphs and documents, and apply block-level attention masks during concatenation so that unrelated tokens from different documents do not interfere with each other and contaminate the long-range dependency signal.
3.3 Effective context ≠ advertised context
The advertised window length and the length a model can actually use are often not the same. Engineering acceptance should be stress-tested in layers:
- Single-point retrieval: place the target information at the beginning, middle, and end, and check whether recall is consistent (mid-context decay is a common phenomenon);
- Multi-hop reasoning: require connecting several facts scattered across opposite ends of the window to answer;
- Distractor pressure: embed highly similar decoys in the context to test whether the model is misled;
- Refusal ability: when the answer is not in the context, the model should clearly say it does not know, rather than fabricating.
Only after running these four layers on your own data can you determine “how long this window actually reaches for our task.”
3.4 Long context vs. retrieval augmentation: where they intersect
The two are not substitutes but an intersection of cost curves:
- Structured, frequently updated data that needs citation traceability (orders, document libraries, knowledge entries) → prioritize retrieval augmentation; tokens are precise, cost is low, and it is auditable;
- Data that needs whole-file global understanding and whose semantic connections are hard to chunk (an entire codebase, long meeting transcripts, full contract comparisons) → long context is a better fit;
- The most common production combination: coarse-grained retrieval recalls candidates → long context loads them all at once for global re-ranking and reasoning.
def assemble_prompt(query, retriever, budget=24_000):docs = retriever.search(query, top_k=200) # coarse recall, allows noiseselected, used = [], 0for d in sorted(docs, key=lambda x: x.score, reverse=True):n = d.tokensif used + n > budget: # explicit budget controlbreakselected.append(d)used += n# Key point: place the most relevant chunks at the start and end# to counter mid-context attention decayselected.sort(key=lambda d: 0 if d is selected[0] else (1 if d is selected[-1] else 2))context = "\n\n".join(f"[{i}] {d.text}" for i, d in enumerate(selected))return f"Reference material:\n{context}\n\nQuestion: {query}\nWhen answering, cite the reference numbers.", used
3.5 Code capability: the most direct beneficiary of long context
Code tasks are the most natural landing spot for the long-context route, because they inherently have three traits: “cross-file, long dependencies, verifiable.” There are three engineering points: repository-level indexing and symbol-reference retrieval (feeding long context to the few files that are “actually relevant” rather than the whole repo), fill-in-the-middle training objectives (getting the model used to completing in the middle rather than only continuing the end, which is directly tied to the IDE completion experience), and a compile-and-test feedback loop (feeding failure information back to the model to iterate, which is more effective than simply piling on parameters).
4. Route 3: Lightweight and Inference Engineering — the Most Underrated Advantage
4.1 “Lightweight” actually means three different things
These are often lumped together, but they should be separated:
- Lightweight model: a dense model with fewer parameters; low deployment barrier and can be self-hosted, but with a limited ceiling;
- Lightweight structure: sparse-activation architectures, i.e. the MoE idea described above;
- Lightweight serving: the model stays the same, and cost is driven down through inference engineering — this category often has the highest return on investment.
4.2 Four levers on the inference side
- Quantization: a combination of weight quantization and activation quantization, with full grouping, per-channel scaling, and a calibration set that covers the real distribution. In practice, for precision-sensitive tasks, use layered quantization (keeping key layers at higher precision) and separately regression-test numerical stability.
- Distillation and data mix: use a strong model to produce high-quality instruction and explanation data, then train a target small model; at the same time, adjust the corpus mix according to the business distribution. Small models do have a lower capability ceiling, but on narrow tasks, with good data, they often capture 80% of the benefit.
- Decoding-side optimization: continuous batching (new requests can be inserted without waiting for the whole batch to finish), paged KV cache management (reducing memory fragmentation and raising concurrency), prefix reuse (KV cache hits for the system prompt and fixed context), and speculative decoding (draft model / self-drafting + verification and acceptance, mainly to cut decode-stage latency).
- Scheduling and caching: semantic caching, similar-query hits, and routing by task complexity to models of different sizes.
4.3 The metrics that matter, not just one
You must look at five metrics at once — optimizing any single one can just shift the cost elsewhere:
- TTFT (time to first token): decides the perceived “lag” of an interaction, and is most affected by prefill length and queueing;
- TPOT / inter-token output interval: decides how smoothly long text streams;
- System throughput (tokens/s): decides unit cost;
- Max concurrency per card / SLA attainment rate: decides capacity planning and the safety margin;
- Cost per effective task: only after folding in retry rate, failure rate, and human fallback is it the real cost.
5. Route 4: Multimodality and Knowledge Enhancement
The mainstream paradigm for multimodal deployment is a three-stage “perception encoder + projection layer + language model”: a vision encoder extracts features, the projection layer aligns them to the language model’s representation space, and the language model does the reasoning. Another exploration direction is to cut images into patches, unify them into a token sequence, and fuse them early. The most practical cost variable here is resolution and the token budget: dynamic-resolution tiling preserves the readability of small text and tables, but the number of tiles multiplies directly into token cost. So document-understanding scenarios must do “layout analysis + on-demand tiling,” not unconditional high resolution.
Knowledge enhancement solves a different problem: probabilistic generation is inherently fuzzy at factual boundaries. Injecting the deterministic relations of a knowledge graph and authoritative snippets retrieved into the context can effectively narrow those fuzzy boundaries. Notably, alignment and controllability have two complementary paths — one builds a rule-based value-constraint framework inside the model, and the other does localized-context optimization and knowledge-graph enhancement at the training-data level. In engineering, the two should be stacked, and before launch you should run regression on a fixed evaluation set to avoid any change on either side causing overall behavioral drift.
6. Scenario-Based Selection Matrix
| Scenario trait | Preferred route | Key constraint |
|---|---|---|
| High concurrency, long-running online service | Sparse architecture | Total memory, communication topology, expert load balancing |
| Global understanding of long documents / contracts / codebases | Long context + retrieval hybrid | KV memory, measured effective window, citation traceability |
| IDE completion, low-latency interaction | Lightweight dense model + inference engineering | TTFT, speculative decoding, prefix caching |
| Self-hosted deployment, data cannot leave the domain | Small-parameter dense model + quantization | Single-card memory, offline throughput, controllability |
| Understanding receipts / charts / scanned documents | Multimodality + dynamic resolution | Token budget, layout analysis, structured output |
| Strong factual question answering | Retrieval augmentation + knowledge graph | Recall quality, timely updates, refusal ability |
The recommended order when getting started is: build your own evaluation set first → then define the constraint boundaries (latency / cost / compliance) → and only then pick a route. Doing the reverse — picking an architecture first and then hunting for a scenario — almost always leads to rework.
7. Deployment Architecture and Observability
A recommended design has four layers, turning “swapping the model” into a replaceable configuration rather than a rebuild:
- Model serving layer: mount models of different sizes by task tier, with support for canary rollout and rollback;
- Retrieval layer: hybrid vector and keyword retrieval, retaining source text and citation location;
- Orchestration layer: workflows, model routing, caching, retries and idempotency, and structured-output validation;
- Governance layer: call logging, quality and cost monitoring with alerts, quotas and rate limiting, and an evaluation regression pipeline.
When deploying on the cloud, this can be carried by a set of general-purpose categories: a model serving platform for unified access to and invocation of multiple model types, object storage for archiving corpora and training/inference artifacts, a container platform for elastically scaling inference services, a message queue for decoupling long-running tasks, and logging, monitoring, and alerting to connect quality metrics with cost accounting. Baidu AI Cloud can support these general-purpose capability categories, and developers can use them to build the full “model–retrieval–orchestration–governance” chain. Refer to the official documentation for specifics.
8. Six Common Pitfalls
- Looking only at public leaderboards, not your own scenario evaluations: leaderboards and business distributions almost never match.
- Treating long context as a cure-all: ignoring the KV memory multiplier, mid-context decay, and rising cost.
- Adopting a sparse architecture blindly: without a clear memory and communication budget, the real cost can actually be higher.
- One-size-fits-all quantization: using the same precision across the whole model, so sensitive layers collapse in accuracy, with no regression testing done.
- Shipping without an evaluation set: every subsequent optimization then loses its basis for judgment.
- Over-relying on swapping the model: the payoff from investing in prompt design, retrieval quality, and output validation is often higher than switching to a larger model.
9. Action Checklist
- Build an evaluation set from 100–300 real business samples, annotating expected outputs and key constraints;
- Measure TTFT, TPOT, throughput, failure rate, and cost per task separately for each candidate solution;
- Run the four-layer stress test on long-context solutions to determine the real usable window;
- Calibrate quantization and batching parameters on the inference service, and lock in the SLA safety margin;
- Establish a dual loop of offline regression + online monitoring — any change to the model or prompt must pass the evaluation set first.
