ERNIE 5.5 Flash Base Explained: A Developer's Guide to Baidu's Open-Source Sparse MoE Model

1. Release at a Glance

On November 5, 2026, Baidu officially released and open-sourced ERNIE 5.5 Flash Base, a base model. The key facts are:

• Architecture: sparse Mixture-of-Experts (sparse MoE)
• Parameter count: 280B total parameters, with only 14B activated per token
• Pre-training: trained from scratch on 33.5T tokens
• Context window: a maximum training context of 128K, and up to 1M tokens for fine-tuning and inference
• Official benchmarks: Baidu reports state-of-the-art results for its size class (SOTA) on knowledge, reasoning, math, code, multilingual and long-context benchmarks. The model is also said to lead other open-source base models of the same size class on several capabilities.

Headline facts alone tell you only that the model is bigger, its context is longer and its scores are better. Developers also need to know:

1. What do the 280B and 14B figures actually constrain?
2. How should you understand and validate the 1M-token context capability?
3. How should you interpret the “SOTA for its size class” claim?
4. What engineering problems come up during deployment and fine-tuning?

The rest of this article answers these four questions.

2. Reading the Numbers: 280B Total, 14B Activated

2.1 What sparse activation means

In a standard Transformer decoder, every token passes through all of the model’s parameters. MoE splits the feed-forward network (FFN) into several parallel “experts” and adds a lightweight router network. For each token, the router picks only a few experts to do the work and leaves the rest idle.

ERNIE 5.5 Flash Base activates 14B parameters per token, about 5% of the total. In other words, only 1 in every 20 parameters is used. Parameters decide where compute goes, and sparse activation decouples “knowledge capacity” from “per-token compute”:

• Total parameters determine how much knowledge the model can hold and how much memory it takes to deploy.
• Activated parameters determine how much computation each token actually costs.

The diagram below shows the basic flow of a sparse MoE forward pass. The experts receive different numbers of tokens, which is one of the main things engineering work has to deal with.

Input token vector x
      │
      ▼
 ┌───────────┐   Score + Top-K selection
 │  Router   │ ─────────────────► Selected experts 1, 3, 7, …
 └───────────┘
      │ Dispatch with weights
      ▼
 ┌────┬────┬────┬────┬────┐
 │ E1 │ E2 │ E3 │ …  │ En │   Only selected experts compute
 └────┴────┴────┴────┴────┘
      │ Combine with weights
      ▼
   Weighted-sum output

2.2 What each number constrains

Dimension	Total parameters (280B)	Activated parameters (14B)
Main resources constrained	Memory capacity, memory bandwidth, distributed communication	Compute, inference latency, training compute
Impact on pre-training	Overhead of optimizer states and sharded gradients	Directly sets the compute per token
Impact on inference	Minimum number of GPUs needed; whether the model fits on a single server	Sets compute cost and the lower bound on decode latency
Impact on fine-tuning	Full-parameter fine-tuning needs a lot of memory	Optimizer states apply only to the parameters that are updated

A simplified order-of-magnitude estimate shows how this plays out:

• Pre-training compute: total training compute is roughly 6 × activated parameters × training tokens. Plugging in 6 × 14B × 33.5T gives about 2.8×10^24 FLOPs, or 2800 ZFLOPs. This is an ideal dense lower bound. Real training adds routing communication overhead and wasted cycles from load imbalance.
• Inference memory: at BF16 (16-bit floating point), the weights alone take 280B × 2 bytes = 560 GB. That needs at least 7 accelerator cards with 80 GB of memory.
• Inference memory access: during decoding, the activated expert weights are read from memory for every generated token, about 14B × 2 bytes = 28 GB.

These figures show that a small number of activated parameters does not mean a low resource barrier. Sparse MoE trades memory and bandwidth for compute efficiency. When deploying, you have to assess “does it fit?” and “is it fast enough?” separately.

2.3 The benefits and costs of a high sparsity ratio

A sparsity ratio of about 20:1 puts this model on the high-sparsity path. The benefits:

1. Greater knowledge capacity: for the same compute budget, more total parameters usually means more knowledge can be stored.
2. Per-token compute close to a 14B dense model: decode-time compute is low, which suits high-concurrency inference.
3. Relatively controllable training compute: each token only touches a small part of the model, so the same compute budget can cover more data.

The costs are equally clear:

• High memory pressure: the model does not fit on a single card and needs distributed approaches such as tensor parallelism and expert parallelism.
• Decode bound by memory bandwidth: compute per step is small, but each step reads a large number of weights, so decoding is often bottlenecked by memory access.
• Load imbalance: the router can send a large share of tokens to a few “hot experts,” leaving some devices idle.
• Higher fine-tuning risk: after pre-training, the experts are highly specialized. Full-parameter fine-tuning on a small dataset can easily break the routing balance and cause catastrophic forgetting.

3. From 33.5T Tokens to a 1M-Token Context

3.1 What 33.5T tokens means

33.5T tokens is one of the core investments in training from scratch. It matters on three levels:

• Data scale: by a common text tokenization rule of thumb, one Chinese character is about one token. That makes 33.5T tokens roughly equivalent to tens of billions of Chinese characters of text. A corpus this size can’t come from a single source. It usually combines web text, books, code, math and multilingual data with carefully filtered synthetic data.
• Data engineering: what really determines model quality is usually not the token count but the data mixture and cleaning quality. That includes deduplication, quality filtering, harmful-content filtering and privacy de-identification. General-domain data provides breadth, high-quality code and math data shapes reasoning and coding ability, and multilingual data determines cross-lingual transfer. The mix has to be set with ablation experiments.
• Training stability: large-scale MoE pre-training runs into two common problems. One is router “collapse,” where a few experts absorb most of the tokens. The other is loss spikes partway through training. Common industry mitigations include an auxiliary load-balancing loss, router noise, expert capacity limits, and higher numerical precision for routing computations.

3.2 128K training context vs. 1M fine-tuning and inference context

There are two kinds of “context” to distinguish:

• Training context: the longest sequence the model saw during pre-training, which is 128K here.
• Inference and fine-tuning context: the longest input the model can accept when deployed or fine-tuned, which is 1M here.

Anything beyond the training length is essentially extrapolation. The mainstream approach is to do most pre-training at a relatively short context length. The window is then extended by adjusting the position encoding (for example, scaling or interpolating the frequencies of rotary position embeddings) and continuing training on long samples. So a 1M-token window does not mean the model is as reliable at 1M tokens as it is within 128K. Real-world performance follows the “effective context” rule:

1. Accuracy usually drops as length increases.
2. Key information in the middle of a long text is more likely to be missed.
3. For tasks that need exact global retrieval or long chains of reasoning, a longer window does not mean stronger capability.

The diagram below shows a typical accuracy decay pattern in long-context tasks:

Accuracy
  │  ████████████████
  │  ███████████████
  │  ████████████
  │  ████████
  │  ████
  └────────────────────────►  Input length
    Short text      Medium-long     Near the limit
    (stable)        (decaying)      (clear decay)

The table below shows the main costs of a 1M-token context:

Cost	Growth	Main impact	Common optimizations
Attention compute	Quadratic in length	Time to first token (TTFT) rises sharply	Chunked attention, sliding window, sparse attention
KV cache memory	Linear in length	Fewer concurrent requests	KV quantization, paged cache management, prefix reuse
Decode-phase compute	Linear in length	Higher latency per token	Compressing history, hierarchical summarization

A 1M-token window does not remove these costs. The usual engineering practice is to retrieve only the relevant chunks when a task doesn’t need the full context, or to summarize in chunks and then aggregate the summaries. Treating the 1M window as a fallback rather than the default way of working is usually more economical.

3.3 How to test long-context performance yourself

Design tests along these lines:

• Retrieval and localization: plant key information at the beginning, middle and end of a long document and check whether the answers are accurate. For example, in a collection of contract terms about 500,000 characters long, can the model locate a payment clause around character 200,000?
• Cross-section reasoning: ask the model to combine information scattered across several sections before answering.
• Long-document summarization and code comprehension: feed in a full project description or multiple code files and check whether the summary is complete and the references are accurate.
• Length sensitivity: run the same task at 4K, 32K, 128K, 512K and 1M tokens. Plot the performance curve to find the boundary of the effective context.
• Cost and latency: record TTFT, time per token and memory usage at each length to see whether a long context is actually worth it.

4. How to Read “SOTA for Its Size Class” Critically

According to the official benchmarks, the model achieves best-in-class results on knowledge, reasoning, math, code, multilingual and long-context benchmarks. It also leads comparable open-source base models of the same size class on several dimensions. These results are useful, but you need to understand what they assume:

1. What “same size” means: are models compared by total parameters or by activated parameters? A model with 280B total and 14B activated parameters is not in the same league as a dense model with tens of billions of total and activated parameters. Their memory cost and per-token compute are on different scales. Make resource constraints comparable before drawing conclusions.
2. Base vs. assistant models: a base model is only pretrained or lightly continued-trained. Its output is raw continuation and does not follow instructions. Most mainstream benchmarks for knowledge, math and code are question-and-answer formats designed for instruction-tuned models. Running them directly on a base model tends to underestimate it. The standard practice is few-shot evaluation adapted to base models, or instruction fine-tuning before evaluation.
3. Evaluation setup: scores are very sensitive to setup. That includes whether chain-of-thought prompting is used, the prompt template, the number of few-shot examples, the sampling temperature, and the rules for extracting and scoring answers. A score without these details is hard to compare across models.
4. Benchmark saturation and data contamination: some public benchmarks have been widely trained on and no longer separate models well. More reliable options are private test sets, question banks that are updated regularly, and human evaluation.
5. Average vs. individual capabilities: a high average does not mean the model leads on every dimension. Weight the results according to your business priorities instead of looking only at the overall ranking.

The table below lists the questions to ask for each capability:

Capability	What to examine	Suggested evaluation
Knowledge	Factual accuracy, recency, hallucination rate	General knowledge QA; tests on unpublished or recent facts
Reasoning	Multi-step logic, common sense, instruction following	Logic problem sets, multi-turn instruction tasks
Math	Solution correctness, reliability of the steps	Competition problems, word problems, tasks requiring exact calculation
Code	Completion correctness, bug fixing, repository-level understanding	Unit test pass rate, cross-file modification tasks
Multilingual	Quality in low-resource languages, cross-lingual transfer	Native-language prompting, multilingual translation and summarization
Long context	Effective length, recall of information in the middle	Length-tiered tests, long-document QA

5. Deployment: Balancing Memory, Bandwidth and Concurrency

5.1 A worked example of inference resource estimation

The script below uses only the standard library. It estimates weight memory and the theoretical lower bound on decode latency. The parameters are example assumptions; use the official figures for your actual deployment.

# Rough estimation of inference resources for an MoE base model
# (illustrative only; adjust the parameters to match the model you deploy)
TOTAL_PARAMS_B = 280      # Total parameters (billions)
ACTIVATED_B = 14          # Parameters activated per token (billions)
def estimate(precision_bytes, gpu_mem_gb, gpu_bw_bs, gpus):
    weight_gb = TOTAL_PARAMS_B * precision_bytes           # Weight memory (GB)
    min_gpus = -(-weight_gb // gpu_mem_gb)                 # Round up
    step_ms = weight_gb / (gpus * gpu_bw_bs) * 1000        # Theoretical time to read the activated weights once (ms)
    print(f"Precision {precision_bytes * 8} bit: weights ~{weight_gb:.0f} GB, "
          f"at least {int(min_gpus)} x {gpu_mem_gb:.0f}GB GPUs")
    for batch in (1, 16, 64, 256):
        # Theoretical lower bound: assumes all sequences in the batch share one weight read
        print(f"  batch {batch:>3}: time per token >= {step_ms:>5.1f} ms, "
              f"theoretical throughput <= {batch / step_ms * 1000:>8.0f} token/s")
# Example assumptions: BF16 (2 bytes), 80 GB per GPU, 3 TB/s per GPU, 16 GPUs in total
estimate(precision_bytes=2, gpu_mem_gb=80, gpu_bw_bs=3000, gpus=16)
# Example assumptions: weights quantized to 8 bits (1 byte), other conditions unchanged
estimate(precision_bytes=1, gpu_mem_gb=80, gpu_bw_bs=3000, gpus=16)

Illustrative output:

Precision 16 bit: weights ~560 GB, at least 7 x 80GB GPUs
  batch   1: time per token >=  11.7 ms, theoretical throughput <=       86 token/s
  batch  16: time per token >=  11.7 ms, theoretical throughput <=     1372 token/s
  batch  64: time per token >=  11.7 ms, theoretical throughput <=     5489 token/s
  batch 256: time per token >=  11.7 ms, theoretical throughput <=    21954 token/s
Precision 8 bit: weights ~280 GB, at least 4 x 80GB GPUs
  batch   1: time per token >=   5.8 ms, theoretical throughput <=      171 token/s
  batch  16: time per token >=   5.8 ms, theoretical throughput <=     2743 token/s
  batch  64: time per token >=   5.8 ms, theoretical throughput <=    10974 token/s
  batch 256: time per token >=   5.8 ms, theoretical throughput <=    43907 token/s

The results show two patterns:

• Single-request latency is limited by the activated weight reads. For a single user decoding, latency mainly depends on how long it takes to read the activated weights, regardless of batch size.
• Total throughput depends on concurrent batch size. With more concurrency, the weight reads shared within a batch are spread more thinly, so total throughput goes up.

This is why high-sparsity MoE models are well suited to high-concurrency serving. It is also why capacity planning has to look at latency and throughput at the same time.

These are only theoretical lower bounds. In practice, performance is also affected by:

• Expert-parallel communication: the overhead of sending tokens across cards and nodes can become the bottleneck.
• Load imbalance: the slowest device in a parallel group sets the pace for the whole group.
• KV cache and attention overhead: this is especially significant with long contexts.

5.2 Parallelism and quantization strategies

280B parameters cannot fit on a single card on a single server. The common approach is to combine tensor parallelism with expert parallelism:

• Tensor parallelism: split a single expert’s weights across multiple cards to reduce per-card memory pressure.
• Expert parallelism: place different experts on different devices and use All-to-All communication to dispatch and collect tokens. There are usually far more experts than devices, so each card holds several experts.
• Communication optimization: All-to-All communication is sensitive to network bandwidth. Do as much of the routing communication as possible over high-speed interconnect within a single server, and keep cross-node communication to a minimum.

Quantization can cut costs with almost no loss of quality:

• Quantize the expert weights to 8 bits. This roughly halves weight memory and the theoretical lower bound on decode latency.
• Keep higher precision for sensitive modules such as attention.
• Re-evaluate the model after quantization. Pay particular attention to knowledge QA and multi-step math tasks.

5.3 Engineering notes for long-context serving

For long-context use cases, consider:

• Paged KV cache management: allocate memory in blocks on demand to reduce fragmentation and increase concurrency.
• Prefix caching: in multi-turn conversations or when a system prompt is shared, reuse the KV cache of the common prefix to avoid recomputing it.
• KV quantization: store the cache at lower precision to reduce memory use, but assess the accuracy loss.
• Separating prefill from decode: prefill is compute-intensive and decode is memory-access-intensive. Putting the two types of requests on separate resource pools keeps long inputs from blocking short requests.
• TTFT optimization: chunked processing of very long inputs, sparse attention and similar techniques.
• Output length limits: set a reasonable maximum output length per request so a few requests don’t hold memory for a long time.

5.4 Load-testing checklist before launch

• At your target concurrency, measure the P50/P95/P99 percentiles of TTFT, time per token and end-to-end latency.
• Watch how the system behaves with a mix of long and short requests, so long requests don’t slow everything down.
• Test memory peaks and error rates when approaching the maximum context length.
• Compare output quality across quantization precisions.
• Track cost per unit of throughput (for example, resource cost per million tokens) as input for capacity planning.

6. Fine-Tuning: Typical Pitfalls with MoE Models

6.1 Load balancing and router stability

The most common problem when fine-tuning MoE models is that the load balance breaks:

• Narrow fine-tuning data: a previously balanced distribution becomes heavily skewed. Some experts are overloaded, others sit idle, and inference latency goes up.
• Large router changes: when the learning rate is too high, routing decisions can change drastically. They can even fall into a self-reinforcing loop, where a few experts are used more and more and become stronger and stronger.

Ways to deal with this:

• Use a smaller learning rate, or an even smaller one for the router.
• Keep the auxiliary load-balancing loss in place.
• Monitor the token share of each expert and the router’s output distribution. Roll back promptly if something looks off.

The example below shows how to monitor load balancing:

import math
from collections import Counter
def load_balance_report(router_hits, num_experts):
    total = sum(router_hits.values())
    entropy = -sum(
        (c / total) * math.log2(c / total)
        for c in router_hits.values() if c > 0
    )
    max_ratio = max(router_hits.values()) / (total / num_experts)
    print(f"Load entropy: {entropy:.3f} / {math.log2(num_experts):.3f}")
    print(f"Hottest expert relative to mean: {max_ratio:.2f}x")
    return entropy, max_ratio
# Example data: 64 experts, most of them balanced, a few clearly hotter
hits = Counter({i: 1000 for i in range(60)})
hits.update({60: 4000, 61: 3000, 62: 2500, 63: 2000})
load_balance_report(hits, num_experts=64)

Illustrative output:

Load entropy: 5.705 / 6.000
Hottest expert relative to mean: 2.40x

The closer the load entropy is to its theoretical maximum, the more balanced the distribution. The higher the hottest-expert ratio, the more imbalanced it is. As a rule of thumb, when that ratio stays high for a while, check the router learning rate, the auxiliary loss coefficient and the data distribution.

6.2 Parameter-efficient vs. full-parameter fine-tuning

• Parameter-efficient fine-tuning (PEFT), such as LoRA:
  • It trains only a small number of extra parameters, so it uses little memory and forgets less.
  • LoRA on the attention layers usually pays off.
  • Adding LoRA to the expert layers multiplies the number of adapters and adds routing overhead, so you need to weigh the trade-off.
  • Watch for precision loss when merging adapters back into the model.
• Full-parameter fine-tuning:
  • It suits cases with a lot of data and a large domain shift.
  • Optimizer states and sharded gradients use a lot of memory, so a distributed training framework is usually required.
  • Mix in a share of general-domain data to reduce forgetting.

The “1M tokens for fine-tuning” support mainly targets training tasks that involve very long samples. Such training uses a lot of memory and usually needs techniques such as sequence parallelism and gradient checkpointing.

6.3 Suggested fine-tuning workflow

1. Define the goal: decide which capability to improve, such as domain QA, style alignment or structured output.
2. Prepare the data: deduplicate, clean and standardize the format. Split it into training and validation sets at roughly 8:2, and make sure the distribution matches production.
3. Validate on a small scale: run a quick end-to-end test on a small dataset first.
4. Train in stages: use a larger learning rate first to adapt quickly to the new task, then a smaller one for stable convergence.
5. Monitor throughout: track the loss, validation metrics, expert load distribution and generation quality at the same time.
6. Run regression tests: evaluate on both general and domain-specific test sets to avoid fixing one capability at the expense of another.
7. Roll out gradually: release to a small share of traffic before a full launch.

7. Model Selection Checklist

Overall, ERNIE 5.5 Flash Base fits:

• High-concurrency, high-throughput online services that are sensitive to per-token cost
• Tasks that involve long documents, long conversations or large blocks of code
• Teams planning custom fine-tuning on industry data
• Teams with the capacity to run and operate multi-GPU, multi-node inference

Work through these questions when evaluating the model:

• Can the target hardware handle your peak concurrency and target latency?
• Does your memory budget cover the weights, the KV cache and runtime overhead?
• Do your tasks really need an extra-long window, based on their effective-context requirements?
• Does your evaluation reflect how much you care about knowledge, reasoning, math, code and multilingual ability?
• Do you have your own private test set?
• Does your fine-tuning approach (PEFT or full-parameter) match the size of your dataset?
• Can you monitor expert load, memory and latency?

8. Conclusion

The release of ERNIE 5.5 Flash Base reflects the design philosophy of the sparse MoE path: trade parameters for knowledge and control compute through activation. The 280B total parameters provide knowledge capacity, the 14B activated parameters keep per-token compute in check, pre-training on 33.5T tokens builds the data foundation, and the 1M-token context extends what the model can be used for.

For developers, looking critically at marketing claims is only the first step. The more important step is to validate the model in your own business context:

• Before deployment: estimate memory, bandwidth and concurrency.
• When selecting: measure effective context with your own private test set.
• When fine-tuning: watch router stability and the risk of forgetting.
• After launch: keep monitoring expert load, latency and cost.

Only when these engineering details are handled properly can an open-source base model become stable, controllable and cost-effective enterprise AI capability.