How to Read LLM Technical Reports: MoE, Activated Parameters, and Evaluation Pitfalls
This article provides a practical framework for reading large language model technical reports without being misled by headline numbers. It explains total versus activated parameters, sparse activation, Mixture-of-Experts architecture, training data quality and mixture, post-training, agentic capabilities, benchmark interpretation, and common evaluation pitfalls. Readers will learn what to check in a report and how to choose models based on real use cases rather than rankings alone.
Introduction: How to Read an LLM Technical Report Without Just Looking at the Numbers
Every time a new large language model is released, its technical report is often the main window through which outside readers understand it. However, these reports are full of parameter counts, token counts, benchmark scores, and architecture terms. Without a way to interpret them, it is easy to be misled by the intuition that “bigger is better.”
The real question is not what a specific score is, but what engineering trade-offs sit behind those metrics: how much capacity the model has, how much compute it costs per inference, how it was trained to reason and hold conversations, and how its real capability should be measured. The following sections provide a practical framework for reading LLM technical reports.
1. Total Parameters vs. Activated Parameters: How to Read Sparse Activation?
The first distinction to make in a technical report is between two concepts that are often confused: total parameter count and activated parameter count.
- Total parameter count determines the model’s capacity: the amount of knowledge and patterns it can theoretically hold and store. Some flagship model reports may disclose total parameter counts at the level of hundreds of billions, such as “more than 700 billion.” However, if such a number does not clearly refer to a specific real model, it should be treated as illustrative and not directly cited as factual data.
- Activated parameters are the parameters actually used during each inference. The proportion of sparse activation is usually low, often in the range of 5% to 10%. For example, if a model has 400 billion total parameters and 40 billion activated parameters, the activation ratio is 10%. If it has 500 billion total parameters and 25 billion activated parameters, the ratio is about 5%.
The core idea behind this is sparse activation: although the model “knows” a lot of information, it does not need to use all parameters to answer a specific question. Instead, it prioritizes a small, most relevant subset. This is similar to an expert answering a question: an expert does not activate every part of their brain at once, but focuses on the most relevant experience.
The engineering benefit is critical: total parameters determine the upper bound of capability, while activated parameters determine the actual cost of each inference. Because only a small number of parameters are activated each time, very large models can be served within acceptable latency and compute budgets. Therefore, total parameters and activated parameters must be read as a pair; looking at only one of them can easily lead to the wrong conclusion.
2. What Is MoE? How Is Sparse Activation Implemented?
A common engineering implementation of sparse activation is the Mixture of Experts architecture, or MoE. A traditional dense model can be understood as a general practitioner who handles every problem with the same set of parameters. An MoE model is more like a hospital with multiple specialists: when a patient arrives, a triage step decides which few experts should handle the case, and only the selected experts do the work.
A typical mechanism works as follows:
- Each MoE layer contains many parallel experts, and each expert is essentially an independent feed-forward network.
- A lightweight gating network, the Router, scores all experts for each input token.
- Top-K selection picks the K experts with the highest scores, usually K=2, and only these experts participate in computing that token.
- The outputs of the experts are weighted and summed according to the gating weights to produce the final output of the layer.
The following code is an illustrative example for understanding the basic logic of MoE. It helps explain routing and gating, but it is not a production-grade efficient implementation:
import torchimport torch.nn as nnimport torch.nn.functional as Fclass TopKMoELayer(nn.Module):"""Illustrative sparse Mixture-of-Experts layer: logically uses only Top-K experts.Note: For clarity, this implementation evaluates all experts first, then combines them using gate masks.Production systems typically dispatch tokens by expert_id and compute only the selected experts."""def __init__(self, d_model, d_ff, num_experts, top_k=2):super().__init__()self.top_k = top_kself.router = nn.Linear(d_model, num_experts) # Gating networkself.experts = nn.ModuleList([nn.Sequential(nn.Linear(d_model, d_ff),nn.GELU(),nn.Linear(d_ff, d_model),) for _ in range(num_experts)])def forward(self, x): # x: [batch, seq, d_model]logits = self.router(x) # Score every expertweights, idx = torch.topk(logits, self.top_k, dim=-1)weights = F.softmax(weights, dim=-1) # Normalize gating weightsout = torch.zeros_like(x)for k in range(self.top_k):expert_id = idx[..., k]gate = weights[..., k].unsqueeze(-1)for e, expert in enumerate(self.experts):# Illustrative implementation: evaluate all experts first, then combine using gate masks; not efficientmask = (expert_id == e).unsqueeze(-1)out += mask * gate * expert(x)return out
In engineering practice, MoE also needs to solve several key problems:
- Load balancing: If the router always sends tokens to a few popular experts, other experts remain idle and compute is wasted. Auxiliary load-balancing losses are often added during training.
- Capacity factor: To keep training stable, the number of tokens each expert can receive may be capped. Tokens beyond the limit need to be dropped, reassigned, or handled by other strategies.
- Routing collapse: If the model degenerates into always selecting only a few experts, expert diversity decreases and generalization can suffer.
- Expert parallelism: In real deployment, the number of experts is often large, so different experts need to be split across devices for training and inference.
When reading a technical report, therefore, it is not enough to check whether the model has MoE. It is also important to see whether the report explains the number of experts, the activation strategy, load balancing, capacity factor, and parallel deployment.
3. Training Data: How to Judge Scale, Quality, and Mixture?
The amount of training data is another core metric in a technical report. For example, a report may say that the model was trained on 28.5 trillion tokens.
How large is that? It can be roughly converted as follows: if an ordinary book has about 300,000 words and one token is roughly approximated as one word, 28.5 trillion tokens is roughly equivalent to nearly 100 million books. If a book is estimated at 100,000 words, it could be several hundred million books. Such conversions are only to help understand the order of magnitude and should not be treated as exact facts.
When interpreting training data, it is not enough to look only at “big.” Three dimensions matter:
- Scale: The total number of tokens affects how rich the language phenomena the model has seen are, which influences the breadth of knowledge coverage.
- Quality: Filtered, deduplicated, and cleaned high-quality data is more valuable than simply piling up noisy text. Low-quality repeated data is not only unhelpful but can also harm model performance.
- Mixture: The proportion of different sources, such as code, mathematics, multilingual text, dialogue corpora, and web text, can significantly shape the model’s capabilities.
A common misconception is that “more data is always better.” In reality, when the compute budget is fixed, model size and training tokens need to be scaled together; an imbalance can waste compute. For example, if code and mathematics corpora make up a larger share, the model may perform better at programming and logical reasoning. If multilingual corpora are more sufficient, the model may have an advantage on cross-lingual tasks. Therefore, a report that discloses data scale, source mixture, and cleaning strategy is usually more credible.
4. Post-Training: From “Being Able to Predict” to “Being Able to Reason and Converse”
Pre-training gives the model a language foundation, but it is essentially performing next-token prediction and is not naturally good at following instructions, multi-turn dialogue, or rigorous reasoning. Post-training is the key stage that makes the model usable and reliable.
Post-training usually includes several stages:
- Supervised fine-tuning, SFT: Training with high-quality instruction-response pairs helps the model learn to answer in the format and style expected by humans.
- Preference optimization: Using human preference signals that rank multiple responses helps the model output values that are more useful, harmless, and honest.
- Reasoning-oriented reinforcement learning: For verifiable tasks such as mathematics, code, and logical reasoning, outcome correctness is used as a reward signal to strengthen the model’s chain-of-thought capability.
Post-training is similar to job training before starting work. Even a highly educated newcomer needs targeted training before becoming truly competent in a specific role. When interpreting a report, the more detailed the post-training description, especially the emphasis on verifiable reasoning reinforcement learning, the more it usually indicates that the model is more usable in practical tasks.
5. Agentic Capabilities: How to Judge Whether a Model Can Complete Tasks Autonomously?
One important dimension for measuring how advanced a large model is its agentic capability, or the ability to act as an intelligent agent. Simply put, it is whether the model can complete complex tasks autonomously, like a smart intern.
Traditional conversational models operate in a passive mode: they answer one question at a time. A model with agentic capabilities can face open-ended requests, such as “build me an e-commerce website,” and complete the whole process autonomously:
- Planning: Break down the task and design the overall architecture.
- Coding: Write the front-end interface and back-end services.
- Debugging: Run the code, locate bugs, and fix them.
- Verification: Execute test cases and confirm that the functionality is correct.
This closed-loop capability, from instruction to delivery, is an important step for large models to move from “tool” to “collaborator.” When reading a report, it is useful to check whether it provides quantifiable evaluations, such as:
- Tool-call success rate;
- Long-horizon task completion rate;
- Average number of steps to completion;
- Error recovery ability;
- Tool usage cost or latency;
- Retry and correction ability in failed tasks.
If a report only vaguely claims “agentic capability” without giving task types, evaluation methods, or result metrics, its credibility should be treated cautiously.
6. Benchmarks: How to Scientifically Interpret Capability Comparisons?
Technical reports usually list a set of benchmark scores to compare with similar models. Understanding these numbers is more important than simply memorizing them. Common benchmarks can be roughly divided into several categories:
- Software engineering benchmarks based on real code repositories: These evaluate the model’s ability to fix bugs and understand codebases in real projects, and best reflect engineering practicality.
- High-difficulty comprehensive reasoning benchmarks: These have extremely difficult questions and are used to distinguish the ceiling of models in deep knowledge and reasoning.
- Deep browsing and retrieval benchmarks: These evaluate the model’s ability to use search tools to find and integrate answers from large amounts of information.
- Terminal/command-line operation benchmarks: These evaluate the model’s practical ability to execute multi-step tasks in command-line environments.
When interpreting benchmark scores, it is especially important to watch out for the following pitfalls:
| Comparison dimension | With tools | Without tools | How to interpret |
|---|---|---|---|
| Browsing and retrieval | Can call search and web scraping | Relies on the model’s internal knowledge | Scores cannot be directly compared side by side |
| Code repair | Can run tests and read repositories | Can only generate code statically | Scores with tools are closer to real engineering practice |
| Reasoning tasks | May call a calculator or code | Pure model reasoning | Check whether external capabilities are enabled |
| Evaluation protocol | Prompts and decoding strategies may differ | Protocols may be inconsistent | Cross-report comparisons should be cautious |
In practice, it is important to check three points:
- Whether tools are enabled: For the same benchmark, scores with tools and without tools cannot be directly compared. If the report clearly labels this, it is more credible.
- Whether the evaluation method is consistent: Different reports may use different evaluation protocols, prompts, and decoding strategies, so cross-report score comparisons should be cautious.
- Whether there is a risk of data contamination: If the training data may contain evaluation questions, the authenticity of the scores may be affected.
From an engineering perspective, a responsible report will usually state that, on many general capabilities, top models may already be quite close. However, in specific directions such as browsing and retrieval or reasoning with tool use, some models may gain relative advantages through specialized optimization. Therefore, users should not look only at the overall ranking. They should choose the model that best matches their actual scenario, such as programming, retrieval, reasoning, or terminal operation.
7. Practical Checklist for Reading Technical Reports
Based on the above dimensions, the following checklist can help quickly grasp the key points when reading a large model technical report:
- What are the total parameter count and activated parameter count? Are they disclosed as a pair?
- What is the activation ratio? Sparse activation is commonly around 5% to 10%, but different models should not be directly assigned the same number.
- Are flagship parameter numbers in the report illustrative, or do they clearly correspond to a real model?
- Does the model use MoE? Are the number of experts, Top-K activation strategy, load balancing, and capacity factor explained?
- Are the scale, source mixture, and cleaning strategy of the training data disclosed? Is the order-of-magnitude conversion clearly defined?
- Does post-training include verifiable reinforcement learning oriented toward reasoning?
- Do agentic capabilities have quantifiable metrics, such as tool-call success rate, task completion rate, and average number of steps?
- Do the benchmarks indicate whether tools are enabled, what evaluation protocol is used, and whether there is a data contamination risk?
Conclusion: Judging Real Capability Beyond Numbers and Terminology
The core value of a technical report is not how high the numbers it presents are, but whether it shows a coherent set of engineering trade-offs: using sparse activation to balance capability and cost, using high-quality data as the foundation, using post-training to refine usability, using agentic capabilities to expand application boundaries, and using rigorous evaluation to verify real capability.
With this framework, readers can cut through terminology and marketing language, truly understand the capability of a large model, and make rational technical selection decisions.
