Trillion-Parameter MoE Models and Agent Capability Evaluation: From Leaderboard Scores to Task Completion

1. Shifting evaluation focus: from answering well to completing tasks

Over the past two years, the mainstream metrics for model capability evaluation have been the accuracy of single-turn question answering, the pass rate of reasoning chains, and the execution pass rate of code snippets. These metrics share a common assumption: task boundaries are clear, context is self-contained, and feedback is immediate. Once a task requires the model to autonomously plan multiple steps, call external tools, produce side effects in an environment, and adjust its strategy based on the results, the original evaluation paradigm begins to fail.

In recent public industry discussions, one notable phenomenon is that a leading sparse MoE model has entered the same range as top closed-source models from outside China on several agent-oriented benchmarks for the first time, while still showing a clear gap in pure code generation and terminal command execution. This kind of structural difference is more informative than a single aggregate score: it shows that capability growth is uneven across task families.

Therefore, the truly valuable goal is not to memorize a score, but to understand three things:

1. Where the cost structure of trillion-parameter sparse MoE models comes from.
2. What engineering constraints are involved in full-stack domestic-compute training.
3. What agent benchmarks actually measure, and what shortcomings they expose.

2. 1.6 Trillion Total Parameters and About 48 Billion Activated Parameters per Token: The Benefits and Costs of Sparsification

2.1 The roles of the two parameter counts

Sparse MoE decouples “total parameter count” from “compute used in a single inference pass”:

• Total parameter count determines the upper bound of knowledge capacity. It is almost entirely resident in memory as weights, affecting VRAM usage, storage cost, and bandwidth cost.
• Activated parameter count determines the compute cost of a single forward pass. In each layer, only the small number of experts selected by the router participate in computation.

With about 48 billion activated parameters relative to 1.6 trillion total parameters, the activation ratio is roughly 3%. This means the inference compute is approximately equivalent to a dense model at the 50-billion-parameter level, but the knowledge coverage comes from a parameter pool more than 30 times larger. On the training side, the estimate of forward and backward compute follows the same sparse accounting:

# Rough estimate of training compute: FLOPs ≈ 6 × activated parameters × training tokens
activated_params = 48e9          # activated parameters per token
training_tokens = 20e12          # number of training tokens (example value)
flops = 6 * activated_params * training_tokens
# 1e24 FLOPs is approximately 1 YFLOP
print(f"{flops / 1e24:.2f} YFLOPs")

A frequently overlooked engineering conclusion is that sparsification reduces compute, but it does not reduce memory. Full expert weights must remain resident, making Expert Parallelism almost a hard requirement. Routing also introduces cross-device all-to-all communication. The essence of architecture design is to find a balance between “compute savings” and “communication and memory overhead.”

2.2 Routing illustration

def moe_layer(x, gate_w, experts, top_k=8, capacity_factor=1.25):
    # x: [tokens, hidden]
    logits = x @ gate_w                    # [tokens, num_experts]
    topk_logits, topk_idx = logits.topk(top_k, dim=-1)
    weights = softmax(topk_logits, dim=-1)  # normalized gating weights
    y = zeros_like(x)
    for e in active_experts(topk_idx):      # dispatch only the experts that are hit
        token_mask, slot = assign(x, topk_idx, e, capacity_factor)
        y[token_mask] += weights[token_mask, e].unsqueeze(-1) * experts[e](x[token_mask])
    return y

The truly difficult engineering problem is load balancing. If capacity_factor is too large, compute is wasted; if it is too small, tokens are dropped. Issues such as auxiliary load-balancing loss, routing jitter, and expert hotspots directly affect training stability. These details determine whether “nominal activated parameters” and “actual average compute” remain consistent.

3. Full-Stack Domestic-Compute Training: Where the Constraints Are and Where the Benefits Are

“Training entirely on domestic compute” is a statement with substantive information: it is both a proof of capability and a set of engineering constraints.

Constraint 1: Mapping parallel strategies to communication primitives. Data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism must be implemented on concrete collective communication operators. When the operator coverage of the communication library and the maturity of the surrounding ecosystem differ, teams often need to compensate for bandwidth loss through communication overlap, computation reordering, and topology-aware partitioning strategies.

Constraint 2: Operators and numerical stability. If fused operators for attention, normalization, or routing are missing, they must be implemented and validated for numerical consistency. Problems such as overflow, gradient explosion, and loss spikes under low-precision training are amplified during long training cycles.

Constraint 3: Stability engineering. In training at the scale of tens of thousands of accelerators, hardware failures and performance fluctuations are the norm rather than “running once without interruption.” Effective practices include hierarchical checkpointing, asynchronous saving, fault-domain isolation, automatically removing slow nodes based on step frequency, and anomaly detection based on gradient norms and loss curves.

# Minimum viable set of training monitoring metrics (illustrative)
metrics = {
    "loss": loss.item(),
    "grad_norm": grad_norm,
    "router_entropy": compute_router_entropy(),   # whether routing converges prematurely
    "expert_load_imbalance": load_cv(),           # load imbalance coefficient
    "step_time_p95": ..., "tokens_per_sec": ...,
}
if metrics["grad_norm"] > threshold or is_nan(loss):
    trigger_checkpoint_and_restart(step)

The benefits are also clear: autonomy over compute and toolchains reduces supply-chain risk, iteration speed is no longer constrained by external quotas, and the cost and delivery cycle per unit of compute become more predictable. For cloud vendors building domestic compute stacks, these training practices also push the maturity of heterogeneous compute adaptation, distributed training framework optimization, and engineering toolchains—such as object storage, logging services, monitoring and alerting, and containerized scheduling—and provide a more stable foundation for upper-layer model services.

4. What Agent Benchmarks Actually Measure

Treating “agent capability” as a single overall metric is misleading. It includes at least three separable capability dimensions.

4.1 Automated business workflow execution

Given a business goal and a set of tools—forms, approvals, database queries, and message notifications—the model must autonomously decide the call order, handle exceptional branches, and fall back when failures occur. Scoring is usually based on both result correctness and process efficiency:

def eval_workflow(task, agent, verifier, budget):
    steps, tokens, ok = 0, 0, False
    for _ in range(budget.max_steps):
        action = agent.step(task.state)
        result = dispatch(action)
        task.state = observe(task.state, result)
        steps += 1; tokens += action.tokens
        if verifier.is_done(task.state):
            ok = True; break
    return {
        "success": ok,
        "steps": steps,              # fewer steps are better
        "tokens": tokens,            # cost
        "tool_error_rate": ...,      # tool call failure rate
        "recovery_rate": ...,        # post-failure recovery success rate
    }

4.2 Broad and deep web retrieval tasks

These tasks test whether the model can construct a retrieval path when it does not know what to search for: query rewriting, cross-validation of results, long-document extraction, and conflict resolution across multiple sources. They do not reward memorization; they reward search strategy and evidence organization. Scoring is usually aligned with verifiable factual points, so the benchmark is resistant to fabrication.

4.3 3D and spatial code generation

These tasks require converting descriptions of spatial structures into executable geometry or scene code, testing understanding of coordinate systems, transformation matrices, and topological constraints. Accuracy on these tasks is difficult to improve through language pattern matching alone, so they better distinguish whether a model truly has structured reasoning ability.

5. Why Pure Code Generation and Terminal Command Execution Remain Weak Areas

Combining public evaluation feedback, the most stable weak areas are pure code generation and terminal/command-line instruction execution. This is not accidental; it can be explained by task characteristics:

1. Missing feedback loops. Web and business tasks have clear tool return values and verifiable end states. Terminal-task feedback is often implicit side effects—files changed, process state changed—so training signals are sparse and noisy.
2. Nearly open action space. Command combinations, pipes, permissions, and environment differences form an extremely high-dimensional action space with a severe long-tail distribution, making it difficult for any dataset scale to cover it completely.
3. Strict format consistency. A mistake in parameter order or an escaping character can cause failure. There is less room for semantic tolerance than in natural-language tasks.
4. Mismatch between data distribution and real environments. Scripts in training corpora are often idealized examples, while real terminal environments contain version differences, path quirks, and legacy configuration.
5. Coupling between correctness and safety. Destructive operations require the model to learn to refuse or request confirmation, which creates tension with an evaluation orientation that rewards completing as much as possible.

Corresponding mitigation directions include: using executable sandboxes to build reinforcement-learning environments with real feedback; creating reproducible command-level test sets; introducing structured decoding and syntax constraints for tool calls; and evaluating “dangerous-operation recognition” as an independent metric rather than mixing it into task success rate.

6. What Enterprises Should Do: A Practical Evaluation Framework

Do not treat vendor leaderboards as the basis for selection. Build your own layered evaluation set:

• Layer 1: capability smoke testing. Cover the high-frequency tasks in your actual business. Twenty to fifty examples are enough for rapid elimination.
• Layer 2: task-level evaluation. Design tests by the three dimensions described above, and record five metrics: success rate, number of steps, token cost, tool error rate, and recovery rate.
• Layer 3: stability and safety. Inject disturbances such as tool exceptions, timeouts, insufficient permissions, and malicious prompts, and observe whether the model can fall back and refuse.

The key principle is cost normalization: compare performance under different parameter sizes and inference configurations by unit token cost and end-to-end latency. Otherwise, the “strongest model” is often not the optimal choice under budget constraints. In implementation, general cloud capabilities such as model serving, containerized deployment, logging and tracing, and monitoring/alerting should be included in the evaluation scope, because they determine post-launch observability and failure-localization efficiency. This often matters more than a fraction of a point on a leaderboard in determining production success or failure.

7. Conclusion

The significance of trillion-parameter sparse MoE is that it uses lower inference compute in exchange for greater knowledge capacity. The significance of full-stack training on domestic compute is that it puts the engineering controllability of this path in one’s own hands. The value of agent benchmarks is that they turn “whether a model can complete tasks” from an abstract description into a quantifiable problem.

Scores may become outdated, but several analytical frameworks are more durable than any single leaderboard ranking: decoupling total parameters from activated parameters, understanding the engineering cost of sparse routing, decomposing agent tasks into three dimensions of capability, and explaining code and terminal shortcomings through feedback loops.