Open-Source 2.8T-Parameter MoE Model: Architecture and Long-Horizon Agent Capabilities

1. Background: What Long-Horizon Agents Demand from a Model

When an agent moves beyond single-turn Q&A into multi-step task execution, the challenge shifts qualitatively. A single terminal operation may involve dozens of commands. A code fix requires iterative location, modification, and verification. A CAD generation task demands balancing geometric constraints against design intent. These tasks share three traits: long trajectories, many intermediate states, and error accumulation.

Four core requirements emerge:

1. Context capacity — the model must hold full task history, tool return values, and intermediate artifacts.
2. Adjustable reasoning depth — simple steps should not waste compute, while complex steps need a generous thinking budget.
3. Stable tool calling — parameter formats and call intent must stay consistent across many turns.
4. Long-horizon coherence — after hundreds of steps, the model must still remember the initial goal and key constraints.

The technical approach discussed here targets all four: a MoE architecture with 2.8T total parameters and 104B activated parameters, a 1M-token context window, native image-text input, sparse attention, and joint reinforcement learning post-training. Target scenarios include long-horizon code agents, software engineering, 3D design, web retrieval, financial research, and office document processing.

2. Architecture: Sparse Activation and Sparse Attention

2.1 MoE: Trading Total Parameters for Capacity, Activated Parameters for Cost

The model uses a Mixture-of-Experts (MoE) architecture with 2.8T total parameters and roughly 104B activated parameters per forward pass. This decouples “knowledge capacity” from “per-token compute”:

• Total parameters determine how much knowledge and pattern diversity the model can store.
• Activated parameters determine the compute cost per token.

The activation ratio is approximately 3.7%, meaning only a small fraction of expert networks participate in any single inference step. For developers, this has direct implications for GPU memory planning and throughput: weight storage scales with total parameters, while compute cost scales closer to activated parameters. When planning deployment, evaluate “weight-loading cost” and “per-token compute cost” separately rather than relying on a single parameter figure.

2.2 From Gated MLA to Gated DSA

Compared with the previous version, the new release replaces 24 layers of Gated MLA with Gated DSA (Dynamic Sparse Attention), combined with 69 layers of Kimi Delta Attention. DSA dynamically selects 2,048 historical tokens for each attention computation.

The core idea is attention sparsification. With a 1M-token context window, full attention over all history would make compute and memory grow rapidly with sequence length. By selecting a fixed number of historical tokens per query, attention cost becomes tied to the selection count rather than to total length, making ultra-long contexts engineering-feasible.

Importantly, sparse attention is not a simple sliding-window truncation. A fixed window discards distant but critical information; dynamic selection preserves the ability to “pull relevant history on demand.” The 2,048 figure is a tunable parameter—actual effectiveness depends on the selection strategy and task profile. For long-horizon tasks that must revisit early constraints, selection quality matters more than selection quantity.

# Context organization for long-horizon tasks (pseudocode)
def build_context(task_history, current_step):
    # Preserve global goals and key constraints so they survive sparse selection
    anchors = extract_anchors(task_history, keys=["goal", "constraint", "interface"])
    # Keep recent trajectory intact for local coherence
    recent = task_history[-N:]
    return anchors + recent + current_step

This illustrates a practical point: even with sparse selection at the model level, the application layer should explicitly retain “global anchors” (goals, constraints, interface contracts) to reduce long-horizon drift.

2.3 Native Image-Text Input

The model supports native image-text input, which is especially valuable for 3D design and CAD tasks. Sketches, screenshots, and annotated diagrams often convey more precise information than text descriptions. Native multimodal input avoids the information loss of “convert-to-text-first” pipelines and lets the agent review render results or UI states during execution, closing the loop.

3. Training Strategy: Joint Reinforcement Learning and Quantization

3.1 Joint Reinforcement Learning Across Hundreds of Downstream Domains

The model undergoes large-scale joint reinforcement learning post-training spanning hundreds of downstream domains. This approach aligns “general capability” with “task patterns” in a single optimization pass. Traditional pipelines—general pretraining followed by single-scenario fine-tuning—risk capability fragmentation. Joint training lets the model learn transferable strategies across the task distribution, such as stable tool-call formats, retry-after-failure policies, and multi-step decomposition habits.

The takeaway for developers: agent capability depends heavily on the task distribution designed during post-training. To improve performance on a specific business use case, construct training and evaluation data that mirrors real trajectories—including success paths, failure recovery, and edge cases—rather than relying on prompt engineering alone.

3.2 MXFP4 Weight Quantization

Training uses MXFP4 weight quantization. Low-bit quantization directly reduces weight storage and bandwidth pressure, a critical step for deploying large-parameter models. Engineers should note that quantization introduces numerical precision loss. Evaluation must track both “capability metrics” and “stability metrics” (e.g., format error rate and tool-call failure rate in long-horizon tasks), not just single-turn accuracy.

4. Capability Map and Task Fit

4.1 Code Development and 3D Creation

For long-horizon code agents, software engineering, and 3D design, official evaluations cover terminal operations, code repair, and CAD generation with tool-assisted multi-step execution. These scenarios share an “execute–observe–correct” loop:

• Terminal operations: command output feeds the next turn; the model must parse errors accurately and adjust strategy.
• Code repair: requires localization, minimal modification, and regression testing; long-horizon coherence is paramount.
• CAD generation: strong geometric constraints; success hinges on the interplay of image-text input and tool calls.

4.2 Agents and Knowledge Work

The model supports web retrieval, financial research, document processing, and tool calling. The challenge here is not single-step reasoning but information integration and citation consistency: retrieved content must align with existing conclusions, and document processing must preserve structural integrity. A recommended application-layer pattern is “intermediate artifact persistence + periodic summarization”—compressing long trajectories into structured state to ease context pressure.

5. Thinking Budget: low / high / max Tiers

Inference offers three thinking-budget tiers—low, high, and max—configurable per task. This is a practical engineering interface. A suggested tiered approach:

Budget Tier	Typical Scenarios	Configuration Advice
low	Format conversion, simple retrieval, status queries	Prioritize batch concurrency; keep latency low
high	Code repair, document processing, multi-step tool calls	Default tier; balance quality and cost
max	CAD generation, complex planning, failure recovery	Limit concurrency; pair with timeout and retry logic

# Select thinking budget by task type (pseudocode)
BUDGET = {"query": "low", "patch": "high", "cad": "max"}
def run(task):
    return agent.invoke(task, thinking_budget=BUDGET[task.kind],
                        max_steps=task.max_steps, timeout=task.timeout)

Avoid “max for everything” in practice: higher budgets increase per-step latency and compute, and total cost scales quickly over long trajectories. A more sound strategy is dynamic tiering—high budget during planning, lower during execution, temporary elevation after failures.

6. Engineering Recommendations

1. Layered context: Use a three-tier structure—global anchors + recent trajectory + tool-result summaries—rather than a single monolithic context.
2. Tool contracts first: Define explicit input/output schemas and error codes for every tool to minimize format drift across long call chains.
3. Observability: Log per-step thinking budget, tool calls, latency, and failure causes to pinpoint long-horizon degradation points.
4. Evaluation aligned with real trajectories: Beyond single-turn metrics, add multi-step success rate, average step count, and recovery success rate.
5. Re-evaluate after quantization: Re-run long-horizon benchmarks post-deployment with low-bit weights; focus on stability, not just averages.

7. Summary

The core logic of this open-source approach can be distilled into four pillars: MoE for expanded capacity, sparse attention for long contexts, joint reinforcement learning for task-shape alignment, and adjustable thinking budgets for quality–cost balance. For developers, the model’s raw capability is only the starting point. What truly determines long-horizon agent effectiveness is how you design context management, tool contracts, and evaluation systems. Understanding these structural choices matters more than memorizing parameter counts.