Million-Token Context and Trillion-Parameter MoE Models: Engineering Practices
This article breaks down the engineering behind a new class of flagship large language models built on a roughly 3-trillion-parameter mixture-of-experts (MoE) architecture, with a 1-million-token context window and native visual understanding. Rather than treating the model's headline numbers as the story, it explains why the gains come mainly from post-training and what that means for teams working on a fixed base model. It walks through sparse activation and its cost implications, effective long-context utilization, native multimodality, and verifiable-reward reinforcement learning, then turns to practice: interpreting benchmark scores cautiously, engineering agents with sandboxing, triple budgets, state machines, and automated verifiers, and accounting for real costs. It closes with a phased adoption plan—from selection and context engineering to internal evaluation loops and open-weights privatization—so readers can turn single-task benchmark results into measurable, production-ready service levels.
Introduction: The engineering question behind the latest model upgrade
A Chinese AI vendor recently launched a new-generation flagship large language model along with a higher-throughput “high-speed” variant. The vendor positions it as its most capable model to date—a “3-trillion-parameter-class” model. It supports a 1-million-token context, offers native visual understanding, and targets code, 3D content creation, financial research, and long-horizon agents as the focus of this upgrade.
For engineering teams, three signals in the announcement are worth unpacking:
- Where the capability change comes from: the upgrade comes from post-training investment, not from scaling up parameters.
- Under what conditions the capability holds: benchmark scores are measured around a single task, in resource-constrained terminal environments.
- How the capability translates into business: you need budget control, sandbox isolation, and automated verification to turn a “single-task score” into a “production-measurable service level.”
This article follows these three layers: first, the technical foundations behind this class of model; then, what the size of the upgrade means and why it does not equal ready-to-ship; and finally, phased recommendations for adoption.
1. Technical foundations: understanding trillion-parameter models correctly
1.1 Official claims, actual parameters, and post-training as the driver
This release is a “same base, retrained” iteration. The previous generation had roughly 2.8 trillion total parameters (officially described as “3-trillion-parameter-class”); the new generation keeps the same total parameter count, activates about 104 billion parameters per inference pass, supports a 1-million-token context window, and accepts text and image inputs. This structure can be summarized as follows:
| Dimension | Details |
|---|---|
| Parameter count | MoE architecture, ~2.8 trillion total parameters, ~104 billion activated per inference pass (officially “3-trillion-parameter-class”) |
| Context window | 1 million tokens |
| Input modality | Text + image, native visual understanding |
| Variants | Standard / high-speed (vendor claims comparable intelligence, ~2-3x output speed, ~2x price) |
| Access channels | Vendor website, mobile app, office suite, AI coding tools, and open APIs |
| Weights plan | Vendor plans to open-source the full weights within roughly two weeks; license not yet published |
The vendor emphasizes that the gains come mainly from the post-training stage: post-training compute increased significantly, and reinforcement learning was expanded to hundreds of downstream domains. This shows that pretraining sets the upper bound of a model’s capability, while post-training determines how much of that bound can be reliably invoked. An important engineering implication follows: with a fixed base model, you can still gain a lot through refined prompt engineering, tool design, context management, and retry strategies. Rather than continuing to pile on prompts while the model has yet to be replaced, it is better to first systematically diagnose why your agent is failing.
1.2 Sparse activation: total parameters determine capacity, activated parameters determine cost
The core idea of the mixture-of-experts (MoE) architecture is “sparse activation.” It contains a large number of expert subnetworks, and a router selects a few experts to participate in computation for each token.
- Knowledge capacity: the total parameter count roughly determines the knowledge capacity the model can hold.
- Per-token compute: the compute per token depends mainly on the activated parameter count, roughly estimated as “2× activated parameters” floating-point operations (including attention overhead)—in this case, about 200 billion FLOPs per token.
- Expert load: the memory footprint, inter-expert communication overhead, and load balancing introduced by many experts mainly affect the deployment side, not the billing of a single call.
There is a common misreading here: 2.8 trillion total parameters does not mean every call consumes compute for 2.8 trillion parameters; actual consumption is closer to the activated parameters. From this it follows that judging whether a model upgrade is worthwhile cannot rely on total parameter count alone—it should look at three metrics:
- Effective metrics: task pass rate, field-extraction accuracy, and human-review rate on your own business dataset.
- Economic metrics: average token consumption and average cost per task.
- Experience metrics: first-token latency, output speed, and stability under long context.
Many teams only compare the first item, and eventually find that cost went up while perceived quality did not change much—the reason lies precisely here.
1.3 The million-token context: window size ≠ effective utilization
1 million tokens is roughly equivalent to several million Chinese characters, enough to fit all of the following in a single pass:
- a medium-sized code repository and its commit history;
- a full set of product documentation, contract files, or annual reports;
- a long-horizon task trace spanning several hours.
But there is still a gap between “supports” and “uses well,” and developers need to pay attention to four levels:
- Memory footprint: the intermediate state of attention grows quadratically with sequence length. Mainstream inference frameworks generally use optimizations such as paged KV cache, chunked prefill, and fused attention kernels to mitigate this, but memory pressure does not disappear.
- Effective utilization: whether long text is used well depends on capabilities such as “needle-in-a-haystack” single-point retrieval, multi-hop reasoning across paragraphs, and structured extraction from long documents. These content-dense repository-level code tasks and financial research report tasks are exactly where long-context capability is tested.
- Cost amplification: with pay-as-you-go billing, input tokens are usually billed by length and are often priced higher than output tokens. The cost increase from longer input grows roughly linearly, and the longer the returned content, the higher the cost.
- Performance degradation: the longer the sequence, the higher the first-token latency and the lower the per-request throughput.
Therefore, the correct way to use long context is not “stuff everything in without thinking,” but “first narrow the scope with retrieval, then use the long window to carry cross-file, cross-section associative reasoning.”
1.4 Native visual understanding: from image-text splicing to unified representation
Native visual understanding means the vision encoder and the language model are trained jointly, images are encoded as tokens, and processed in the same sequence as text tokens—rather than first bolting on an OCR module and then feeding it to the language model.
This has three direct impacts on engineering practice:
- Visual information such as charts, UI screenshots, architecture diagrams, and handwritten annotations can directly participate in reasoning;
- Tasks like analyzing table screenshots and verifying invoices and receipts can skip the “OCR plus regex” pipeline;
- Image resolution and slicing affect the number of tokens consumed, and token count directly determines cost.
Usage recommendation: for scenarios rich in unstructured visual information, try native multimodality first; for stable-format receipts and tables, structured parsing is often cheaper and more controllable, so you can route by task type.
1.5 Post-training: verifiable-reward-driven capability expansion
The vendor attributes the gains to increased post-training compute and to the expansion of reinforcement learning to hundreds of domains, which is clearly interpretable from an engineering standpoint:
- Tasks such as code, terminal operations, and structured data extraction are outcome-verifiable. Whether unit tests pass, whether a command’s exit code is 0, and whether JSON passes schema validation can all serve directly as reward signals.
- The more objective the feedback, the more easily reinforcement learning converges, and the more stable the model’s “analyze first, then act, then correct based on feedback” behavior becomes in real environments.
The same applies on the application side: building automated verifiers (unit tests, schema validation, rule checks) for business tasks both improves agent success rates and becomes a foundational asset for later data feedback, fine-tuning, and even training your own small models.
2. Capability observation: the size of the upgrade in code and long-horizon agents
2.1 Scores and cost distribution on terminal benchmarks
Code and long-horizon agents are the most obvious upgrade directions this round. On a coding-agent benchmark aimed at real, resource-constrained terminal environments, the new flagship model scores about 65.7%:
- higher than a mainstream closed-source flagship model at 58.2%;
- within 0.7 percentage points of the top-scoring closed-source model (66.4%).
This benchmark tests a coding agent’s ability to complete complex tasks in real, resource-constrained terminal environments—for example, diagnosing build failures, locating bugs, and writing scripts to process data—with each task equipped with a dedicated test. The vendor’s in-house long-horizon code benchmark scores about 61.2%, up from 55.95% in the previous generation.
Three points can be read from the two figures (score comparison and cost-score distribution):
- Capability has reached the first tier. On real terminal tasks, the new model is only a few tenths of a percentage point behind the top-scoring closed-source model.
- Cost-performance has become a new competitive dimension. The cost distribution shows that different models consume very different amounts of tokens and incur very different call costs at the same score. A terminal agent often needs dozens of tool calls per task, and the cost gap is amplified many times over.
- The same capability exists as a speed-for-price variant. The high-speed version offers about 2-3x output speed at about 2x the price; whether it is worth it depends on how sensitive the business is to latency and throughput.
2.2 How to objectively view self-test scores
When reading scores like these, keep three points of caution in mind:
- A benchmark only describes the average. A 65.7% pass rate means about one-third of tasks fail, so a production system must design fallback paths for failures, such as human takeover, graceful degradation, or an explicit failure return.
- Self-test results need independent verification. Vendor self-tests are easily influenced by the choice of evaluation set, prompt templates, and scaffolding implementation, so they should be re-measured in your own scenarios.
- Benchmark scores cannot replace scenario evaluation. Enterprise-internal code style, private frameworks, and domain terminology differ from public benchmark distributions, and the gap may be far larger than the gap between models.
2.3 Failure modes determine architecture: why long-horizon tasks fall at the finish line
The challenge of long-horizon tasks lies not in single-step capability but in error accumulation:
- If step 1 makes a wrong judgment, all subsequent steps unfold on a false premise;
- The longer the context, the more easily early constraints are “forgotten”;
- When a tool returns a large amount of logs, key information gets buried;
- When the task space is too large, the agent falls into repeated attempts.
The corresponding engineering countermeasure is: break long-horizon tasks into short-range subtasks with checkpoints, save progress with external state, and judge convergence with executable verification conditions.
3. From leaderboard to production: the developer’s adoption path
3.1 Selection: profile the scenario first, then match it
Rather than “one model to rule them all,” a more robust approach is to route by task characteristics. Here is a selection matrix for reference:
| Task characteristics | Recommended form | Key considerations |
|---|---|---|
| High-volume, non-real-time (log analysis, document ingestion, report generation) | Standard version + batch tasks | Unit cost first; can combine batch APIs and caching |
| Human interaction (AI coding assistant, customer service) | High-speed version | First-token latency and output speed; judge whether the premium is worth it |
| Cross-file refactoring, complex bug localization, long-document analysis | Long context + long-horizon agent framework | Context management, verification loop, budget caps |
| Screenshots, charts, receipts and forms understanding | Native visual understanding | Image token cost, structured parsing routing |
| Sensitive data that must stay in-domain, needs continuous fine-tuning | Open-weights private deployment | License terms, compute scale, inference framework tuning |
3.2 Context engineering: trading structure for accuracy
Long context is not “paste all the materials in,” but structured, layered organization:
- System layer: role, output format, safety constraints, and prohibitions—keep it concise and stable;
- Task layer: goals, acceptance criteria, constraints, and deadlines;
- Knowledge layer: retrieved code snippets, document paragraphs, and prior conclusions, sorted by relevance and annotated with sources;
- Trace layer: tool calls and their returns, keeping only what is relevant to the current decision.
Parameters to control include:
- reserve a token budget for output;
- truncate and summarize long tool-return logs;
- replace raw history with compressed summaries;
- enable caching for fixed prefixes to reduce repeated billing and latency.
3.3 Agent engineering: sandbox, budget, state machine, and automated verification
Coding and terminal agents operate directly on user code and system environments, so from day one they must be designed as “external programs that may execute arbitrary commands.”
Sandbox and least privilege:
- Run in an isolated container, restrict network egress, and mount non-essential directories read-only;
- Set quotas for CPU, memory, disk, and execution duration;
- Force human confirmation for dangerous operations (deletion, privilege escalation, data exfiltration);
- Validate the arguments of model-generated commands to avoid command injection.
Triple budget: Set separate budgets for tokens, steps, and execution duration; when any is exhausted, terminate and return a structured failure reason.
State machine and failure recovery: Maintain task states (to-do, in progress, pending verification, completed, abandoned); on failure, preserve a snapshot of the scene and support retry from a checkpoint.
Automated verifiers: Prefer executable signals such as unit tests, schema validation, and static checks, rather than letting the model “judge for itself whether it did it right.”
The following illustrative code shows a minimal viable skeleton:
# Illustrative code: minimal control skeleton for a terminal coding agent# Note: model, sandbox, and verify are self-implemented abstract interfaces# that can be swapped for the model service and runtime environment you usefrom dataclasses import dataclass, field@dataclassclass Budget:max_steps: int = 40 # Step budget: prevents falling into an invalid loopmax_input_tokens: int = 600_000 # Input budget: leaves headroom within the 1-million-token windowmax_seconds: int = 1800 # Duration budget: caps the wall-clock time of a single task@dataclassclass Usage:steps: int = 0input_tokens: int = 0output_tokens: int = 0# Each tool should declare: whether it is read-only, whether it is dangerous, and its max execution timeTOOL_SPECS = {"list_dir": {"read_only": True, "danger": False, "timeout": 10},"read_file": {"read_only": True, "danger": False, "timeout": 10},"write_file": {"read_only": False, "danger": False, "timeout": 10},"run_command": {"read_only": False, "danger": True, "timeout": 300},}def run_task(task, model, sandbox, budget: Budget):usage = Usage()state = {"done": False, "need_human": False, "reason": "max_steps"}while not state["done"] and not state["need_human"]:if usage.steps >= budget.max_steps:state["reason"] = "max_steps"break# 1) Assemble layered context: system prompt + task goal + relevant code/docs + compressed tracemessages = build_messages(task, usage)reply = model.generate(messages, tools=TOOL_SPECS)usage.steps += 1usage.input_tokens += reply.input_tokensusage.output_tokens += reply.output_tokens# 2) Execute tool calls one by one, doing permission and safety checks firstfor call in reply.tool_calls:spec = TOOL_SPECS[call.name]if spec["danger"] and not is_safe(call.arguments):state.update(need_human=True, reason=f"unsafe:{call.name}")return stateresult = sandbox.execute(call, timeout=spec["timeout"], read_only=spec["read_only"])# 3) Write only the compressed key info back to the trace, so logs do not drown the contextappend_trace(call, summarize(result))# 4) Judge convergence with executable signals: tests pass / validation passes / explicit failureif verify(task):state.update(done=True, reason="verified")elif usage.input_tokens >= budget.max_input_tokens:state["reason"] = "token_budget"# 5) Return a structured result for monitoring, cost accounting, and human takeoverreturn {"status": state, "usage": usage}
The key of this skeleton is not the code itself, but four design principles:
- The model is only responsible for “deciding what to do next”; permissions, budgets, and verification are controlled by the outer program;
- Each tool has clear read-only, dangerous, and timeout attributes;
- Convergence judgment relies on an external verifier, not on model self-assessment;
- All calls produce structured logs, providing data for later cost attribution and quality analysis.
3.4 Cost accounting: breaking the bill into optimizable items
Taking a terminal agent as an example, the single-task cost is mainly composed of four parts:
- Input tokens: the system prompt, code context, and tool returns accumulate repeatedly;
- Output tokens: the reasoning process and code generation;
- Number of calls: retries, multi-candidate sampling, and multi-agent collaboration all amplify the call volume;
- Environment overhead: sandbox containers, image pulls, and test execution time.
When calling the model per invocation, the single-call cost is approximately:
single-call cost ≈ input tokens × input unit price + output tokens × output unit pricesingle-task cost ≈ single-call cost × average number of calls
The two formulas correspond to two optimization directions: the former relies on context compression and caching, and the latter on reducing invalid calls and failed retries. Note that vendor-side and third-party statistics differ in scope—for example, whether reasoning tokens are included, and how batch-call discounts are calculated. When comparing costs, be sure to unify the scope and rely on self-test data.
3.5 Evaluation loop: rebuild a “mini benchmark” with business data
It is recommended that teams build a repeatable internal evaluation set that includes at least:
- Task set: covering high-frequency real business scenarios, each task equipped with an automated verifier;
- Metrics: pass rate, first-token latency, single-task cost, human-takeover rate, and number of security incidents;
- Regression mechanism: automatically re-run after every change of model, prompt template, or tool set, forming trend curves;
- Failure sample library: categorized by failure mode, such as context loss, tool misuse, and failed verification, as input for the next round of optimization.
3.6 Privatization and compliance: notes on deploying open weights
The vendor plans to open-source the full weights within about two weeks, and the license has not yet been published. For teams, this means an additional route:
- Data stays in-domain: sensitive code and business data can be processed within your own environment;
- Sustainable fine-tuning: you can do domain continued training based on your own data;
- Controllable cost: under stable high-concurrency scenarios, the long-term unit cost of self-hosted inference may be lower.
But you must clearly recognize the cost structure:
- High memory threshold: even low-precision weights require overall memory on the order of terabytes to hold all expert parameters, usually relying on multi-machine, multi-GPU, and expert parallelism;
- High engineering threshold: you need to solve problems such as expert load balancing, cross-node communication, and KV cache management;
- High optimization threshold: throughput, latency, and quantization strategies require dedicated tuning to approach the effect of the official service.
Compliance also needs to be assessed in advance:
- License review: after the license is published, focus on confirming commercial-use rights, usage-scale limits, and distribution and attribution requirements;
- Input/output compliance: desensitize data involving personal information, and establish input filtering and output review mechanisms;
- Content labeling: apply necessary labeling to AI-generated content;
- Filing requirements: when offering generative AI services to the public, complete the corresponding regulatory filing procedures required by law;
- Audit trails: fully record the call chain, tool execution results, and human-intervention records.
4. Phased action recommendations
| Phase | Time reference | Core actions | Deliverables |
|---|---|---|---|
| Preparation | Before weights are open-sourced | Profile scenarios, build the evaluation set and automated verifiers, and set up the sandbox and budget framework | Scenario list, evaluation baseline report |
| Onboarding | 1-2 weeks after weights are open-sourced | Re-test the standard and high-speed versions on your own data, complete license and security review, and connect observability and cost dashboards | Selection conclusion, cost and latency comparison |
| Optimization | Ongoing | Context engineering, tool-set streamlining, failure-sample feedback, and, if necessary, private deployment and fine-tuning | Pass-rate improvement curve, single-task cost reduction curve |
Conclusion
The most noteworthy thing about this model upgrade is not the “3-trillion-parameter-class” number, but three structural changes:
- Sparse activation allows the capacity of trillion parameters and controllable inference cost to coexist;
- The million-token context and native vision make repository-level code understanding and mixed image-text input possible;
- Post-training-driven reinforcement learning expansion pushes code and long-horizon agents from “usable in a demo” to “usable on a benchmark.”
But benchmark scores are only the starting point. What really determines business returns is the engineering system of sandbox isolation, triple budgets, state-machine management, automated verifiers, and observable cost dashboards. It is recommended that development teams build the evaluation set, verifiers, and failure-sample library as long-term assets, and, once the license and weights are in place, make the final selection based on self-test data rather than leaderboard numbers.
