LLM Architecture and Systems Engineering for AI Agents and Coding: Sparse MoE, Million-Token Context, and Production Deployment

    This article breaks down the architecture and systems engineering behind a new class of large language models purpose-built for AI agents and coding. It examines a 320B-parameter sparse Mixture-of-Experts model with roughly 18B activated parameters, a million-token multimodal context, and a permissive open-source license, then explains where its efficiency comes from: expert sparsity that decouples knowledge capacity from per-step compute, hybrid sparse-and-linear attention that makes long context affordable, and a visual self-check loop for agent tasks. It also covers distributed parallelism (TP/EP/PP/DP), inference optimization, agent runtime and context management, cost-aware routing, and deployment on domestic accelerator hardware. Readers will find guidance on reading benchmark results, KV-cache sizing formulas with runnable code, and practical checklists for turning leaderboard numbers into controllable, observable, measurable production systems.

    1. From Anonymous Blind Test to Identity Reveal

    For a while, the AI community was tracking a mystery model known only by a codename. On a mainstream model aggregation and routing platform it took anonymous blind-test traffic, processing roughly 44 trillion tokens in a single week and topping the weekly call-volume chart. The team behind it was then officially revealed: a native multimodal large model from a Chinese team, with a core configuration of a 320-billion-parameter MoE architecture, million-level multimodal context, and a permissive open-source license that allows commercial use and redistribution. What drew even more attention was that it had already sustained real, global production-grade traffic on a purely domestic 100,000-accelerator compute cluster.

    For engineering teams, the real value is not the chart-topping rank but three questions: how does sparsification cut both training and inference cost at the same time; can million-level context run stably; and can domestic compute support scaled service. The rest of this article addresses these three points.

    2. Core Capabilities at a Glance

    • High-sparsity MoE: 320 billion total parameters, roughly 18 billion activated, an activation ratio under 6%.
    • Million-level context: native support for text, visual, and audio input, holding up to 1 million tokens in a single inference.
    • Permissive open-source license: commercial use and redistribution allowed, so enterprises can self-host, fine-tune, and embed it in their own products.
    • Domestic compute adaptation: the first to carry real production traffic on a purely domestic 100,000-accelerator cluster.
    • Execution-oriented: close to top closed-source flagship models on coding, agent, and office tasks, with an official claim of roughly one-tenth the per-call cost.
    • Visual self-check feedback: as the core model of an intelligent coding tool, it can autonomously correct frontend and 3D output based on runtime screenshots and rendered results.

    3. Reading the Benchmarks: A Model Built to Get Work Done

    The published benchmark results are as follows:

    Benchmark area Representative metric Result
    Coding & agents Software-engineering repair, terminal-task evaluations 63.4 / 84.3, first tier
    Automated tasks Multi-step execution and tool-calling evaluations Previous gen 26.2 → 48.8
    Real work tasks Real economic-value task evaluation 1773, ranked first
    Hard reasoning (with tools) Hard knowledge-reasoning evaluation 54.7 → 55.3

    Three takeaways emerge from this data:

    1. The improvement is uneven. Automated tasks nearly doubled, coding and terminal tasks reached the first tier, yet hard reasoning improved by only 0.6 points. This suggests the progress comes mostly from “behavioral optimization in the post-training stage” rather than an across-the-board leap in baseline intelligence.
    2. The optimization focus is long-horizon execution. In multi-step tasks the model must continuously plan, call tools, read errors, and self-correct. Standard pretraining rarely covers such trajectories; this capability is usually built through fine-tuning on agent-trajectory data and reinforcement learning with verifiable rewards (such as unit-test pass rate).
    3. The real-task score points to deployment scenarios. A high “real work tasks” score means greater reference value for long-workflow scenarios like enterprise agents and digital employees. But leaderboards cannot replace business evaluation; run A/B validation on your own task set before going live.

    Selection guidance: prioritize this class of model for coding, automation pipelines, long-document processing, and multimodal tasks; for knowledge-QA-heavy scenarios, a distilled small model can reduce latency and cost.

    4. Architecture: Where the Efficiency Comes From

    4.1 Expert Sparsity: Dividing Labor Between Total and Activated Parameters

    A MoE (Mixture-of-Experts) model splits the feed-forward layers into several “experts,” and each token activates only a few of them through a routing network. Its design logic is to decouple “knowledge capacity” from “per-step compute”:

    • Knowledge capacity is carried by total parameters: expert parameters keep accumulating domain knowledge during training, like a set of categorized bookshelves.
    • Compute cost is determined by activated parameters: each token passes through only a few experts, so FLOPs scale with the number of activated parameters.

    With roughly 18 billion activated parameters, the compute per token is on the order of a 10-to-20-billion-parameter dense model, while the parameter capacity far exceeds a dense model of the same scale. In other words, it trades “20-billion-parameter-class” compute for “300-billion-parameter-class” knowledge capacity.

    The costs are equally clear:

    • Memory must hold all expert weights, so deployment cost is calculated on total parameters;
    • Load imbalance is the norm; when popular experts get overloaded, the capacity factor hits its ceiling and tokens get dropped;
    • Expert parallelism depends on high-frequency All-to-All communication and is sensitive to cluster interconnect bandwidth.

    4.2 Hybrid Attention: Making Long Context Affordable

    Attention compute grows quadratically with sequence length, and it is the single largest cost source for million-token context. Let the sequence length be n and the model dimension be d; the cost of three attention types is:

    Attention type Compute complexity Per-step decode cost Main characteristics
    Full attention O(n²d) Proportional to history length Strong global modeling, high cost on long sequences
    Sparse attention (local window, etc.) O(nwd), w = window Constant Good at local dependencies; global coverage relies on cross-layer composition
    Linear attention O(nd²) Constant Compresses history into a fixed-size state; complexity near-linear

    The idea behind mixing sparse and linear attention is to have most layers use the low-cost mechanism and keep only a few layers for global information integration. Two mainstream combinations each have trade-offs:

    • Sparse window + a few global layers: the global layers handle cross-segment retrieval.
    • Stacked linear layers + periodic dense layers: a regular attention layer is inserted every few layers to periodically refresh the global state.

    This yields three benefits:

    • Long-sequence prefill is near-linear;
    • Decode-stage attention cost is essentially constant and does not balloon as context grows;
    • Long-text training and inference cost drops substantially.

    A few engineering notes:

    • Linear attention struggles to precisely backtrack to any arbitrary history fragment, so precise-location tasks are best paired with retrieval augmentation;
    • Different layers use different KV-cache strategies (some layers keep only in-window cache), so memory planning must be done layer by layer;
    • Domestic accelerators need focused adaptation for the recursive state updates, block matrix multiplications, and gating computations involved in linear attention.

    4.3 The Cost of Million-Token Context and How to Solve It

    The bottleneck of long context lies not only in compute but more in memory and bandwidth. The size of a single layer’s KV cache is:

    KV_bytes = 2 × L × k × d × s × n

    where 2 accounts for the two copies K and V, L is the number of layers, k is the number of KV heads, d is the per-head dimension, and s is the bytes per element. Taking a dense model with L=60, k=8, d=128, FP16 as an example, just 128K context needs about 30GB of memory, and 1M context approaches 230GB — before counting model weights and activations.

    1. # Single-layer KV cache (bytes): 2 stands for the two copies, K and V
    2. def kv_per_token(layers, kv_heads, head_dim, dtype_bytes=2):
    3. return 2 * layers * kv_heads * head_dim * dtype_bytes
    4. # L=60, KV heads=8, head_dim=128, FP16
    5. per_tok = kv_per_token(60, 8, 128)
    6. print(per_tok * 128_000 / 1024**3) # 128K context: ~29.3 GB
    7. print(per_tok * 1_000_000 / 1024**3) # 1M context: ~228.9 GB

    A single linear-attention layer’s state size is roughly 2 × d_k × d_v × s, usually on the order of KB. The total cache of a hybrid architecture is “a few global layers growing linearly with length + most linear layers staying fixed,” and this is precisely the prerequisite that makes million-token context feasible.

    Supporting engineering techniques:

    1. Paged KV management: allocate in blocks to reduce fragmentation and support cross-request prefix-cache sharing.
    2. Prefix-cache reuse: stable prefixes such as system prompts, code-repository context, and tool descriptions are computed only once.
    3. Chunked prefill: split long prompts into chunks and interleave them with decoding, so long requests do not block short ones.
    4. KV quantization: compress the KV cache to INT8/FP8 and evaluate the accuracy loss.
    5. Layered context strategy: store raw content in object storage and feed the model only summaries and relevant snippets, avoiding the “context stuffed full means effective” fallacy.

    4.4 Native Multimodality and the Visual Self-Check Loop

    Native multimodality means vision and audio are modeled jointly with text from pretraining onward, rather than being bolted on afterward. On the vision side, a vision encoder typically extracts features first, then a projection layer maps them into the language model’s embedding space, where they become visual tokens that participate in attention. Engineering-wise, watch for token inflation from resolution and frame sampling; high-frequency video input in particular needs token compression.

    “Visual self-check” is a typical case of agent capability extending to the perception end: after the model generates an interface or 3D scene, it automatically renders a screenshot, then uses its own vision to check the result against the requirements and iteratively revises any problems it finds.

    1. def build_with_visual_feedback(spec, max_iter=5):
    2. artifact = generate(spec) # generate a frontend page / 3D scene
    3. for i in range(max_iter):
    4. # render multi-view screenshots; multi-viewport for frontend, multi-camera for 3D
    5. shots = render(artifact, views=["front", "top", "perspective"])
    6. report = inspect(shots, spec) # model visually reviews against the spec
    7. if report.passed:
    8. return artifact, i + 1
    9. artifact = refine(artifact, report.issues) # make targeted fixes from the issue list
    10. raise RuntimeError("Max iterations exceeded; escalate to human review")

    Key design points:

    • Acceptance criteria must be quantifiable (layout alignment, element spacing, proportions, and color scheme);
    • Set a maximum iteration count to avoid infinite loops;
    • Decouple review from generation to reduce the risk of “lenient self-scoring.”

    5. Systems Engineering Guide

    5.1 Distributed Parallelism Strategy

    Dimension Typical deployment Role Main cost
    Tensor parallelism (TP) Intra-node high-speed interconnect Splits a single layer’s matrix multiplications Multiple All-Reduce per layer; depends on inter-card bandwidth
    Expert parallelism (EP) Cross-node Distributes experts across devices High-frequency All-to-All; load imbalance
    Pipeline parallelism (PP) Cross-node Splits by layer to reduce per-card memory Pipeline bubbles; needs 1F1B-style scheduling
    Data parallelism (DP) Training / multi-replica inference Scales throughput Gradient sync or request-routing overhead

    Practical guidance:

    • Inference favors EP + TP: the key to MoE inference is expert load balancing and overlapping All-to-All communication with computation;
    • Training uses 3D parallelism: combine data, tensor, and pipeline parallelism with ZeRO-style memory optimization;
    • Long sequences introduce sequence parallelism: for example, Ring-Attention-style schemes that distribute attention compute and activation memory.

    5.2 Deploying on Domestic Accelerator Clusters

    On domestic accelerators, the framework and kernels are often not “install and run”; low-level adaptation is required:

    • Operator coverage: check training and inference pipelines against an operator inventory, fill missing operators with high-performance implementations, or substitute them via computation-graph rewriting;
    • Numerical consistency: under mixed precision, unify the accumulation order and rounding strategy across vendor libraries, compare outputs layer by layer, and locate accuracy deviations;
    • Communication adaptation: align collective-communication implementations such as All-Reduce and All-to-All, run bandwidth and congestion tests, and enable topology-aware scheduling so communication-heavy parallel groups sit within the same switching domain;
    • Compilation and execution: use graph compilation, operator fusion, and memory pooling on the domestic software stack to cut kernel-launch overhead;
    • Large-scale stability: a 100,000-accelerator cluster needs silent-data-corruption detection, automatic slow-node isolation, checkpoint resume, and elastic fault tolerance;
    • Pre-launch evaluation: run end-to-end throughput, time-to-first-token, and long-context accuracy regression, comparing item by item against mainstream hardware baselines.

    Reference checklist:

    • All key operators have high-performance implementations;
    • The parallelism strategy matches the cluster topology;
    • The inference engine is adapted to the target hardware;
    • Single-card failures can be isolated automatically, with checkpoint resume support.

    5.3 Inference Service Optimization

    1. Continuous batching: dynamically batch requests to raise GPU utilization.
    2. Prefill/decode separation: long-prompt prefill is compute-intensive while decoding is memory-intensive; deploying them separately allows independent scaling and keeps them from interfering.
    3. Speculative decoding: use a draft model or a built-in multi-token prediction head to verify several tokens at once, accelerating decoding.
    4. Layered caching: a prefix-cache hit can substantially cut repeated computation.
    5. Elastic scaling: auto-scale on multiple metrics — queue length, GPU utilization, P99 latency — and pair it with a message queue to smooth out spikes.

    5.4 Agent Runtime and Context Management

    An agent system should consist of three parts: the model, the tool layer, and the runtime. The core of context management is “keep effective information within the window while controlling cost”:

    • Rolling summary: compress earlier conversation once a threshold is exceeded;
    • Tool-result externalization: write large files and logs to external storage, passing only summaries and references;
    • Structured memory: store long-term preferences and project conventions in a vector store or structured storage;
    • Plan externalization: maintain the task plan separately so it is not diluted by summarization.
    1. # Pseudocode: agent main loop + rolling summary
    2. async def agent_loop(task, max_steps=40):
    3. plan = await planner(task) # produce a checkable list of steps
    4. summary = "" # rolling summary, updated as the window grows
    5. for step in range(max_steps):
    6. if budget_exceeded():
    7. return escalate_to_human(plan, summary) # escalate on timeout or over-budget
    8. action = await model.decide(plan, summary, last_results)
    9. result = await executor.run(action) # tool call / terminal exec / file IO
    10. last_results.append(compress(result)) # compress / externalize large results first
    11. if len(last_results) > K:
    12. summary = await model.summarize(summary, last_results[: len(last_results) - K])
    13. last_results = last_results[-K:]
    14. plan = update_plan(plan, result)
    15. if plan.done():
    16. return verify(plan) # run tests / visual acceptance
    17. return escalate_to_human(plan, summary)

    Engineering points:

    • Constrain tool descriptions with a JSON Schema, and add parameter validation and idempotent design;
    • High-risk operations need human confirmation: irreversible actions such as deletion, deployment, and transfers follow “two-step confirmation + operation audit”;
    • Run terminal commands in a sandbox, with resource and time limits;
    • Design for failure: on failure, feed the full error back to the model and set a maximum retry count;
    • Verification over self-assessment: judge completeness by objective signals such as tests, compilation, and visual acceptance.

    5.5 Cost and Routing Strategy

    This class of model is positioned to “deliver near-flagship performance at a lower per-call cost,” but that does not mean every request should go to it. A tiered routing strategy is recommended:

    • Lightweight requests (classification, summarization, formatting) go to a small model;
    • Hard requests (complex coding, long-horizon automation) go to the large model;
    • Offline batch jobs use a low-priority resource pool;
    • Real-time interactions use a dedicated high-priority resource pool.
    1. import hashlib
    2. def route(prompt: str, task_type: str) -> str:
    3. # high-risk / complex tasks go to the large model, lightweight tasks to the small model
    4. if task_type in {"code", "agent", "multimodal"}:
    5. return "large-model"
    6. if len(prompt) < 2000 and task_type in {"summary", "classify"}:
    7. return "small-model"
    8. # remaining traffic is split by ratio; the bucket is stable, which helps A/B experiments
    9. bucket = int(hashlib.md5(prompt.encode()).hexdigest(), 16) % 100
    10. return "large-model" if bucket < 30 else "small-model"

    Cost formula:

    Per-request cost = input tokens × input price + output tokens × output price + cache hit rate × cache discount + retry loss

    For self-deployment, GPU utilization must also be factored in: when utilization is too low, economies of scale are canceled out.

    5.6 Open-Source Self-Deployment and Compliance

    A permissive open-source license is an important advantage of this class of model: it can be deployed privately, keeps data within your own domain, and supports domain fine-tuning. Points to note in practice:

    • Although the license permits commercial use, you must still comply with terms such as attribution and including the license;
    • There is a responsibility boundary between model output and the open weights; services aimed at external users should provide clear content labeling and disclaimers;
    • When building agent services on a public cloud, you can combine general-purpose capabilities such as container services, object storage, message queues, and logging/monitoring/alerting to complete deployment and operations, while following the platform’s content-safety and data-security standards.

    6. Common Pitfalls and a Go-Live Checklist

    Common pitfalls

    1. Treating “320 billion total parameters” as the basis for inference cost: compute is actually determined by activated parameters, while memory is determined by total parameters.
    2. Treating “supports million-token context” as “million-token context is usable”: the effective usable length is constrained by task type, position effects, and cost.
    3. Choosing a model by leaderboard alone: coding and execution metrics cannot be extrapolated directly to a business scenario.
    4. Understanding domestic-compute adaptation as a driver swap: the real barriers are operator coverage, numerical consistency, and communication performance.
    5. Treating visual self-check as “unlimited autonomy”: you must set acceptance criteria, an iteration cap, and a human fallback.

    Go-live checklist

    • Task and model capabilities match, validated against a business evaluation set;
    • Context budget, prefix cache, and summary strategy are configured;
    • The parallelism strategy matches the hardware topology and the inference engine is adapted;
    • Continuous batching, prefill/decode separation, and elastic scaling are in place;
    • Tool calling has sandboxing, auditing, and human-confirmation mechanisms;
    • Cost and routing strategy are live, with dashboards and alerts established;
    • License obligations and content-safety responsibilities are clearly defined.

    7. Conclusion

    From anonymous blind test to official reveal, the signal from this model is clear: competition among large models is shifting from “smarter” to “better at getting work done, cheaper to compute, and easier to deploy.” Architecturally, expert sparsity decouples parameter capacity from compute, hybrid attention makes long context economical, and visual self-check gives agents a closed-loop error-correction capability. On the engineering side, the successful adaptation to domestic compute shows that hardware diversity is gradually becoming an option rather than a constraint. For developers and enterprises, the key is to turn leaderboard capabilities into controllable, observable, measurable production systems, and to measure value by business metrics rather than a single score.