LLM Engineering for Production: Sparse MoE, Long Context, and Domestic AI Accelerators

    This article lays out the engineering path for moving a new-generation general-purpose LLM into production. It explains how sparse Mixture-of-Experts (MoE) architecture decouples memory footprint from compute cost, why long-context (around 200K-token) and token-efficiency techniques matter in practice, and how agentic coding turns a model into an agent that plans, calls tools, and delivers working code. It then covers building a three-tier evaluation system, mixed FP8+INT4 quantization, adapting models to domestic AI accelerators, and serving the model through unified gateways with cost and observability governance. Readers get a four-layer architecture reference and a phased implementation plan organized around two core metrics: task success rate and cost per task.

    Introduction: What the New Generation of General-Purpose LLMs Is Good At — and Where It Struggles

    The new generation of general-purpose LLMs is shifting its capability focus from “answering questions” to “completing tasks.” Parameter counts keep growing, yet the actual compute per inference stays controlled because of sparse activation. Input length has moved into the 100K-token range, and usage has shifted from single-turn Q&A toward coding agents that autonomously plan, call tools, and deliver results step by step.

    This shift creates two kinds of challenges. On the capability side, a model must reach a usable engineering level across code generation, mathematical and logical reasoning, long-document understanding, and information retrieval — all at once. On the engineering side, it must control token cost and run stably across different types of compute. This article follows an “architecture → capability → evaluation → deployment → governance” thread and offers design ideas you can actually put to work.

    1. Technical Foundation: How Sparse MoE Balances Capacity and Compute

    New-generation general-purpose LLMs commonly adopt a Mixture of Experts (MoE) architecture. It consists of a router and multiple parallel “expert” subnetworks, and each token is routed to only a few experts for computation. According to public sources, flagship general-purpose models have reached hundreds of billions of total parameters — typically around 355B total with about 32B activated. In other words, each token activates less than one-tenth of the parameters.

    This design decouples memory footprint from compute cost:

    • Memory cost is determined by the total parameter count. All expert weights must reside in memory (or in tiered storage), so deployment needs more devices or larger memory.
    • Compute cost is determined by the activated parameter count. Sparse activation sharply reduces the compute for a single forward pass, unlocking higher throughput potential.

    The following code illustrates the routing mechanism only:

    1. import torch
    2. import torch.nn.functional as F
    3. class TopKRouter(torch.nn.Module):
    4. """MoE router sketch: each token activates only the top-K experts."""
    5. def __init__(self, hidden_size: int, num_experts: int, top_k: int = 2):
    6. super().__init__()
    7. self.gate = torch.nn.Linear(hidden_size, num_experts, bias=False)
    8. self.top_k = top_k
    9. def forward(self, x: torch.Tensor):
    10. # x: [num_tokens, hidden_size]
    11. probs = F.softmax(self.gate(x), dim=-1)
    12. weights, indices = torch.topk(probs, self.top_k, dim=-1)
    13. weights = weights / weights.sum(dim=-1, keepdim=True) # normalize
    14. return weights, indices # used for per-expert computation

    Engineering essentials:

    1. Routing load balancing. Uneven expert load creates “hot experts” that drag down overall throughput. Training usually adds an auxiliary load-balancing loss, while production requires continuous monitoring of each expert’s activation distribution.
    2. Communication cost. Experts are typically spread across devices, so distributing tokens and aggregating results needs efficient collective communication. When expert parallelism is stacked on top of tensor and pipeline parallelism, communication efficiency often determines the real throughput ceiling of MoE inference.
    3. Tiered storage. In compute-constrained scenarios, you can adopt strategies like tiered weight loading and on-demand swapping of cold experts, trading throughput against resource usage.

    MoE lowers compute cost but does not eliminate memory pressure. Memory optimization is exactly the prerequisite for putting long context into production.

    2. Long Context: From “It Fits” to “You Can Afford It”

    The context window of new-generation general-purpose models has expanded to around 200K tokens, enough to cover an entire technical manual or the core files of a mid-sized codebase in one pass. But a bigger window does not automatically mean affordable or effective.

    Engineering challenges of long context:

    • Attention cost grows with length. Standard attention’s compute and memory cost scale quadratically with sequence length.
    • Middle-of-context information decays. When key information sits in the middle of a long context, the model tends to overlook it — the well-known “lost in the middle” problem.
    • Repeated computation of the cache. In multi-turn conversations and agent loops, prefixes are highly repetitive, and recomputing them every time is expensive.

    Main optimization techniques:

    • Efficient attention implementations: chunked computation, fused operators, and similar techniques that reduce the compute and memory-access cost of long sequences;
    • Paged KV-cache management: managing the attention cache the way an operating system manages memory pages, reducing fragmentation and redundancy;
    • Context compression: summarizing or trimming tool-call records and retrieved snippets before they enter the model;
    • Front-loading key information: placing system constraints and output-format requirements prominently in the prompt to lower the chance they get ignored.

    Token Efficiency can be understood as “the number of tokens consumed to complete the same task.” The vendor claims a token-efficiency improvement of more than 30%, meaning comparable results with fewer tokens — which directly lowers the cost of long-running tasks. In engineering practice, you can pursue token efficiency through:

    • Structured output constraints (such as limiting output to a JSON Schema) to cut down on ineffective explanatory text;
    • Deduplication and refinement of retrieval results;
    • Reasoning-budget control to avoid over-thinking on simple tasks.

    3. Agentic Coding: From Code Completion to Autonomous Delivery

    Traditional code completion predicts the next snippet at the cursor. Agentic coding instead asks the model to take a fuzzy requirement like “implement this feature” and break it into a complete flow: reading the repository, locating files, modifying code, running tests, and iterating based on errors. Capability gains show up along this chain:

    • The vendor claims a code-generation improvement of about 27% over the previous generation, on par with leading international closed-source models;
    • On the real-world software-engineering benchmark LiveCodeBench v6, it trades wins and losses with frontier international models;
    • On a coding benchmark for real repository fix tasks, a new-generation general-purpose model has surpassed representative open-source models of the same generation;
    • In a hands-on test across 74 real-world coding scenarios, the new-generation general-purpose model led overall.

    Agentic coding capability roughly breaks into four layers:

    1. Single-file code generation and explanation;
    2. Cross-file dependency understanding and refactoring;
    3. Self-repair based on errors and logs;
    4. End-to-end development tasks in a sandbox that call tools such as terminals, tests, and search.

    The following agent main-loop sketch helps clarify how the task loop closes:

    1. import json
    2. def run_agent(llm, tools: dict, task: str, max_steps: int = 20):
    3. """Agent main-loop sketch: plan-execute-observe, until the task is done or the budget runs out."""
    4. history = [{"role": "user", "content": task}]
    5. for step in range(max_steps):
    6. reply = llm.chat(history, tools=list(tools))
    7. history.append(reply)
    8. # The model uses structured output to signal "call a tool" or "finish the task"
    9. if reply.get("finish"):
    10. return {"result": reply["content"], "steps": step + 1}
    11. name, args = reply["tool"], reply["arguments"]
    12. if name not in tools:
    13. observation = {"error": f"unknown tool: {name}"}
    14. else:
    15. try:
    16. observation = tools[name](**json.loads(args))
    17. except Exception as exc:
    18. observation = {"error": str(exc)}
    19. # Return only key information (e.g., a diff summary, failing cases) to control context growth
    20. history.append({"role": "tool", "content": json.dumps(observation)})
    21. return {"result": None, "steps": max_steps, "status": "budget_exhausted"}

    Practical essentials:

    1. Describe tool interfaces clearly. Use stable data structures for parameters and return values so the model can call them correctly.
    2. Sandbox and least privilege. File reads/writes, command execution, and network access should all be isolated and whitelisted. Model-generated code is untrusted input; run static checks before execution, and prohibit destructive operations without human confirmation (such as deleting data or changing production environments).
    3. Context management. As steps accumulate, tool output expands quickly and must be summarized promptly so key information is not drowned out.
    4. Human in the loop. Keep a human review checkpoint for critical changes; the model’s job is to “get the work to a reviewable state,” not to auto-deploy with no supervision.

    Once the capability boundaries are clear, the next question is how to measure them scientifically.

    4. Capability Evaluation: Leaderboards Inform, Real Use Cases Decide

    According to public sources, the new-generation general-purpose model performs strongly on several public benchmarks:

    • On LiveCodeBench v6, AIME 25, Humanity’s Last Exam, BrowseComp, and similar benchmarks, it trades wins and losses with Claude Sonnet 4/4.5 and often ranks near the top of important leaderboards such as MathArena;
    • On composite arena leaderboards, it consistently ranks near the top, in the overall top four;
    • In dedicated coding listings, its coding ability is on par with several leading international models.

    For developers, leaderboards are a quick entry point to understanding capability boundaries, but they should not be the sole basis — for three reasons:

    • Leaderboard questions are limited and susceptible to data contamination and overfitting;
    • Leaderboards do not reflect your own business corpus, toolchain, or output specifications;
    • Versions iterate frequently, so snapshot rankings age quickly.

    Recommended three-tier evaluation system:

    Evaluation tier Content Metrics of focus
    L1 General benchmarks Public benchmarks for math, code, multi-subject knowledge, etc. Absolute capability and version regression
    L2 Domain cases Your own codebase fixes, domain-document Q&A, business SQL generation, etc. Task success rate, manual-edit rate
    L3 Online canary Small-traffic A/B tests and user feedback Retention, latency P95, cost per task

    Each tier should cover both correctness and efficiency:

    • Correctness: functional test pass rate, factual consistency;
    • Efficiency: first-token latency, tokens output per second, average tokens consumed.

    Also set regression gates: before a new model or new inference configuration goes live, it must pass L1 and L2 to prevent capability regressions. Once capability clears the bar, the next problem is running it cheaply and stably.

    5. Inference Deployment: Mixed Quantization and Domestic Accelerator Adaptation

    5.1 Mixed Quantization: Balancing Precision and Memory

    FP8 (8-bit floating point) and INT4 (4-bit integer) are common low-precision data types in inference deployment. They have different characteristics and suit different roles:

    Data type Characteristics Suitable for
    FP8 Has exponent bits, giving a wider dynamic range than fixed-point of the same bit width Activations, key-layer weights
    INT4 High storage density, more sensitive to quantization error Weights that are less sensitive to precision

    Take the “FP8 + INT4” scheme used by a new-generation general-purpose model as an example. Newer domestic AI chips already support FP8 computation natively, so the model does not need to drop to FP16 first — it uses FP8 directly as the compute-precision baseline. Selected weights are then compressed to INT4 to reduce memory footprint and memory access. The vendor claims this scheme significantly lowers inference cost with essentially no loss of precision.

    Engineering advice:

    • Choose precision by layer sensitivity. Attention projections and the language-model output head are more sensitive to error and should retain higher precision; some feed-forward layers or expert weights can be compressed more aggressively.
    • Use a calibration set to determine quantization parameters. Quantization scales should be derived from representative calibration data, and the calibration set should closely match your real business distribution.
    • Regress precision across the full chain. After calibration, you must regress on L1/L2 cases, not just look at a perplexity metric.

    Mixed quantization also pairs naturally with MoE: total parameters determine memory footprint, and expert weights are the largest space for compression; activated parameters determine compute, and FP8 is well suited to lowering compute cost on constrained devices.

    5.2 Domestic Accelerator Adaptation: From “It Runs” to “Production-Ready”

    A new-generation general-purpose model has achieved end-to-end deployment with FP8+INT4 mixed quantization on a domestic AI inference chip, forming an integrated model-and-chip solution. It has also been adapted to a new-generation compute card from a domestic GPU vendor through a mainstream open-source inference framework. The vendor states that its adaptation, based on a self-developed general-purpose GPU architecture, validates ecosystem compatibility and can run stably at native FP8 precision.

    These cases show that domestic accelerator adaptation has entered the engineering stage. Whether adaptation proceeds smoothly depends on four layers:

    Adaptation layer Key capabilities Risk points
    Driver & compiler toolchain Graph compilation, operator fusion, memory management New-operator compile errors, performance regressions
    Base operator library Matmul, attention, quant/dequant, collective communication, etc. Missing operators or no FP8 support
    Parallelism & scheduling Tensor/pipeline parallelism, continuous batching, paged KV cache Partition-strategy differences causing throughput drops
    Inference-framework adaptation Hardware backends for mainstream inference frameworks New features (e.g., speculative decoding) lagging behind

    Recommended adaptation process:

    1. Precision baseline. Run the model on general-purpose hardware and freeze the evaluation cases and metrics.
    2. Operator alignment. Verify key operators layer by layer on domestic hardware and pinpoint the sources of numerical differences.
    3. End-to-end stress testing. Cover different input/output lengths, concurrency levels, and batch sizes.
    4. Long-run stability and failure drills. Run continuously for tens of hours while monitoring memory fragmentation, leaks, and performance decay.
    5. Canary launch. Start with internal traffic and gradually increase the share.

    When selecting hardware, also evaluate “migration cost.” Implementations that depend on specific hardware interfaces are costly to migrate, so prefer a general-purpose inference framework that supports multiple backends — this avoids deep lock-in to a single vendor.

    6. Serving and Cost Governance: Making the Model as Available as Utilities

    Model capability ultimately has to be delivered as a service. A new-generation general-purpose model is delivered through a Model-as-a-Service (MaaS) platform and, once released to a mainstream open-source community, can also be deployed privately by enterprises. On the serving side, three points deserve attention.

    Unified access gateway. Wrap authentication, rate limiting, quotas, and metering behind a single API gateway that hides interface differences between models. This makes canary switching easy and cost accounting straightforward.

    1. # Service call with retry and fallback (interfaces are pseudo-code)
    2. from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
    3. class StableClient:
    4. def __init__(self, primary, fallback=None):
    5. self.primary, self.fallback = primary, fallback
    6. @retry(stop=stop_after_attempt(3),
    7. wait=wait_exponential(multiplier=1, min=1, max=8),
    8. retry=retry_if_exception_type((TimeoutError, ConnectionError)))
    9. def complete(self, messages, **kwargs):
    10. try:
    11. return self.primary.chat(messages, **kwargs)
    12. except Exception:
    13. if self.fallback is None:
    14. raise
    15. return self.fallback.chat(messages, **kwargs) # degrade to fallback model

    Performance and cost control. First-token latency directly affects the interactive experience, while throughput sets the concurrency ceiling. Real cost is determined by “token unit price × tokens consumed.” Beyond choosing a cost-effective model, you should also:

    • Cache and reuse common prefixes such as system prompts and retrieval results;
    • Schedule long-context requests and short requests into separate pools so they do not interfere with each other;
    • Route lightweight tasks such as summarization and classification to lighter models.

    A composite arena leaderboard once showed that a new-generation general-purpose model’s API price is roughly one-seventh that of leading international closed-source models, with comparable capability. For cost-sensitive businesses, this means unit-token cost can drop significantly. Cost and capability are not an either/or: multi-model routing lets you balance quality and budget.

    Observability and reliability. Build three layers of monitoring:

    • Business layer: task success rate, manual-edit rate;
    • Model layer: input/output token distributions, average steps;
    • System layer: QPS, latency P95/P99, GPU utilization, memory level, error rate.

    Combine this with logging for full-chain tracing and with alerting to catch capacity bottlenecks early — the standard way to keep a service running stably.

    7. Architecture Reference: A Four-Layer System from Training to Agent Applications

    Layer Main components Key design
    Access API gateway, authentication, rate limiting Multi-model routing, quota management
    Orchestration Agent framework, RAG retrieval, tool sandbox Task planning, context management, human approval
    Inference Inference framework, quantization engine, domestic-chip backend Mixed quantization, parallel strategy, continuous batching
    Data & Ops Object storage, message queue, logging & monitoring Data versioning, async tasks, full-chain tracing
    • Data & Ops layer. The data layer manages training data, fine-tuning data, and knowledge bases; the ops layer handles release, canary rollout, and alerting.
    • Inference layer. This is the cost core and determines per-token cost and the throughput ceiling.
    • Orchestration layer. This layer turns model capability into deliverable business flows.
    • Access layer. This layer unifies the external surface and forms a stable contract for callers.

    Take a coding scenario as an example. A typical request flows like this: a code repository index is stored in object storage → the task is dispatched asynchronously via a message queue → the inference service runs on domestic compute → generated results enter a sandbox for testing → logs from the entire process flow into the logging service, while latency and success-rate metrics feed into monitoring and alerting.

    Trend outlook:

    1. Sparsification and mixed quantization become standard. Larger capacity at a lower per-token cost makes it possible to deploy models on more domestic accelerators.
    2. Agentic capability becomes the core dividing line for general-purpose models. The fusion of code, tool calling, and long context determines whether a model can truly “get work done.”
    3. Co-design of model and compute. Jointly optimizing model structure, quantization schemes, and chip characteristics is becoming the main path to lowering inference costs.
    4. Evaluation moves from static leaderboards to dynamic, task-based evaluation. Evaluation that closely reflects real business needs will become the baseline for model selection and version upgrades.

    Phased implementation recommendations:

    • Phase 1 (selection and validation, ~1–2 weeks): Compare candidate models with L1/L2 cases and set quality thresholds and a cost ceiling.
    • Phase 2 (deployment optimization, ~2–4 weeks): Complete quantization calibration and domestic accelerator adaptation, and stress-test to determine concurrency specs and elasticity strategy.
    • Phase 3 (serving launch, ~2–4 weeks): Build the unified gateway and observability system, run a small-traffic canary, then open up fully.
    • Phase 4 (continuous operation): Establish a version-regression mechanism, track new model releases, and periodically review per-task cost and success rate.

    Conclusion

    The competition among new-generation general-purpose LLMs is shifting from “how large the parameters are” to “how well tasks get completed and how cheaply they run.” Sparse MoE architecture provides capacity; long context and token-efficiency optimization provide usability; mixed quantization and domestic accelerator adaptation provide affordability; and agentic coding brings the model into real engineering workflows. For developers and enterprises, the key is to build an engineering system measured by two yardsticks — task success rate and cost per unit — so that model capability is reliably converted into business value.