Sparse Architectures and Training Infrastructure for Open Large Models

    A public technical report is used as a sample for reading a hundred-billion-parameter sparse open model: total versus activated parameters, compressed and dynamic sparse attention, multi-token prediction, fully asynchronous agent reinforcement learning, and the training and serving systems behind reproducibility and cost. The figures explain engineering tradeoffs, not guaranteed capability in every setting.

    Sparse Architectures and Training Infrastructure for Open Large Models

    Faced with a large-model technical report that runs to dozens of pages, developers often fall into one of two reading habits. One is to look only at leaderboard rank and treat a temporary lead as the best choice for production. The other is to stare at parameter and configuration tables while ignoring the training objectives, system design, and cost constraints behind those numbers. A more useful approach is to read the report in four layers: base scale and sparse structure, attention and training stability, training objectives and long-context efficiency, and reinforcement learning together with engineering infrastructure. The discussion below uses a configuration disclosed in a public technical report as an observation sample and examines the design logic of one class of hundred-billion-scale sparse open models. The numbers are used to explain engineering tradeoffs. They are not a capability promise for any particular scenario.

    1. What a technical report is actually worth reading for

    A model report usually contains three kinds of information at once:

    1. Capability information: parameter scale, training data, evaluation results, context length, and task performance.
    2. Method information: attention structure, optimizer adaptation, training objectives, sparse routing, and the reinforcement-learning framework.
    3. Systems information: distributed training, data pipelines, inference serving, failure recovery, and the evaluation stack.

    Many discussions focus only on the first category. What usually decides whether a model can be reproduced, served, and run at a controllable cost is the second and third. A very large total parameter count with a smaller activated count does not automatically mean low inference cost. A smaller activated count with a large number of experts does not automatically mean simple deployment. Sparse models separate compute from memory, communication, and scheduling complexity. Developers have to evaluate those pieces separately.

    Claims such as “the best open model” should be read as stage-specific conclusions under a particular time, evaluation set, prompt, and cost constraint. The value of a technical report is not an absolute ranking across every scenario. It is an account of the tradeoffs the team made in architecture, data, training objectives, and systems to reach the reported result.

    2. Base design: what 744B total and 40B activated parameters mean

    One typical configuration in the public report is about 744B total parameters, about 40B activated per inference step, 256 experts, 80 layers, and about 28.5T training tokens, of which pretraining is about 27T and mid-training is about 1.5T. The report also describes a generational change: total parameters rise from about 355B to 744B, and activated parameters rise from about 32B to 40B. Taken together, those figures support several practical judgments.

    2.1 Total parameters and activated parameters should be evaluated separately

    The core of a mixture-of-experts design is decoupling capacity from compute. Total parameters set the knowledge and representation space the model can hold. Activated parameters set the main scale of computation that each token actually uses. With 744B total and 40B activated, the activation ratio is on the order of five percent. For inference, the dominant matrix compute per token is closer to the activated scale. Memory footprint, expert-parallel communication, routing overhead, and cache management still depend on the total parameter count and the number of experts.

    It is therefore not accurate to say that “activating 40B only requires the resources of a 40B model.” If every expert must stay resident in memory, or if experts are sharded across machines and devices, cross-node communication becomes a critical bottleneck. Real serving cost depends on all of the following:

    • Memory capacity and bandwidth
    • Expert parallelism and all-to-all communication
    • Routing hotspots and load balance
    • Batch size and tail latency
    • Key-value cache under long context
    • Quantization, cache reuse, and scheduling policy

    2.2 More experts amplify routing and communication problems

    A design with 256 experts and 80 layers increases model capacity and also enlarges the routing space. The router’s job is to send different tokens to different experts so the model can hold more capacity under a fixed compute budget. In practice, familiar problems appear: some experts are chosen too often and become hotspots; some stay idle for long periods, so parameter utilization falls; and during training, gradient scales differ across experts and affect convergence.

    Common responses include auxiliary load-balancing losses, expert capacity limits, routing-temperature adjustments, expert replication, dynamic batching, and fine-grained monitoring. On the serving side, what matters most is whether hotspot experts create long-tail latency, not the average. A system with high average throughput can still be unsuitable for online traffic if P99 latency jitters badly because of expert hotspots.

    2.3 More training tokens are not the same as simply stacking data

    Training tokens rising from about 23T to 28.5T, with about 1.5T split out as mid-training data, shows that the data pipeline is not “more is always better.” Mid-training typically strengthens high-quality data, adapts the model to long context, fills domain gaps, and shapes format and tool-calling habits. For large models, data quality, deduplication, mixture ratios, curriculum, and stage goals often matter more than a single total volume.

    Teams that build their own continued pretraining should check in particular:

    • Whether deduplication and contamination detection are sufficient
    • Whether pretraining, mid-training, and post-training have clear data goals
    • Whether the mix of long-context and ordinary samples is reasonable
    • Whether domain data is quality-tiered
    • Whether evaluation sets are isolated from training data

    3. Low-rank compressed attention and per-head orthogonalization: stability matters more than a paper figure

    A key change in the report is low-rank compressed attention, used to reduce key-value cache cost on long text. Training also exposed a mismatch between the optimizer and the attention layout, which was improved by orthogonalizing updates within groups of attention heads. This is worth attention because structural changes are not isolated. The optimizer, parameter layout, and gradient update have to be designed together.

    3.1 Gains and costs of low-rank compressed attention

    In conventional attention, long generation caches keys and values for every token, every layer, and every head. Longer context means a larger key-value cache and more pressure on memory and bandwidth. Low-rank compressed attention projects keys and values into a lower-dimensional latent representation and reconstructs or computes jointly at use time, which reduces cache dimension. The direct benefits are lower long-context memory use, less cache traffic, and room for higher concurrency or longer context.

    Compression is not free. The latent path adds projection compute. The chosen rank affects representational capacity. Gradients pass through compression and reconstruction and can amplify numerical instability. When heads share or group projections, specialization across heads can also be disturbed. The design has to be judged together on training stability, long-context evaluation, and serving-side cache savings.

    3.2 Why per-head orthogonalization is needed

    Orthogonalized updates are often used to improve the condition of a matrix update, spread update directions more evenly, and stop a few directions from being amplified too much. If an entire projection matrix is orthogonalized as one block, while that matrix is actually a concatenation of attention heads, gradients from different heads are mixed. The more specialized the heads are, the more that mixing can interfere, showing up as loss spikes, slower convergence, or weaker downstream results.

    Splitting by head and then orthogonalizing respects each head’s parameter subspace. Updates stay stable inside each head and avoid excessive coupling across heads. The idea can be sketched as:

    1. for projection in attention_projections:
    2. for head_slice in split_by_head(projection):
    3. grad = head_slice.grad
    4. update = orthogonalize(grad)
    5. head_slice.weight -= learning_rate * update

    Implementation has to respect parameter memory layout. If weights are packed so that all heads sit in one contiguous region, splitting can add extra copies. If weights are stored in per-head blocks, updates are more efficient, but load and communication logic is more complex. Details that look like “only an optimizer change” often decide whether large-scale training stays stable.

    4. Multi-token prediction: extending one-step supervision to several steps

    Multi-token prediction is a training objective that is easy to underestimate. A standard language model predicts the next token at each position. Multi-token prediction adds auxiliary heads on the same hidden state to predict several future tokens. It can supply a denser training signal, push the representation to encode information several steps ahead, and provide a basis for draft-style or speculative decoding.

    A simplified sketch:

    1. for position t:
    2. hidden = model(prefix[:t])
    3. loss = cross_entropy(head_0(hidden), token[t])
    4. for step in 1..K:
    5. loss += weight[step] * cross_entropy(head_step(hidden), token[t + step])
    6. loss.backward()

    Potential benefits include:

    • More prediction signal at each training position, so samples are used more efficiently
    • Hidden states that carry short-horizon planning information
    • A path, at inference time, to combine with draft generation and verification decoding and reduce decode latency

    The idea should not be overread. Multi-token prediction does not guarantee faster online serving. Acceptance rate of the auxiliary heads, verification cost, batching efficiency, and output consistency all affect the net gain. Training also has to balance loss weights, auxiliary-head depth, memory overhead, and the main task against the auxiliary tasks. If supervision on future tokens is too strong, next-token prediction can suffer. If it is too weak, the extra representation may never form. It is better treated as a combined training objective and decoding tool, not as a standalone acceleration switch.

    5. Dynamic sparse attention: long context depends on selective compute

    The central tension in long context is that attention cost grows quickly with sequence length. Full attention models every token pair and is expressive but expensive. Fixed-window attention is cheaper but can drop distant information. Dynamic sparse attention tries to sit between them: keep a local window and a small set of global anchors, and dynamically select the most relevant blocks or tokens for the rest of the compute.

    A block-level sketch:

    1. for query_block in split_queries(hidden):
    2. local_output = attend(query_block, local_window)
    3. global_output = attend(query_block, global_sink_tokens)
    4. routing_scores = route(query_block, candidate_blocks)
    5. selected_blocks = topk(routing_scores, k)
    6. sparse_output = attend(query_block, selected_blocks)
    7. output = merge(local_output, global_output, sparse_output)

    The hard part is whether sparsity actually saves time. Irregular sparse patterns fragment memory access and can lower GPU utilization instead of raising it. If routing differs between training and inference, quality can swing. If the long-context distribution in training data differs sharply from production, sparse selection can bias toward position patterns that were common in training.

    Evaluating dynamic sparse attention therefore cannot stop at theoretical complexity. It also has to cover:

    1. Whether block size is aligned with hardware memory access
    2. Whether routing scores are stable, and whether load balance or capacity limits are needed
    3. Whether the long-context task actually depends on distant information
    4. Whether the training-token budget matches the context-length distribution
    5. Whether online P95 latency and memory use improve together

    Discussing attention efficiency together with the training budget is a reminder that long-context ability does not come from one structure alone. It is the joint result of data, positional encoding, training stage, sparse policy, and inference cache.

    6. Fully asynchronous agent reinforcement learning: throughput gains and stability costs

    Agent reinforcement learning differs from ordinary text reinforcement learning. An agent calls tools, reads environment feedback, and takes actions over multiple turns, so trajectory length varies widely. In synchronous training, one policy update often waits until every environment finishes a full batch of trajectories. Slow tool calls leave a large amount of compute idle. A fully asynchronous framework decouples trajectory generation, policy learning, environment execution, and the replay buffer so the trainer can keep consuming samples and raise overall throughput.

    An abstract loop:

    1. while training:
    2. policy_version = learner.current_version()
    3. trajectories = rollout_workers.collect(env, policy_snapshot=policy_version)
    4. replay_buffer.add(trajectories, version=policy_version)
    5. batch = replay_buffer.sample(max_staleness=K)
    6. loss = clipped_policy_gradient(batch, learner.current_policy())
    7. learner.update(loss)

    Asynchrony creates two immediate problems: stale policies and shifted sample distributions. The policy that generated a trajectory may lag the current learner. Using old samples without limit biases the policy-gradient estimate and can make training oscillate. Engineering practice usually needs to:

    • Set a maximum staleness K and drop trajectories that are too old
    • Use importance sampling or a clipped objective to limit update size
    • Monitor the mix of samples by version and the distribution of advantage estimates
    • Clean failed trajectories, timeouts, and abnormal rewards
    • Sandbox environment execution, with rate limits, retries, and timeouts
    • Record full interaction traces for reproduction and audit

    “Fully asynchronous” is not simply putting training and inference in two processes. It requires versioned weights, backpressure, partial-trajectory recovery, failure isolation, and observability. For agent tasks, sparse rewards, uncertain tool returns, and environments that cannot be fully reproduced all amplify instability. High-quality reinforcement-learning infrastructure often spends most of its effort on data cleaning, trajectory versioning, and filtering abnormal samples, not on a single algorithm formula.

    7. Training infrastructure: the hidden capability behind scale and reproduction

    The disclosed model scale, expert count, training tokens, and reinforcement-learning framework point to one fact: competition among large models is no longer only about model structure. It is also competition among data pipelines, distributed training, inference serving, and experiment management.

    7.1 What to watch on the training side

    • Data pipeline: deduplication, quality filtering, mixture ratios, curriculum, and long-context sampling
    • Distributed parallelism: the combination of data, tensor, pipeline, and expert parallelism
    • Communication: all-to-all traffic, gradient compression, and overlapping compute with communication
    • Stability: loss-spike monitoring, gradient norms, expert load, and numerical precision
    • Failure recovery: checkpoints, elastic training, and node-failure isolation
    • Experiment management: binding data versions, code versions, hyperparameters, and evaluation sets

    7.2 What to watch on the inference side

    Serving a mixture-of-experts model is not just loading weights. Expert parallelism adds cross-device or cross-machine communication. Routing hotspots create tail latency. Long context increases key-value cache pressure. Multi-token prediction or draft decoding changes the shape of batches. A usable serving system usually needs capacity planning, dynamic batching, expert caching, a quantization policy, request prioritization, and end-to-end monitoring.

    On a cloud such as Baidu AI Cloud, a similar pipeline can be carried by general capabilities such as a container platform, object storage, message queues, logging, and monitoring and alerting. The concrete design still depends on data scale, SLA, budget, and the team’s operations capacity. The point is not to stack cloud products. It is to connect data, training, evaluation, deployment, and observation into a pipeline that can be rolled back.

    8. How to read evaluations and choose a model rationally

    Public aggregate benchmarks, text arenas, and code arenas are useful references. They do not replace evaluation on your own workload. A model that ranks high on a general leaderboard is not automatically best for your domain, prompts, tool chain, and cost constraints. A selection check should at least cover:

    1. 1. Task split: generation, question answering, code, retrieval augmentation, and tool calling, evaluated separately
    2. 2. Long context: P95 latency and accuracy under the real document-length distribution
    3. 3. Structured output: stability of JSON, function calls, and format constraints
    4. 4. Deployment cost: memory, bandwidth, expert parallelism, batching, and scale-out policy
    5. 5. Safety and compliance: data isolation, content safety, log audit, and access control
    6. 6. Observability: error rate, timeout rate, routing hotspots, cache hit rate, and rollback

    Also watch for evaluation contamination, prompt sensitivity, version drift, and the trap of averages. A more reliable approach is a held-out set of your own plus a small-traffic A/B test, comparing quality, latency, cost, and stability on the same table. For open models, also assess license, weight distribution, fine-tuning cost, community maintenance, and long-term reproducibility. For a hosted service, assess data boundaries, quotas, rate limits, and failover.

    9. Closing

    The most useful lesson in this technical report is not any single number. It is the ability to connect structure, training objectives, optimization stability, sparse long-context compute, asynchronous reinforcement learning, and systems engineering. 744B total parameters and 40B activated parameters show capacity decoupled from compute. Low-rank compressed attention and per-head orthogonalization show that structural changes have to fit the training optimizer. Multi-token prediction extends supervision to several future positions. Dynamic sparse attention tries to control long-context cost with selective compute. Asynchronous agent reinforcement learning puts throughput and stability in the same system and trades them off together.

    For developers, the right way to read a report of this kind is to separate capability claims from engineering mechanisms, then validate both on your own business data. Leaderboards can point a direction. What decides system value is whether the architecture matches the load, whether training can be reproduced, whether inference cost is controllable, and whether evaluation is close to real tasks. Only when those factors are read together does a technical report become an engineering judgment that can be put into practice.