770B Total Parameters, 49B Active: Next-Generation Sparse LLM Architecture, 1M Context, and Inference Costs
This article analyzes a next-generation sparse large language model preview with 770B total parameters and 49B active parameters per token. It explains the mixture-of-experts design, gated sparse attention with hierarchical index caching, the 4-way residual information flow, native multi-token prediction, FP8 support, and the practical implications for 1M-token context, inference cost, evaluation, and production deployment.
Introduction: How to Evaluate a Long-Context Large Language Model
At the end of August, a vendor released and open-sourced the preview version of a next-generation large language model. The model has 770B total parameters, activates 49B parameters per token, claims a 1M context window, and targets real production scenarios such as code, office tasks, and scientific work. Its architecture has been almost completely reworked compared with the previous generation. The parameter scale increased significantly because of AI team reorganization, and the overall complexity also increased.
If we break down the public information, the main theme of this upgrade is: using less compute per token to support longer context and stronger agent capabilities. Three points are especially worth attention:
- Parameter and activation scale expanded substantially, bringing inference cost challenges.
- The attention mechanism was rebuilt from conventional grouped-query attention into “gated sparse attention + hierarchical index cache.”
- Official benchmarks show broad leads, but real-world experience with similar preview models may not match leaderboard scores.
The following sections follow the main thread: architecture, capabilities, cost, evaluation, and deployment.
1. Read the Parameter Table First: Why Total and Active Parameters Must Be Separate
This type of model uses a mixture-of-experts (MoE) architecture. Each layer contains hundreds of experts with different parameter counts, and a lightweight router selects a small number of experts to compute for each token. The weights of the other experts do not participate in that token’s computation. This creates two key metrics:
- Total parameters: determine the memory required to store the full weights. More experts or larger experts make total parameters grow faster.
- Active parameters per token: determine the compute for a single forward pass. This is not simply the sum of attention, shared components, and selected expert parameters. Routed experts account for only part of the computation. If each layer also has a resident shared expert, its parameters affect all tokens and must be counted separately.
| Dimension | Previous preview | New preview | Change |
|---|---|---|---|
| Total parameters | 295B | 770B | About 2.6x |
| Active parameters per token | 21B | 49B | About 2.3x |
| Context length | 256K | 1M | About 4x |
| Layers | 80 | 78 | Slightly reduced |
| Hidden dimension | 4096 | 6144 | About 1.5x |
| Routed experts | 192 | 256 | 64 more experts |
| Activation method | Top-8 routing | Top-8 routing + 1 shared expert | Adds a resident shared expert |
| Attention | Grouped-query attention (GQA) | Gated sparse attention + latent attention + index cache | Fully reworked |
| Residual structure | Conventional single-path residual | 4-way information flow (iHC) | Parallel multi-path information flow |
| Multi-token prediction (MTP) | 3.8B / 0.7B active | 10B | Native draft module |
| Open precision | BF16 | BF16, MXFP8 (FP8) | Adds FP8 weights |
Here is a closer reading of each change:
- Capacity and compute expanded together, but compute growth is slightly lower: 770 / 295 is about 2.61, while 49 / 21 is about 2.33. The growth in active parameters is slightly smaller than the growth in total parameters. The extra capacity exists more in the form of a larger and finer expert pool, used to store knowledge and specialized capabilities, rather than being pushed onto every token.
- The hidden dimension expanded substantially: 4096 to 6144 is about 1.5x, while active parameters increased by about 2.3x. This matches the design logic of MoE models: the hidden dimension is part of the shared “backbone” for all tokens, so its marginal cost on a single token is limited. The actual compute is mainly determined by the number of layers and the amount of expert activation.
- Routing became finer-grained: the expert pool increased from 192 to 256, while the activation strategy remains Top-8. Each expert covers a narrower knowledge scope, and the routing selection space is larger, but roughly 70% or more of the parameters remain dormant per token.
- Layers decreased slightly while the hidden dimension became wider and the residual path became multi-path: 78 layers can hold more effective information, leaving room for long-context reasoning.
- The multi-token prediction module adds an extra cost: it does not change the main model structure, but it can improve decoding throughput.
There is an engineering conclusion that is easy to overlook: the memory required to load the model is determined by total parameters, while the compute required to process each token is determined by active parameters. Using BF16, which takes 2 bytes per parameter, 770B weights require about 1.54TB of memory. A single card cannot hold that amount, so deployment usually requires a combination of tensor parallelism, expert parallelism, and pipeline parallelism, or direct use of cloud-hosted inference. A large total parameter count does not automatically mean high per-token compute cost, but it significantly raises the deployment threshold.
2. Attention Reconstruction: How Gated Sparse Attention Supports 1M Context
The previous generation’s conventional grouped-query attention (GQA) reduced KV Cache by allowing multiple query heads to share a group of key-value heads. It mainly solved the problem of duplicate storage across multiple heads. As sequences get longer, two problems become more important:
- KV Cache grows linearly: when the model generates each new token, it must read and process the key-value vectors of all previous tokens.
- Quadratic complexity: conventional attention needs to calculate the relevance between the current query and all historical keys. The compute grows quadratically with sequence length.
A 128K context is already challenging. A 1M context would be difficult to accept unless the attention computation itself changes. The new approach can be summarized as “filter first, then compute”:
- Lightweight indexer scoring: for each query, a lightweight indexer scores its relevance against all historical tokens. Since sequences under causal attention are processed in blocks, only one aggregate score is needed for each block. This overhead is far lower than full attention.
- Compute only Top-K: the main attention only processes the 2048 highest-scoring positions. Whether the context is 32K or 1M, the compute for a single attention pass is determined by a fixed budget. The main cost that grows with context length is compressed into a small linear term in the indexing stage.
- Adaptive gating: not every layer and not every query head needs to be sparse. The gating module learns a switch for each query head: heads handling content-dense tasks that require precise global comparison fall back to full attention, while other heads take the sparse path. This preserves the accuracy needed for retrieval-style tasks while avoiding the information loss caused by a one-size-fits-all sparse design.
- Latent attention (MLA) compresses KV: together with gated sparse attention, each position’s key and value are compressed into a low-dimensional latent vector. The KV Cache only needs to store one compressed vector per token, rather than expanding it by the number of heads. This reduces storage cost instead of letting it grow with the number of heads.
However, running a full indexer in all 78 layers would still be inefficient. The new approach uses hierarchical index reuse:
- All 78 layers enable sparse attention.
- Only 21 layers run the full indexer: layers 0 and 1, then every 4 layers starting from layer 5 through layer 77.
- The other 57 layers directly reuse the token indices selected by earlier layers and do not score again.
There are two design details here:
- Special handling for layers 0 and 1: these layers are near the network entrance, where representation ability is not yet fully formed. Keeping full indexing helps build a better representation foundation.
- Preference consistency between adjacent layers: representations in adjacent Transformer layers are highly similar, and their “relevant token sets” are often similar as well. Reusing indices causes limited precision loss but reduces indexing overhead to about one quarter of the original cost, or 21 / 78.
The trade-off is also clear. During inference, each token can see at most 2048 historical positions. Semantics beyond that range can only be reflected through the aggregated information of those 2048 positions. For tasks such as long-document detail retrieval or cross-section reasoning, the indexer must “capture” the key information into the candidate set. Otherwise, the model may not even “see” the paragraph that contains the answer. In addition, sparse attention and conventional attention usually cannot share the same KV Cache format and operators. Deployment must carefully distinguish version and precision configurations.
3. 4-Way Residual Stream: Making Room for Greater Depth
Residual connections add the input directly to the output of each sublayer. They are a basic structure for solving information loss in deep networks. A conventional residual connection has only one information channel. When the hidden dimension grows to 6144 and the context length reaches the million-token level, a single-layer subspace becomes increasingly crowded.
The new approach introduces a 4-way residual information flow (iHC). It splits the hidden state into 4 parallel channels, allowing sublayers to read and write on different channels. This is similar in spirit to multi-resolution and multi-rate information processing. It usually brings two benefits:
- Wider information capacity: different channels can carry different types of information, such as long-range semantics and local details, reducing mutual interference.
- More stable gradient paths: deep networks get multiple stable gradient propagation paths, which helps training efficiency and makes greater trainable depth more feasible.
Reducing layers from 80 to 78 while nearly doubling total parameters is essentially a trade-off of “exchanging width for depth”: a wider and more structured residual space carries representation capacity that would otherwise require deeper networks.
4. Native MTP Draft Layer: Turning Prediction into a Trainable Component
Speculative decoding uses a lighter draft model to generate several candidate tokens in a row. The main model then verifies them in parallel. Accepted tokens are directly adopted, while rejected tokens are regenerated from the correction point. A single forward pass can produce multiple tokens, reducing the latency of token-by-token decoding.
The native MTP (multi-token prediction) module shares the vocabulary and backbone representation with the main model. It can serve as a draft layer and be trained together with the main model. The new generation expands it from 3.8B parameters with 0.7B active to 10B parameters. The larger capacity means more room for improving draft quality, but it also means extra memory usage and training overhead.
This module is an optional capability. Before formal deployment, it is important to measure whether “acceptance rate × throughput improvement” is actually worthwhile.
5. Open Precision and Cost: Price Is the Real Variable
| Model (preview / top-tier) | Input price (relative) | Output price (relative) |
|---|---|---|
| New preview model | 6 | 18 |
| Same-tier lightweight preview A | 0.8 | 2.8 |
| Same-tier lightweight preview B | 0.8 | 2.7 |
| Same-tier full version C | 8 | 28 |
| Same-tier flagship version D | 12 | 36 |
Using relative values, where smaller numbers mean lower cost, we can see that:
- Its price is close to the flagship tier, slightly below full version C, and about seven to eight times higher than lightweight preview models A and B.
- The extra budget should bring capability gains. Therefore, evaluation must return to “effect differences on real tasks,” not just leaderboard scores.
There are two additional cost variables:
- Prefix caching: if system prompts, tool definitions, and reference documents for long contexts can be reused, repeated computation can be significantly reduced.
- Sparsity benefits: attention overhead under a 1M context is greatly compressed, turning “long context” from a marketing number into an affordable engineering option.
6. How to Read Evaluations: Don’t Be Captured by Leaderboards and Numbers
1. De-hyping benchmark scores
Public benchmarks have been heavily covered by many models and datasets. There are risks such as training-data contamination, leaked questions, and leaked solutions. Simply comparing scores is hard to reflect true capability. A credible evaluation should at least do the following:
- Use private test sets that are self-built and not public.
- Design metrics according to business tasks: for code, look at compilation pass rate and unit-test pass rate; for office tasks, look at structured-output compliance; for scientific tasks, look at citation accuracy.
- Pay attention to the “effective utilization rate” of long context. Place key information at the beginning, middle, and end, and test each separately. This is commonly known as a “needle in a haystack” evaluation, and it helps avoid a gap between “supports 1M” and “can actually use 1M.”
2. Simple example code
def build_eval_set(seed=0, n=50, needles=(0.1, 0.5, 0.9)):"""Build a long-context retrieval evaluation set with answers placed at the beginning, middle, and end."""import randomrng = random.Random(seed)cases = []for i in range(n):ctx = ["Generic background paragraph"] * 2000key = f"Key fact-{i}"pos = int(len(ctx) * needles[i % len(needles)])ctx[pos] = keycases.append({"prompt": "Search the materials below and answer the question:" + "\n".join(ctx) + f"\nQuestion: What is the content of {key}?","expected": key,"needle_pos": pos / len(ctx),})return casesdef report(pred_ok, total, by_pos):"""Bucket hit rates by position to identify 'middle collapse'."""acc = sum(pred_ok) / totalprint(f"Overall accuracy: {acc:.3f}")for k, v in sorted(by_pos.items()):print(f" Position {k:.1f}: {v[0]}/{v[1]} = {v[0]/v[1]:.3f}")
3. Three rules for self-built evaluation
- Isolate contamination: questions and answers should not be stored in public datasets or uploaded to public platforms.
- Control variables: keep prompts, temperature, and decoding parameters fixed, and change only the model.
- Pay attention to variance: sample the same input multiple times and observe stability rather than relying on a single score.
7. Deployment Advice: When Should You Use a 1M Context?
The goal is not “the longer, the better,” but “the more accurate, the better.” An indexer that can select 2048 relevant positions is often more effective than blindly lengthening the context. In many business scenarios, retrieval-augmented generation plus refined context is often cheaper and more stable than stuffing everything into the context window.
Agent scenarios should first manage a “context budget.” Tool definitions, conversation history, codebase snippets, and retrieval results all need quotas. The following is a directly reusable quota allocator:
class CtxBudget:"""Token budget allocator for long-context scenarios (units: tokens)."""def __init__(self, total=1_000_000, reserved_ratio=0.1):self.total = totalself.usable = int(total * (1 - reserved_ratio)) # Reserve space for generation and system overheadself.quota = {"system": 0.10, "tools": 0.10, "history": 0.25, "retrieval": 0.55}def alloc(self, key, used):limit = int(self.usable * self.quota[key])return min(used, max(0, limit)), limit# Example: prioritize retrieval snippets and compress conversation historybudget = CtxBudget(total=1_000_000)docs, dlimit = budget.alloc("retrieval", used=120_000)hist, hlimit = budget.alloc("history", used=800_000)print(docs, dlimit, hist, hlimit) # 120000 550000 250000 250000
- Pay attention to memory pressure from “long context × high concurrency.” The KV Cache size is approximately:
2 × layers × KV head dimension × precision bytes × sequence length × concurrency
If the sequence length increases by 10x and concurrency doubles, memory usage can increase by 20x. Sparse attention can compress part of this, but peak concurrency still needs stress testing.
Use the preview version for validation, not direct production deployment. Preview APIs, rate limits, and capabilities may change. If the hosted service provides token-based billing, prefix caching, streaming output, or similar features, verify whether they match your business needs, and rely on the official documentation.
Draw an “effect-cost” curve. Run the same task with a lightweight model, a mid-tier model, and a flagship model, then plot quality versus cost. This often reveals a cost-performance turning point, rather than defaulting to the largest model.
8. Summary
The significance of this next-generation sparse large model is not simply that “parameters became larger.” It provides a reusable engineering paradigm:
- Gated sparse attention and hierarchical index caching turn long context from a luxury into an affordable option.
- Multi-way residual paths and a wider hidden state exchange width for representation capacity.
- A native draft module improves decoding throughput.
- FP8 weights and prefix caching help control inference costs.
For developers, the real judgment criteria are only two questions: is the effect on your self-built evaluation set worth the price, and does your business really need a 1M context? Once these two points are clear, model selection will not be driven by leaderboards.
