Sparse MoE Multimodal LLM Architecture and Training: An Engineering Guide
This article explains how large models are moving from “bigger” to “more controllable capability density” by combining sparse mixture-of-experts (MoE), efficient attention, native multimodal fusion, multi-stage pretraining, and agentic post-training. It describes how sparse MoE trades total parameter count for capacity and activated parameter count for efficiency, how routing and load-balancing losses affect stability, how long-context systems reduce KV-cache memory pressure, how multimodal tokenization and alignment objectives should be chosen, and how agentic post-training can improve task completion. It also provides practical guidance on training and inference engineering, model selection, monitoring, and troubleshooting.
Introduction: LLM competition shifts from “bigger” to “more controllable capability density”
As model parameters move from tens of billions to trillions, context windows extend from thousands to millions of tokens, and capabilities expand from pure text to vision, multimodal understanding, tool calling, long-horizon planning, and autonomous execution, the competition among large models is no longer only about “can it be larger.” It is about whether capacity, long context, multimodal capability, and task execution can be organized into deployable systems under constraints of compute, memory, latency, stability, and cost.
This article focuses on five key areas: sparse mixture-of-experts (MoE), efficient attention, native multimodal fusion, multi-stage pretraining, and agentic post-training. The goal is not simply to list concepts but to help readers understand why sparse MoE can increase capacity, how routing affects stability, how million-token context can reduce memory usage, how multimodal tokens should be selected, how agentic post-training rewards and curricula should be designed, and finally how to choose architectures and troubleshoot issues.
1. Sparse MoE: Trading total parameters for capacity and sparse activation for compute efficiency
1.1 Sparse activation is not “free compute,” but an engineering trade-off between capacity and efficiency
In dense models, every token passes through all parameters during forward propagation, so compute cost is roughly proportional to total parameter count. If we want to increase knowledge capacity, semantic expressiveness, or task capability, adding parameters is the most direct approach, but training and inference costs grow at the same time.
Sparse MoE changes this constraint: the model contains many expert subnetworks, but for each token, only a small number of experts selected by a router participate in computation. This introduces two key metrics:
- Total parameter count: determines model capacity, knowledge upper bound, and memory pressure.
- Activated parameter count: determines the actual compute cost of a single forward pass.
For example, suppose an MoE model has 200B total parameters, but each token activates only about 8B parameters, giving an activation ratio of roughly 4%. If a dense model were used to achieve similar capacity, it might require 200B activated parameters, and a single forward pass could require tens of times more compute than the sparse model. Conversely, MoE is not always superior: increasing total parameters leads to higher memory usage, more complex expert loading, heavier All-to-All communication, and more difficult routing-stability management.
Therefore, sparse MoE is more suitable for scenarios such as:
- Cases that need large knowledge capacity while controlling per-token compute cost.
- Training and inference throughput is a priority, and infrastructure supports high-bandwidth communication.
- Experts have strong task separability, such as language, code, vision, and structured data.
- The team can accept more complex training monitoring, capacity control, and load balancing.
It is less suitable for scenarios such as:
- Extremely low-latency or strongly real-time interactive use cases, because expert loading and routing overhead may offset the benefits of sparsity.
- Small-model scenarios with small parameter scale but strongly coupled tasks, because sparse routing may not be more stable than dense networks.
- Cluster environments with limited communication bandwidth where expert parallelism is difficult to schedule efficiently.
- Product systems that require strong interpretability or simple deployment pipelines.
1.2 Routing and gating: load-loss must be aligned with the top-k routing definition
The core of MoE is not merely “having many experts,” but whether routing is stable, balanced, and trainable. The router is usually a lightweight scoring network that computes affinity between each token and each expert, then selects the top-k experts and performs a weighted sum of their outputs.
A unified description can be expressed as follows:
- For each input token, compute gating logits:
logits = gate(hidden_state) - Obtain expert probabilities through softmax:
probs = softmax(logits / temperature) - Select the top-k experts with the highest probabilities.
- Normalize the weights of the selected experts to avoid weight-scale drift.
- Output the weighted sum of the selected expert results.
- Simultaneously compute a load-balancing auxiliary loss to suppress expert collapse.
When computing the load-balancing loss, it is important not to mix “a load penalty generated by an arbitrary function” with the “final implemented importance/load formula.” A common unified formulation is:
importance_i: the average gating probability of expert i over all tokens.load_i: the actual token-assignment ratio received by expert i.N: total number of experts.L_aux = alpha * N * sum(load_i * importance_i)
The meaning of this formula is: if an expert both receives high probability and receives too many tokens, the penalty becomes larger; if an expert consistently has low probability and low load, the overall load is pushed toward greater uniformity. For top-k routing, load_i is usually computed based on “total assignment count,” not single-round token count; otherwise, different top-k settings lead to inconsistent definitions.
The following pseudocode is consistent with the load-balancing formula:
def moe_forward(tokens, gate, experts, top_k=2, aux_alpha=0.01, capacity_factor=1.25):B, T, D = tokens.shapeflat_tokens = tokens.reshape(B * T, D)logits = gate(flat_tokens)probs = softmax(logits, dim=-1)topk_probs, topk_idx = topk(probs, k=top_k, dim=-1)topk_probs = topk_probs / topk_probs.sum(dim=-1, keepdim=True).clamp_min(1e-9)N = probs.shape[-1]importance = probs.mean(dim=0)assignments = topk_idx.reshape(-1)counts = bincount(assignments, minlength=N).float()load = counts / (B * T * top_k)aux_loss = aux_alpha * N * (load * importance).sum()outputs = zeros_like(flat_tokens)capacity = ceil(B * T * top_k / N * capacity_factor)for i, expert in enumerate(experts):mask, slot = select_tokens_with_capacity(topk_idx, i, capacity)outputs[mask] += topk_probs[mask, slot] * expert(flat_tokens[mask])outputs = outputs.reshape(B, T, D)return outputs, aux_loss
Several boundary conditions deserve special attention:
- When an expert receives more tokens than the capacity limit, the system must decide whether to drop tokens, overflow them to the next expert, or increase capacity through the capacity factor. During training, a capacity factor is commonly used to control load peaks.
- When the total number of tokens is small or the number of experts is large, load statistics can be noisy, and the auxiliary loss may become unstable, requiring smoothing, clipping, or a lower weight.
- When the probability after top-k selection is extremely low, normalization should include a numerical stability term to avoid division by zero or weight explosion.
- If the router has not converged early in training, too large an auxiliary-loss weight may hinder expert differentiation; too small a weight may induce collapse.
- Real systems often add router z-loss to suppress uncontrolled logit scale and avoid softmax saturation or gradient oscillation.
1.3 Stability under extreme sparsity: expert collapse, numerical fluctuations, and monitoring checklist
When the number of experts reaches hundreds or even thousands and the activation ratio is pushed very low, training stability becomes a major challenge. Common problems include:
- Expert collapse and load imbalance: the router repeatedly selects a few head experts, causing some experts to receive gradients for a long time and eventually become ineffective.
- Numerical instability: sparse activation combined with deep networks can lead to excessive gradient variance, activation overflow, and loss spikes.
- Capacity overflow: some experts are selected excessively and exceed capacity, triggering token dropping and affecting effective training samples.
- Insufficient routing noise: the router forms a strong preference for a few experts too early, limiting later differentiation.
Engineering practice usually combines the following strategies:
- Load-balancing auxiliary loss: use load and importance aligned with the top-k definition to push expert assignment toward a more uniform target.
- Capacity factor and drop-token strategy: set expert capacity; tokens exceeding capacity may be dropped, rerouted, or retained with a small gradient signal.
- Routing noise or temperature control: add noise or increase temperature early in training to prevent premature fixation of routing preferences.
- Gated activation functions: in deep sparse networks, gated structures are usually smoother than a single activation function, helping gradient propagation and expressiveness.
- Router z-loss: constrain logit scale to reduce numerical explosion and softmax saturation.
Recommended monitoring items include:
- Expert load distribution: whether each expert’s token share is close to uniform.
- Router entropy: whether it approaches the theoretical maximum entropy; persistently low values indicate routing becomes concentrated too early.
- Head-expert share: whether top-1 or top-k experts occupy too high a share over a long period.
- Drop-token rate: if it continues to rise, capacity or load balance may be problematic.
- Gradient norm: whether abnormal spikes or continuous drift occur.
- Loss spikes: whether they are triggered by routing discontinuities, abnormal expert activation, or communication errors.
- Expert gradient coverage: whether the proportion of experts with no gradient over a long period is too high.
There are no absolute universal thresholds for these metrics, but they can be used to judge relative trends. For example, router entropy that is persistently and significantly lower than the theoretical value, a drop-token rate that remains beyond an acceptable range for a long time, or expert gradient coverage that stays near zero for a particular expert usually indicates that the auxiliary loss, capacity factor, or routing temperature needs adjustment.
2. Efficient attention: an engineering path to million-token context
2.1 The compute wall of long context: attention complexity and KV-cache expansion
The computational complexity of standard self-attention grows quadratically with sequence length, while the KV cache grows linearly with sequence length. When the context expands from thousands of tokens to one million tokens, naive attention reaches limits in both compute and memory at the same time.
Therefore, million-token context is not simply a matter of extending the window. It requires redesigning attention mechanisms, caching strategies, and training curricula. Common directions include low-rank compression, latent attention, cross-layer sharing, sparse attention, window differencing, quantized caching, and paged management.
2.2 Low-rank compression and latent attention: reducing KV-cache size
One mainstream approach is to apply low-rank compression to Key and Value, mapping high-dimensional KV representations into a low-dimensional latent space and caching and computing in that space. The core logic is:
- The original KV cache is large, especially during long-context decoding.
- Low-rank projection can significantly compress cache dimensions.
- During decoding, approximate K/V can be recovered from the latent space for attention computation.
Simplified pseudocode is as follows:
latent_kv = project_down(key, value)cache = latent_kvattn = attention(query, project_up(cache))
The benefits of this approach are reduced KV-cache memory usage and lower decoding bandwidth pressure, but the following points need attention:
- Too high a compression rate may cause information loss and affect long-distance retrieval and fine-grained understanding.
- Recovery projection requires extra computation, which may offset some memory savings.
- Position encoding, attention masks, and low-rank representations need to be re-aligned.
- Different layers may need different compression rates; per-layer adjustment is usually more stable than a single global compression rate.
2.3 Cross-layer sharing and residual injection: clarifying targets, locations, and costs
In long-sequence scenarios, fine-grained information extracted in shallow layers can decay as it passes through layers. To mitigate this, some systems introduce cross-layer information sharing or residual injection. However, “attention residual” is not a fully standardized unified term. To avoid misleading implementers, it can be limited to three engineered approaches:
Cross-layer KV-cache sharing: some layers directly reuse KV caches or low-rank caches generated by earlier layers.
- Injection target: K/V, latent states, or compressed context representations.
- Injection location: before attention computation in later layers, as additional context.
- Cost: increased cache management and memory usage, possibly introducing cross-layer communication synchronization overhead.
- Suitable scenarios: long documents, long videos, multi-turn conversations, and other tasks that need to reuse stable historical information.
Inter-layer state recurrence: use the hidden state of a previous stage or a specific stage as an input bias for the next stage.
- Injection target: hidden states, attention outputs, or gating vectors.
- Injection location: layer-input bypass, pre-attention fusion, or output residual.
- Cost: may introduce nonlinear fusion modules and increase memory and compute overhead.
- Suitable scenarios: tasks that need to preserve global summaries, task states, or long-range context.
Residual injection: add key representations from some layers to later layers through learnable gating.
- Injection target: layer outputs, attention outputs, or compressed states.
- Injection location: near residual connections or after attention outputs.
- Cost: requires extra projection and gating parameters and may increase training instability.
- Suitable scenarios: long-dependency preservation, cross-layer information reuse, and ultra-long-context memory enhancement.
In engineering, it is important not to treat cross-layer sharing as a universal mechanism. It does not inherently reduce complexity; on the contrary, if cross-layer caches are too large or synchronization is too frequent, it may increase memory and communication pressure. Whether to enable it should be evaluated jointly based on sequence length, task type, layer depth, memory budget, and inference latency.
2.4 Incremental attention: boundaries, non-reusable scenarios, and memory impact
Incremental attention is often mentioned in long-context training and inference, but it is not a single algorithm. It is a class of engineering methods that “compute only newly added or changed parts.” Common forms include:
- Prefix KV caching: during autoregressive decoding, the KV of historical prefixes has already been computed, so only new tokens need to be computed.
- Paged caching: manage long-context KV caches in blocks and load or evict them on demand.
- Window differencing: compute only changed parts in adjacent windows or sliding windows.
- Sparse incremental updates: update only key tokens or key blocks affected by new information.
- Quantized caching: apply low-bit quantization to historical KV caches to reduce memory and bandwidth usage.
Its core value is avoiding repeated computation of highly overlapping historical context. It is especially suitable for autoregressive decoding, streaming input, and long-prefix reuse scenarios.
However, incremental attention has clear limitations:
- Non-reusable scenarios: if new tokens do not share a common prefix with historical parts, or each sample requires independent full attention, the reuse benefit is limited.
- Training-scenario differences: random-mask training, non-causal attention, and bidirectional understanding tasks may not directly reuse inference-side caches.
- Accuracy impact: low-bit quantization, sparse updates, and approximate retrieval may introduce errors, so fine-grained retrieval tasks require additional evaluation.
- Memory boundaries: the longer the context, the more likely the KV cache becomes a bottleneck; paging, quantization, eviction, or offloading to lower-tier storage may be needed.
- Engineering complexity: cache management, eviction policies, cross-request isolation, and privacy/security requirements all increase system complexity.
Therefore, incremental attention is not a “universal accelerator.” Its value can be realized stably only after task shape, context reuse rate, and memory budget are clarified.
3. Native multimodal: moving from concatenation to a unified token sequence
Early multimodal systems often used a “vision encoder + alignment layer + language model” concatenation architecture. This approach is easy to implement, but vision and language remain disconnected in representation space, making low-level cross-modal joint modeling difficult.
The idea of native multimodality is to treat modalities such as vision as first-class citizens in a unified token sequence and attention computation during pretraining. This allows the model to establish cross-modal associations at the base level rather than relying only on later alignment.
Engineering efforts should focus on the following issues.
3.1 Choosing visual tokenization
Visual tokenization directly determines context length, detail preservation, and compute cost. Common methods include:
- Patch splitting: divide an image into fixed-size patches, a common approach in ViT-style models.
- Sliding windows or adaptive resolution: dynamically adjust input according to image size and task requirements.
- Token merging and pruning: merge or remove redundant visual tokens to reduce sequence length.
- High-resolution chunking: split a large image into multiple sub-images, encode them separately, and concatenate the results.
The number of visual tokens can be roughly estimated as:
N_visual ≈ (H / patch_size) * (W / patch_size)
For example, a 1024×1024 image using 14×14 patches theoretically produces more than 5000 tokens; with 2×2 merging, the number can drop to the thousand level. Higher resolution provides more detail but also intensifies long-context pressure. Therefore, a trade-off is needed among clarity, token budget, and inference cost.
3.2 Image-text alignment objectives cannot rely on a single loss
Native multimodal models usually require multiple objectives to work together:
- Image-text contrastive objective: pull matched image-text representations closer and push mismatched ones farther apart.
- Image-text matching objective: determine whether an image and text correspond to each other.
- Generative objective: describe an image with text or generate explanations based on an image.
- Fine-grained localization objective: recognize regions, object boxes, OCR text blocks, and other elements.
- Visual instruction objective: enable the model to understand instructions such as “what is wrong in this image” or “summarize the trend in this chart.”
If only coarse-grained image-text contrastive learning is used, the model may learn global matching but struggle with local details, spatial relationships, and document structure.
3.3 Data mixture and training-stage planning
Multimodal training cannot focus only on data volume; data structure and curriculum planning matter more. Recommended considerations include:
- Interleaved image-text data: web pages, articles with mixed images and text, product descriptions, and tutorial documents.
- Video and multi-frame data: require temporal dimensions, frame sampling, and temporal alignment.
- Document and chart data: tables, flowcharts, slides, and scanned documents.
- OCR and fine-grained recognition data: improve understanding of text, symbols, and layouts.
- Safety filtering: filter violent content, privacy risks, copyright risks, and low-quality noisy samples.
Training stages can be roughly divided into:
- Modality-encoder pretraining: separately establish vision and language capabilities.
- Modality alignment: build a cross-modal space through image-text matching, contrastive learning, or generative objectives.
- Joint long-context training: place images, text, and video frames into a unified sequence and train long dependencies.
- Instruction and preference tuning: optimize for question answering, summarization, reasoning, tool calling, and other tasks.
- Agentic stage: let the multimodal model perform multi-step tasks under environmental feedback.
The risk of multimodal models is not only “not understanding,” but also “over-reliance on language priors.” If visual tokens are ignored by the language model, the system may degrade into a text model with a shell. Therefore, training and evaluation should include visual perturbation, fine-grained localization, and cross-modal consistency tests.
4. Multi-stage pretraining: long context is not achieved in one step
Million-token context is usually not obtained through a single training run; it is typically expanded progressively through multiple stages. The reasons are that long-context training is expensive, has high numerical-instability risk, and different context lengths correspond to different data distributions.
A typical route is:
- First, establish language, knowledge, and basic reasoning capabilities on shorter contexts.
- Then gradually expand to tens of thousands or hundreds of thousands of tokens, introducing long documents, code repositories, books, and other data.
- Finally, use long-context alignment and instruction data to teach the model to use long-range information for question answering, summarization, retrieval, and tool tasks.
Each stage requires appropriate data mixtures and learning-rate schedules. Common engineering considerations include:
- Bucket data by length to avoid excessive truncation of long samples.
- Reduce the learning rate appropriately during long-context stages to reduce loss spikes.
- Apply quality filtering to ultra-long samples to avoid low-quality concatenated content dominating training.
- Monitor long-distance dependency loss, not only average loss.
- Compare extrapolation capabilities of position encodings across different context stages.
The core of multi-stage pretraining is not merely “letting the model see longer sequences,” but enabling the model to maintain objective consistency, retrieval accuracy, and task stability under longer dependencies.
5. Agentic post-training: from “can answer” to “can act”
Pretraining gives the model knowledge and language capability, but the ability to complete tasks comes more from post-training. Agent-oriented post-training aims to make the model robust in long-horizon, multi-step tasks that require tool calls and self-correction.
5.1 Core checklist for agentic post-training
- Task-level rewards: the primary objective should be whether the final task is completed, not merely single-turn text fluency.
- Process rewards: for long-horizon tasks, provide rewards for key steps such as planning, checking, retrying, and error correction.
- Tool-call success rate: include correct parameter generation, understanding API responses, exception handling, and retry strategies.
- Failure rollback capability: when tools fail or the environment becomes abnormal, the model should be able to recover state and continue the task.
- Trajectory-length control: overly long trajectories lead to inference costs, state drift, and sparse rewards.
- Curriculum design for sparse-reward scenarios: progress gradually from short tasks, single-tool tasks, and low-ambiguity tasks to long tasks, multi-tool tasks, and complex environments.
- Safety constraints: tool calls require permission isolation, sandboxes, audit logs, and interception of dangerous operations.
5.2 Long-horizon reasoning and planning
Agent tasks often span multiple operational rounds, and the model needs to maintain goal consistency. For example, in a chain such as “search for materials—filter information—call a calculator—generate a report,” the model must not only know the next step but also remember what has already been completed.
Therefore, post-training data should contain not only “question—answer” pairs, but also:
- Intermediate-state summaries.
- Parsing of tool-call results.
- Error detection and rollback.
- Multi-turn context compression.
- Self-assessment of task completion.
5.3 Reward design and stability
Common problems in agent training include long trajectories, sparse rewards, and unstable training. If only outcome rewards are used, the model may learn reward hacking; if only process rewards are used, processes may become verbose or dominated by formalistic actions.
A layered reward structure is recommended:
- Outcome reward: whether the task is completed.
- Constraint reward: whether safety, format, and resource limits are followed.
- Efficiency reward: whether step count, time cost, and call cost are acceptable.
- Reflection reward: whether the model identifies failures, corrects errors, and avoids repeated ineffective actions.
Stable training strategies are also needed, such as KL regularization, rejection sampling, trajectory filtering, advantage normalization, and curriculum-based difficulty progression.
6. Engineering implementation: stability, cost, and selection decisions
6.1 Key training and inference engineering practices
- Expert parallelism: very large MoE models are suitable for distributing different experts across devices, but this needs to be combined with data parallelism, tensor parallelism, and pipeline parallelism.
- All-to-All communication: expert routing naturally introduces cross-device communication, so bandwidth, synchronization, and communication overlap need attention.
- Memory hotspots: expert parameters, KV caches, optimizer states, activation caches, and communication buffers may all become bottlenecks.
- Inference cost reduction: use quantization, cache reuse, dynamic batching, expert preloading, and routing caches to reduce overhead.
- Stability monitoring: continuously track expert load, router entropy, gradient norm, loss spikes, and drop-token rate during training.
- Infrastructure support: large-scale training and inference depend on general infrastructure capabilities such as object storage, message queues, container platforms, logging and monitoring, alerting systems, elastic computing, network storage, and job scheduling.
6.2 Selection decision table
| Scenario | Recommended direction | Cost hotspots | Main risks |
|---|---|---|---|
| Low-latency dialogue, lightweight deployment | Dense small models | Training and inference compute | Lower capacity ceiling |
| High-capacity language and code models | Sparse MoE | Memory, communication, routing complexity | Load imbalance, training instability |
| Long documents, code repositories, multi-turn context | Long-context optimization | KV cache, attention compute | Memory overflow, long-dependency decay |
| Image-text understanding, document parsing, video analysis | Native multimodal | Visual tokens, data mixture | Modality imbalance, visual information loss |
| Tool calling, automated execution, complex workflows | Agentic post-training | Environment interaction, reward design | Reward hacking, trajectory oscillation |
6.3 Common troubleshooting checklist
- MoE training loss spikes: check router logits, expert load, gradient norms, learning rate, and data distribution.
- Expert collapse: check auxiliary-loss weight, capacity factor, routing temperature, expert initialization, and data diversity.
- Insufficient memory for long context: check KV cache, low-rank compression, paging strategy, quantization, and sequence-length distribution.
- Long-context performance drops instead of improves: check position encoding, data quality, learning rate, long-sample truncation, and attention dilution.
- Multimodal model ignores images: check visual-token weight, modality dropout, image-text data mixture, and visual-perturbation evaluation.
- Unstable agent tasks: check reward sparsity, tool-return noise, trajectory length, curriculum difficulty, and error-rollback mechanisms.
7. Frequently asked questions and judgment recommendations
7.1 Why is sparse MoE prone to expert collapse?
Because routers are affected by positive feedback: experts that gain a slight advantage early receive more tokens, obtain more gradients, and further increase their probability of being selected. Without load-balancing constraints, this trend can keep strengthening until a few experts become overloaded while most experts degrade. Solutions include load-balancing loss, capacity control, routing noise, expert initialization, and data diversity.
7.2 How can million-token context reduce memory usage?
Usually, no single method is enough. It requires combining low-rank KV compression, latent caching, quantized caching, paged caching, cross-layer sharing, sparse attention, window strategies, and sequence parallelism. The key is to distinguish between training memory and inference memory: training focuses on activations and optimizer states, while inference focuses more on KV caches and batching throughput.
7.3 What are common mistakes in long-context training?
Common mistakes include directly extending context length without adjusting the learning rate, insufficient quality of long-sample data, failure to adequately validate extrapolation of position encodings, mismatch between data-length distribution and inference distribution, average loss masking degradation in long-dependency tasks, and failure to monitor attention dilution and retrieval-capability decline.
7.4 How should multimodal tokenization be chosen?
Resolution and token budget should be decided around the task. If the task is document OCR, chart understanding, or fine-grained visual question answering, higher resolution should be preserved; if the task is image summarization, coarse-grained video understanding, or low-cost inference, tokens should be compressed appropriately. It is recommended to evaluate visual token count, language token count, total context length, and inference latency together, rather than considering image resolution alone.
8. Summary: understanding mechanisms matters more than memorizing metrics
Ultra-large-scale sparse MoE multimodal models represent a complex systems-engineering effort: sparse activation is used to trade for capacity, efficient attention for long context, native multimodality to broaden perception, agentic post-training to improve task completion, and cost reduction, monitoring, and infrastructure support to ensure deployment.
For engineering teams, the real difficulty is not whether individual modules exist, but:
- Whether sparse routing is stable and whether load is balanced.
- Whether long context truly improves task capability rather than merely increasing memory usage.
- Whether multimodality truly uses visual information rather than degrading into language priors.
- Whether agentic post-training truly improves task completion rather than increasing trajectory complexity.
- Whether training, inference, monitoring, and infrastructure form a closed loop.
Understanding these underlying mechanisms is more important than remembering the parameter scale or leaderboard score of a particular model, because together they determine the capability ceiling, cost boundary, and engineering feasibility of the next generation of large models.
