# Trillion-Parameter MoE Multimodal LLMs: Architecture Breakdown and Engineering Notes

    This article unpacks the design of a recently disclosed trillion-parameter multimodal model built on a Mixture-of-Experts (MoE) architecture, and translates it into engineering guidance. It walks through the reported headline figures — 2.8 trillion total parameters, 104 billion activated parameters, native vision support, and a 1M-token context — and explains the three core design choices: gated attention that mixes KDA linear attention with Gated MLA and adds Block AttnRes cross-layer fusion; a lower-bound parameterization of the log-decay that removes a special-case code path from the diagonal-block kernel; and quantile balancing (QB), which replaces auxiliary load-balancing loss with a bounded routing constraint. It then covers the progressive context extension schedule (8K to 1M), million-token agentic RL, transfer costs, cloud deployment considerations, and the limits that matter for model selection.

    Trillion-Parameter MoE Multimodal LLMs: Architecture Breakdown and Engineering Notes

    Disclaimer: This article is organized from public technical-report summaries. Parameter counts, training pipelines, and reported outcomes should be taken from the original reports. This is a technical interpretation and engineering analysis, not an official benchmark or a product review.

    1. Introduction: Three Problems Meet

    Three trends shaped large-model development over the past year: parameter counts climbing toward the trillion mark, context windows expanding from tens of thousands of tokens to millions, and interaction moving from single-turn Q&A to long-horizon agentic tasks. Stacked onto a single model, point optimizations stop being enough:

    • The more parameters, the sparser the activation — and the more fragile expert routing and numerical stability become.
    • The longer the context, the harder the quadratic cost of standard attention is to absorb.
    • The longer the task horizon, the more the model needs memory and multi-step reasoning.

    A recent public technical report disclosed the design of a trillion-parameter multimodal model (summarized here from its public write-up; the report’s name, issuing organization, and publication date should be confirmed in the original, as should its reported results). The headline figures it discloses: 2.8 trillion total parameters, 104 billion activated parameters, native vision multimodality, and a 1M-token context window. This article is not a product review. It abstracts the reusable technical points from that design, explains what problem each one solves and what it costs, and flags what to watch when porting them into your own systems.

    2. Overview: Core Design at a Glance

    In one sentence: the model uses a trillion-parameter MoE backbone that mixes gated linear attention with latent attention, connects layers through a chunked cross-layer attention residual, and handles extreme sparsity on the MoE side with quantile-based load balancing plus bounded activation. The report claims these choices improve long-context efficiency, stability at very large MoE scale, and long-horizon agentic performance (the specific benchmarks, experimental conditions, and measurement methodology should be taken from the original report).

    Dimension Key information Source
    Model positioning Trillion-parameter MoE multimodal model with native vision input and a 1M-token context Disclosed in report
    Parameter scale 2.8T total parameters, roughly 104B activated Disclosed in report; top-k and the non-expert share are not disclosed
    Attention design Mix of KDA (gated linear attention) and Gated MLA; Block AttnRes shares chunked attention residuals across layers Disclosed in report
    MoE design LatentMoE with 896 routed experts in a latent space; SiTU-GLU activation; Quantile Balancing (QB) for dynamic load balancing Disclosed in report
    Pretraining pipeline Context extended stepwise from 8K to 64K, 256K, and finally 1M Disclosed in report
    Post-training Million-token-scale agentic reinforcement learning (Agentic RL) Disclosed in report

    A note on the order of magnitude of activation sparsity

    Dividing 2.8T total parameters by 104B activated parameters gives an activated-to-total ratio of roughly 1:27. Two caveats are worth stating up front. The report does not disclose the number of experts actually routed per token (top-k), nor the share of non-expert parameters such as attention, embeddings, and the router. So the arithmetic below is a rough order-of-magnitude exercise and cannot be used to back out the real top-k:

    • Dividing “2.8T total parameters by 896 routed experts” naively gives about 3B parameters per expert.
    • But the 104B activated parameters are not all expert parameters — they also include attention, embeddings, and other non-expert components. So “the activated parameters correspond to a few dozen experts” is only a rough analogy at the order-of-magnitude level; the actual top-k should be taken from the original report.

    One robust conclusion does follow: for models like this, most parameters take no part in any given forward pass. The training bottleneck is usually not floating-point throughput itself, but whether routing stays balanced, whether numerics stay stable, and whether experts can be parallelized efficiently. The module designs below are best read through that lens.

    3. Attention Design: From Quadratic Cost to Layered Cooperation

    3.1 Hybrid architecture: linear attention for volume, full attention for precision

    The cost of dense attention grows quadratically with sequence length, and the KV cache grows linearly during inference. At a million tokens, both become bottlenecks. Linear attention uses kernel factorization and associativity to bring complexity down to linear, and at inference time maintains a fixed-size recurrent state per step.

    Linear attention is not free, though. A bounded-size state is good at aggregating global information and comparatively weak at exact long-range retrieval. That is why newer designs generally adopt a hybrid strategy: most layers use linear attention to do the bulk of context compression, while a small number of layers retain full attention for precise retrieval and multi-hop reasoning.

    This model adds gating to both attention types. On the linear branch, KDA introduces an input-dependent decay factor alongside the incremental state update, so the model can forget old state on demand. On the full-attention branch, Gated MLA adds an output gate on top of low-rank KV compression. Combined, they make “long context plus multimodality” a trainable and servable configuration rather than a paper metric.

    3.2 Gated MLA: compress the KV, gate for numerics

    The core idea of Multi-head Latent Attention (MLA) is to compress key-value pairs into a low-dimensional latent vector. At inference time, only that latent representation is cached, which sharply reduces KV cache size; at compute time, per-head keys and values are up-projected from the latent as needed. Gated MLA adds an output gate: each attention head’s output first passes through a per-channel sigmoid gate and is then merged into the residual stream. There are two reasons for this:

    1. It suppresses outlier outputs from a small number of anomalous heads, reducing error amplification in low-precision training and quantized inference.
    2. It lets the model decide, channel by channel, how much of the attention result to adopt, improving representational flexibility in deeper networks.

    The tradeoff: gate parameters add a small amount of overhead. For deep, multi-head architectures, that cost is usually acceptable.

    3.3 Why does the diagonal block overflow numerically? — Lower-bounded decay parameterization in KDA

    The pain point: why the diagonal block was forced onto a per-position path

    Linear attention’s sequential computation can be written as a recurrent update with forgetting:

    1. S_t = α_t ⊙ S_{t-1} + β_t · v_t k_tᵀ # state update
    2. o_t = q_tᵀ S_t # output

    Notation (kept consistent with the pseudocode below):

    • S_t: state matrix of shape [d_k, d_v]; α_t ∈ (0,1) is an input-dependent decay (forgetting) factor, a scalar; β_t is the write strength, a scalar.
    • q_t ∈ R^{d_k}, k_t ∈ R^{d_k}, v_t ∈ R^{d_v} are the query, key, and value vectors of the current token.

    Real implementations exploit GPU matrix-multiply units, so the sequence is typically computed in chunks: state is passed between chunks, which is naturally a matmul. Inside a chunk, the decay between adjacent query-key pairs depends on the product of the intermediate tokens’ α, forming the “diagonal block.” Let g_t = log α_t and let the in-chunk accumulation be Γ_i = Σ_{l≤i} g_l; then element (i,j) of the diagonal block must be multiplied by exp(Γ_i − Γ_j). Because g_t ≤ 0, that factor necessarily lies in (0,1], and with only the upper bound in view it looks numerically safe.

    The problem appears when it is factored into matmul form: exp(Γ_i) and exp(−Γ_j) must then be computed separately, and the latter has a positive exponent. When the decay function has no lower bound — a softplus-style function, for instance, which grows roughly linearly for large inputs — Γ plunges quickly into large negative values, and exp(−Γ) overflows. Under low precision such as BF16, this further degenerates into underflow or loss of significant bits. To dodge the risk, many implementations take a separate per-position (position-pair) path for the diagonal block, using elementwise operations to accumulate the intermediate decays explicitly rather than a matmul. The consequences: two execution paths inside a single attention operator; the diagonal block, a locally dense computation, can no longer be served by matmul instructions, so hardware utilization drops; the kernel becomes more complex and its low-precision behavior harder to predict.

    The mechanism: give the log-decay a fixed lower bound

    KDA rewrites the log-decay from an unbounded negative softplus into a lower-bounded, scaled sigmoid parameterization:

    1. g_t = g_min · Sigmoid(f(x_t)), g_min = -5

    Since the sigmoid output falls in (0,1), g_t is locked inside (g_min, 0), and each token’s decay factor α_t never drops below e⁻⁵ (about 0.0067).

    The benefit: every causal block can use a matmul

    • The special case disappears. The diagonal and off-diagonal blocks can share one unified matmul form, with the intermediate decays absorbed naturally. The report states that under the relevant configurations this improves matmul utilization and throughput (specific numbers, benchmarks, and measurement methodology should be taken from the original report), and the kernel simplifies accordingly.
    • NoPE comes for free. In linear attention, positional information is modeled implicitly through the recursive decay, which decouples it from training length. In theory the model can handle sequences longer than those seen in training without any explicit positional encoding; if inference-time optimizations such as truncation or chunking are applied, extrapolation can continue within a bounded numerical range.

    Illustrative pseudocode

    The pseudocode below illustrates where the decay enters the update and how it lands in the diagonal block. It is not any particular implementation.

    1. # Illustrative pseudocode: for understanding only, not a runnable implementation
    2. # Shape conventions: S:[d_k,d_v] k:[d_k] v:[d_v] (single token) g:scalar G:[chunk]
    3. G_MIN = -5.0 # lower bound on the log-decay
    4. def log_decay(x, w): # decay strength: current token mapped through a linear layer
    5. return G_MIN * sigmoid(x @ w) # equivalent to keeping the forget gate in [0,1)
    6. def delta_update(S, k, v, beta, g):
    7. """Delta rule with forgetting: decay the old state, then write a correction toward the new value."""
    8. S = exp(g) * S + beta * (v - S @ k) @ k.T
    9. return S
    10. def attn_block(q, k, v, g, chunk=16):
    11. """Diagonal block: fold the intermediate decays into two scaling matrices so it becomes a matmul."""
    12. G = cumsum(g).reshape(-1, chunk) # in-chunk cumulative log-decay, shape [..., chunk]
    13. q_scaled = q * exp(G)[..., None] # exp(G) <= 1
    14. k_scaled = k * exp(-G)[..., None] # for bounded chunk length, exp(-G) <= exp(chunk * |g_min|)
    15. A = causal_mask(q_scaled @ k_scaled.T)# lower-triangular mask
    16. return A @ v

    Numerical boundary: chunk size and the decay bound must be designed together

    The real constraint for numerical safety is that the absolute value of the in-chunk cumulative log-decay, chunk × |g_min|, must stay below some “safe log ceiling” L, and L must in turn sit below the representable range of the low-precision format. BF16 and FP32 have comparable exponent bits, with a maximum representable value near 3.4×10³⁸, corresponding to a log ceiling of about ln(3.4×10³⁸) ≈ 88.7. Hence:

    • With g_min = -5 and chunk = 16, chunk × |g_min| = 80, so the scaling factor is bounded by e⁸⁰ ≈ 5.5×10³⁴ — roughly three orders of magnitude (about 6000×) below the BF16 maximum of about 3.4×10³⁸, leaving comfortable headroom.
    • Inverting, chunk ≤ L / |g_min|. With a safe ceiling of L = 80, for example, chunk ≤ 16.

    The key point is that chunk size and |g_min| must be tuned jointly: the closer g_min is to 0, the longer the chunk you can afford; the more negative g_min is, the shorter the chunk must be. Since BF16 has only 7 mantissa bits, avoiding lost significant digits matters as much as avoiding overflow, so in practice the safe ceiling should be set more conservatively than the theoretical limit.

    Applicability boundary

    Constraints always come at a cost. Forcing every token to decay by at least e⁻⁵ limits how well “distant history” is preserved, so exact long-range retrieval still depends on the full-attention layers.

    3.4 Why do deep networks degrade? — Block AttnRes

    Deep Transformers have two long-standing problems. First, the deeper the stack, the more information tends to be diluted as it flows layer by layer. Second, under tensor and pipeline parallelism every layer requires its own reductions and communication, so memory and bandwidth costs accumulate linearly with depth.

    Block AttnRes works by chunking each attention block’s output by type and fusing it, with learnable input-dependent weights, against representations from different layers, so each layer can dynamically select the information useful to it. The gains are threefold: representationally, it mitigates degradation in deep stacks, letting shallow local detail complement deep semantic summaries; resource-wise, it spreads aggregation over fewer communication and memory-access operations, lowering both memory and communication cost; training-wise, cross-layer gradient paths are shorter and more stable, which matters especially at trillion-parameter scale.

    4. MoE Design: Keeping a Thousand Experts Stable and Useful

    4.1 Structure: latent-space experts and bounded gated activation

    LatentMoE: routed experts do not act on the full-dimensional hidden state. Most of the transformation happens in a reduced latent space, after which the result is projected back to the hidden space. This cuts both the parameters and the compute per expert, and it shrinks the write surface of routing. When the expert count scales to 896, total parameters can grow while per-token activation cost grows only modestly.

    SiTU-GLU activation: a variant of the gated linear unit (GLU) that replaces the SiLU gate branch of standard SwiGLU with a sigmoid-based bounded gate.

    Aspect SiLU gate (SwiGLU) Bounded sigmoid gate (SiTU-GLU)
    Gate output range Unbounded; keeps increasing with the input Bounded; typically in (0,1)
    Behavior under extreme sparsity Gate values can swing sharply, amplifying variance in the activation distribution Gate saturates, naturally constraining single-expert output
    Potential benefit Mature, general-purpose default Fewer activation outliers, gradient spikes, and routing oscillations

    In a trillion-scale sparse model, a single expert’s input distribution drifts with the routing outcome. A bounded gate keeps the output bounded even for “extreme” inputs — effectively a soft numerical safeguard inside the module. The cost is a slight contraction of expressiveness, which has to be compensated by techniques such as output rescaling to keep magnitudes comparable to the original activation.

    4.2 Load balancing without an auxiliary loss — Quantile Balancing (QB)

    Conventional MoE relies on an auxiliary load-balancing loss that couples “expert balance” with “language modeling” in one objective. If the auxiliary loss weight is set poorly, it interferes with convergence of the main task — a problem that becomes especially visible in very large-scale, long-duration training. Existing loss-free schemes generally maintain a per-expert updatable bias to steer routing choices, but they typically offer no explicit constraint (bounded or unbounded is left unclear).

    QB’s approach is to impose a quantile constraint directly during routing, based on the historical load distribution. Each expert is assigned a historical quantile in a dynamically maintained load distribution, and expert scores are mapped within the limit corresponding to that quantile, so the gate output naturally falls in a controllable range. It adds no extra loss term, adapts quickly to shifts in expert “specialties,” and suits routing at the scale of 896 experts. It is complementary to gated activation: the gate acts on the activation side, while QB constrains the routing side.

    Note: what follows is a generalized engineering recommendation that turns the report’s idea into something implementable. Concrete values need tuning against your own model and do not represent specific numbers disclosed in the original report.

    Illustrative pseudocode (idea only):

    1. # Illustrative pseudocode: how quantile balancing and bounded gating work together,
    2. # not a runnable implementation.
    3. def route(hidden, experts, load_hist, block_size=128, top_k=8, temperature=1.0):
    4. # 1. Chunk-wise load statistics: estimate each expert's current position in the load distribution
    5. quantile = estimate_quantile(load_hist, block_size) # target quantile (e.g., 0.8)
    6. logits = hidden @ experts.router_weight # routing scores [tokens, n_experts]
    7. limit = quantile_bound(logits, quantile) # bound routing scores by the quantile limit
    8. gate = sigmoid((logits - limit) / temperature) # bounded routing gate
    9. top_idx, top_w = topk(gate, k=top_k) # sparse selection
    10. return top_idx, top_w

    Implementation parameters and cautions:

    • Load-statistics window: use a sliding window (for example, the last N batches). Too short and it jitters; too long and it cannot keep up with drifting expert specialties. Tune it per training stage.
    • Quantile-limit formula: maintain a historical load series per expert, take the threshold at target quantile p (for example 0.8) as limit, and map the logit through a sigmoid to a bounded gate in (0,1). Higher p gives better balance but may suppress head experts.
    • Temperature / top-k: temperature controls how soft routing is — larger values smooth it out. top_k sets the number of experts activated per token and is the core tradeoff between sparsity and quality; match it to the total parameter scale.
    • Cold-start exploration: leave exploration room for newly created or low-load experts (a minimum gate floor, or a small amount of random perturbation), so freshly initialized experts are not marginalized immediately.
    • Cross-device sync frequency: load statistics require cross-device synchronization, so weigh statistic freshness against communication cost. Typically synchronize every few steps rather than every step.

    5. Training Pipeline: Staged Context Extension and Agentic RL

    5.1 Progressive context extension: 8K → 64K → 256K → 1M

    Why stage it: training directly at a million-token length is compute-expensive, and attention matrix size grows quadratically with length. Linear-attention states also need to “learn” from short to long how to allocate capacity. On top of that, high-quality long samples are relatively scarce, so each stage needs control over the long-sample ratio while retaining enough general corpus.

    Adjustments required at each stage:

    1. Data mix: progressively add long documents, code repositories, multi-hop reasoning traces, and interleaved image-text material, and verify that loss declines smoothly on medium-to-long contexts rather than merely “completing a run.”
    2. Parallelism strategy: short-sequence stages rely mainly on combinations of data, tensor, and expert parallelism; long-sequence stages need additional sharding such as sequence parallelism, paired with activation-checkpointing recomputation to control activation memory.
    3. Position handling: the base frequency of positional encoding on the full-attention branch is adjusted stage by stage as length grows; the linear branch relies on decay parameters that learn an appropriate timescale from long-context data.
    4. Training stability: each expansion stage usually needs a brief learning-rate rollback and re-warmup to avoid loss spikes caused by the abrupt length change.

    5.2 Million-token agentic reinforcement learning

    Once the context reaches 1M, the post-training objective shifts from “can it answer” to “can it complete work autonomously over a long task.” The Agentic RL approach mentioned in the report emphasizes three things: multi-turn tool use (learning across long trajectories of the form “read a file — run a command — observe the result”); long-horizon rewards (trajectories can span hundreds of thousands of tokens, with sparse and delayed rewards); and transferable experience (sharing experience across tasks and environments rather than memorizing individual trajectories).

    Three practical levers are worth attention: process rewards (feedback on the soundness of intermediate steps, which eases credit assignment over long trajectories), trajectory filtering (dropping samples that “succeeded by luck” to avoid reinforcing the wrong policy), and sequence management for training and inference (sampling cost and memory footprint of long trajectories directly determine RL iteration speed).

    5.3 Multimodality: native, not bolted on

    Native multimodality means the vision encoder and the language backbone are pretrained jointly, with images expressed as tokens that share the same positional and attention machinery as text. This reduces the conversion loss in image-text alignment and makes “long context plus vision” possible — reading an entire illustrated manual and locating a detail, reasoning continuously inside a long session that contains screenshots, or handling the very long visual sequences produced by video frame sampling. For long contexts, visual token expansion is the main source of pressure; common mitigations include resolution adaptivity, visual token compression, and dynamic sampling.

    6. Engineering View: Three Highly Reusable Pieces and What to Watch

    The three most reusable techniques in the report all share a property: a local change with global benefit.

    6.1 Lower-bounded decay parameterization (KDA)

    • Porting cost: usually just a change to how decay is parameterized inside the linear-attention kernel.
    • Benefit: removes the special-case diagonal-block branch and, in principle, improves matmul utilization (measure on your own configuration).
    • Cautions: the new numerical range requires recalibrating low-precision inference and quantization strategies; when choosing a bounded decay kernel, match the chunk length to the safe boundary; for short-context workloads, run an ablation first to confirm the bounded decay causes no perceptible loss.

    6.2 Quantile Balancing (QB)

    • Porting cost: add load statistics and quantile estimation at the routing layer, and compare against your existing load-balancing scheme.
    • Benefit: removes the auxiliary loss term and the interference it creates with the main task.
    • Cautions: window length affects how fast balancing converges; cross-device sync frequency must be controlled; long training runs need continuous monitoring of per-expert activation rates to catch “zombie experts” and “over-dominant gates.”

    6.3 Block AttnRes

    • Porting cost: change the inter-layer wiring and the computation-graph structure of the training framework.
    • Benefit: breaks the deep-representation bottleneck and lowers memory and communication cost.
    • Cautions: cross-layer representations must be kept for the backward pass, so an activation-checkpointing strategy is mandatory; verify consistency under gradient accumulation and pipeline parallelism; initialize the cross-layer fusion weights sensibly to avoid early-training oscillation.

    6.4 Deployment notes on the cloud (general capability view)

    1. Training side: trillion-parameter MoE training requires mature container orchestration, high-performance networking, and distributed storage working together. General capabilities offered by mainstream cloud platforms — container services, object storage, monitoring and alerting — can be used to build elastic scheduling, checkpoint persistence, and anomaly-detection paths for training jobs (this is a generic description of capabilities, not a reference to a specific vendor).
    2. Inference side: linear attention’s fixed-size state, latent KV compression, and chunked residuals are useful reference designs for inference services. For mixed long/short request traffic, KV cache management, and memory reuse, plan resource isolation and capacity carefully.
    3. Evaluation side: build scenario-specific evaluation sets for long context, multimodality, and agentic tasks, so a single benchmark score does not hide real capability differences.

    7. A Sober Reading: Strengths, Boundaries, and Selection Advice

    7.1 Three structural strengths

    1. Precise problem framing: the three core designs each target a real pain point in engineering practice — diagonal-block kernel implementation, load balancing in very large MoE, and deep representation plus communication cost.
    2. System-level coherence: mixing linear and full attention, pairing gated activation with quantile balancing, and optimizing inter-layer communication via chunked residuals all serve the same objective — “trillion-scale sparsity plus million-token context” — rather than being stacked on top of each other.
    3. Full-lifecycle thinking, training through inference: staged context expansion in pretraining, long-horizon RL in post-training, and native multimodality together form a complete engineering loop.

    7.2 Boundaries to keep in mind

    • Parameters ≠ capability: 2.8T total parameters does not mean every parameter is used at every step; the actual behavior is determined by how the few dozen activated experts combine.
    • Context ≠ effective memory: a 1M window does not mean any detail at any position is retrievable. The state-compression characteristics of linear attention mean exact long-range retrieval still depends on the full-attention layers.
    • Architecture ≠ tuning outcome: the report’s stability conclusions rest on a specific data mix and hyperparameter set, and may not carry over linearly to your own data distribution.
    • Benchmarks ≠ business results: run A/B tests on your own task distribution, focusing on the stability and cost curve of long-trajectory tasks rather than leaderboard position.

    7.3 Selection and adoption advice

    Scenario What matters Recommendation
    Long-document / code-repository analysis Context length, retrieval precision Check the number and placement of full-attention layers; measure long-range retrieval accuracy
    Multimodal image-text understanding Quality of joint image-text pretraining Evaluate interleaved image-text, chart parsing, resolution adaptivity, and similar dimensions
    Agentic tasks Tool use, long-horizon planning Look at the data scale and reward design behind Agentic RL, not a single task’s pass rate
    Cost and capacity planning Activated parameters, KV compression ratio Estimate inference cost from activation scale rather than total parameters
    On-premises / cloud deployment Engineering retrofit cost Assess how the linear-attention operator, expert routing, and cross-layer communication fit your existing framework

    7.4 FAQ

    • How do I estimate KV cache quickly? Under dense attention, KV cache ≈ layers × 2 × sequence length × number of KV heads × head dimension × bytes per element. With MLA, estimate from the low-rank latent dimension instead, which typically brings a substantial reduction; the linear-attention branch maintains a fixed-size recurrent state that is decoupled from length.
    • How do I diagnose zombie experts? Monitor per-expert activation frequency and load variance; zero or near-zero activation over a long period marks a candidate zombie expert. Handle it with QB-style cold-start exploration, load reallocation, or expert resets.
    • How do I validate long-context retrieval accuracy? Use needle-in-a-haystack-style probes that place target information at different depths and lengths and check recall. Also evaluate multi-hop and distractor-resistant variants rather than testing only single-point insertion.

    8. Conclusion

    A model of this kind — “trillion-parameter MoE multimodal” — is best understood on three levels. Engineering: lower-bounded decay parameterization, quantile balancing, and Block AttnRes represent the current mainstream thinking in long-context sparse-model optimization, trading structured constraints for engineering feasibility. Architecture: the combination of linear attention, latent attention, and residual structures reflects the industry’s ongoing search for balance between capacity and efficiency. Ecosystem: progressive context extension and long-horizon agentic RL show that the training pipeline itself is becoming part of the model’s capability.

    For developers and architects, what matters more than “how big the model is” is the reasoning behind these tradeoffs: under a fixed compute and memory budget, how to obtain the largest stability gain from the smallest change. Understanding these constraints and tradeoffs is what makes a sound technical choice possible in your own business context. (Once more: the figures and conclusions in this article should be taken from the original public technical report.)