Inference Cost, MoE Deployment, and Reverse-Engineering Model Architecture

A technical discussion based on blackboard equations for large-model training and inference reveals an important fact: the architectures, costs, and pricing models of leading AI systems are not isolated product decisions. They are jointly constrained by accelerator hardware, interconnect topology, batch scheduling, and model-parallel execution strategies. This article uses first-principles reasoning to organize these constraints into a reusable technical framework.

1. The First Principles of Inference Cost: Why Batching Creates Order-of-Magnitude Differences

Large-model inference is generally divided into two stages: prefill and decoding. Prefill is dominated by parallel computation, while decoding generates tokens sequentially and is strongly affected by memory bandwidth, KV-cache access, and scheduling overhead.

When an accelerator serves only one user request at a time, its compute capacity is often poorly utilized. A significant portion of execution time is spent reading weights, accessing caches, and waiting for communication. Grouping multiple requests into batches increases arithmetic intensity. The same weight reads can then serve more tokens, spreading fixed costs across a larger workload and reducing the cost per token.

A key conclusion from the original discussion is that processing requests without batching can make the cost of a single inference up to 1,000 times higher. This figure is not necessarily an exact production benchmark; it illustrates how strongly batching can affect the cost structure, far beyond what individual kernel optimizations can achieve. A simplified cost model is:

def cost_per_token(batch_size, compute_cost, memory_cost, scheduling_cost):
    # Illustrative model: batching amortizes fixed costs,
    # while latency constraints limit the maximum batch size.
    return (compute_cost + memory_cost + scheduling_cost) / batch_size

A real system must also account for time to first token, tail latency, KV-cache capacity, and the distribution of request lengths. Continuous batching, paged caching, priority queues, and preemptive scheduling are all designed to enlarge the effective batch size while keeping latency within an acceptable range.

The reason a fast-serving mode may carry a price premium greater than its speed improvement is that it sacrifices batching scale and resource reuse in exchange for lower queueing latency and higher priority. In contrast, a slower mode can use larger batches, lower priority, preemptive scheduling, and longer queueing windows to reduce the unit price.

For example, if a fast mode costs six times more while being only 2.5 times faster, the difference demonstrates that low latency is not free. It disrupts the cost-optimal batching curve.

2. Pretraining Data Volume: Why It Can Exceed the Theoretical Optimum

Under a fixed compute budget, model size and training-token volume have an empirically optimal relationship. However, some frontier models may use as much as 100 times the data volume suggested by a theoretical optimum.

This may appear wasteful, but it can reflect several engineering and product considerations:

• More data can improve knowledge density and generalization.
• Overtrained models may be more cost-efficient during inference.
• A stronger base model provides a better foundation for reinforcement learning, distillation, and domain adaptation.
• Data quality, repeated training, and curriculum-learning strategies can change the effective value of the training data.

In other words, a theoretical scaling rule answers the question, “How can a model be trained most efficiently under a given compute budget?” A production model must also answer, “How can one training run support large-scale, low-cost serving over the long term?”

Pretraining data volume, model size, and inference cost therefore need to be optimized together rather than evaluated using a single training-stage metric.

3. The Core MoE Tension: Expert Activation Versus All-to-All Communication

A Mixture-of-Experts architecture uses routing so that each token activates only a subset of the available experts. One class of MoE models may contain 256 experts while activating only a small fraction—such as 32 experts—for each inference step. This makes it possible to maintain a large total parameter count while controlling the computation required per token.

The cost is communication. Tokens must be routed to the devices hosting the selected experts, creating an all-to-all communication pattern. Any accelerator may need to send tokens to other accelerators. This pattern naturally fits high-bandwidth interconnect domains within a rack. Once experts are distributed across racks, some tokens must travel over a slower general-purpose network.

A key physical boundary discussed in the original analysis is that a typical high-bandwidth, in-rack interconnect domain may contain around 72 accelerators, while cross-rack communication can be approximately eight times slower. This difference directly limits the scalable size of an expert layer.

If experts are distributed across two racks, roughly half of the token traffic may cross the rack boundary. The slower link can quickly become the bottleneck. As a result, rack size places an upper bound on how far an expert layer can scale. MoE deployment therefore cannot focus only on increasing the number of experts; it must also account for network topology.

4. Deployment Strategies Under Rack-Level Constraints

Several strategies are commonly combined to manage rack boundaries:

• Expert grouping and topology-aware routing: Place frequently communicating experts within the same high-bandwidth domain to reduce cross-rack all-to-all traffic.
• Hierarchical all-to-all communication: Aggregate traffic within a rack, exchange data across racks, and then redistribute it within each rack to reduce pressure on slower links.
• Redundant experts and load balancing: Replicate hot experts across multiple racks to reduce routing skew.
• Capacity factors and token-dropping policies: Limit the number of tokens assigned to each expert so that long-tail traffic does not dominate tail latency.
• Hybrid parallelism: Combine data, tensor, pipeline, and expert parallelism, mapping different communication patterns to the appropriate levels of the network topology.
• Prefill-decoding separation and elastic scheduling: Deploy prefill and decoding in separate resource pools, then select batch sizes and priorities according to the required SLA.

The shared objective is to place high-frequency communication on high-bandwidth links while restricting slower links to traffic that can be aggregated, compressed, or delayed without violating service requirements.

The observation that rack size limits expert-layer scale is fundamentally a requirement for co-designing model architecture, parallelism, and physical topology. Some teams may achieve pretraining breakthroughs earlier not because of a single algorithmic advantage, but because their training clusters, interconnects, compilers, and model-parallel strategies reached a mature state sooner through joint optimization.

5. Inferring Architecture and Scheduling from API Pricing

Public pricing provides a useful window into underlying architectural constraints. If a fast mode costs six times more but is only 2.5 times faster, the low-latency tier likely requires reserved capacity, smaller batches, and higher scheduling priority. Its cost per token therefore rises naturally.

A slower mode can reduce costs through larger batches, lower priority, preemption, and longer queueing windows. The pricing difference reflects different SLAs rather than a simple change in the price of the same model.

In practice, developers can build their own cost models through load testing and observability. Useful measurements include:

• Batch size
• Time to first token
• Tail latency
• Cache hit rate
• Cross-rack traffic ratio
• Expert load-balance quality

These measurements can then be combined with the cost of accelerator capacity to evaluate alternative scheduling policies.

On Baidu AI Cloud, developers can use general-purpose capabilities such as container platforms, object storage, logging, and monitoring alerts to build evaluation environments. The core engineering challenge, however, remains topology-aware scheduling, batching policy, and capacity planning.

6. Conclusion

The architecture, cost, and pricing of large models ultimately reflect several physical constraints:

• Batching determines the inference cost curve.
• Data allocation affects long-term serving efficiency.
• MoE all-to-all communication is constrained by the high-bandwidth domain within a rack.
• Cross-rack bandwidth differences limit the scalable size of expert layers.
• Fast and slow serving modes represent a trade-off between SLA requirements and batching efficiency.

Understanding these constraints helps explain why modern large-model systems have their current architectures and pricing structures. It also provides a practical basis for making better deployment, parallelism, and scheduling decisions.