From Training to Inference: How LLM Compute Breaks Through the Memory Wall

1. The Structural Shift: From Training-Heavy to Inference-Heavy Workloads

Over the past few years, investment in large model computing has been concentrated almost entirely on the training side: larger parameter counts, more training data, and higher cluster throughput. However, as models move into production environments, especially with the spread of generative AI and AI Agent applications, the industry is undergoing an irreversible shift: inference is becoming the dominant form of compute consumption.

This shift leads to an important conclusion: training and inference place different demands on hardware. Training aims to process massive datasets in a given period of time and therefore emphasizes throughput. Inference, especially online inference for user-facing interactions, aims to produce tokens quickly, smoothly, and cost-effectively. Optimizing both goals within a single architecture often results in a suboptimal solution.

2. Core Metrics in the Inference Era: Latency, Throughput, and Cost

In generative inference, several metrics directly determine whether a commercial application can succeed:

• TTFT (Time To First Token): The time from request arrival to the first generated token. This determines the response speed perceived by users.
• ITL (Inter-Token Latency): The interval between adjacent tokens. This determines whether the output stream feels smooth.
• Throughput (tokens/s): The total number of tokens produced by the system per unit of time. This directly affects the cost per token.
• Energy efficiency (tokens produced per joule): In large-scale deployment, this directly affects operating costs and sustainability.

AI Agent workflows further increase the importance of these metrics. A single agent task may require dozens of tool calls and iterative reasoning steps within seconds. Any high latency introduced by one call can be amplified through serial execution. Therefore, for online business systems, deterministic low latency is often more important than peak compute.

3. The Memory Wall: The Architectural Bottleneck of Autoregressive Decoding

To understand why traditional massively parallel architectures are not always optimal for inference, it is necessary to first understand the memory wall.

Autoregressive decoding generates tokens one by one. Each time a new token is generated, the system must read the entire model weights and the continuously growing KV cache. In this process:

• Arithmetic intensity is very low: Each weight read corresponds to only a small number of multiply-accumulate operations.
• The bottleneck shifts from compute to memory bandwidth: Many floating-point compute units remain idle, while data movement becomes the dominant overhead.
• The KV cache grows with context length, further increasing bandwidth pressure and consuming memory capacity.

As a result, during the decoding phase, expensive high-performance compute units are underutilized. This raises latency and reduces energy efficiency. The memory wall is not simply a problem that can be solved by adding more compute. It is a mismatch between data movement patterns and hardware architecture.

4. The Limits of Massively Parallel Architectures

Massively parallel accelerators, represented by GPUs, are nearly indispensable during training. With thousands of compute cores and high-bandwidth memory, they can maximize hardware utilization for compute-intensive, large-batch, highly parallel workloads.

However, their advantages are reduced during inference decoding:

• Decoding is memory-bound rather than compute-bound, leaving many compute units idle.
• Flexible cache hierarchies and scheduling mechanisms, designed for general-purpose workloads, introduce additional latency.
• Designs optimized for throughput are not naturally suited to online services that are sensitive to tail latency.

In short, the architecture that is best for training is not necessarily the architecture that is best for inference.

5. Deterministic Dataflow Architectures: Designed for Token Generation

In contrast to the approach of simply adding more parallel compute, another path is compiler-centric, software-defined hardware. A compiler maps the computation graph onto the chip statically, allowing data to flow along deterministic paths and minimizing access to external memory.

These inference-oriented architectures, often referred to in the industry as dataflow architectures or deterministic architectures, typically focus on the following design points:

• Large on-chip SRAM: Keeping model weights and activations on-chip as much as possible reduces reliance on HBM/DRAM and directly mitigates the memory wall.
• Deterministic scheduling: Predictable execution timing is well suited to token generation with low latency and low jitter.
• Compile-time optimization: Operator fusion and dataflow orchestration are completed during compilation, reducing runtime overhead.

The trade-off is reduced flexibility. Such architectures are more sensitive to specific model structures and operators and are less general than GPUs. Therefore, they are not a replacement for GPUs, but a complement to them.

6. Heterogeneous Collaboration: Running the Right Workload on the Right Chip

The more valuable question is not “which architecture should be used,” but “how can different architectures work together?” A typical approach is to split the inference pipeline into two stages and optimize each separately:

• Prefill: Processes the input context. This stage is compute-intensive and parallelizable, making it suitable for massively parallel architectures to reduce TTFT.
• Decode: Generates tokens one by one. This stage is memory-intensive and latency-sensitive, making it suitable for deterministic low-latency architectures to reduce ITL.

A simplified scheduling logic can be expressed as follows:

def serve_request(request):
    # Prefill stage: compute-intensive; assign to high-throughput parallel architecture
    kv_cache = parallel_engine.prefill(request.prompt)
    # TTFT is mainly determined by this stage
    first_token = parallel_engine.first_token(kv_cache)
    # Decode stage: memory-intensive; assign to deterministic low-latency architecture
    tokens = [first_token]
    while not done(tokens):
        next_token = deterministic_engine.decode(kv_cache, tokens)
        tokens.append(next_token)  # Each token determines ITL
    return tokens

This idea of prefill/decode disaggregation is essentially cluster-level workload profiling: routing each type of workload to the hardware that handles it best. This can optimize overall TTFT, ITL, throughput, and cost, rather than pursuing extreme performance on every metric with a single type of hardware.

7. Engineering Recommendations for Deployment

1. Design from metrics: First define the business targets for TTFT, ITL, throughput, and cost, then choose the architecture accordingly. Avoid a compute-only mindset.
2. Focus on end-to-end performance, not peak scores: A single-chip benchmark result does not equal online service quality. Tail latency, such as P99, is often the real bottleneck for user experience.
3. Use batching and caching effectively: Techniques such as continuous batching, KV cache reuse, and quantization can significantly improve inference efficiency without changing the hardware.
4. Keep the architecture evolvable: Inference workloads and model structures are changing rapidly. The system should be able to schedule and switch between different acceleration forms.

For developers and cloud users, these principles apply in the same way. Whether a deployment uses general-purpose GPU resources, dedicated inference acceleration resources, or cloud-hosted elastic inference and model services, the core objective remains the same: systematic optimization for stable token production with lower latency, higher throughput, and lower cost, rather than blindly chasing a single hardware parameter.