High Bandwidth Flash (HBF): Can It Solve the AI Memory Wall?

1. The Real Bottleneck in AI Compute: The Memory Wall

Over the past few years, the development of large models has long focused on stacking compute resources: more compute units, higher clock speeds, and greater parallelism. However, as model parameter counts and context lengths continue to grow, the system bottleneck has shifted from “being too slow to compute” to “being unable to feed data fast enough.” When a flagship accelerator can provide hundreds of TFLOPS, the memory subsystem must move data to the compute units at a matching speed. Otherwise, a large amount of expensive compute capacity will remain idle.

The memory wall is essentially a dual constraint of bandwidth and capacity. Today’s mainstream High Bandwidth Memory (HBM) achieves high-density interconnect by vertically stacking multiple DRAM dies and using through-silicon vias (TSVs). It performs strongly in bandwidth, with a single stack capable of delivering data throughput ranging from hundreds of GB/s to TB/s-class performance. However, the core weakness of HBM lies in capacity and cost: the amount of HBM that can be installed on a single accelerator card is limited, and the cost per unit of capacity is high. As a result, HBM alone is difficult to fully support the complete workload of ultra-large models.

2. KV Cache: The Underrated Capacity Killer

For autoregressive large language models, inference needs to cache the key and value tensors of historical tokens, known as the KV cache, to avoid repeated computation over historical sequences. The size of the KV cache is proportional to the number of model layers, the key/value dimension of attention heads, the sequence length, and the batch size. When the context window expands from thousands of tokens to millions, the KV cache grows sharply and may even exceed the model weights themselves.

Take a typical open-source model with hundreds of billions of parameters as an example. At FP8 precision, the model weights are on the order of hundreds of GB. When the context expands to one million tokens, the precomputed KV cache alone may reach hundreds of GB. If the context is pushed further to tens of millions of tokens, cache requirements can climb to the TB level. The following illustrative code helps estimate the scale of the KV cache:

def kv_cache_bytes(num_layers, num_kv_heads, head_dim,
                   seq_len, batch, dtype_bytes=1):
    """
    Estimate the KV cache size in bytes.
    The factor 2 corresponds to the Key and Value tensors.
    """
    per_token_per_layer = 2 * num_kv_heads * head_dim
    return per_token_per_layer * num_layers * seq_len * batch * dtype_bytes
# Example: dozens of layers, several KV heads per group, million-token context
size_gb = kv_cache_bytes(
    num_layers=126, num_kv_heads=8, head_dim=128,
    seq_len=1_000_000, batch=1, dtype_bytes=1
) / (1024 ** 3)
print(f"KV cache ≈ {size_gb:.1f} GB")

This is a concrete representation of the memory wall. If HBM alone had to hold such massive caches, it would require stacking dozens of accelerator cards, and cost and power consumption would increase linearly or even superlinearly.

3. The HBF Concept and Its Promise

In response to the capacity gap, the industry has proposed High Bandwidth Flash (HBF), an attractive idea: if HBM can achieve high bandwidth by vertically stacking DRAM, can the same TSV stacking process be used to stack NAND flash dies into an “HBM-like” form?

Under this concept, HBF aims to provide bandwidth approaching HBM while leveraging NAND flash’s much higher storage density than DRAM to achieve an order-of-magnitude capacity increase. Even more attractive is the cost: the cost per bit of NAND flash is typically only a fraction of DRAM or HBM. If mature stacking and packaging processes can be reused, HBF may raise available capacity by an order of magnitude while maintaining similar bandwidth. For capacity-constrained inference scenarios, especially KV cache offloading for long-context and large models, this looks like a tailored solution. Some storage vendors have claimed they will introduce samples soon, with the goal of achieving commercial deployment in the near future.

However, turning a seemingly perfect solution into reality is far more complex than paper analysis. A closer examination of related architecture research shows that HBF still has considerable distance to practical use.

4. From Ideal to Reality: Reexamining HBF

4.1 Limits of the Read-Only Workload Assumption

Many HBF performance projections are based on the assumption that flash mainly carries read-only data. For example, frozen model weights or historical KV caches may be written to flash, and inference only requires sequential reads. The problem is that real large-model inference is not always read-only.

In scenarios such as dynamic batching, paged KV cache management, frequent cache swap-in and swap-out, and alternating prefill and decode phases, cache content is continuously written, overwritten, and reclaimed. These operations require the underlying medium to support strong random write and update capabilities. However, flash is known for “erase-before-write,” block-based operations, and significant write amplification. Once the workload shifts from read-only to mixed read/write, HBF’s performance and endurance models need to be substantially revised.

4.2 The Physical Latency Gap of NAND

A more fundamental obstacle comes from the physical characteristics of NAND. The access latency between DRAM and NAND differs by about one to two orders of magnitude. DRAM is typically in the nanosecond range, while NAND is in the microsecond or even millisecond range. This means that even if bandwidth can be raised through stacking, latency cannot be compensated for by stacking more layers.

For inference workloads that are latency-sensitive and have scattered access patterns, tail latency can be amplified stage by stage. When the decode phase requires accessing the KV cache for each generated token, any slow access that falls to NAND may slow down the entire inference pipeline. As a result, “high bandwidth” may be offset by “high latency” in the end-to-end experience.

4.3 Hidden Costs of SRAM Buffers and Complex Controllers

To hide NAND’s high latency, HBF must place a sizable high-speed buffer in front of the medium. This usually involves tens of MB of SRAM as a read cache, a certain amount of DRAM as an intermediate layer, and a complex controller with address mapping, garbage collection, wear leveling, prefetching, and error correction capabilities.

This patch directly erodes HBF’s initial cost promise:

• Large-capacity SRAM is expensive in advanced process nodes and consumes scarce chip budget.
• Multi-level storage hierarchies introduce additional data movement and consistency maintenance overhead.
• A complex controller increases design difficulty, verification cost, and power consumption.

In other words, to make “cheap NAND” actually perform well, a large amount of expensive high-speed storage and control logic must be added around it. This significantly weakens the cost advantage per unit of effective capacity and creates an internal contradiction with HBF’s original selling point of cheap NAND.

4.4 Yield, Thermal, and Reliability Barriers

From a manufacturing and engineering perspective, HBF also faces multiple practical challenges:

• Yield: The more layers are stacked, the higher the probability that a single die defect causes the entire chip to fail. The yield and cost model for stacked flash is not optimistic.
• Thermal: High stacking density causes heat concentration, while NAND is temperature-sensitive. High temperatures can worsen data retention degradation and increase error rates, making thermal design more difficult than for logic-oriented HBM.
• Reliability: Flash wear, data retention characteristics, and error rates under frequent long-term read/write operations require rigorous validation for inference scenarios that run online for long periods.

These engineering barriers mean that HBF’s path from samples to mass production requires more time and uncertainty than expected.

5. The Race Among Alternative Solutions

The biggest challenge for HBF does not come entirely from the technology itself. Alternative solutions are maturing faster:

• Memory expansion and pooling interconnect: Cache-coherent memory expansion and pooling interconnect technologies allow large amounts of DRAM to form memory pools outside the host, expanding capacity at lower cost, with rapid ecosystem and standardization progress.
• Next-generation high-bandwidth memory: HBM iteration continues, with steady improvements in per-stack capacity and bandwidth. Finer stacking and packaging processes continue to relieve capacity pressure.
• Software-layer optimization: KV cache quantization, paged management, cache compression and eviction strategies, operator fusion, and scheduling optimization can significantly reduce memory usage without replacing hardware.
• Tiered storage and offloading: Offloading cold data to host memory or persistent storage, combined with intelligent prefetching, can achieve near-infinite capacity expansion at the software level.

For most deployment teams, software optimization and mature interconnect standards offer a higher return on investment and faster deployment. This further compresses HBF’s market window.

6. Conclusion

HBF addresses a real problem: the memory capacity bottleneck in AI inference. Its direction of using NAND to trade for capacity and cost is indeed valuable. However, from an engineering reality perspective, it still needs to cross the physical latency gap of NAND, the applicability boundary of read-only assumptions, the hidden costs introduced by SRAM and controllers, and manufacturing barriers such as yield, thermal behavior, and reliability. At the same time, memory pooling, next-generation HBM, and software optimization are approaching the same goal through shorter paths.

Therefore, a more rational judgment is that HBF deserves continued attention as an exploratory technology, but it is unlikely to become a mainstream solution in the short term. For teams that want to relieve the memory wall, starting with software methods such as KV cache optimization, quantization, and tiered offloading can often provide meaningful capacity and cost benefits today. Whether HBF can find a real balance among latency, reliability, and cost will be the key to its commercial success.