Running a 2.8-Trillion-Parameter Model Without a GPU Using Persistent Memory

    This article describes a CPU-based setup that runs a 2.8-trillion-parameter model without a GPU. It uses large-capacity persistent memory as a warm weight tier alongside a DRAM cache. The article covers the memory capacity problem, a three-tier memory hierarchy, quantization and on-demand expert activation, measured token speeds, the reasons throughput stays in the single digits, and the practical limits of this approach for offline or low-concurrency workloads.

    1. The Problem: A Capacity Wall Beyond GPU Memory

    Large-scale model inference is usually treated as a GPU memory problem. However, when a model reaches 2.8 trillion parameters, the weights alone remain very large, even at extremely low bit widths. A rough 1-bit estimate puts 2.8 trillion parameters at about 350 GB of raw weight data. At 3-bit quantization, the total approaches 1 TB. Runtime buffers, token caches, intermediate activations, and framework overhead add to that figure.

    For many home labs and cost-sensitive environments, the constraint is not only compute. Capacity is also insufficient, and expanding it is expensive. GPU memory is limited, and its expansion is constrained by hardware topology. General-purpose server CPU platforms, however, can reach a larger address space through high-capacity memory modules and persistent memory. This leads to an approach that trades capacity for throughput and time for feasibility. Instead of targeting real-time interaction, it tests whether a very large model can complete inference without a GPU.

    The value of this kind of experiment is not in replacing high-throughput GPU clusters. It shows that memory hierarchy design, quantization, and CPU inference software can work together as a complete system under extreme capacity limits. It also makes one point clear: the engineering limits of LLM inference depend not only on compute, but also on memory capacity, memory bandwidth, and cache hit rate.

    2. System Architecture: CPU, Persistent Memory, and DRAM Cache

    The core hardware in this experiment is a general-purpose multi-core server processor paired with large-capacity persistent memory. Persistent memory, also called non-volatile memory, sits between DRAM and local storage. Its capacity is far larger than DRAM, its data is retained after power loss, and its latency and bandwidth are better than those of ordinary storage media. It suits cold or warm weights, while DRAM serves as the hot-data cache.

    The inference system can be viewed as a three-tier memory hierarchy:

    Tier Media Type Main Role Key Risk
    Hot tier DRAM Caches frequently accessed layers, active experts, and recently used tensors Limited capacity; the cache replacement policy has a large impact
    Warm tier Persistent memory Holds most model weights and provides a large address space Higher latency and lower bandwidth than DRAM; NUMA topology needs attention
    Cold tier Local storage Holds compressed model packages, backups, and cold data Poor random-read performance; should not be swapped in and out frequently

    The experiment runs inference on a multi-core CPU, so memory access patterns matter a great deal. When threads access remote NUMA nodes, latency rises noticeably. When the weight layout does not match the memory channels, bandwidth utilization drops. In practice, engineers should use thread binding, memory binding, page size tuning, and sequential prefetching where possible, so the weights needed for each matrix multiplication are read contiguously by layer or by expert.

    The following pseudocode illustrates the approach:

    1. # Pseudocode: tiered weight access and quantized inference
    2. weights = {
    3. "hot": DRAM_CACHE,
    4. "warm": PERSISTENT_MEMORY,
    5. "cold": LOCAL_STORAGE
    6. }
    7. def get_tensor(layer_id, expert_id):
    8. key = build_key(layer_id, expert_id)
    9. if key in hot_cache:
    10. return hot_cache[key]
    11. if key in persistent_memory:
    12. tensor = persistent_memory.read(key)
    13. if is_hot(key):
    14. hot_cache.put(key, tensor)
    15. return tensor
    16. return load_from_storage(key)
    17. def decode_step(input_ids):
    18. x = embed(input_ids)
    19. for layer in model.layers:
    20. active_experts = select_active_experts(layer, x)
    21. for expert in active_experts:
    22. w = get_tensor(layer.id, expert.id)
    23. x = quantized_matmul(x, w)
    24. return sample_token(x)

    The key idea in this logic is on-demand reading. For each token generated during autoregressive decoding, the system accesses only the weights needed for the current layer and the active path. It does not load all parameters into DRAM at once. For highly sparse, very large models, this on-demand access can significantly reduce pressure on hot data. However, it moves the bottleneck to random memory access and cache hit rate.

    3. Why It Runs: Quantization, Compression, and On-Demand Activation

    Running a 2.8-trillion-parameter model without a GPU starts with reducing the model’s size. 1-bit and 3-bit quantization are common approaches. The 1-bit approach compresses weights to very low bit widths and greatly reduces capacity, but its quality loss and recovery cost must be evaluated. The 3-bit approach needs more capacity but usually preserves numerical precision better.

    Raw experiment data shows that with 1-bit quantization and compressed model data, the initial generation speed was about 0.85 tokens per second. After turning off direct-access mode and letting frequently used data benefit from the DRAM cache, the speed rose to 1.50 tokens per second. A separate 3-bit quantization test reached about 0.75 tokens per second, and generating 50,000 tokens took about 18.5 hours. These results show two things. First, extremely low-bit quantization is the prerequisite for the model to run at all. Second, the caching strategy has a direct effect on performance.

    Direct-access mode presents persistent memory to software as a separate data space. This reduces some copy and mapping overhead, but frequently used data may not fully use the DRAM cache. With direct-access mode off and the cache layer enabled, hot data hits DRAM, and token speed nearly doubles. This is a typical trade-off. Direct-access mode is predictable and has a short data path. Cache mode depends on data locality, but when the hit rate is high, latency drops noticeably.

    Very large models often use sparse activation, so each token passes through only some layers or some experts. On-demand activation means the system does not scan all weights at every step. It does create an uneven expert heat problem. A few popular experts are accessed often and suit DRAM residency, while long-tail experts are accessed sparsely and can stay in persistent memory. In practice, placement can follow access frequency rather than simply splitting the model by layer.

    4. Performance: Why Token Speed Is in the Single Digits

    At 0.75 to 1.50 tokens per second, each token takes roughly 0.67 to 1.33 seconds. That speed is not suitable for real-time conversation. It does work for offline generation, long batch jobs, feasibility checks, and low-concurrency experiments. Four factors limit performance:

    1. Memory bandwidth limits: Each CPU inference step reads large amounts of weight data. Persistent memory has lower bandwidth than DRAM, so matrix multiplication is easily bandwidth-bound.
    2. Access latency and randomness: On-demand reads of expert weights create irregular access patterns. When the cache misses, latency increases sharply.
    3. Quantization decoding overhead: Low-bit weights must be unpacked, dequantized, or looked up before computation. The CPU instruction overhead is not negligible.
    4. Cache replacement cost: DRAM capacity is limited. A poor eviction and loading policy causes cache thrashing.

    The optimization goal, therefore, is not simply adding CPU cores. It is raising effective bandwidth and cache hit rate. Some practical steps include:

    • Keep high-frequency layers and popular experts pinned in DRAM.
    • Reorder weights by access order.
    • Use large pages to reduce address translation overhead.
    • Keep batch sizes reasonable so the KV cache does not crowd out the weight cache.
    • Monitor the share of remote NUMA accesses and the page fault rate.

    5. Deployment Guidance and Scope

    If you want to reproduce or adapt this approach, define the goal first. Decide whether you need to confirm that a model can run at all, or whether you need usable throughput. The first goal accepts extremely low-bit quantization and jobs that run for hours. The second requires a much more careful evaluation.

    Capacity planning should account for model weight size, runtime buffers, token cache, and a safety margin. 1-bit quantization saves space, but its quality risk must be verified with an evaluation set. 3-bit quantization gives more stable quality but needs more capacity and higher bandwidth. For caching, a tiered layout works well: frequently used layers stay in DRAM, cold layers remain in persistent memory, and cold backups go on local storage. Record the cache hit rate, memory bandwidth, page faults, CPU utilization, and token latency percentiles.

    This approach fits offline batch processing, low-concurrency long-text generation, teaching experiments, and cost-sensitive feasibility checks. It is not suitable for high-concurrency online services, low-latency interaction, or strict real-time workloads. Also consider the supply and firmware compatibility of discontinued hardware, the reliability of persistent memory, ECC and data consistency, long-running stability, and whether the quality of a quantized model meets your