Running Trillion-Parameter LLMs on CPUs with Persistent Memory: A Deep Dive

    This article examines a CPU-only inference approach that uses discontinued second-generation persistent memory (PMem 200 series) to hold trillion-parameter model weights. It covers the memory capacity and cost tradeoffs, byte-addressable mapping through App Direct mode, mandatory active cooling, measured throughput under 1-bit and 3-bit quantization, and offline use cases where low latency is not required. It concludes with engineering lessons on heterogeneous memory tiers, matching quantization to the workload, and evaluating physical constraints.

    1. Background: The Memory Capacity Bottleneck in LLM Inference

    With the arrival of trillion-parameter models, the barrier to inference deployment has shifted from raw compute to memory capacity. Take a 2.8-trillion-parameter model as an example: even with aggressive low-bit quantization, loading the model weights into memory still requires about 610 GB. That is only the weights. Once KV cache, activations, and runtime overhead are included, actual requirements are even higher.

    Traditional approaches usually follow one of two paths: stack multiple high-end GPUs to increase VRAM, or add many DDR memory modules to expand capacity. With memory prices continuing to rise, building more than 600 GB of capacity from DDR4/DDR5 is extremely expensive. Multi-GPU setups often start at tens of thousands in cost, which is not friendly to individual developers or small teams.

    2. The Workaround: Replacing Conventional Memory with Persistent Memory

    A developer recently validated a workaround: using second-generation persistent memory (PMem 200 series), which has long been discontinued, to hold model weights. This storage-class memory has several key characteristics that make it a good fit for LLM inference:

    1. High capacity per module. These persistent memory modules cover a per-module capacity range of 128 GB to 512 GB. A small number of modules can reach hundreds of gigabytes, far exceeding the per-module capacity limit of ordinary DDR memory.

    2. Clear cost advantage. Because the original manufacturer has stopped iterating on the product line, supply in the second-hand market is abundant, and prices are only a fraction of same-capacity conventional memory, making the cost-effectiveness outstanding.

    3. Non-volatility. Data is not lost after power failure. With App Direct mode, persistent memory can be mapped as an independent byte-addressable memory region, allowing software to directly and precisely control where model tensors are placed. This is highly beneficial for memory layout optimization in inference frameworks.

    For deployment, the developer used a 40-core third-generation Xeon-class processor, which can be understood as an enterprise server CPU platform that supports persistent memory. The CPU and persistent memory are connected directly through the memory controller, avoiding the I/O bottleneck of an NVMe SSD approach. This point is critical because the random read latency of SSD paging is much higher than that of byte-addressable persistent memory.

    3. Key Risk: Cooling Is a Hard Requirement

    The biggest risk in this setup is not performance but heat. Persistent memory consumes considerable power during sustained full-load reads and writes, and densely packed memory slots allow heat to accumulate quickly.

    The developer used two 12W high-power blower fans blowing directly at the memory slots and explicitly warned that active cooling is a hard requirement and cannot be omitted. Without this cooling, the high temperatures generated by persistent memory can damage adjacent conventional DDR memory and, in severe cases, even melt the motherboard DIMM slots, causing irreversible hardware damage.

    For readers who want to reproduce this setup, cooling design should take priority over any performance optimization. Recommendations:

    • Provide an independent airflow path for each group of persistent memory slots to avoid heat transfer between DIMMs;
    • Monitor memory-area temperatures and, if necessary, throttle or reduce speed at the inference framework layer;
    • Plan chassis airflow and memory cooling together rather than relying only on chassis fans.

    4. Performance: Slow but Cost-Effective

    With 1-bit quantization, the system’s inference speed is about 0.85 tokens/s. At 3-bit quantization, speed drops further to 0.75 tokens/s. Higher precision means more data must be read, so speed naturally decreases.

    At this rate, generating a 50,000-token task may take more than 13 hours. This speed is completely unusable for real-time conversation, but the key point is that not all inference scenarios require low latency.

    Suitable scenarios include:

    • Late-night offline batch processing, such as large-scale corpus cleaning and annotation preprocessing;
    • Batch summarization and structured extraction for long documents;
    • Synthetic data generation and other pipeline tasks that do not prioritize timeliness but do prioritize throughput cost.

    In these scenarios, the cost-effectiveness advantage of replacing hardware costing far more with low-cost second-hand persistent memory is fully amplified.

    5. Practical Lessons

    The value of this case is not the specific hardware combination itself, but the general engineering approach it reveals:

    1. Capacity bottlenecks can be mitigated with heterogeneous storage tiers. When model size exceeds VRAM/RAM budgets, byte-addressable persistent media is a cost-effective middle tier between memory and SSD.
    2. Quantization precision should match the scenario. Real-time services should choose higher precision, while offline tasks can relax to 1-bit to 3-bit in exchange for capacity and cost advantages.
    3. Unconventional solutions must evaluate physical constraints. Power consumption, cooling, and motherboard load are “non-software issues” that are often the real keys to success or failure for such solutions.

    For developers with limited budgets who need to run very large models, this “slow but cheap” CPU-only inference path offers an option worth serious evaluation for offline inference scenarios. If the business is latency-sensitive, it should still return to mainstream approaches such as multi-GPU inference or cloud GPU instances.