Running Trillion-Parameter LLMs on Pure CPU: Persistent Memory and Thermal Challenges

    Inference bottlenecks for large language models are shifting from compute to memory capacity. This article examines a cost-driven engineering path for serving trillion-parameter models on pure CPU hardware backed by large-capacity persistent memory. It covers the memory footprint of trillion-scale models, how persistent memory enables low-cost capacity expansion, hardware deployment considerations, the often-underestimated cooling requirements, real-world performance numbers, software optimization strategies, and the risks and limitations to weigh before adopting this approach. Readers will learn when this "slow but cheap" architecture makes sense for offline inference workloads — and when it does not.

    As large language model parameter counts keep climbing from hundreds of billions toward the trillions, the resource bottleneck of inference systems has gradually shifted from “not enough compute” to “nowhere to put the memory.” GPU VRAM offers extremely high bandwidth, but per-card capacity is limited, and scaling across multiple cards brings steep hardware and interconnect costs. Meanwhile, traditional DRAM prices remain high, and few teams can afford stacking hundreds of gigabytes of conventional memory. This has brought a seemingly “retro” route back into engineering view: running trillion-parameter models on pure CPU hardware with large-capacity persistent memory. This is not wishful thinking, but a difficult trade-off across capacity, cost, thermals, and performance.

    Background: The Memory Footprint of Trillion-Parameter Models

    A 2.8-trillion-parameter model still requires roughly 610GB of resident space for its weights, even under extreme low-bit quantization. For GPU-based solutions, this far exceeds the VRAM of a single card; even multi-card tensor parallelism requires an expensive compute cluster. CPU solutions, while weaker per core than GPUs, offer a much wider memory address space and more flexible storage expansion. As long as memory capacity is sufficient, virtually any model can be loaded and run in theory. The question is how to acquire that much memory at an acceptable cost.

    Persistent Memory: The Key to Low-Cost, High-Capacity Storage

    Persistent Memory (PMem) is a storage medium that combines memory-like access characteristics with non-volatility. It supports byte-granular reads and writes like DRAM while retaining data after power loss, giving it significant advantages in capacity density and cost per gigabyte. In practice, persistent memory can be configured in “App Direct mode,” where PMem regions are mapped as a separate memory address space, letting applications directly control where model tensors reside and avoiding data copies through the file system layer.

    This mechanism fits large model inference particularly well: model weight files can be persistently mapped into memory, eliminating the need to reload them after a system restart and saving substantial time. More importantly, persistent memory modules can reach hundreds of gigabytes each, and combining multiple channels easily pushes total capacity past the terabyte level — a physical carrier for trillion-parameter models.

    Of course, persistent memory has shortcomings: random access latency is higher than DRAM, and write bandwidth is relatively low. However, weight reads during large model inference are dominated by large-scale sequential streaming access, so the actual performance penalty does not reach unacceptable levels.

    Hardware Deployment: Capacity, Channels, and Firmware All Matter

    Building a pure-CPU inference platform capable of running trillion-parameter models requires careful hardware planning. In one publicly documented setup, the core is a 40-core server-class processor with a multi-channel memory controller, populated with large-capacity persistent memory modules as needed. The motherboard and BIOS must correctly detect and enable persistent memory; otherwise the modules can only be used as block devices, forfeiting the benefits of byte-level access.

    Capacity planning is not a matter of “the more, the better.” Beyond the model weights themselves, runtime requires temporary storage for activations, the KV cache, and other dynamic data. Placing everything on persistent memory not only drags down access speed but also accelerates media wear from frequent writes. In practice, deployments commonly adopt a hybrid strategy: persistent memory carries the weights, while a smaller amount of DRAM holds hot data — balancing capacity and performance.

    Active Cooling: A Non-Negotiable Requirement

    Persistent memory modules generate substantial heat under full load — far more than ordinary memory. Inadequate cooling leads not only to memory throttling and system instability, but heat can also conduct to adjacent DIMM slots and potentially damage the motherboard itself. In one test, multiple high-power turbine fans had to blow directly onto the memory area continuously just to keep temperatures within a safe range. This detail makes one thing clear: any expansion scheme based on persistent memory must treat cooling as a core design element, not an afterthought.

    Cooling retrofits typically include: adjusting chassis airflow to add dedicated intake/exhaust channels for the memory area; installing high-airflow fans covering all persistent memory modules; or, where feasible, adding temperature monitoring with automatic throttling policies. Ignoring cooling means, at best, interrupted inference and, at worst, destroyed hardware — a net loss either way.

    Real-World Performance: Extremely Slow, but Extremely Stable

    In actual testing, the pure-CPU setup can only be described as “slow and steady.” At 1-bit quantization, inference ran at roughly 0.85 tokens per second; at 3-bit quantization, it dropped further to about 0.75 tokens per second. Generating a 50,000-token piece of text could take more than 13 hours of continuous operation. That speed is clearly unsuitable for real-time conversation or interactive applications — but for offline tasks with no time sensitivity, it delivers exceptional cost-effectiveness.

    The slowness stems from two factors: CPU memory bandwidth is far below that of GPUs, and persistent memory latency is higher than DRAM. At trillion-parameter scale, computing each token requires traversing massive amounts of weights, amplifying memory access overhead indefinitely. Pure-CPU inference therefore does not suit high-concurrency, low-latency online services, but it is well suited for individuals or small teams running overnight batch evaluation, data synthesis, long-document summarization, and similar tasks.

    Cost Advantages and Suitable Use Cases

    With DRAM prices staying high, conventional memory of equivalent capacity is expensive. By comparison, used persistent memory can cost a fraction of ordinary memory per gigabyte. For scenarios that require repeatedly loading large models, this cost difference matters enormously. Suitable use cases include:

    • Offline batch processing: hour-level latency is acceptable, such as summarizing entire books or large-scale corpus cleaning;
    • Model evaluation and benchmarking: the same weights must be loaded repeatedly, and the non-volatility of persistent memory significantly reduces load times;
    • Low-priority long-form generation: queued overnight, results collected the next morning, without disrupting daily development;
    • Quantization tuning: validating model quality repeatedly across different bit widths without bearing expensive memory upgrade costs.

    Software Optimization: From “Runs” to “Runs Efficiently”

    To make this setup deliver real value, memory management must be purpose-built at the software layer. The core ideas are:

    • Map model weights directly onto the persistent memory device instead of copying them into DRAM first;
    • Prioritize hot data — activations and intermediate results — in DRAM, leaving cold data on persistent memory;
    • Implement batching and pipeline parallelism so other computations proceed while weight reads are pending;
    • Align, compress, and prefetch quantized weights to reduce random access patterns.

    A sketch in pseudocode:

    1. weights = pmem_map(model_path) # Map weights to persistent memory
    2. dram_cache = allocate_dram(cache_size) # Keep activations and hot data in DRAM
    3. for batch in data_loader:
    4. activations = compute_activations(weights, batch, dram_cache)
    5. write_output(activations)
    6. check_temperature_and_throttle() # Thermal monitoring and throttling

    In practice, the implementation depends on the memory management interfaces of the inference framework, and developers may need a degree of custom wrapping. Overall, the goal of software optimization here is not to speed up individual tokens, but to lower total cost of ownership and improve task throughput.

    Risks and Limitations

    While the persistent memory approach is highly attractive on cost, its limitations must be faced squarely:

    • Performance ceiling: it cannot meet real-time requirements and only suits low-throughput, large-output scenarios;
    • Media lifespan: persistent memory has limited write endurance; frequent overwrite cycles consume lifespan, so logs, temp files, and similar data should not be written to persistent memory;
    • Cooling retrofit complexity: additional fans bring noise, power draw, and space demands, and long-term operation requires regular maintenance;
    • Ecosystem maturity: native persistent memory support in inference frameworks is still evolving, requiring strong low-level systems expertise from developers.

    This approach is therefore an “economics play” suited to specific workloads — not a universal answer for every AI inference need.

    Conclusion

    Under the dual pressure of VRAM and DRAM costs, expanding capacity with persistent memory and driving trillion-parameter models on pure CPUs has become an engineering path worth watching. It breaks through the physical limits of GPU VRAM, making trillion-parameter models runnable at extremely low cost — while simultaneously introducing challenges in thermals, performance, and ecosystem maturity. For developers, turning this path into reusable offline inference productivity requires thoroughly understanding the hardware characteristics, honestly assessing the workload, and getting cooling and fault-tolerance design right. As non-volatile memory technology iterates and inference software stacks improve, setups like this may become an important complement for deploying ultra-large models.