Running Extremely Large Models on Pure CPU: Persistent Memory, Capacity Planning, and Cooling
This article examines the engineering trade-offs of running extremely large models on pure CPU systems equipped with large-capacity persistent memory. It covers memory capacity estimation for weights, runtime buffers, KV cache, and OS overhead; the limitations and role of persistent memory; memory tiering; how low-bit quantization affects speed; cooling and stability concerns; suitable offline and experimental use cases; and a pre-deployment checklist covering capacity, NUMA, threading, monitoring, and thermal risk.
In recent years, model parameter counts have continued to grow, and inference-stage requirements for VRAM and system memory have risen accordingly. Once models reach the trillion-parameter scale, even with low-bit quantization, relying solely on conventional GPU VRAM or ordinary server memory can lead to capacity shortages, high costs, and hardware expansion difficulties.
Against this backdrop, some developers have begun experimenting with pure CPU systems paired with large-capacity persistent memory to run extremely large models. The core idea is not to pursue real-time response but to trade lower hardware cost for the possibility of running the model at all. This approach mainly targets offline inference, batch generation, long-running tasks, and experimental validation.
1. Why Extremely Large Models Are Capacity-Bound First
During model inference, memory usage is not equal to the size of the parameter file alone. A reasonably complete capacity estimate usually includes at least the following parts:
- Model weights
The number of model parameters multiplied by the number of bytes per parameter is the primary source of memory usage. With low-bit quantization, each parameter may theoretically require only 1 to several bits, but actual storage also includes group scales, zero points, alignment padding, and other metadata. As a result, actual usage is usually higher than the ideal value.
- Runtime buffers
Inference frameworks need extra space for matrix operations, weight decoding, thread scheduling, and temporary tensors. Memory overhead varies significantly across implementations, so the final capacity cannot be derived simply by multiplying parameter count by quantization bits.
- KV cache
During autoregressive generation, the attention mechanism stores context information that has already been processed. The longer the context and the more concurrent requests, the more noticeable the KV cache usage becomes. For long-text tasks, it can even become the second-largest memory consumer after model weights.
- Operating system and program overhead
With pure CPU inference, the operating system, runtime, file cache, and multithreading libraries all consume memory. If model capacity is already close to the physical memory limit, a slight increase in context length or concurrency can trigger swapping, allocation failures, or even process crashes.
Therefore, a statement that a model needs “several hundred GB of memory” is usually only a minimum threshold for running. To maintain stable operation, additional headroom should be reserved to prevent the system from frequently reclaiming cache or using disk swap under high load.
2. Why Persistent Memory Can Be a Low-Cost Alternative
Traditional server memory offers lower latency and higher random-access performance, but large-capacity configurations often mean expensive hardware costs. Persistent memory sits between memory and storage. It usually offers larger capacity and a lower cost per GB, while still being accessible by the operating system or applications in a manner close to memory.
In an appropriate operating mode, persistent memory can be mapped as a separate memory region, and applications can place model weights, caches, or data structures in a designated area. Compared with handing all data to the operating system for unified management, this approach makes memory tiering and resource control easier.
However, persistent memory is not a complete replacement for ordinary memory. Its main limitations include:
- Access latency is usually higher than that of traditional memory.
- Bandwidth and response may be limited during sustained writes.
- Different platforms have compatibility requirements for capacity, channels, and operating modes.
- Not all software can automatically recognize and reasonably utilize persistent memory.
- When used improperly, performance may be noticeably lower than expected.
For large-model inference, model weights are usually read-mostly, and after low-bit quantization they need to be decoded before computation. Therefore, although persistent memory introduces access latency, in capacity-first scenarios it can still make it possible to run a model that would otherwise fail to start.
3. Memory Tiering Matters More Than Simply Stacking Capacity
Treating all memory as a single type of resource is often not optimal. A more reasonable approach is to tier data according to access frequency:
- Place buffers that are frequently read and written at runtime in faster ordinary memory.
- Place the largest model weights in higher-capacity persistent memory.
- Place infrequently accessed caches or checkpoints on slower storage media.
- Avoid placing high-frequency write data in higher-latency regions.
For quantized models, decoded temporary tensors can be placed in faster memory while compressed weights remain in the large-capacity region. This can reduce the impact of persistent memory access on overall performance without significantly increasing hardware cost.
It is important to note that the effectiveness of memory tiering depends heavily on the inference framework implementation. If the framework frequently copies weights, repeatedly allocates temporary buffers, or cannot control where data resides, the advantages of persistent memory may be offset by extra data movement.
4. The Relationship Between Quantization Precision and Inference Speed
Low-bit quantization can significantly reduce model capacity, but lower bit width does not necessarily mean faster speed.
With 1-bit, 2-bit, or 3-bit quantization, for example, model weights are smaller, but inference usually requires additional decoding, scaling, and reconstruction operations. If the CPU lacks suitable vectorized instructions, or if memory access has already become the main bottleneck, the capacity benefit of extremely low-bit quantization may not fully translate into a speed benefit.
The main factors affecting speed include:
- Memory bandwidth
Large-model inference often requires repeatedly reading weights. When compute units are waiting for data, CPU utilization may appear high, but actual throughput can remain limited.
- Quantization decoding overhead
The more compressed the weights, the higher the runtime restoration cost. The quantization format, group size, and decoding implementation all affect final performance.
- Context length
The longer the context, the more attention-related computation and KV cache access are involved, and generation speed may gradually decline.
- Threading strategy
More threads are not always better. Too many threads increase synchronization overhead, cache contention, and memory bandwidth contention, and can even reduce performance.
- NUMA layout
In multi-socket servers, different processors may have their own memory regions. If threads frequently access remote memory, latency increases further. Therefore, model sharding, thread binding, and memory allocation strategies all need to be designed together.
In some experimental deployments, generation speed with extremely low-bit quantization may be less than one token per second. At that rate, generating tens of thousands of tokens could take more than ten hours. Such performance is clearly unsuitable for online chat, but it can be used for overnight batch processing, offline summarization, data pre-generation, or model capability validation.
5. Cooling Is a Critical Part of Persistent Memory Deployments
Large-capacity persistent memory does not mean low power consumption. During model loading and inference, memory controllers, memory modules, and processors can remain under high load for extended periods. If chassis airflow design is inadequate, heat may accumulate locally.
At minimum, deployments should address the following:
- Ensure cold air flows directly over memory modules.
- Avoid blocking the area around memory slots with cables or expansion cards.
- Use active fans capable of providing sustained static pressure.
- Monitor temperatures in memory modules, processors, and motherboard regions.
- Prepare throttling and task-pause strategies for high-temperature environments.
- Clean dust regularly to prevent cooling capability from gradually degrading.
This is especially important on open-air platforms or modified servers. The absence of an obviously hot surface on a module does not mean cooling is sufficient. Local hot spots may occur inside modules, in power delivery areas, or near slots. Prolonged high-temperature operation can not only reduce performance but also cause system reboots, data errors, slot aging, and other issues.
For experimental equipment, it is advisable to run long-duration stress tests before formal operation and to record temperature, power consumption, error counts, and inference throughput at the same time. If temperatures continue to rise or speed drops noticeably over runtime, the system is usually already affected by thermal protection or memory stability issues.
6. Suitable Use Cases for Running Extremely Large Models on CPU
The value of this approach lies not in speed but in lowering the entry barrier. It is relatively suitable for the following tasks:
1. Offline batch generation
For example, generating large amounts of text, code drafts, test samples, or structured data overnight. The task does not require immediate return, as long as it can eventually complete reliably.
2. Model capability validation
Researchers can use low-cost equipment to verify whether a model can handle a specific task without first purchasing a large-scale accelerator cluster.
3. Long-running automated workflows
For workflows that can be split into multiple independent tasks, queue scheduling can keep the system running continuously and make full use of idle time on low-speed hardware.
4. Deployment in special environments
Some environments cannot use high-performance accelerators but have large-capacity server memory or persistent memory. In such cases, a pure CPU approach has practical value.
In contrast, the following scenarios are not suitable for this architecture:
- Online customer service and real-time conversation
- High-concurrency API services
- Applications with strict first-token latency requirements
- Services that need to switch frequently among multiple large models
- Deployments that are highly sensitive to power consumption and data center space
7. Pre-Deployment Capacity and Risk Checklist
Before building the system, evaluate it in the following order:
- Calculate total capacity for weights, runtime buffers, and KV cache.
- Reserve memory headroom for the operating system and abnormal fluctuations.
- Confirm that the processor platform supports the target memory type and operating mode.
- Evaluate memory channels, slot count, and NUMA topology.
- Choose a suitable quantization format and run small-scale benchmarks.
- Measure throughput and latency under different thread counts.
- Design independent, continuous active cooling for persistent memory.
- Set up monitoring for temperature, power, memory errors, and system logs.
- Run stability tests for several hours before running long-text tasks.
- Split tasks into recoverable batches to avoid losing all work when a single run fails.
Conclusion
Running extremely large models on pure CPU with large-capacity persistent memory is essentially an engineering approach that trades time for capacity and cost for speed. It cannot replace high-performance accelerators, nor is it suitable for real-time services, but it can make some models runnable that would otherwise be limited by VRAM capacity.
What ultimately determines success is not only whether the model can be loaded into memory, but also memory tiering, quantization format, NUMA scheduling, thread configuration, cooling design, and task scheduling. Only by optimizing these factors as a whole can relatively stable inference be achieved on a limited budget. For users who do not require immediate response but want to complete offline processing of extremely large models at lower cost, this architecture still has experimental and practical value.
