CPU-Only Inference for Trillion-Parameter Models: A Deep Dive into Low-Cost Persistent Memory
A community experiment demonstrated that a 2.8-trillion-parameter model could run locally without GPUs by combining extreme low-bit quantization, general-purpose CPUs, and high-capacity persistent memory. This article explains the memory requirements, storage-tiering strategy, cooling challenges, real-world performance, and the offline workloads where this low-cost architecture can make practical sense.
CPU-Only Inference for Trillion-Parameter Models: A Deep Dive into Low-Cost Persistent Memory
Introduction: When VRAM Becomes the Bottleneck
The parameter count of large language models continues to grow at an extraordinary pace. Leading open-source mixture-of-experts models have already reached the trillion-parameter scale. Even with aggressive low-bit quantization, storing the model weights alone can require hundreds of gigabytes of memory. The conventional approach is to combine multiple professional accelerators and pool their VRAM, but hardware costs that can reach tens of thousands of dollars put this option out of reach for most individual developers and small teams.
Recently, a particularly instructive community experiment attracted significant attention. A developer successfully ran a 2.8-trillion-parameter model locally without a GPU, using only general-purpose processors and a large amount of persistent memory. Although inference was extremely slow, the experiment revealed a highly cost-effective path for offline, batch-oriented workloads.
This article examines the technical principles, engineering details, and practical limitations behind this approach.
1. Why Trillion-Parameter Models Consume So Much Memory
Before discussing the hardware configuration, it is important to understand where the memory requirement comes from.
During large language model inference, the complete set of model weights must reside in memory that can be accessed directly by the compute units. For a 2.8-trillion-parameter model, the approximate requirements are:
- FP16: 2 bytes per parameter, requiring approximately 5.6 TB;
- INT8 quantization: 1 byte per parameter, still requiring approximately 2.8 TB;
- 4-bit quantization: approximately 1.4 TB, which remains beyond the capacity of typical servers;
- Extreme 1- to 2-bit quantization: compressed to the hundreds-of-gigabytes range, such as the approximately 610 GB used in the reported experiment.
Extreme low-bit quantization is therefore a prerequisite for making this type of deployment accessible. Modern quantization methods, including non-uniform approaches based on importance matrices, can preserve a meaningful portion of a model’s inference capability even at 1- or 2-bit precision. Output quality does decline noticeably, but it may remain acceptable for many batch-processing tasks.
Mixture-of-experts architectures also require an important distinction. Although only a subset of experts is activated during each forward pass, the weights of all experts still need to remain resident in memory. Activation sparsity reduces computation, not storage requirements. This is why these models demand enormous capacity while being relatively less dependent on peak memory bandwidth—a characteristic that creates an opportunity for slower memory media.
2. Persistent Memory: An Overlooked High-Capacity Option
As the cost of conventional memory remains high, building a system with more than 600 GB of standard DDR memory can be prohibitively expensive. The reported experiment instead used a class of hardware positioned between system memory and storage: persistent memory.
This type of media, based on technologies such as phase-change memory, offers several relevant characteristics:
- Byte-addressable access: The processor can access it through ordinary load and store instructions, without passing through the block-device I/O stack;
- High capacity: Individual modules can offer between 128 GB and 512 GB, far exceeding the capacity of standard memory modules from the same period;
- Persistence across power loss: Data is retained when power is removed, although this solution primarily uses the capacity rather than the persistence feature;
- Low cost: Because some of these products have been discontinued, second-hand prices can be only a fraction of the cost of conventional memory with comparable capacity.
Persistent memory commonly supports two operating modes:
- Memory Mode: Persistent memory serves as the primary memory, while conventional memory acts as a high-speed cache. This configuration is relatively simple, but cache misses can introduce unpredictable latency.
- App Direct Mode: Persistent memory is exposed as an independent persistent memory region. Applications can control data placement more precisely through mechanisms such as memory-mapped files.
The reported configuration used App Direct Mode to distribute model tensors explicitly across persistent and conventional memory. Frequently accessed layers and active attention-related weights were placed in faster memory, while less frequently activated expert weights were placed in persistent memory. This created a more deliberate storage hierarchy.
Explicit control over tensor placement is one of the key engineering techniques that makes the system workable. The general layout can be illustrated as follows:
Illustrative memory layout├── Conventional memory (low latency): attention weights, shared experts, KV cache└── Persistent memory (high capacity): infrequently activated routing experts↑ Accessed directly through memory mapping during inference and managed through the page cache
3. Cooling: A Critical Requirement That Is Easy to Underestimate
Persistent memory can consume significantly more power and produce substantially more heat than conventional memory. Under full load, heat can accumulate heavily around the memory slots. The experiment produced a serious warning: active cooling is not optional; it is a strict requirement.
In the reported setup, two high-power 12 W blower fans were aimed directly at the memory area to provide forced airflow. The reasoning is straightforward. Without sufficient cooling, the heat generated by persistent memory can cause throttling or even hardware damage. It may also heat adjacent conventional memory modules to dangerous levels. In extreme cases, the plastic components around the motherboard slots could soften or deform.
Developers considering a similar setup should pay attention to the following cooling measures:
- Install dedicated, high-static-pressure fans for the memory area so that airflow can pass through the gaps between modules;
- Monitor memory-device temperatures through the sensor interfaces provided by the platform and configure alert thresholds;
- Design the chassis airflow path so that the memory region does not become a dead zone;
- Run a low-load temperature stress test before starting long, full-load batch jobs.
4. Performance in Practice: Slow, but Potentially Usable
The performance figures for this approach must be viewed realistically:
- At 1-bit quantization, inference ran at approximately 0.85 tokens per second;
- At 3-bit quantization, performance fell further to approximately 0.75 tokens per second.
At these speeds, generating 50,000 tokens would take more than 13 hours. The main bottlenecks are the much lower bandwidth and higher latency of persistent memory compared with accelerator memory, as well as the large performance gap between CPU matrix operations and specialized AI hardware.
This performance is not meaningful for real-time conversations or online services. However, it can have a distinct value proposition for offline workloads that are not time-sensitive, including:
- Overnight batch processing: document summarization, bulk translation, data cleaning, and annotation, with results collected the next morning;
- Offline evaluation and research: behavioral analysis and benchmarking of very large models without occupying expensive accelerator resources;
- Cost-sensitive prototyping: testing whether a model’s capabilities match a business requirement before committing to production-grade compute infrastructure.
If the incremental electricity cost is only a few cents per hour, while cloud compute may cost tens of currency units per hour, the economics can work for these specific scenarios.
5. Engineering Lessons and Further Implications
This experiment offers a broader lesson for infrastructure design: inference optimization is not only about purchasing faster chips. It is also about matching workload characteristics to the right hardware hierarchy.
- The “low computation, high storage” profile of mixture-of-experts models naturally fits a tiered architecture that combines a large amount of slower memory with a smaller amount of faster memory;
- Explicit data placement, rather than relying entirely on the operating system’s default policies, is essential for extracting performance from a tiered memory system;
- Progress in low-bit quantization continues to reduce the memory barrier for deploying very large models.
For teams that need to run such models in production, an extreme local configuration is better viewed as a platform for validation and experimentation. Production online inference is still generally better suited to elastic cloud infrastructure. For example, general cloud capabilities such as container platforms and object storage from Baidu AI Cloud can be used to build on-demand inference services, host model weights in cloud storage, and combine deployment with monitoring and alerting. This avoids the cooling, hardware lifespan, and maintenance burden of operating the system locally.
Conclusion
Running a 2.8-trillion-parameter model on general-purpose CPUs and discontinued persistent memory is an impressive engineering experiment. It demonstrates that the lower bound of large-model inference costs can be pushed surprisingly far, while also making the limitations clear: generation times measured in hours, demanding cooling requirements, and a use case largely restricted to offline batch processing.
The key lesson is not to copy the exact hardware configuration. It is to understand the underlying principles:
Quantization lowers the memory barrier, tiering lowers the cost, and workload characteristics determine the architecture.
As compute resources become increasingly expensive, this workload-aware approach to system design can be more valuable than any single piece of hardware.
