Memory Expansion for Trillion-Parameter LLM Inference Without GPUs
This article explains how to run inference on ultra-large language models using only CPUs and large memory capacity, without GPUs. It covers the role of persistent memory, capacity estimation for model weights at different quantization levels, the bandwidth bottlenecks that limit decoding speed, and practical deployment guidance including memory tiering, NUMA-aware binding, and inference parameter tuning for latency-insensitive offline workloads.
1. Background: Why GPU-Free Inference Is Worth Exploring
Large model inference typically relies on GPUs because their parallel compute capability and high-bandwidth memory meet the demands of matrix operations. However, in many scenarios, GPU procurement costs, power consumption, and supply lead times become real constraints. Some teams already own servers with large memory capacity and want to use existing hardware for offline inference, batch text processing, or model evaluation. This raises a natural question: can a model with trillions of parameters run on CPUs and large-capacity memory alone?
The answer depends on two things: whether the model fits into available memory, and whether generation speed meets business latency requirements. The first is a capacity problem; the second is a bandwidth and latency problem. We examine each below.
2. The Role of Persistent Memory
Persistent memory is a class of devices that sits between traditional memory and storage. Data survives power loss, and it can be accessed byte-by-byte in a memory-like fashion. Compared with standard DRAM, persistent memory typically offers larger capacity per module and lower cost per gigabyte, but with higher read/write latency and lower bandwidth. It is commonly used in database caching, large-scale key-value stores, and in-memory databases.
Using persistent memory for LLM inference is essentially trading some performance for capacity. During inference, model weights are accessed repeatedly in a read-only pattern, which places little demand on write endurance—making this use case viable.
2.1 Two Common Usage Modes
- App-direct mode: The operating system exposes persistent memory and DRAM as two separate address spaces. The application must explicitly decide which data goes where. The advantage is predictable behavior; the downside is that the application layer needs modification.
- Tiered mode: Persistent memory acts as a slower second tier of memory, with the system or runtime deciding data placement. Frequently accessed data stays in DRAM; less-accessed data resides in persistent memory. The advantage is near-transparency to the application; the downside is that performance depends heavily on cache hit rates.
3. Capacity Estimation: Where Model Size Comes From
Memory occupied by weights can be estimated with a simple formula:
Weight memory (bytes) ≈ parameter count × bits per parameter ÷ 8
For a 2.8-trillion-parameter model:
- 1-bit quantization: ~350 GB
- 3-bit quantization: ~1050 GB
- 4-bit quantization: ~1400 GB
This covers weights only. At runtime you also need the KV cache, the framework’s own memory overhead, and space for the OS and other processes. The KV cache grows with context length and can become substantial when processing long texts. In addition, low-bit quantization sacrifices some accuracy, so you should compare output quality on a fixed evaluation set rather than only checking whether the model fits.
Here is a sample estimation function:
def estimate_weight_gb(params_billion: float, bits: float) -> float:"""Estimate memory occupied by weights (GB), excluding cache and runtime overhead."""total_bytes = params_billion * 1e9 * bits / 8return total_bytes / 1e9print(estimate_weight_gb(2800, 1)) # 350.0
4. The Speed Bottleneck: Why Generation Slows Down
LLM inference has two phases. The prefill phase processes all input tokens at once; it is compute-intensive and parallelizes well. The decode phase generates output tokens one by one, and each token requires reading the model weights again. At this stage, the main bottleneck is usually not compute but memory read bandwidth.
A rough theoretical upper bound can be expressed as:
Tokens per second ≈ effective memory bandwidth ÷ bytes read per token
For dense models, each token requires reading nearly all weights. For mixture-of-experts models, each token activates only some experts, reducing the actual read volume—but the overhead from expert routing and cross-memory-node access must still be considered.
Persistent memory has lower bandwidth and higher access latency than DRAM, so during decoding its generation speed is typically well below that of a DRAM-based setup under otherwise identical conditions. In publicly available experimental tests, some configurations produced fewer than two tokens per second, with noticeable differences across quantization methods and cache settings. These figures come from specific test environments and should not be treated as universal benchmarks, but they indicate where this approach fits: latency-insensitive offline tasks are far more tolerant.
5. Deployment Practices
5.1 Evaluate Models and Quantization Schemes
First determine the minimum accuracy your business can accept, then choose the quantization bit width. It is recommended to compare outputs from different quantized versions on the same sample set, recording metrics such as accuracy, format compliance, and hallucination rate—rather than simply pursuing the smallest footprint.
5.2 Plan Memory Tiering
The core principle of memory tiering is: put write-intensive, latency-sensitive data in DRAM, and read-only, bulky data in persistent memory.
- KV cache: write-intensive; should reside in DRAM.
- Model weights: read-only and the largest component; well suited for persistent memory.
- Small, frequently accessed structures such as embedding tables: prioritize DRAM.
5.3 Bind Compute and Memory Topology
In multi-socket systems, different memory regions are at different distances from CPU cores, and cross-node access latency increases significantly. Inference processes should be pinned to cores on the same node as the memory they use, with monitoring to confirm there is no heavy cross-node traffic.
5.4 Tune Inference Parameters
- Thread count should generally not exceed the number of physical cores; too many threads intensify memory contention.
- Set context length according to actual business needs—do not maximize it arbitrarily—so KV cache growth stays under control.
- Batch size affects the trade-off between throughput and per-request latency and should be tested against real workloads.
5.5 Establish Benchmarks
Before going live, benchmarks should be consolidated into a repeatable test suite.
