Running a Trillion-Parameter LLM Without a GPU: A Practical Look at Persistent Memory
GPU-free trillion-parameter model inference isn't just about "skipping the graphics card." It moves the model weights that normally live in VRAM into large-capacity persistent or storage-class memory, then uses server-class CPUs to do the forward pass. Written for developers and architects, this article answers three questions: why this approach exists, where its performance bottlenecks are, and which engineering capabilities you need to build out for selection, tuning, and production. It walks through a real-world experiment that ran a 2.8-trillion-parameter model on a ~40-core server CPU with large DRAM and persistent memory, explains why memory capacity and bandwidth—not raw compute—are the first constraint, and covers memory tiering, quantization trade-offs, the counterintuitive effect of disabling DAX, an engineering checklist, and the associated risks and compliance considerations.
1. Introduction: The Realistic Path to GPU-Free Trillion-Parameter Inference
Running a trillion-parameter model without a GPU doesn’t simply mean “not using a graphics card.” It means moving the model weights that were previously held in VRAM into large-capacity persistent memory or storage-class memory, then letting a server-class CPU handle the forward computation. This article is aimed at developers and architects, and focuses on answering three questions: why this approach exists, where its performance bottlenecks are, and which engineering capabilities you still need to build out for selection, tuning, and production.
In a home-lab-style experiment, a developer ran an ultra-large language model with as many as 2.8 trillion parameters without a discrete GPU—relying instead on a server-class scalable processor of roughly 40 cores, large-capacity DRAM, and persistent memory. The value of this case isn’t how fast it runs; it’s that it clearly exposes the first constraint of trillion-parameter inference: memory capacity and memory bandwidth. For readers who want to evaluate the feasibility of non-GPU inference, design an offline batch-inference pipeline, or understand the memory bottleneck of ultra-large models, this article offers a practical reference.
One thing to state upfront: because the original experiment did not fully disclose the model name, the inference framework version, quantization format details, or the exact measurement methodology, some of the speed figures here are treated as “example figures.” They illustrate optimization trends and should not be taken as a unified performance benchmark.
2. Conclusion First: It’s a Fallback, Not a Universal Replacement
If you keep only one takeaway: CPU plus persistent memory can run trillion-parameter models, but it’s usually suited only to offline, low-throughput scenarios that tolerate high latency.
It fits:
- Experimental environments without an accelerator card that still need to validate an ultra-large model architecture;
- Tasks with low real-time requirements that can tolerate long generation times for a single request;
- Storage-oriented inference that needs to host ultra-large model weights on a limited budget;
- Offline batch processing, log analysis, model self-testing, internal evaluation, and other scenarios that don’t demand low online latency.
It does not fit:
- Online interactive dialogue;
- High-concurrency API services;
- Generative applications sensitive to first-token latency;
- Product environments that need high-throughput, real-time inference.
So the essence of this approach is “trading capacity for speed.” When VRAM capacity is unavailable or too expensive, it provides a workable alternative path—but it can’t replace the GPU’s advantages in parallel compute and high-bandwidth memory.
3. Why Bypass the GPU: Comparing VRAM, DRAM, Persistent Memory, and SSD
The core conflict of trillion-parameter inference is that the weights are simply too large. Take a 2.8-trillion-parameter model as an example: even with a relatively low-bit quantization, the weights can still reach the multi-terabyte range. The traditional approach—multi-card parallelism, tensor splitting, and pipeline parallelism—brings hardware cost, communication overhead, and deployment complexity.
The comparison across media can be summarized as follows:
| Medium | Typical capacity | Latency | Bandwidth | Cost | Typical use |
|---|---|---|---|---|---|
| GPU VRAM | Tens to hundreds of GB | Low latency | High bandwidth | High | Online inference, training, high-concurrency generation |
| DRAM | Tens of GB to TB-scale | Low latency | High bandwidth | Relatively high | Hot data, frequently used weight shards, cache tier |
| Persistent memory | Hundreds of GB to multi-TB | Higher than DRAM, lower than SSD | Limited by channels and platform | Medium-low | Hosting ultra-large model weights, byte-addressable storage |
| SSD | TB to PB-scale | High latency | Block-device access | Low | Weight archiving, cold storage, pre-load staging |
One point deserves special attention: persistent memory is not a single, vaguely defined concept. It can refer to a DIMM-form, byte-addressable, persistence-capable memory-expansion device. Different hardware forms differ significantly in compatibility.
Common forms include:
- Server persistent-memory products that were once widely used but face limited supply or gradual discontinuation;
- NVDIMM-class solutions based on DIMMs or memory slots;
- Memory-expansion devices over CXL, such as CXL Type-3 memory expansion or persistent memory solutions;
- Certain dedicated devices that map storage-class media into the memory address space.
When selecting, you must confirm: whether the CPU platform supports it, whether it’s available in BIOS, whether the OS supports DAX or memory expansion, whether the file system supports direct mapping, whether the NUMA topology is balanced, and whether per-channel capacity and bandwidth meet requirements. You can’t equate all “persistent memory” with DRAM, nor can you assume every server platform enables these capabilities automatically.
4. Memory Tiering: The App Direct Mode of DRAM + Persistent Memory
Persistent memory is typically used in two common ways.
The first is App Direct mode. DRAM and persistent memory are presented to the application as two independent data spaces. The application itself decides which data goes in DRAM and which goes in persistent memory.
The second is memory-expansion mode. Persistent memory acts as a transparent extension of DRAM, with hardware or firmware managing cache coherency and data placement. The application side is largely unaware, but there’s relatively limited room for tuning.
For ultra-large model inference, the more common and easier-to-control approach is App Direct mode. The reason: model weights are large, but access patterns across layers, tensors, and token stages differ. If the application manages data placement explicitly, it can put the hottest layers, the weights most relevant to the current context, or frequently used shards in DRAM, and place the less frequently accessed weight shards in persistent memory.
This can be abstracted as:
┌─────────────────────────────────────────────┐│ DRAM: small capacity, high bandwidth; holds hot data / common weight shards │├─────────────────────────────────────────────┤│ Persistent memory: large capacity, slightly higher latency; holds massive model weights │└─────────────────────────────────────────────┘
The goal of tiering isn’t to make all data slow to read, but to let high-bandwidth DRAM absorb hot accesses as much as possible, hiding the higher latency of persistent memory behind prefetching, pipelined computation, and improved page-cache hit rates.
5. How CPU Inference Works and the Forward-Pass Bottleneck
The typical CPU inference flow isn’t complicated:
- Load model weights into the address space via memory mapping;
- Perform matrix and vector multiplications layer by layer;
- Maintain the KV cache;
- Generate a new token at each step;
- Feed the new token back into the next step’s context.
For ordinary models, the weights can stay resident in DRAM or cache. But for a trillion-parameter model, generating each token in principle requires reading a large amount of weights. If the weights live mainly in persistent memory, memory bandwidth becomes the absolute bottleneck.
The following is conceptual pseudocode, meant only to illustrate the basic idea of memory mapping and layer-by-layer forward computation. It does not include the engineering details of tokenization, positional encoding, sampling strategy, KV management, and so on:
# Conceptual sketch: mmap-load quantized weights and run a layer-by-layer forward passweights = mmap_load(model_path) # weights may live in persistent memory or DRAMkv_cache = allocate_kv_cache(max_seq_len)for token_pos in range(max_new_tokens):hidden = embedding[input_ids[token_pos]]for layer in model.layers:q, k, v = layer.attention(hidden, weights) # weights may be streamed in from persistent memorylayer.update_kv_cache(k, v, kv_cache, token_pos)hidden = layer.feed_forward(q, weights)next_token = sample(layer.output_head(hidden))input_ids.append(next_token)
In this loop, every layer reads a large amount of weights. If the weights reside mainly in persistent memory, the CPU compute units often have to wait for data to be moved across the memory channels. In that situation, raising clock frequency or adding a few cores won’t scale generation speed linearly. What truly determines throughput is memory bandwidth, NUMA access efficiency, page-cache hit rate, quantization compression ratio, and data prefetch strategy.
6. Quantization and Speed: How Bit-Width Affects Memory Footprint and Generation Speed
In ultra-large model inference, quantization is the core enabler of whether the model can run at all. Quantization represents weights with fewer bits, directly reducing memory footprint and bandwidth pressure.
Common descriptions include:
- 1-bit quantization: each weight occupies theoretically just 1 bit, compressing memory footprint to an extremely low level;
- 3-bit quantization: usually better accuracy than a 1-bit scheme, but memory footprint is several times that of the 1-bit approach.
The original experiment saw speed descriptions similar to 0.85, 1.50, and 0.75 tokens/second. Because of the missing model, framework, hardware, and test conditions, this article labels them as example figures and they should not be treated as a general benchmark.
You can understand them under the following example conditions:
- Model: a trillion-parameter-class language model, roughly 2.8 trillion parameters;
- Inference hardware: a ~40-core server-class CPU;
- Memory: large-capacity DRAM (for example, TB-scale) together with multi-TB persistent memory;
- Quantization format: 1-bit or 3-bit weights;
- Inference framework: a general-purpose open-source CPU inference framework, version not fixed;
- Input prompt length: about 512 tokens;
- Batch size: 1;
- Concurrency: 1;
- Measurement methodology: count tokens/second after warmup, or measure the time to generate roughly 50,000 tokens.
Under these example conditions, with 1-bit quantization directly accessing persistent memory, the initial generation speed was about 0.85 tokens/second. When direct access was turned off and more frequently used data was allowed to enter the DRAM page cache, speed improved to about 1.50 tokens/second. After switching to 3-bit quantization, speed may fall back to about 0.75 tokens/second, and generating roughly 50,000 tokens could take about 18.5 hours.
This set of figures shows that quantization bit-width, cache strategy, and inference speed are directly coupled. More aggressive quantization buys you the ability to get it running at all, but it may also degrade model quality. Architects can’t look only at tokens/second; they must also evaluate the quality loss from quantization at the same time.
When evaluating aggressive quantization, it’s recommended to focus at least on:
- Perplexity change: compare PPL before and after quantization on a fixed validation set;
- Task quality: test on math, code, Q&A, summarization, instruction following, and other tasks;
- Long-context behavior: watch for forgetting, repetition, or grammatical degradation under long inputs;
- Factual risk: low-bit quantization may amplify factual errors;
- Human evaluation: manually sample key business cases;
- Production risk: for high-stakes scenarios, use 1-bit or overly aggressive quantization schemes cautiously.
7. Direct Access vs. Cache: Why Disabling DAX May Actually Be Faster
The original case mentioned a counterintuitive phenomenon: turning off direct access actually improved speed. Here the concept needs to be stated clearly.
“Direct access” usually corresponds to DAX (Direct Access). With DAX enabled, the application or file system can bypass the traditional page cache, directly mapping the persistent-memory region and avoiding one copy. It sounds more efficient, but in trillion-parameter inference the problem isn’t just the copy cost.
If weight access has a clearly random pattern and the hot set isn’t stable enough, enabling DAX can cause a large number of accesses to land directly on higher-latency persistent memory. The CPU still has to wait for the data. In contrast, turning DAX off and falling back to the page-cache or block-device path lets the system cache hot weights in DRAM. As long as the hit rate is high enough, the cache benefit may outweigh the extra copy overhead.
It should be noted that “turning off direct access” isn’t simply flipping a switch; it depends on platform and system configuration. Common approaches include:
- Disabling DAX and using the page-cache and block-device path;
- Configuring persistent memory in memory-expansion mode, managed by hardware;
- Adjusting file-system mount parameters;
- Adjusting NUMA policy to avoid cross-node access;
- Checking whether BIOS, the OS, and the file system support persistent-memory direct mapping.
Hardware and OS prerequisites typically include:
- The motherboard and CPU support the corresponding memory-expansion or persistent-memory form;
- The OS can recognize the persistent-memory device;
- The file system supports the corresponding mount method;
- If using DAX, confirm compatibility configurations for file systems such as xfs, ext4, or others;
- If using CXL memory, confirm firmware, BIOS, and OS support.
When tuning, it’s recommended to measure the following metrics:
- Page-cache hit rate;
- Persistent-memory read bandwidth;
- DRAM bandwidth utilization;
- Cross-NUMA access ratio;
- Per-request token generation speed;
- First-token latency;
- Performance degradation during long-sequence generation;
- Peak memory usage and stability.
The conclusion isn’t that “turning off DAX is always better.” Rather: in scenarios that are capacity-constrained, sensitive to media latency, and have unstable access patterns, cache strategy must be validated with real workloads. You can’t simply assume that more direct is faster.
8. From Home Experiment to Engineering: A Checklist to Close the Gaps
To move GPU-free trillion-parameter inference from an experiment to an operable offline system, you can’t rely on a single machine and a script. You need to build out several categories of capability.
First, weight loading and storage preparation.
- Weight files should be stored as shards for easy parallel loading and verification;
- It’s recommended to keep the model card, quantization config, tokenizer version, and inference framework version;
- For mmap loading, evaluate whether the file system is suitable for persistent memory or large-page caching;
- Verify before loading to avoid inference anomalies caused by corrupted weights;
- Clarify the source and license of the model weights.
Second, compute resource planning.
- Estimate DRAM and persistent-memory capacity based on total weight size;
- Reserve enough space for the KV cache, runtime buffers, and page cache;
- Pay attention to the NUMA topology, and keep memory accesses as close as possible to the corresponding CPU node;
- For long-sequence tasks, test KV cache growth separately;
- For high-load scenarios, consider rate limiting and queuing.
Third, offline batch processing and asynchronous scheduling.
In GPU-free inference, throughput is often low. So that users don’t face long waits directly, requests can be made asynchronous with a message or task queue. Object storage is well suited to holding TB-scale model weights, compute instances handle loading and inference, and monitoring systems track metrics.
You can combine:
- Object storage: store model weights, input samples, and output results;
- Large-memory compute instances: host the weights and page cache;
- Message queue: queue offline requests and smooth out processing speed;
- Monitoring, alerting, and logging services: record token generation rate, memory bandwidth, cache hit rate, error rate, and task duration.
The point here isn’t a particular cloud brand, but the capability combination: storage, compute, queue, and monitoring must all be designed around the goals of weight loading and offline batch processing.
Fourth, quality and operational metrics.
At minimum, it’s recommended to collect:
- Average tokens generated per task;
- Time per task;
- Tokens/second;
- First-token latency;
- Page-cache hit rate;
- Persistent-memory read bandwidth;
- DRAM and persistent-memory capacity levels;
- Task failure rate;
- Timeout rate;
- Sampled output-quality results.
9. FAQ: Can You Run a Trillion-Parameter Model Without a GPU?
Q: Can you run a trillion-parameter model without a GPU?
A: You can try, but it usually works only as an offline, low-throughput solution. The keys are memory capacity, memory bandwidth, quantization strategy, and how the model is loaded.
Q: What speed can you expect for GPU-free trillion-parameter inference?
A: Speed is usually far below GPU solutions. In the example, it may be only a few tokens per second, and generating tens of thousands of tokens could take several to tens of hours. Actual speed depends on the CPU, memory bandwidth, quantization bit-width, cache strategy, and prompt length.
Q: What’s the difference between persistent memory and SSD?
A: Persistent memory usually offers byte-addressability closer to memory and lower access latency, making it suitable for hosting weights; SSD offers larger capacity and lower cost but higher latency, and is usually better suited for cold storage and pre-load staging.
Q: How do you choose persistent memory for GPU-free trillion-parameter inference?
A: First look at platform compatibility, then capacity, channels, latency, NUMA, file-system support, and whether you can enable DAX or memory-expansion mode. Don’t select on capacity alone.
Q: What are common misconceptions?
A: Common misconceptions include: equating persistent memory with DRAM; assuming that turning off DAX is always worse; ignoring the page-cache hit rate; ignoring quantization quality; testing only first-token latency without long sequences; and not recording the model version and measurement methodology.
Q: What risks should you watch for?
A: Watch for persistent-memory data-residency risk, the compliance of the model-weight source, inference-data privacy, log redaction, quantization quality degradation, and task backlogs caused by low throughput.
10. Risk and Compliance Notes
Although the GPU-free inference approach is experimental-leaning, you still need to pay attention to compliance and risk.
Persistent memory has the characteristic that data may persist after power loss. So when testing sensitive data or model weights, you should evaluate data-erasure, encryption, and access-control strategies. If you use persistent memory to cache user inputs or intermediate inference results, define the data-retention period and cleanup mechanism clearly.
The model-weight source must also be compliant. Before use, confirm the license, commercial-use restrictions, redistribution restrictions, and model-card statements. For weights fine-tuned on internal data, they should also be brought into the asset-management and audit process.
If the system processes user data, avoid logging complete sensitive inputs. Monitoring metrics should prioritize retaining aggregated data—such as latency, throughput, error rate, and cache hit rate—rather than recording raw user text directly.
11. Summary
The core value of GPU-free trillion-parameter inference isn’t proving that it can replace the GPU; it’s validating that when VRAM becomes a hard constraint, large-capacity persistent memory, CPU inference, aggressive quantization, and memory tiering can still get an ultra-large model running.
There are three key lessons for architects:
First, the first constraint of large-model inference is usually memory capacity and memory bandwidth, not raw compute peak.
Second, quantization bit-width, page-cache strategy, the DAX switch, NUMA layout, and memory tiering are the main battlegrounds for tuning—and they must be validated repeatedly with real workloads.
Third, every approach has boundaries. CPU plus persistent memory suits offline validation and storage-oriented inference, but not online, high-concurrency, low-latency scenarios. Understanding the boundaries is what lets you make rational trade-offs among compute, capacity, and cost.
