# Running Trillion-Parameter LLMs Without a GPU: A CPU and Persistent Memory Inference Path

    This article examines a deliberately low-cost approach to very large model inference: instead of sharding weights across GPUs, run a trillion-parameter-class model on a multi-core server CPU backed by byte-addressable persistent memory. It works through the capacity math for 1-bit, 3-bit, and 8-bit quantization at a 2.8-trillion-parameter scale, explains how memory mode versus application-direct mode changes data placement and whether DRAM participates as a cache, and shows how effective memory bandwidth and per-token weight reads govern generation speed. You will also find the capability limits of this setup, an engineering checklist, common measurement pitfalls, and guidance on reproducing the pattern in the cloud.

    1. Beyond the VRAM Bottleneck: Why CPU Plus Persistent Memory Is Worth Considering

    As model parameters move from the hundreds of billions into the trillions, the first deployment problem is not algorithmic — it is where the weights live and how fast they can be read for every generated token. If a single GPU cannot hold all the weights, the conventional answer is multi-card sharding, high-speed interconnect, and non-trivial communication engineering. For teams with limited compute budgets, researchers, and individual experimenters, that route is expensive and complex.

    So it is worth revisiting a more basic question: can you skip the GPU entirely and get a very large model running on a multi-core server CPU paired with byte-addressable persistent memory?

    The goal of this path is not to replace GPU inference. It is to offer a low-barrier way to validate things when you do not have enough accelerators. The value is in lowering the hardware entry point: you can observe inference behavior, validate quantization schemes, and measure memory access bottlenecks — all of which provide evidence for later cloud migration or hardware upgrades.

    2. Capacity Math: How Much Memory a Trillion-Parameter Model Actually Needs

    The first step toward feasibility is getting the capacity numbers right. Storage demand can be approximated as:

    Model capacity ≈ parameter count × bits per parameter ÷ 8

    Taking 2.8 trillion parameters as the example:

    Quantization width Capacity needed for 2.8T parameters Quality risk When it fits
    1 bit ~350 GB Higher risk to output quality; needs task-level evaluation Extreme compression, offline validation, resource-constrained experiments
    3 bit ~1.05 TB Still lossy, but generally better than 1 bit Offline batch processing, algorithm research, non-interactive inference
    8 bit ~2.8 TB Closer to a high-precision baseline Quality first, sufficient capacity, tolerating lower throughput

    From a pure capacity standpoint, the appeal of persistent memory is that it provides a large, fast memory pool that sits between DRAM and conventional disks. Ordinary DRAM is more expensive and smaller in capacity; a traditional SSD is large but typically has higher latency and weaker random access than memory-class media. Persistent memory is deployed in memory form, offering hundreds of gigabytes to several terabytes of capacity, which makes it suitable for holding model weights that do not fit in DRAM.

    The caveat matters: enough capacity only means the model fits. It says nothing about generating quickly. Real inference experience is determined by memory bandwidth, the quantization strategy, the access path, and caching behavior.

    3. Hardware and Software Architecture: CPU, Persistent Memory, and Access Patterns

    A typical low-cost experimental platform has three parts:

    1. Processor — a multi-core server CPU, using a large thread count to absorb matrix computation and memory access pressure.
    2. Memory — large-capacity persistent memory holding the bulk of the model weights.
    3. Software stack — an inference framework that supports CPU inference, quantized weight loading, and multi-threaded batching.

    Persistent memory is generally used in one of two ways.

    Memory mode. Persistent memory is transparent to the software; the operating system treats it as an extension layer of DRAM. Applications simply see a large amount of memory, but the read and write policies are managed by the OS, and programmers have relatively weak control over data placement.

    Application direct mode. Here DRAM and persistent memory appear to the software as two separate address spaces, so the application can explicitly decide which weights live in persistent memory and which weights or caches live in DRAM. For very large model inference, application direct mode is easier to tune, because the framework can make fine-grained decisions based on weight size, access frequency, and caching policy.

    One clarification is important: application direct mode does not mean “bypass the operating system cache entirely.” Persistent memory can be exposed to software in different ways. With DAX direct mapping, data can be accessed without going through the traditional page cache path. Without DAX — using ordinary filesystem page cache or a regular mapping path instead — the OS may cache some of the pages it has touched in DRAM.

    So when tuning, do not simplify “application direct” to “all data must go straight to persistent memory.” What actually matters is which data path the inference framework ends up using, whether DRAM participates in caching, and what is being cached — weight pages or temporary intermediate results.

    4. Performance Tuning: Two Levers and a Common Baseline

    In bandwidth-bound CPU inference, generation speed is roughly:

    Generation speed (tok/s) ≈ effective memory bandwidth ÷ weight bytes read per token

    That gives two tuning directions: compress the weights to shrink the denominator, or optimize the access path to grow the numerator — that is, raise effective memory bandwidth.

    To avoid mixing up results from different setups, performance numbers should share a common baseline. At minimum, record: CPU model and core count, DRAM capacity and bandwidth, persistent memory model and capacity, inference framework version, quantization implementation, whether page caching is enabled, and whether you measured prefill or steady-state decode. Without these conditions aligned, different tok/s figures cannot be compared directly.

    Lever 1: Compress the Weights, Shrink the Denominator

    Quantization is the most direct compression mechanism. The lower the bit width, the fewer weight bytes must be read per generated token, so throughput should rise in theory.

    Here one common misconception needs to be avoided: 3-bit quantization is not merely “a little slower” than 1 bit. If the bytes read per token strictly triple and everything else stays constant, speed should in theory fall to roughly one third.

    For example, assuming a purely bandwidth-bound case with all other conditions identical, if 1-bit steady-state decode runs at about 0.85 tok/s, the theoretical figure for a 3-bit model should be around 0.28 tok/s. If some test record shows 3 bit still reaching 0.75 tok/s, that usually means the run was not on the same baseline as the 1-bit measurement. Possible reasons: DRAM page caching was enabled for the 3-bit run, the 1-bit run was limited by dequantization or compute logic, the measurement included the prefill phase, or the model was not fully read but had sparse activation.

    Under the common baseline used in this article, then, 3 bit should not be described as “only 12% slower than 1 bit.” The accurate statement is: in steady-state decode that is purely bandwidth-bound with caching disabled, 3 bit is noticeably slower than 1 bit; if the measured gap is small, the caching, compute, or measurement conditions must be explained.

    Quantization is also not a case of “more aggressive is better.” Choosing among 1 bit, 3 bit, and 8 bit means weighing speed, capacity, and output quality together. Offline validation can tolerate the accuracy loss from low bit widths, but for question answering, code generation, or high-quality long-form writing, the quality risk has to be reassessed.

    Lever 2: Optimize the Access Path, Grow the Numerator

    Whether DAX is enabled changes effective memory bandwidth substantially.

    With DAX enabled, persistent memory can be mapped and read directly, shortening the data path — but it usually also means less use of DRAM as an acceleration layer. When weights are enormous and DRAM is too small to cache all of them, frequently reading persistent memory adds noticeable latency.

    With DAX disabled, the OS can keep some weight pages in DRAM through the page cache. What gets cached here is not “hot data” in the conventional sense, because autoregressive decode reads all weights for every token and there is no stable temporal locality. A more precise description is that the system caches whatever shard of the weights fits in DRAM. Those shards are served from fast DRAM, which raises overall effective bandwidth.

    On the same baseline: if a 1-bit model runs at about 0.85 tok/s with DAX enabled and no DRAM caching, versus about 1.50 tok/s with DAX disabled and DRAM page caching allowed to participate, that is roughly a 76% improvement. It shows that when capacity allows caching part of the weights, a DRAM caching layer is often more effective than chasing “zero-copy direct access.”

    The gain is not unbounded, though. The larger the DRAM and the more weight shards it can hold, the more visible the caching benefit. If DRAM can only cache a small fraction of the weights, the improvement is limited; if weight layout and access order make cache hits unstable, the result will fluctuate.

    5. Capability Boundaries: What This Path Fits and What It Does Not

    The value of this route is clear, but so are its limits.

    Speed is still low. Even at roughly 1.5 tok/s after optimization, it is far below the tens of tok/s that interactive chat applications usually require. It suits offline batch processing, algorithm validation, long-form generation, and model behavior analysis — scenarios that do not need real-time response.

    The bandwidth ceiling is hard to raise. Sustained read bandwidth from persistent memory is typically lower than high-end GPU memory, and it depends on channel count, NUMA topology, memory slot layout, framework implementation, and quantization format. The bigger the model, the more is read per token, and the more pronounced the slowdown.

    Quantization cuts both ways. Low bit widths reduce both capacity and read volume but can damage output quality. The 1-bit scheme in particular needs caution and must be calibrated through perplexity, task accuracy, human evaluation, or A/B testing.

    This is not a production-grade high-concurrency inference design. If the goal is a large-scale online service for end users, you still need GPUs, dedicated accelerators, or a well-optimized cloud inference cluster.

    Its proper positioning, therefore, is low-cost feasibility validation: using limited hardware to observe the behavior of a very large model, not claiming to replace GPUs entirely.

    6. Implementation Checklist and Common Questions

    To make this easier to reuse, here is an engineering-oriented checklist.

    Which scenarios suit persistent memory inference?

    Offline analysis, batch text generation, model capability validation, quantization strategy research, and long-context experiments. If the workload demands low latency, high concurrency, or a stable SLA, choose GPUs or elastic cloud inference instead.

    How do you evaluate the quality loss from 1-bit quantization?

    Build a small evaluation set covering common-sense QA, instruction following, code generation, and long-document summarization. Record perplexity, human preference, and error categories alongside it. Do not judge only by whether the output “sounds human” — judge by whether the key tasks remain usable.

    How do you estimate your own tok/s?

    First size the model capacity, then measure achievable effective memory bandwidth. If bytes read per token is B and effective bandwidth is BW, theoretical throughput is roughly BW ÷ B. Real results vary with caching, NUMA, thread scheduling, and the quantization implementation, so measure it under a common baseline.

    What performance pitfalls come up often?

    Not separating prefill from steady-state decode; assuming DRAM caching still applies after enabling DAX; poor persistent memory slot layout; incorrect NUMA binding; a quantization implementation with extra dequantization overhead; and measuring a single generation pass instead of taking a steady-state value.

    7. Reproducing the Pattern in the Cloud and Selection Advice

    If you want to reproduce the “large memory plus tiered storage” idea in the cloud, you can use object storage as the weight repository and a high-memory compute instance as the DRAM caching layer, then add monitoring and alerting to watch memory bandwidth, throughput, queue depth, and error rates. For quick validation, a general-purpose elastic compute plus storage combination works too: complete small-sample inference first, then decide whether to scale up resources.

    Cloud selection is not simply a matter of buying the most memory. It has to be matched to model capacity, quantization bit width, concurrency, and throughput targets. Persistent memory inference is better understood as a resource-constrained experimental path than as a default production architecture.

    8. Conclusion

    Running a trillion-parameter model on a multi-core CPU with persistent memory is about more than any single tok/s figure. The real lesson is that the CPU inference bottleneck usually lies not in peak compute but in effective memory bandwidth, quantization layout, and the caching path. When capacity, access mode, and quantization strategy are combined sensibly, a GPU-free environment becomes a low-cost way to explore large model behavior.