No GPU Needed: Running Trillion-Parameter LLM Inference with Persistent Memory

    This article explains why memory capacity, rather than raw compute, often becomes the first bottleneck for large language model inference. It describes how server-grade persistent memory can provide multi-terabyte addressable memory for CPU-based servers, covers App Direct, Memory Mode, and DAX access patterns, shows a memory-mapped quantized inference example, and discusses quantization tradeoffs, observed token-generation performance, and the scenarios where this approach is practical.

    1. Background: Why GPU Memory Becomes the First Wall for LLMs

    In LLM inference, a commonly overlooked fact is that the first constraint on deploying very large models is often not compute but memory capacity. Autoregressive inference requires the full set of model weights to be placed in addressable memory, and weight size grows linearly with parameter count.

    For a trillion-parameter-class model, a rough estimate shows the pressure:

    • FP16 (2 bytes per parameter) requires several TB.
    • 8-bit quantization is still on the order of 1 TB.
    • 4-bit quantization still requires hundreds of GB.
    • Even aggressive 1-bit quantization requires hundreds of GB of addressable space.

    Mainstream consumer GPUs typically provide tens of GB of VRAM. Even data-center accelerators often cannot host such a model on a single card by themselves. Multi-GPU parallelism is possible, but it introduces communication overhead, partitioning strategies, cost, and operational complexity. This naturally raises a question: if GPUs are temporarily unavailable, can a large model still be run through another path?

    2. Key Premise: Expanding Addressable Memory with Persistent Memory

    The core idea is to change the memory hierarchy, not the compute unit.

    Server platforms support a large number of memory slots. Server-grade persistent memory, a byte-addressable non-volatile memory module, can provide hundreds of GB per module on the memory bus. By mixing persistent memory with ordinary DRAM, a general-purpose multi-core server can be equipped with multi-terabyte addressable memory. This is the physical basis for running a very large model without a GPU.

    Two distinctions should be emphasized:

    1. Capacity first, not bandwidth first. Persistent memory has high capacity density and lower cost, but its access bandwidth and latency are usually not as good as DRAM. It solves the “can it fit” problem, not the “does it run fast” problem.
    2. The CPU must support the required memory control and addressing. Such configurations typically target server processor platforms with multiple cores and multiple memory channels. Whether a specific system supports them depends on hardware generation and BIOS/firmware configuration.

    3. Memory Tiering and Access Patterns: App Direct and DAX

    How persistent memory is used directly affects inference performance. The industry commonly discusses the following concepts:

    • App Direct mode: DRAM and persistent memory appear to software as two independent data spaces. The application decides which data belongs to which tier. This offers strong control, but the application must explicitly manage data placement.
    • Memory Mode: persistent memory is presented as volatile memory capacity, while DRAM acts as a high-speed cache. This is transparent to software, but how well hot data hits the cache depends on access locality.
    • Direct Access (DAX): allows memory-mapped files to bypass the operating system page cache, letting the processor read and write file data in persistent memory directly and avoiding an extra copy.

    In a typical setup, developers map the model file into the address space with memory mapping (mmap) and load weight pages on demand. The following pseudocode illustrates the general structure of “memory mapping + quantized inference”; it is not the implementation of any specific framework:

    1. # Illustrative example: load quantized weights on demand through memory mapping
    2. import mmap
    3. # Open a quantized model file (weights are compressed to low bit widths)
    4. model_file = open("weights quantized.bin", "rb")
    5. mm = mmap.mmap(model_file.fileno(), 0, access=mmap.ACCESS_READ)
    6. # During inference, read layer weights from the mapping; the OS handles paging as needed
    7. for layer in model_layers(memory_view=mm):
    8. out = layer.forward(input_tensor) # Execute matrix operations on the CPU
    9. input_tensor = out
    10. # When DAX is disabled, frequently used pages can remain in DRAM cache to reduce repeated access overhead

    4. Quantization: Turning “It Fits” into “It Runs”

    Persistent memory solves capacity, but making a trillion-parameter model truly fit into limited memory also requires quantization to shrink the weights. Common approaches include low-bit quantization such as 1-bit, 3-bit, and 4-bit:

    • Lower bit widths reduce memory usage, but model accuracy and generation quality may decline.
    • Quantization is often combined with group-wise scaling factors (scale/zero-point) to mitigate accuracy loss.
    • Inference frameworks need to dequantize weights during loading, which introduces additional compute overhead.

    In other words, quantization and memory tiering work together: quantization compresses the model into the memory budget, while the tiering strategy tries to keep frequently accessed data in the faster layer.

    5. Performance Characteristics: Why “It Runs” Does Not Mean “It Is Usable”

    A home-lab case shared by a developer community is representative (the numbers are observations for that specific configuration and do not represent a general benchmark): on a general-purpose multi-core server, a trillion-parameter-class model with 1-bit quantized weights was hosted in large-capacity persistent memory and inferred through memory mapping. The initial generation speed was about 0.85 tokens per second. After disabling direct access so that frequently used data could benefit from DRAM cache, the speed increased to about 1.50 tokens per second. When switching to 3-bit quantization, the speed instead dropped to about 0.75 tokens per second, and generating 50,000 tokens took more than 18 hours.

    These numbers reveal a key conclusion: autoregressive decoding for large models is fundamentally a memory-bandwidth-bound workload. Each token requires reading a huge number of weights, and the bandwidth bottleneck of persistent memory is amplified at each step. Three engineering rules follow:

    1. Higher bit widths mean more data to read and slower speed. This explains why 3-bit quantization can be slower than 1-bit in this setup.
    2. Cache hits can significantly improve speed. Placing hot data in DRAM can noticeably increase throughput.
    3. It is a capability validation, not a productivity solution. It is suitable for feasibility tests, debugging, and offline batch processing, not low-latency online serving.

    6. Applicability Boundaries: When Is This Approach Worth It?

    The value of this approach is to exchange low-cost, high-capacity memory for “runnability.” The applicable and non-applicable scenarios are clear:

    • Applicable: local or offline experiments, model architecture validation, accuracy comparisons, teaching and research, emergency use when accelerators cannot be procured, and environments where memory is severely constrained and low speed is acceptable.
    • Not applicable: user-facing low-latency online inference, high-concurrency services, and high-throughput batch-processing scenarios. These should still prioritize accelerators or mature inference clusters.

    7. General Guidance for Enterprise Deployment

    To deploy very large model inference in production, general cloud capabilities can help build a robust pipeline: use elastic compute to scale compute resources on demand, object storage to store model weights and data versions, container platforms to standardize deployment and orchestration of inference services, and logging services with monitoring and alerting to track memory usage and latency metrics. For example, Baidu AI Cloud offers general capabilities such as elastic compute, object storage, container platforms, and monitoring/alerting that can be combined into a model hosting and inference service approach. The final choice should be evaluated against the model size, latency requirements, and cost budget.

    8. Summary

    The essence of “running a trillion-parameter model without a GPU” is not the discovery of new compute, but a re-examination of the fundamental bottleneck: memory hierarchy and capacity constraints. By expanding addressable space with persistent memory, compressing weights with quantization, and optimizing access paths with memory tiering and caching, developers can validate the feasibility of very large model inference on general-purpose servers. Understanding three points—memory-bandwidth-bound decoding, the tradeoff between quantization bit width and speed, and applicability boundaries—is more important than pursuing specific hardware models. They are the truly general criteria for very large model deployment decisions.