Running Trillion-Parameter Model Inference Without Accelerators

    This article presents an engineering approach for running trillion-parameter language models without dedicated accelerator cards. It explains how to use large byte-addressable memory for model weights, faster volatile memory for hot runtime data, low-bit quantization to reduce memory footprint, and CPU-side optimization to make oversized models practical for validation and offline workloads. It also clarifies why “can run” and “interactive” are fundamentally different performance targets.

    Capacity Comes Before Compute

    Discussions about deploying very large language models have long focused on two questions: whether dedicated accelerators are available and whether their device memory is large enough. Once a model reaches the trillion-parameter scale, the first problem becomes capacity. The weights alone may require several times the memory available on a typical accelerator card. Activations, key-value (KV) cache data, and runtime buffers add further overhead.

    The absence of an accelerator does not mean that computation is impossible. It means that execution must fall back to general-purpose processors and the host memory hierarchy. Throughput is then determined jointly by memory bandwidth, access locality, and the amount of data moved after quantization.

    This article abstracts that type of deployment into a reusable engineering method:

    • Use large byte-addressable memory to hold model weights.
    • Use faster volatile memory for frequently accessed runtime data.
    • Use low-bit representations to reduce the memory footprint.
    • Accept that getting a model to run and making it interactive are two different goals.

    The Capacity Bottleneck Usually Appears First

    The main inference loop reads weights, performs multiply-accumulate operations with the input or cached state, and writes the updated state. For extremely large models, the amount of weight data touched during a forward pass can exceed the amount of data that the processor can usefully process within the same time window.

    In other words, the system is often blocked first by moving data, not by completing arithmetic. Dedicated accelerators benefit from both high-bandwidth device memory and specialized matrix units. However, if the model does not fit in accelerator memory, even substantial on-chip compute capacity cannot solve the problem. Moving data repeatedly between the host and the device can also introduce additional overhead.

    For a no-accelerator deployment, the first objective is therefore not to match interactive generation speed. It is to create a sufficiently large space where software can address model weights at byte granularity.

    Persistent memory sits between conventional DRAM and block storage. Its capacity is typically larger than the system’s DRAM capacity, while its latency and bandwidth are substantially better than loading a model from disk page by page. It is suitable for a large weight region whose access pattern is relatively sequential or can be prefetched. It is not an ideal destination for every type of runtime state.

    Assigning Roles in a Two-Tier Address Space

    The system should divide data into cold weights and hot state instead of placing everything in the same type of memory.

    Cold data consists primarily of the parameters in each layer. Even after low-bit quantization, the weights can remain extremely large. Their access pattern is usually layer-oriented and can often be prefetched or scanned sequentially. Persistent memory, or another large byte-addressable medium, is a suitable location when capacity and stable bandwidth are the primary concerns.

    Hot state includes:

    • Activations for the current layer
    • Attention-related KV cache data
    • Small workspaces for tokenization and sampling
    • Runtime structures that cannot be replaced by the operating system’s page cache

    These data structures are reused frequently and are more sensitive to latency, so they should remain in DRAM whenever possible.

    If the platform allows software to treat persistent memory and DRAM as two separate regions, placement should be explicit:

    • Put model weights in the large-capacity region.
    • Put caches and runtime state in the faster region.

    If persistent memory is instead exposed as a larger conventional memory pool, hardware or the operating system may automatically decide what remains in the faster cache. This simplifies configuration, but makes hotspot behavior less transparent. An extremely large model can easily fill the limited fast-memory capacity, causing performance to degrade toward the speed of the slower medium.

    Direct access allows the application to read and write persistent memory at its native granularity, removing one layer of page-cache copying and shortening the data path. However, once the working set exceeds DRAM, slower accesses become directly visible.

    By contrast, allowing frequently used pages to pass through DRAM caching adds copying and consumes fast memory, but can improve effective bandwidth when the same portions of the weights are scanned repeatedly.

    Neither approach is universally better. The choice depends on:

    • Whether the quantized model is larger than fast memory
    • Whether the generation process repeatedly touches the same pages
    • How much locality the runtime can expose

    Testing should use fixed thread counts, batch sizes, and context lengths. Measure both time to first token and steady-state generation speed. Do not rely on a single result that may have benefited from an accidental cache hit.

    Quantization Is a Capacity Requirement, Not a Decoration

    Without adding more memory modules, reducing weights from higher-precision floating point to very low-bit representations is the only broadly applicable way to reduce the footprint by an order of magnitude. Three factors must be evaluated together.

    1. Footprint

    The number of bits per parameter approximately determines the size of the weights. However, vocabulary data, normalization parameters, alignment padding, and other metadata do not necessarily shrink at the same rate.

    Actual memory consumption should therefore be measured rather than estimated only by multiplying parameter count by bit width.

    2. Bandwidth

    Generation is often performed with a small batch size or even a single request. Each generated token may require scanning a large portion of the weights.

    Lower bit widths allow the same memory bandwidth to deliver more effective parameters to the compute units. However, dequantization, table lookup, and bit unpacking also consume processor cycles. Once the cost of expansion exceeds the time saved by moving fewer bytes, further reducing the bit width may no longer improve performance.

    3. Quality

    Extremely low-bit representations can harm long-context consistency and precision-sensitive tasks such as numerical reasoning and code generation.

    Offline summarization, rough retrieval, and batch labeling may tolerate this degradation. Multi-turn user-facing conversations should treat very-low-bit output as a candidate result rather than assuming production-level quality.

    A more robust strategy is to use mixed precision:

    • Quantize the largest feed-forward weights more aggressively.
    • Keep a smaller set of output-sensitive parameters at higher precision.
    • Store the KV cache at a precision that DRAM capacity can support.

    The KV cache grows linearly with context length. With sufficiently long contexts, it can exceed the weights and become the new capacity bottleneck. This must be budgeted separately during system design.

    Organizing the CPU-Side Inference Loop

    Without an accelerator, computation runs on the vector units of general-purpose processors. The most predictable optimizations are limited in scope and should focus on preventing stalls rather than adding layers of unrelated tuning.

    Align Threads with Memory Nodes

    In multi-socket servers, both persistent memory and DRAM are attached to specific processor nodes. Reading weights across nodes can reduce effective bandwidth further.

    The inference process should be bound to the node where the weights are located. The number of worker threads should not exceed what that node’s physical cores and memory channels can feed. Adding too many threads usually increases contention instead of improving throughput.

    Prefer Sequential Access and Prefetching

    During layer-by-layer execution, weight addresses should remain as contiguous as possible. Block sizes should be compatible with cache lines and the memory-channel interleaving scheme.

    The following pseudocode illustrates the placement relationship without binding the design to a particular runtime:

    1. for layer in model.layers:
    2. w = persistent_view(layer.weight_offset, layer.weight_bytes)
    3. h = dram_buffer(layer.hidden)
    4. kv = dram_kv(layer, seq_pos)
    5. h = mlp(dequant(w.ffn), h) # Large weights, sequential reads
    6. h = attention(w.attn, h, kv) # Smaller state, more random access
    7. write_kv(kv, h)

    The persistent view is read-only and contains the weights. DRAM holds the hidden state and KV cache. If dequantization can be performed while data is being read, avoid first expanding a complete full-precision copy in DRAM. Otherwise, the capacity advantage of quantization may be lost.

    Batch Size and Context Length Matter

    Interactive workloads often use a batch size of one, which produces poor bandwidth utilization. This is also where the lower bound of generation speed tends to appear.

    Offline workloads should combine multiple short requests when possible. A single scan of the weights can then serve multiple sequences, reducing the amortized cost of data movement.

    As the context becomes longer, KV-cache reads become more expensive, and generation speed decreases with the amount of text already processed. Evaluation should therefore report results across a range of context lengths instead of publishing only a single point measurement.

    Page and Cache Policies

    Large pages can reduce page-table overhead. Prefetching can help when scanning weight data sequentially, but may pollute the cache when accessing KV data with a wide stride or irregular pattern.

    Direct access, prefetch-window size, and related settings should be tested as controlled comparisons rather than enabled or disabled based on assumptions. A claim that disabling a feature is always faster depends on whether the working set fits in DRAM and how the model accesses memory.

    Performance Expectations and Appropriate Use Cases

    In public discussions, configurations of this type often show generation speeds far below interactive thresholds. Smooth streaming output typically requires each token to arrive within tens of milliseconds. If each token takes close to a second or longer, generating tens of thousands of tokens becomes an operation measured in hours.

    This gap is primarily a consequence of memory bandwidth and does not depend on any single benchmark result. The most reasonable use cases for this approach are:

    • Verifying that a very large model can be loaded and that its numerical results are correct on existing host hardware
    • Running overnight or queued offline generation, evaluation, and data-production jobs
    • Providing a capacity fallback when accelerator resources are unavailable

    For online services, inference clusters with large amounts of high-bandwidth device memory should remain the preferred option. Elastic inference and managed model-hosting services can also be used when interactive latency must be handled by dedicated resources.

    A persistent-memory host configuration is better suited as a supplementary option when cost, hardware inventory, or data-center constraints prevent the addition of accelerators.

    If DRAM already has enough capacity for the quantized model, persistent memory will not improve speed. It will only add complexity and additional failure modes.

    Correctness, Persistence, and Operations

    Persistence does not automatically guarantee correctness.

    Inference weights should be treated as read-only whenever possible. Verify their integrity before loading, and prevent temporary runtime state from being written into the weight region during execution.

    After a power failure, data stored on persistent media may remain available, but the KV cache and unfinished requests held in volatile memory will be lost. Recovery should therefore be designed around replayable input rather than assuming that an incomplete generation can resume in place.

    Capacity planning should list the following items separately:

    • Quantized model weights
    • Runtime code and libraries
    • The KV-cache limit for each concurrent sequence
    • Space reserved for page caching or direct access
    • Memory consumed by the operating system itself

    If concurrency or context limits exceed the budget, the result is often not a graceful performance reduction. It may instead be paging storms or allocation failures.

    Monitoring should expose:

    • Bandwidth utilization on each memory node
    • The ratio of local to remote memory accesses
    • Large-page hit behavior
    • Temperature and power consumption during long generation jobs

    A model that starts successfully is not necessarily a model that can complete a long-running workload reliably.

    Security also requires attention. Large persistent-memory regions may contain plaintext weights and intermediate results in addition to data protected by conventional disk-encryption practices. Before hardware is retired or relocated, data must be cleared according to the characteristics of the medium. Deleting files alone may not be sufficient.

    On multi-tenant hosts, untrusted workloads should not be allowed to map into the same address space that carries the model.

    Deployment Checklist

    1. Calculate the footprint before selecting hardware. Parameter count, bit width, vocabulary data, and the KV-cache limit jointly determine the minimum capacity.
    2. Define the roles of the two memory tiers. Put weights in the large-capacity region and runtime state in the faster region. Do not assume automatic caching will understand the model’s access pattern.
    3. Compare direct access with cached access under fixed workloads, and record steady-state generation speed at multiple context lengths.
    4. Validate quantization together with task quality. Reserve extremely low-bit formats for offline tasks that can tolerate the associated error.
    5. Bind processes to the appropriate memory node, control the thread count, maintain sequential weight layout, and avoid unnecessary full-precision expansion.
    6. Set separate expectations for interactive latency and offline throughput. Hour-scale long-form generation may be acceptable for batch processing, but not for a conversational product.
    7. Keep weights read-only, verify integrity, ensure persistent data can be securely cleared, and enforce hard limits on concurrency and context length.

    Running a trillion-parameter model without an accelerator demonstrates that memory hierarchy design and quantization can make “fits in memory” a reality. It does not mean that a general-purpose processor has suddenly acquired accelerator-class bandwidth.

    The practical design is to separate cold and hot data, position the system as an offline workload platform, and include quality validation and capacity budgeting in the same plan. If stable low latency is a real business requirement, high-bandwidth inference resources remain the appropriate choice. The value of the persistent-memory approach is that, when accelerators are unavailable, it keeps experimentation and batch processing from being blocked entirely by capacity.