Cost and Engineering Practices for Local Deployment of Trillion-Parameter LLMs

I. Background: Private Deployment Enters the Engineering Delivery Phase

Over the past two years, the private deployment of large models has largely remained in the “Proof of Concept” (PoC) phase. Typical challenges faced by enterprises include unstable supply of high-end accelerator cards, mismatches between operator libraries and model structures, and insufficient support for ultra-long contexts in inference frameworks. These factors have prevented many projects from being delivered on time, even after budget approval.

With the gradual maturation of the large model and computing power ecosystem, this situation is undergoing a substantial change. Model weight formats are becoming standardized, mainstream inference frameworks increasingly support the Mixture of Experts (MoE) architecture, and the memory capacity and interconnect bandwidth of accelerator cards have reached a level capable of handling trillion-parameter models. The core conflict of private deployment has shifted from “unable to acquire cards or debug operators” to “how to rationally plan resources and budgets.”

This article systematically analyzes the memory constraints, cost tiers, and engineering practices of local deployment around a typical open-source trillion-parameter model event, providing a reference decision framework for technical teams.

II. Hard Constraints: Weight Size vs. Memory Capacity

2.1 Logic for Calculating Weight Size

The weight size of a trillion-parameter model can be estimated using the following formula:

Weight Size ≈ Number of Parameters × Bytes per Parameter

Taking mixed precision (e.g., BF16) storage as an example, each parameter occupies 2 bytes. If the number of model parameters reaches approximately 780 billion (0.78 trillion), the weight size is roughly:

780 Billion × 2 bytes ≈ 1.56 TB

This is consistent with the weight scale of common trillion-level models in the industry. It should be noted that actual deployment requires additional memory for:

• KV Cache: Grows linearly with context length and concurrency;
• Activations: Intermediate tensors during forward propagation;
• Communication Buffers: Gradient or activation transfers during multi-card interconnection;
• Framework Overhead: The inference engine’s own memory usage.

Therefore, available memory must be significantly larger than the weight size, rather than exactly equal to it.

2.2 The Real Bottleneck of Single-Card Memory

The memory capacity of current mainstream accelerator cards is primarily concentrated in the 64GB to 192GB range. Even when using 192GB memory models, a single card cannot accommodate the 1.56TB weights. This means:

• Single-card deployment is unfeasible: Weight sharding via multi-card parallelism is mandatory;
• Single-node multi-card has limits: A single server typically supports 8 to 16 accelerator card slots, with total memory ranging from 1TB to 3TB;
• Cross-node interconnection becomes essential: When weights exceed the memory capacity of a single node, high-speed network interconnects must be introduced.

Taking a specific accelerator card as an example, a single node configured with 16 cards provides a total memory capacity of roughly 1.15TB, which cannot accommodate the complete weights. This leads to the deployment tier classifications in the next section.

III. Deployment Tiers and Cost Composition

Based on memory capacity, interconnect bandwidth, and inference performance requirements, local deployment plans can be divided into three tiers. The cost estimates below are for reference only; actual prices are affected by supply chains, procurement channels, and deployment scale.

3.1 Tier 1: Entry-Level (Approx. 8.2 - 8.5 Million RMB)

Configuration: Two 16-card servers interconnected via a high-speed network.

Total Memory: Approximately 2.3TB (combined for dual nodes). After deducting framework overhead and KV Cache, it can barely support inference services with short contexts and low concurrency.

Applicable Scenarios:

• Internal technical validation and prototype development;
• Single-turn conversations or short text generation;
• Model fine-tuning experiments in non-production environments.

Limitations:

• Limited concurrency, struggling to support multi-user simultaneous access;
• KV Cache rapidly consumes memory in long-context scenarios;
• Cross-node communication latency may become a performance bottleneck.

This tier can be viewed as an “entry ticket”—proving the team possesses the foundational capability to deploy trillion-parameter models, but still far from production-ready.

3.2 Tier 2: Comfortable-Level (Approx. 16 Million RMB)

Configuration: A cluster of four 32-card servers.

Total Memory: Approximately 4.6TB, leaving ample margin for weights, KV Cache, and activations.

Applicable Scenarios:

• Medium-scale production inference services;
• Support for longer contexts (e.g., 32K to 128K tokens);
• Multi-user concurrent access with reasonable throughput.

Advantages:

• Sufficient memory margin to enable more aggressive batching strategies;
• More regular cross-node interconnect topology, resulting in higher communication efficiency;
• Supports online expansion for smooth upgrades as business grows.

This tier is the rational choice for most enterprises transitioning from validation to production.

3.3 Tier 3: Recommended-Level (Approx. 31 - 32 Million RMB)

Configuration: An eight-node, 64-card super-node cluster.

Total Memory: Approximately 9.2TB, capable of handling complex scenarios such as high concurrency, long contexts, and multimodal inputs.

Applicable Scenarios:

• Large-scale production environments serving hundreds to thousands of concurrent users;
• Ultra-long contexts (e.g., 256K+ tokens);
• Mixed workloads of model fine-tuning and inference;
• Scenarios with strict SLA requirements for latency and throughput.

Key Capabilities:

• Super-node architecture provides high-bandwidth, low-latency inter-card interconnection;
• Supports combined strategies of tensor parallelism, pipeline parallelism, and expert parallelism;
• Features fault isolation and hot-swapping capabilities to improve service availability.

This tier is the “standard posture” recommended for enterprises intending to use trillion-parameter models as core infrastructure.

IV. Engineering Practices: From “Running” to “Running Well”

4.1 Memory Offloading and Multi-Level Caching

When memory is insufficient to hold all weights, a memory offloading strategy can be adopted to temporarily store part of the weights or KV Cache in host memory or local NVMe storage. Typical approaches include:

• Layered Weight Loading: Offloading inactive expert modules to host memory and swapping them in on demand;
• Tiered KV Cache Storage: Keeping hot data in GPU memory, warm data in host memory, and cold data on disk;
• Prefetching and Asynchronous Transfer: Utilizing the overlap of computation and transfer to hide swap-in/swap-out latency.

The following is a pseudocode illustration:

# Illustration of layered weight loading
class LayerOffloadManager:
    def __init__(self, gpu_mem_limit, host_mem_limit):
        self.gpu_cache = LRUCache(gpu_mem_limit)
        self.host_cache = LRUCache(host_mem_limit)
    def get_layer(self, layer_id):
        if layer_id in self.gpu_cache:
            return self.gpu_cache[layer_id]
        if layer_id in self.host_cache:
            layer = self.host_cache.pop(layer_id)
            self.gpu_cache.put(layer_id, layer)
            return layer
        # Load from disk
        layer = load_from_disk(layer_id)
        self.host_cache.put(layer_id, layer)
        return self.get_layer(layer_id)

4.2 Parallelism Strategy Selection

The parallelism strategy for trillion-parameter models requires comprehensive consideration of model structure, hardware topology, and business workload:

Parallelism Method	Applicable Scenario	Communication Overhead
Tensor Parallelism	Single-layer parameters too large, need splitting across cards	High, requires All-Reduce
Pipeline Parallelism	Many layers, can be split by layer	Medium, requires point-to-point communication
Expert Parallelism	MoE architecture, large number of experts	Medium, requires All-to-All
Data Parallelism	Improving throughput, replica redundancy	Low, requires gradient synchronization

In actual deployment, a hybrid parallelism strategy is typically adopted. For example, a combination of expert parallelism + tensor parallelism + pipeline parallelism to strike a balance between memory, communication, and computational efficiency.

4.3 Operator Adaptation and Graph Optimization

Operator adaptation between accelerator cards and mainstream inference frameworks is critical to deployment success. Common issues include:

• Missing Custom Operators: Operators like FlashAttention and RMSNorm need to be rewritten for specific hardware;
• Insufficient Graph Optimization: Operator fusion, memory reuse, and other optimizations require deep collaboration between framework and hardware;
• Precision Alignment: The numerical consistency between mixed-precision training and inference needs verification.

It is recommended to conduct an operator coverage check before deployment to identify missing operators and evaluate their performance impact. For critical operators, collaboration with hardware vendors or framework communities can be sought for implementation.

4.4 Servitization and Observability

Production environment deployment requires comprehensive service capabilities:

• Request Queue and Batching: Dynamic batching (Continuous Batching) can significantly increase throughput;
• Rate Limiting and Degradation: Ensuring core business operations during overload;
• Monitoring and Alerting: Metrics such as memory usage, KV Cache hit rate, Time-To-First-Token (TTFT) latency, and throughput must be collected in real-time;
• Logging and Tracing: To facilitate problem localization and performance analysis.

V. Decision Framework: When to Choose Local Deployment

Local deployment is not suitable for all scenarios. The following decision framework can serve as a reference:

1. High Data Privacy Requirements: Sensitive data cannot leave the enterprise boundary;
2. Cost-Sensitive Workloads: Under long-term, high-frequency usage, the marginal cost of local deployment is lower;
3. Strong Customization Needs: Requires fine-tuning, distillation, or deep integration with internal systems;
4. Strict Latency Requirements: Needs stable, low-latency responses unaffected by public network fluctuations;
5. Compliance and Auditing: Requires complete control over model versions and the inference process.

Conversely, if the business is in the early exploration stage, has a low call frequency, or has a continuous need for the latest model capabilities, prioritizing cloud API calls is advisable.

VI. Conclusion

The local deployment of trillion-parameter large models is transitioning from a “technical adventure” to “engineering delivery.” The core challenges lie in:

• Memory Constraints: Weight size far exceeds single-card capacity, necessitating multi-card, multi-node parallelism;
• Cost Tiers: Entry, comfortable, and recommended tiers correspond to different business stages;
• Engineering Details: Memory offloading, parallelism strategies, operator adaptation, and servitization are all indispensable.

For technical teams, the key is not pursuing the “lowest cost” but finding a resource plan that matches business requirements. Before budget approval, it is recommended to complete a small-scale validation to clarify the actual boundaries of memory, latency, and throughput, and then formulate a deployment plan accordingly. The private deployment of trillion-parameter models is essentially a systems engineering problem that requires collaborative optimization across models, computing power, frameworks, and operations.