Cost and Architecture Analysis of Deploying Open-Source Large Models Locally

1. Why “downloadable weights” does not mean “locally runnable” for large sparse models

Many teams see open-source weights released and immediately think: download them and deploy a local instance. But for models with hundreds of billions or even trillions of parameters, the weight file size is only the first obstacle. The real factors that determine whether deployment is feasible include:

1. Total GPU memory required for weights
2. Activation memory and KV cache during inference
3. Communication bandwidth across multiple machines
4. Additional scheduling overhead caused by sparse expert routing
5. Non-linear memory growth in long-context scenarios

Local deployment therefore cannot be evaluated solely by model capability. It must be treated as a distributed inference system.

Take as an example a model with roughly 2.8T total parameters, a sparse MoE architecture, and native MXFP4 weight format. Its weight size is approximately 1.56TB. Loading it on a single machine is nearly impossible with conventional GPU servers. Even if the weights fit, activation memory and long-context caching during inference can quickly exceed the GPU memory limit.

More importantly, although a sparse MoE model only activates part of its parameters for a single inference step, all expert weights still need to remain resident in GPU memory. Otherwise dynamic routing cannot be completed. This means its memory requirement is far higher than that of a dense model with the same number of activated parameters.

2. Memory estimation: from weights to total inference footprint

2.1 Weight memory

Assuming a model weight size of 1.56TB stored in MXFP4 format, at least 1.56TB of usable GPU memory is required in theory. In practice, deployment also requires additional consideration of:

• Memory management overhead in the inference framework
• Alignment and padding after weight sharding
• Multiple replicas or redundant caches
• Communication buffers and intermediate results

As a result, engineering practice usually reserves 1.2 to 1.5 times the theoretical value. This means the actual memory requirement for weights alone may reach 1.9TB to 2.3TB.

2.2 Activation and KV cache

Long context is often a major selling point for such models. If a 1-million-token context window is supported, KV cache can become a memory black hole.

Even if a hybrid structure combining linear attention and sparse attention is used to significantly reduce KV growth for long sequences, a 1-million-token context may still generate hundreds of gigabytes or more of cache usage. The exact figure depends on:

• Number of layers
• Number of attention heads
• Cache precision
• Whether multi-token prediction is used
• Whether paged cache or compression strategies are adopted

Therefore, when designing a deployment plan, a separate memory pool must be reserved for KV cache rather than calculating total memory based only on weight size.

2.3 Total memory estimation formula

A simplified estimation is as follows:

Total memory requirement ≈ weight memory × redundancy factor
                           + activation memory
                           + KV cache
                           + communication and framework overhead

For a large sparse model with 1.56TB of weights, using a 1.3x redundancy factor means weights alone exceed 2TB. Adding activation, cache, and communication buffers easily pushes the total memory requirement into the 2.5TB to 3.5TB range.

3. Estimating GPU count and node scale

3.1 Usable memory per GPU

GPU memory capacity varies significantly. Among common high-performance accelerators, memory per card may range from 80GB to 192GB. For estimation purposes, they can be grouped as follows:

• Mainstream high-performance cards: about 80GB per card
• High-memory training cards: about 141GB to 192GB per card
• High-performance inference cards: potentially higher, but with significantly higher procurement cost

Assuming 180GB of memory per card, the actual usable amount after firmware and runtime overhead may be about 160GB.

3.2 Rough card count based on memory

If the total memory requirement is 2.5TB to 3.5TB, calculated at 160GB usable memory per card:

2500GB / 160GB ≈ 16 cards
3500GB / 160GB ≈ 22 cards

This only means the model can fit in memory. If a single server has only 8 cards, at least 2 to 3 servers are required.

But fitting in memory does not mean inference speed is acceptable. Compute and communication must also be evaluated.

3.3 Evaluating compute capacity

Large model inference speed is constrained by compute, memory bandwidth, and communication together. For a sparse MoE model, generating each token requires:

• Routing decisions
• Forward computation on activated experts
• Attention computation
• Multi-token prediction
• All-to-all communication across machines

If the number of cards is too small, the weights may load successfully, but generation speed may drop to only a few tokens per second, which is hard to meet production requirements. As a result, actual deployments often choose configurations with memory headroom and more compute capacity.

4. Why recommended configurations are much higher than minimum configurations

4.1 Minimum configuration is not a usable configuration

A minimum configuration usually only solves the problem of “can the model be loaded,” not “can it serve traffic reliably.” Real production environments also require:

• Throughput targets, such as how many requests per second
• Time to first token
• Average generation latency
• Stability under long-context scenarios
• Isolation for concurrent requests

These requirements push a system from a minimum memory configuration to a performance-qualified configuration.

4.2 Communication overhead in sparse MoE

Experts in a sparse MoE model are distributed across multiple cards or even multiple machines. For each forward pass, tokens need to be routed to different experts, generating substantial cross-card communication.

If experts are located within the same 8-card server, communication goes through NVLink or high-bandwidth interconnect, and latency is still acceptable. Once experts are distributed across machines, the system must rely on InfiniBand or high-speed Ethernet. At that point, communication latency can become the inference bottleneck.

Therefore, when deploying large sparse models, teams usually aim to:

• Keep frequently used experts on the same node
• Reduce cross-machine all-to-all communication
• Increase inter-node bandwidth
• Use faster interconnect solutions

4.3 Estimation logic for recommended configurations

Once all engineering factors are considered, the recommended configuration typically rises from a minimum of 16 cards to a much larger scale. For example:

• Minimum experimental configuration: satisfies weight loading, but with low speed
• Recommended production configuration: guarantees throughput and latency
• High-performance configuration: targets high concurrency or heavy long-context scenarios

If a single server has 8 cards, the recommended configuration may require multiple machines forming a supernode with high-speed interconnect. Hardware cost then rises quickly.

5. Budget composition: not just GPUs

Many teams only calculate GPU procurement cost when estimating budgets, but the real budget also includes:

1. Full server cost
2. High-speed network equipment
3. Storage systems
4. Inference frameworks and operations platforms
5. Data center power and cooling
6. Spare parts and long-term maintenance

5.1 Full server cost

GPU servers cannot be evaluated by accelerator card price alone. A full server also includes:

• CPU
• Memory
• Motherboard
• Power supply
• Cooling module
• High-speed network interface card

For high-end configurations, the full server price may be 1.2 to 1.5 times the bare GPU price or more.

5.2 Network equipment

Multi-machine deployment requires high-speed networking. Common options include InfiniBand or RoCE. A multi-node inference cluster may require:

• High-speed switches
• Optical modules
• Cables
• Network interface cards

These components account for a significant share in large clusters, especially in high-bandwidth, low-latency scenarios.

5.3 Storage systems

A 1.56TB weight file is not a one-time load. You also need to consider:

• Multi-replica deployment
• Version management
• Fast distribution
• Snapshots and rollback

This requires high-performance shared storage or a distributed file system, which adds to the budget.

5.4 Operations and platform costs

After the model goes live, you also need to address:

• Inference service orchestration
• Canary releases
• Monitoring and alerting
• Log collection
• Automatic scaling
• Failure recovery

These can be built on general cloud-native capabilities such as container platforms, log services, monitoring and alerting, and object storage. If built in-house, they also incur labor and resource costs.

6. Deployment examples: from experimentation to production

6.1 Experimental deployment

Goal: validate model inference functionality without pursuing high throughput.

Configuration approach:

• Multiple 8-card servers
• At least 80GB memory per card
• Sharded weight loading
• Disable some long-context optimizations
• Use shorter test sequences

This approach lowers initial investment, but generation speed may fluctuate significantly and is not suitable for external services.

6.2 Production-grade deployment

Goal: reliable service with moderate concurrency and longer contexts.

Configuration approach:

• Use high-memory accelerators
• 8 cards per server forming a high-bandwidth domain
• High-speed interconnect across machines
• Fine-grained weight sharding by experts and layers
• Independent KV cache management
• Inference service with dynamic batching

Hardware cost is significantly higher than the experimental approach, but performance is more controllable.

6.3 High-performance deployment

Goal: target heavy long-context or high-concurrency scenarios.

Configuration approach:

• More GPUs for compute redundancy
• Faster inter-node interconnect
• Multi-token prediction for acceleration
• Prefill and decode disaggregation
• Localized optimization of expert routing

This approach requires the largest investment but can unlock the model’s potential for long contexts and high throughput.

7. Additional costs in long-context scenarios

7.1 Cache grows with sequence length

Even when the model uses a hybrid attention structure to reduce complexity, long contexts still create additional memory pressure. For example, a 1-million-token request may consume hundreds of gigabytes more cache than a typical 8K-token request.

If multiple concurrent requests all use long contexts, cache requirements multiply. Simply adding GPUs may not solve the problem linearly. Additional measures include:

• Cache compression
• Chunked prefill
• Memory tiering
• Request scheduling and queuing

7.2 Time to first token increases

The prefill phase for long contexts consumes substantial compute resources. Even with an optimized model architecture, time to first token still increases with sequence length.

Therefore, service-level design should distinguish between:

• Short-text interactive scenarios
• Medium and long document analysis scenarios
• Ultra-long context inference scenarios

Different scenarios may require different hardware configurations or service queues.

8. Reporting budget upward: a suggested evaluation checklist

When requesting budget from management, technical teams should not simply state “we need this many GPUs.” A more effective structure is:

1. Business goals and reasons for model selection
2. Differences between minimum and recommended configurations
3. Estimates for three key resources: memory, compute, and communication
4. Impact of long-context and concurrent scenarios
5. Complete cost including hardware, network, storage, and operations
6. Phased investment plan, such as experimentation before production
7. Cloud elastic resources as a comparison option

For example, the summary can be framed like this:

Minimum configuration:
- 2 servers with 8 cards each
- 80GB per card
- Suitable for functional validation
- Low generation speed
Recommended configuration:
- Multiple 8-card servers
- At least 180GB per card
- High-speed interconnect
- Supports production-grade throughput
Budget range:
- Hardware cost alone may reach tens of millions of RMB

This approach is more persuasive than simply saying “the model is impressive and needs a lot of money.” It also demonstrates the technical team’s evaluation capability.

9. Tradeoffs between cloud and on-premises deployment

For most enterprises, a one-time purchase of a large on-premises cluster is not necessarily the best first step. A more pragmatic approach is:

• Start with pay-as-you-go high-performance GPU instances in the cloud
• Validate model performance and business value
• Once usage stabilizes, evaluate whether local deployment is more cost-effective

Advantages of cloud deployment:

• No large upfront purchase
• Low cost of hardware upgrades
• Elastic scaling
• Faster validation of multiple models

Advantages of on-premises deployment:

• Potentially lower unit cost under long-term high-frequency usage
• Data remains fully internal
• Meets specific compliance requirements

If the business is still in the validation phase, it is not advisable to invest tens of millions of RMB immediately in an on-premises cluster. Elastic cloud inference can first confirm business value before deciding whether to move to local deployment.

10. Conclusion

The open-sourcing of a very large sparse MoE model does not mean any team can easily deploy it locally. Weight size, long-context cache, multi-machine communication, and inference throughput together determine the real cost.

From an engineering perspective, there is a huge gap between minimum and recommended configurations. In some cases, a minimum configuration can only load the model but cannot provide stable service. A recommended configuration is required to meet production requirements for throughput, latency, and concurrency.

Therefore, when evaluating model deployment, the question should not only be “how capable is this model?” but also:

1. What level of inference performance does the business require?
2. Is long context necessary?
3. How large is the concurrency scale?
4. Is this short-term validation or long-term production?
5. Are cloud elastic resources or on-premises fixed resources more suitable?

Only by answering these questions clearly can teams avoid the problem of “powerful model, uncontrollable deployment budget.” For technical teams, building a quantifiable deployment evaluation methodology is far more valuable than chasing benchmark scores for any single model.