Running Open-Source LLMs Locally: Performance, Cost, and Limitations in Practice

1. Background: Open-Source Models Close the Gap, and Local Deployment Becomes a Serious Option

Over the past two years, the capability gap between open-source and closed-source flagship models has continued to narrow. Three structural factors are driving this shift:

1. Architectural scale has changed. A new generation of open-source models widely adopts sparse mixture-of-experts (MoE) architectures: total parameter counts can reach hundreds of billions, but only a few billion to a dozen or so billion parameters are actually activated per token. The capability ceiling is determined by total parameters, while inference cost is determined by activated parameters—the two are decoupled.
2. Native multimodality. Image understanding is built directly into the weights rather than bolted on via external vision modules, improving both end-to-end quality and deployment simplicity.
3. A maturing inference ecosystem. Major open-source inference frameworks have progressively engineered support for multi-token prediction, KV Cache quantization, and low-bit weight loading, significantly amplifying the practical value of consumer-grade hardware.

A recent community test round is representative: a used consumer-grade GPU with roughly 22GB of VRAM, costing about 2,000 RMB, paired with a mature open-source inference framework, can smoothly run a mid-sized open-source model while maintaining a context window in the hundreds of thousands of tokens. “Running a large model locally” has thus graduated from a geek hobby to a technical option worth serious evaluation.

2. Key Technical Breakdown: Why Local Deployment Suddenly Works

1. Sparse MoE: Splitting a Large Model into On-Demand Expert Groups

MoE architectures distribute parameters across a number of “expert” networks, with a router selecting only a small subset of experts to participate in computation for each token. For local deployment, VRAM still needs to hold all weights, but the compute bottleneck is substantially relieved. Combined with low-bit quantization, models at the “hundreds of billions total parameters, teen billions activated” level can run on a single card at speeds approaching the feel of a much smaller model.

2. Weight Quantization: Q4 Is the Sweet Spot Between Quality and VRAM

At 4-bit quantization, each parameter occupies roughly 0.5 bytes, so the weights of a 27B-class model can be compressed to under 16GB—landing neatly within the comfortable range of 22GB VRAM. Community experience generally holds that Q4-level precision loss is imperceptible on most tasks; Q3 is an aggressive tier that is friendlier to sparse models with larger total parameters and smaller activation, but for small-to-mid-sized dense models it produces degradation such as reduced instruction following and weaker logical consistency. A caveat: quantization releases from different producers are not fully equivalent, and quantized models of the same nominal size may show different measured behavior. Cross-validation before going live is advisable.

3. KV Cache Quantization: The VRAM Relief Valve for Long Contexts

One of the major VRAM consumers in autoregressive inference is the KV Cache, whose size scales linearly with context length:

KV Cache VRAM ≈ 2 × layers × KV heads × head dimension × context tokens × bytes per element

Take a model with roughly 60 layers, 8 KV head groups, and a head dimension of 128: a 100K context requires about 12GB at 16-bit floating point, and about 6GB when quantized to 8 bits—freeing up space equivalent to half the weights. Combined with designs such as grouped-query attention, the KV Cache growth of sparse models can be made even gentler—this is precisely the key to small-activation models being able to “carry” a 256K context.

4. Multi-Token Prediction (MTP): Half-Price Acceleration for Throughput

Traditional autoregression produces only one token per step; multi-token prediction allows the model to produce multiple candidate tokens in a single forward pass and then confirm them in batch through a verification mechanism. When the acceptance rate is high, effective throughput can nearly double. In community testing, after upgrading to framework versions supporting this feature, generation speed on comparable configurations generally saw double-digit percentage gains. In addition, GPU-native compute backends and cross-platform general-purpose compute backends each trade off compatibility against peak performance, and should be compared one by one during deployment.

3. Measured Reference: What 22GB of VRAM Can Deliver

The following is a set of typical configuration data circulating in the community, for reference only (actual performance depends on hardware, drivers, framework version, and parameter settings):

Model Size	Quantization	Context	Measured Throughput
~27B	Q4 weights + 8-bit KV Cache + multimodal	100K	Steady ~23 token/s, short-context peak ~30
~35B (sparse, ~3B activated)	Q3 weights + 8-bit KV Cache + multimodal	256K	~97 token/s
~27B, dual concurrent	Q4	—	~40 token/s combined

On quality, the community’s subjective assessment is: the 27B-class version is considered to have matched first-tier closed-source flagships in everyday generation quality, with agent-style benchmark performance also reaching flagship level, and is “usable and pleasant” in long-form dialogue, creative writing, and similar scenarios; the 35B model with 3B activated is slightly behind but extremely fast, suitable as the default choice for high-frequency daily tasks; both support image understanding. Converted into cost: a one-time hardware investment of about 2,000 RMB buys near-flagship-level unlimited calls—an equation that is enough to change many teams’ technology choices.

4. Cost and Structure: Local Deployment Disrupts More Than Just Experience

The value of local deployment goes beyond “saving money”:

• Privacy and compliance: Data never leaves the premises, which is nearly the only viable option for classified and air-gapped scenarios;
• Unlimited calls: No rate limits or billing anxiety, well suited to high-density batch processing and offline analysis;
• Deep controllability: Sampling parameters, system prompts, and model versions are fully under your control, and behavior is reproducible.

From an industry perspective, when mid-sized open-source models can cover most needs locally, the commercial logic of “billing by token” comes under real pressure—at that point only the very top ultra-large model tokens retain a scarcity premium, and the value center of cloud services shifts from “selling capability” to “selling elasticity, compliance, and enterprise-grade assurance.” For users, the more rational approach is a hybrid architecture: sensitive data and high-frequency tasks stay local, while elastic concurrency and top-tier capability go to the cloud. Enterprise scenarios can evaluate model inference services and GPU compute offerings from cloud providers such as Baidu AI Cloud, treating the local cluster as the data foundation and the cloud as the capability ceiling—the two are complementary rather than mutually exclusive.

5. Limitations Not to Overlook: Overthinking and Infinite Loops

The current generation of reasoning models is not perfect in local use. Three typical problems stand out:

1. Overly long chains of thought. Complex questions routinely consume 5,000+ tokens of thinking, and the model often has already reached the correct answer yet still verifies itself repeatedly a dozen or more times;
2. Infinite loops on trick questions. Faced with brain-teaser-style questions, the model easily falls into a “tear down—re-derive” loop, burning time and compute, with a final result that may not even beat “no thinking, direct answer”;
3. Sensitivity to repetition. Repetition penalty and other sampling parameters need manual tuning per official recommended values, and the problem is more pronounced the smaller the model—especially below 10B.

As for causes, one reasonable hypothesis is that benchmark-oriented training taught the model the shortcut that “longer reasoning chains ≈ higher benchmark scores,” over-incentivizing long thinking and generalizing it to scenarios that don’t need long chains. Engineering mitigations include:

• Calibrate sampling parameters (repetition penalty, temperature, top-p, etc.) per the official documentation rather than carrying over experience values from older models;
• Set a token budget ceiling for the thinking phase, and explicitly disable thinking mode for simple tasks;
• Instruct in the prompt to “give the conclusion first, then the reasoning” to compress unnecessary verification;
• Add loop detection at the application layer, automatically resetting the session or changing the random seed when repetitive patterns are detected.

6. Deployment Checklist: VRAM Planning and Decision Path

Do a bit of arithmetic before you start:

Total VRAM ≈ weights (parameters × bytes per parameter)
           + KV Cache (see formula above)
           + activation and framework buffers (reserve 1–2GB)

Decision recommendations, in order:

1. First set the target context length and work backward to the KV Cache footprint;
2. Prefer Q4 for weights; if VRAM is tight and the model is a sparse architecture, drop to Q3;
3. Always enable 8-bit KV Cache quantization—the payoff is largest in long-context scenarios;
4. If throughput is insufficient, first upgrade the framework version (often a free speedup), then consider multi-token prediction and concurrent batching;
5. Before going live, run regression tests on your own real tasks, focusing on instruction following and repetition.

7. Conclusion: A Moment for Local, and a Repositioning for the Cloud

The claim that “a 2,000 RMB GPU delivers near-flagship experience” holds only because model architecture, quantization technology, and inference frameworks have all crossed the threshold at the same time. For developers and teams, the sensible strategy today is layered: lightweight high-frequency work stays local, while the capability ceiling goes to top-tier cloud models and enterprise-grade services (such as the inference and compute capabilities offered by Baidu AI Cloud). Local and cloud are not substitutes but a division of labor—what is truly being phased out is the single commercial imagination of “selling models by token alone.”