Local LLM Deployment: MoE vs Dense Model Selection, 32GB RAM, and Q4 Trade-offs
This guide covers how to choose between MoE and Dense models for local LLM deployment, whether 32GB of memory can run a 35B-A3B model, and whether Q4 quantization noticeably hurts quality. It explains how total vs. activated parameters, quantization, context length, and KV cache shape hardware requirements, then walks through a practical workflow from model selection and quantization to launch parameters, monitoring, and troubleshooting.
1. Four Questions to Answer Before Local LLM Deployment
In enterprise private deployments, “more parameters means a better model” is not always true for local LLM deployment. A more sound approach is to answer four questions first:
- How should you choose between MoE and Dense models for local deployment?
- Can 32GB of memory run a 35B-A3B model?
- Does Q4 quantization noticeably hurt quality?
- What mistakes do teams most often make?
To judge whether a model can run locally, look at these factors together:
- total parameter count
- activated parameter count
- quantization precision
- context length
- KV cache usage
- batch size and concurrency
- target throughput
- acceptable quality loss
- runtime headroom
Total parameter count mainly determines how much memory or VRAM the model weights occupy. Activated parameters affect the compute per forward pass and, as a result, inference speed. Take MoE as an example: the model may have a very large total parameter count, but it activates only a small subset of experts on each pass. This can lead to better throughput on some devices.
Local model selection should therefore not be based on the number in the model name alone. For example, 35B-A3B means about 35 billion total parameters and about 3 billion activated parameters. Whether it can run depends first on whether the 35B weights can be loaded. How fast it runs depends on the compute efficiency of the 3 billion activated parameters.
2. MoE Architecture: Sparse Activation Improves Speed, but Weights Must Stay Resident
The core idea of the Mixture of Experts (MoE) architecture is to invoke experts on demand. The model contains multiple expert sub-networks. For each input, a router activates only a small subset of experts to do the work.
Common naming conventions include 35B-A3B, 122B-A10B, and 397B-A17B. The first number is the total parameter count; the second is the number of parameters activated during inference:
- 35B-A3B: about 35 billion total parameters, about 3 billion activated during inference
- 122B-A10B: about 122 billion total parameters, about 10 billion activated
- 397B-A17B: about 397 billion total parameters, about 17 billion activated
The key benefit of sparse activation is lower compute per inference. All expert weights still need to be loaded, but each forward pass only goes through the activated paths. Under the same hardware, an MoE model can therefore sometimes run faster than a Dense model with a comparable total parameter count.
However, MoE has several practical risks:
- All expert weights must stay resident in memory or VRAM, so capacity pressure is driven mainly by the total parameter count.
- Expert routing can be imbalanced: some experts are called frequently while others rarely participate, leading to uneven resource utilization.
- Under batching or high concurrency, throughput may fluctuate as the routing distribution changes.
- MoE “saves” mainly on compute and speed. This is not the same as a major reduction in VRAM usage.
When evaluating MoE, look at both activated and total parameters. Also consider the inference framework, batching strategy, context length, and business concurrency.
3. Dense Architecture: All Parameters Take Part, Predictable and Stable
Dense models compute with all of their parameters. The activated parameter count equals the total parameter count. Take a compact Dense model in the 27B class: its 27 billion parameters are fully loaded and used on every inference.
Dense models offer a regular structure and predictable behavior. Without routing decisions that introduce uncertainty, deployment is usually simpler, output stability is easier to control, and memory usage is more linear.
As a result, Dense models are often easier to manage and sign off on in enterprise scenarios that require:
- stable service
- clear SLAs
- observability
- controllable operations
The trade-off is that compute per unit of work grows with model size. Larger Dense models generally require more compute per inference.
Dense models therefore fit scenarios where:
- hardware compute is relatively sufficient
- output stability is a high priority
- teams want to minimize tuning complexity
In practice, think of Dense models as the steady choice and MoE as the efficiency-oriented choice. If the goal is quick validation, Dense models are usually easier to get into production. If you want to improve response speed while retaining relatively strong model capability, MoE is a strong candidate.
4. Capability Evaluation: Fewer Parameters Doesn’t Mean a Cliff Drop, but Conclusions Must Be Conditional
One common misconception is that a smaller parameter count always leads to a cliff-like drop in capability. In reality, some mid-sized models may approach certain frontier closed-source models on some public benchmarks or specific tasks. This still does not mean they have “caught up” across the board.
A more rigorous statement is that on some public benchmarks or specific tasks, mid-sized models may reach a competitive level. Real-world results, however, depend on:
- model version
- evaluation benchmark
- quantization method
- context length
- task type
- prompts
- sampling parameters
- post-processing strategy
For example, some models perform well on general Q&A, coding assistance, document summarization, and structured extraction. On complex reasoning, long-context understanding, low-latency simple tasks, or strict output-format requirements, they may still show long thinking times, verbose output, or unstable formatting.
For deployment decisions, rely on small-scale evaluation with real business samples rather than parameter counts or vendor claims alone. Your evaluation should cover at least:
- real business input samples
- the required output format
- acceptable response time
- acceptable quality loss
- long-text and short-text scenarios
- single- and multi-concurrency scenarios
- before-and-after quantization comparison
- behavior at different context lengths
The conclusions from this kind of evaluation are far more suitable for production deployment decisions.
5. Quantization Strategy: Q4 vs. Q8, and Whether Q4 Hurts Quality
Quantization is a key technique in local deployment for reducing model size, improving throughput, and lowering hardware requirements. It reduces the bit width of each parameter, trading a controlled amount of quality loss for resource and speed gains. Q4 usually refers to 4-bit quantization and Q8 to 8-bit quantization.
In general:
- Q4 models are smaller, faster, and friendlier to memory and VRAM.
- Q8 models are larger and more resource-intensive, but usually stay closer to the original model’s quality.
The quality gap between Q4 and Q8 cannot be reduced to a fixed number. Some evaluations have shown a performance difference of roughly 1.5% to 3% between Q4 and Q8. This is not a universal guarantee. Actual results vary with task type, quantization implementation, evaluation methodology, context length, and model version.
Whether Q4 “loses quality” needs to be verified with business samples. Q4 is usually worth trying first for:
- interactive Q&A
- content summarization
- customer service assistance
- real-time code completion
For scenarios with high demands on formatting, factual accuracy, code correctness, or domain terminology, evaluate Q8 or higher-precision options first.
The table below is illustrative only. It shows trends across architectures and quantization levels and is not a reproducible promise:
| Model size | Quantization | Indicative speed (tokens/s) | Preconditions |
|---|---|---|---|
| 35B-A3B | Q4 | ~53 | Unified-memory terminal device, single concurrency, context length unspecified, quantization format unspecified |
| 27B Dense | Q4 | ~40 | Unified-memory terminal device, single concurrency, context length unspecified |
| 27B Dense | Q8 | ~19-20 | Unified-memory terminal device, single concurrency, context length unspecified |
| 122B-A10B | Q4 | ~22 | Unified-memory terminal device, single concurrency, context length unspecified |
Treat these numbers as directional guidance only. Actual speed also depends on:
- device model
- chip architecture
- memory bandwidth
- quantization tools and format
- context length
- batch size
- sampling parameters
- inference framework
- concurrency
Without standardized measurement conditions, a single figure should not be treated as a deployment guarantee.
On some unified-memory devices, a Q4 MoE model can achieve higher throughput than a larger Dense model because it activates fewer parameters. This does not mean MoE is always faster, or that Q4 will always meet your needs. The final decision should be based on target tokens/s, P95 latency, time to first token, and business quality metrics.
6. Hardware Requirements: VRAM Depends on Total Parameters, Speed on Activated Parameters and Quantization
Different model sizes have clear minimum memory or VRAM requirements, and these directly determine whether local deployment is feasible. Hardware estimates should not simply state “model X needs about Y GB.” Specify these details as well:
- quantization precision
- context length
- KV cache
- batch size
- runtime headroom
- operating system overhead
Take a model in the 24B class. With Q4 quantization, it typically uses fewer resources than Q8. However, long context, multiple concurrent users, or large batches can significantly increase memory pressure through the KV cache. A 24B-class model therefore needs its quantization level stated, such as Q4 or Q8. A blanket claim like “about 32GB” is not enough.
Use the following as a reference:
| Model type | Recommended quantization | Memory/VRAM reference | Notes |
|---|---|---|---|
| 24B-class Dense | Q4 or Q8 | Q4: typically start evaluating at the 32GB level | Q8 requires more; long context increases KV cache |
| 27B Dense | Q4 or Q8 | Q4: typically start evaluating at the 32GB level | Q8 resource requirements rise noticeably |
| 35B-A3B | Q4 or Q8 | Relatively moderate hardware requirements | Memory/VRAM is driven mainly by total parameters; speed is driven by activated parameters |
| 122B-class MoE/Dense | Q4 or higher compression | Usually 128GB or above | Also evaluate context, batching, and throughput targets |
| 397B-A17B | Not recommended for typical local deployment | High hardware barrier | Better suited to teams with ample compute budgets and operations capabilities |
Be clear about 35B-A3B. Its memory or VRAM usage is driven mainly by total parameters, because all expert weights must be loaded. Its speed advantage comes mainly from having fewer activated parameters. It does not need only the memory of a 3B model just because it activates 3 billion parameters.
Leave runtime headroom when choosing hardware. In real deployments, several components share the available memory and VRAM:
- model weights
- KV cache
- the inference framework
- concurrent requests
- logging and monitoring
- system services
A budget that sits right at the minimum can easily lead to out-of-memory (OOM) errors, sudden throughput drops, or service crashes.
Use this practical selection checklist:
- What is the total parameter count?
- What is the activated parameter count?
- Will you use Q4, Q8, or FP16?
- What is the target context length?
- Do you need multi-user concurrency?
- What are the target tokens/s or P95 latency?
- How much quality loss is acceptable?
- Have you reserved memory for the KV cache and runtime?
- Does the inference framework support the model and quantization format?
- Do you have fallback and rollback plans?
7. Executable Deployment Checklist: From Download and Quantization to Startup and Monitoring
Local deployment should go beyond conceptual selection and become an actionable checklist. The following is a general workflow.
1. Model selection
Choose Dense models for stability, simplicity, and predictability. Choose MoE models if you want higher speed at a large total parameter count. Prefer models with strong community support, mature quantized builds, and good inference framework compatibility.
2. Downloading and converting the model
The commands below are general examples. Adjust them for the specific model repository and inference framework:
# Download the modelhuggingface-cli download <model_id> --local-dir ./models/<model_name># Convert to GGUF formatpython convert.py ./models/<model_name> --outfile model-f16.gguf --outtype f16# Quantize the modelllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
If you use frameworks such as vLLM, TensorRT-LLM, SGLang, Ollama, or LM Studio, the quantization formats and startup methods may differ. Before deployment, confirm that the framework supports the target model, quantization format, and context length.
3. Startup parameter examples
Here is an example of launching a vLLM-style OpenAI-compatible server:
python -m vllm.entrypoints.openai.api_server \--model ./models/<model_name> \--max-model-len 8192 \--gpu-memory-utilization 0.90 \--dtype auto
When running MoE models, also check:
- Whether expert parallelism or tensor parallelism is enabled
- Whether the batch size is appropriate for the GPU
- Whether the context length exceeds what the hardware can handle
- Whether the KV cache causes out-of-memory errors
- Whether routing becomes imbalanced under multiple concurrent requests
When running Dense models, check:
- Whether the number of inference threads matches the CPU and GPU
- Whether batching improves throughput
- Whether long context causes the KV cache to grow sharply
- Whether Q8 is more stable than Q4
- Whether output formatting is controllable
4. Monitoring metrics
After launch, monitor at least these metrics:
- tokens/s: tokens generated per unit of time
- Time to first token: the speed users perceive first
- P95 latency: tail latency that reflects the real user experience
- P99 latency: extreme outlier latency
- OOM count: memory or VRAM exhaustion events
- Truncation rate: share of inputs or outputs that are cut off
- Error rate: share of requests that fail or time out
- Peak memory/VRAM usage: to gauge resource headroom
- Queue length: to gauge concurrency pressure
- Sampled output quality score: to check whether the quantized model still meets business requirements
5. Common troubleshooting
Insufficient memory or VRAM:
- Switch from Q8 to Q4
- Reduce the context length
- Lower the number of concurrent requests
- Reduce the batch size
- Use lower precision or a more aggressive quantization format
- Check whether the KV cache is too large
- Make sure other large model services are not running at the same time
Sudden throughput drops:
- Check whether the CPU is a bottleneck
- Check whether the inference framework supports the target model
- Check whether batching is disabled or misconfigured
- Check whether MoE expert routing is imbalanced
- Check whether the context is too long
- Check for heavy repetitive generation or abnormal sampling parameters
Declining output quality:
- Compare Q4 with Q8
- Compare with the original FP16 or BF16 precision
- Check whether the prompts have changed
- Check sampling parameters such as temperature, top_p, and repetition penalty
- Check whether the output is being truncated
- Check whether long-text inputs exceed the context window
Truncated output:
- Increase the maximum output length
- Increase the context length, but evaluate the KV cache impact
- Split long tasks into multiple passes
- Add summarization and rewriting logic at the application layer
Service crashes:
- Increase memory or VRAM headroom
- Cap the maximum concurrency
- Add request timeouts
- Add automatic restart on OOM
- Add health checks
- Establish a model version rollback process
8. Common Mistakes and Acceptance Criteria
Here are the most common mistakes in local LLM deployment:
- Looking only at total parameters and ignoring activated parameters
- Looking only at activated parameters and overlooking that all expert weights must be loaded
- Treating a speed screenshot from one device as a general performance guarantee
- Estimating hardware requirements without specifying quantization precision
- Overlooking the impact of context length on the KV cache
- Ignoring batching, concurrency, and runtime headroom
- Testing only single requests without measuring P95 latency
- Focusing only on speed without accepting output quality against business requirements
- Not monitoring OOM events, truncation rate, and error rate
- Having no fallback, rollback, or manual override plans
A practical acceptance criterion is this. With defined hardware, quantization format, context length, concurrency level, and sampling parameters, the service should meet the business’s targets for:
- tokens/s
- P95 latency
- output quality
- stability
- cost budget
Conclusion
MoE and Dense architectures each have their strengths. MoE uses sparse activation to improve compute efficiency, but its weights still need to be fully loaded. Dense uses all parameters for every inference, delivering stable, linear, and predictable output.
The real key to local deployment is not chasing the largest parameter count. It is matching activated parameters, quantization precision, context length, KV cache, target throughput, and hardware resources.
View parameter count and model capability rationally. Validate quantization with real business samples, and use monitoring metrics to keep the service stable. That is how large models can move into production with costs under control.
