Local LLM Deployment: MoE vs Dense Model Selection, 32GB RAM, and Q4 Trade-offs

    This guide covers how to choose between MoE and Dense models for local LLM deployment, whether 32GB of memory can run a 35B-A3B model, and whether Q4 quantization noticeably hurts quality. It explains how total vs. activated parameters, quantization, context length, and KV cache shape hardware requirements, then walks through a practical workflow from model selection and quantization to launch parameters, monitoring, and troubleshooting.

    1. Four Questions to Answer Before Local LLM Deployment

    In enterprise private deployments, “more parameters means a better model” is not always true for local LLM deployment. A more sound approach is to answer four questions first:

    • How should you choose between MoE and Dense models for local deployment?
    • Can 32GB of memory run a 35B-A3B model?
    • Does Q4 quantization noticeably hurt quality?
    • What mistakes do teams most often make?

    To judge whether a model can run locally, look at these factors together:

    • total parameter count
    • activated parameter count
    • quantization precision
    • context length
    • KV cache usage
    • batch size and concurrency
    • target throughput
    • acceptable quality loss
    • runtime headroom

    Total parameter count mainly determines how much memory or VRAM the model weights occupy. Activated parameters affect the compute per forward pass and, as a result, inference speed. Take MoE as an example: the model may have a very large total parameter count, but it activates only a small subset of experts on each pass. This can lead to better throughput on some devices.

    Local model selection should therefore not be based on the number in the model name alone. For example, 35B-A3B means about 35 billion total parameters and about 3 billion activated parameters. Whether it can run depends first on whether the 35B weights can be loaded. How fast it runs depends on the compute efficiency of the 3 billion activated parameters.

    2. MoE Architecture: Sparse Activation Improves Speed, but Weights Must Stay Resident

    The core idea of the Mixture of Experts (MoE) architecture is to invoke experts on demand. The model contains multiple expert sub-networks. For each input, a router activates only a small subset of experts to do the work.

    Common naming conventions include 35B-A3B, 122B-A10B, and 397B-A17B. The first number is the total parameter count; the second is the number of parameters activated during inference:

    • 35B-A3B: about 35 billion total parameters, about 3 billion activated during inference
    • 122B-A10B: about 122 billion total parameters, about 10 billion activated
    • 397B-A17B: about 397 billion total parameters, about 17 billion activated

    The key benefit of sparse activation is lower compute per inference. All expert weights still need to be loaded, but each forward pass only goes through the activated paths. Under the same hardware, an MoE model can therefore sometimes run faster than a Dense model with a comparable total parameter count.

    However, MoE has several practical risks:

    1. All expert weights must stay resident in memory or VRAM, so capacity pressure is driven mainly by the total parameter count.
    2. Expert routing can be imbalanced: some experts are called frequently while others rarely participate, leading to uneven resource utilization.
    3. Under batching or high concurrency, throughput may fluctuate as the routing distribution changes.
    4. MoE “saves” mainly on compute and speed. This is not the same as a major reduction in VRAM usage.

    When evaluating MoE, look at both activated and total parameters. Also consider the inference framework, batching strategy, context length, and business concurrency.

    3. Dense Architecture: All Parameters Take Part, Predictable and Stable

    Dense models compute with all of their parameters. The activated parameter count equals the total parameter count. Take a compact Dense model in the 27B class: its 27 billion parameters are fully loaded and used on every inference.

    Dense models offer a regular structure and predictable behavior. Without routing decisions that introduce uncertainty, deployment is usually simpler, output stability is easier to control, and memory usage is more linear.

    As a result, Dense models are often easier to manage and sign off on in enterprise scenarios that require:

    • stable service
    • clear SLAs
    • observability
    • controllable operations

    The trade-off is that compute per unit of work grows with model size. Larger Dense models generally require more compute per inference.

    Dense models therefore fit scenarios where:

    • hardware compute is relatively sufficient
    • output stability is a high priority
    • teams want to minimize tuning complexity

    In practice, think of Dense models as the steady choice and MoE as the efficiency-oriented choice. If the goal is quick validation, Dense models are usually easier to get into production. If you want to improve response speed while retaining relatively strong model capability, MoE is a strong candidate.

    4. Capability Evaluation: Fewer Parameters Doesn’t Mean a Cliff Drop, but Conclusions Must Be Conditional

    One common misconception is that a smaller parameter count always leads to a cliff-like drop in capability. In reality, some mid-sized models may approach certain frontier closed-source models on some public benchmarks or specific tasks. This still does not mean they have “caught up” across the board.

    A more rigorous statement is that on some public benchmarks or specific tasks, mid-sized models may reach a competitive level. Real-world results, however, depend on:

    • model version
    • evaluation benchmark
    • quantization method
    • context length
    • task type
    • prompts
    • sampling parameters
    • post-processing strategy

    For example, some models perform well on general Q&A, coding assistance, document summarization, and structured extraction. On complex reasoning, long-context understanding, low-latency simple tasks, or strict output-format requirements, they may still show long thinking times, verbose output, or unstable formatting.

    For deployment decisions, rely on small-scale evaluation with real business samples rather than parameter counts or vendor claims alone. Your evaluation should cover at least:

    • real business input samples
    • the required output format
    • acceptable response time
    • acceptable quality loss
    • long-text and short-text scenarios
    • single- and multi-concurrency scenarios
    • before-and-after quantization comparison
    • behavior at different context lengths

    The conclusions from this kind of evaluation are far more suitable for production deployment decisions.

    5. Quantization Strategy: Q4 vs. Q8, and Whether Q4 Hurts Quality

    Quantization is a key technique in local deployment for reducing model size, improving throughput, and lowering hardware requirements. It reduces the bit width of each parameter, trading a controlled amount of quality loss for resource and speed gains. Q4 usually refers to 4-bit quantization and Q8 to 8-bit quantization.

    In general:

    • Q4 models are smaller, faster, and friendlier to memory and VRAM.
    • Q8 models are larger and more resource-intensive, but usually stay closer to the original model’s quality.

    The quality gap between Q4 and Q8 cannot be reduced to a fixed number. Some evaluations have shown a performance difference of roughly 1.5% to 3% between Q4 and Q8. This is not a universal guarantee. Actual results vary with task type, quantization implementation, evaluation methodology, context length, and model version.

    Whether Q4 “loses quality” needs to be verified with business samples. Q4 is usually worth trying first for:

    • interactive Q&A
    • content summarization
    • customer service assistance
    • real-time code completion

    For scenarios with high demands on formatting, factual accuracy, code correctness, or domain terminology, evaluate Q8 or higher-precision options first.

    The table below is illustrative only. It shows trends across architectures and quantization levels and is not a reproducible promise:

    Model size Quantization Indicative speed (tokens/s) Preconditions
    35B-A3B Q4 ~53 Unified-memory terminal device, single concurrency, context length unspecified, quantization format unspecified
    27B Dense Q4 ~40 Unified-memory terminal device, single concurrency, context length unspecified
    27B Dense Q8 ~19-20 Unified-memory terminal device, single concurrency, context length unspecified
    122B-A10B Q4 ~22 Unified-memory terminal device, single concurrency, context length unspecified

    Treat these numbers as directional guidance only. Actual speed also depends on:

    • device model
    • chip architecture
    • memory bandwidth
    • quantization tools and format
    • context length
    • batch size
    • sampling parameters
    • inference framework
    • concurrency

    Without standardized measurement conditions, a single figure should not be treated as a deployment guarantee.

    On some unified-memory devices, a Q4 MoE model can achieve higher throughput than a larger Dense model because it activates fewer parameters. This does not mean MoE is always faster, or that Q4 will always meet your needs. The final decision should be based on target tokens/s, P95 latency, time to first token, and business quality metrics.

    6. Hardware Requirements: VRAM Depends on Total Parameters, Speed on Activated Parameters and Quantization

    Different model sizes have clear minimum memory or VRAM requirements, and these directly determine whether local deployment is feasible. Hardware estimates should not simply state “model X needs about Y GB.” Specify these details as well:

    • quantization precision
    • context length
    • KV cache
    • batch size
    • runtime headroom
    • operating system overhead

    Take a model in the 24B class. With Q4 quantization, it typically uses fewer resources than Q8. However, long context, multiple concurrent users, or large batches can significantly increase memory pressure through the KV cache. A 24B-class model therefore needs its quantization level stated, such as Q4 or Q8. A blanket claim like “about 32GB” is not enough.

    Use the following as a reference:

    Model type Recommended quantization Memory/VRAM reference Notes
    24B-class Dense Q4 or Q8 Q4: typically start evaluating at the 32GB level Q8 requires more; long context increases KV cache
    27B Dense Q4 or Q8 Q4: typically start evaluating at the 32GB level Q8 resource requirements rise noticeably
    35B-A3B Q4 or Q8 Relatively moderate hardware requirements Memory/VRAM is driven mainly by total parameters; speed is driven by activated parameters
    122B-class MoE/Dense Q4 or higher compression Usually 128GB or above Also evaluate context, batching, and throughput targets
    397B-A17B Not recommended for typical local deployment High hardware barrier Better suited to teams with ample compute budgets and operations capabilities

    Be clear about 35B-A3B. Its memory or VRAM usage is driven mainly by total parameters, because all expert weights must be loaded. Its speed advantage comes mainly from having fewer activated parameters. It does not need only the memory of a 3B model just because it activates 3 billion parameters.

    Leave runtime headroom when choosing hardware. In real deployments, several components share the available memory and VRAM:

    • model weights
    • KV cache
    • the inference framework
    • concurrent requests
    • logging and monitoring
    • system services

    A budget that sits right at the minimum can easily lead to out-of-memory (OOM) errors, sudden throughput drops, or service crashes.

    Use this practical selection checklist:

    1. What is the total parameter count?
    2. What is the activated parameter count?
    3. Will you use Q4, Q8, or FP16?
    4. What is the target context length?
    5. Do you need multi-user concurrency?
    6. What are the target tokens/s or P95 latency?
    7. How much quality loss is acceptable?
    8. Have you reserved memory for the KV cache and runtime?
    9. Does the inference framework support the model and quantization format?
    10. Do you have fallback and rollback plans?

    7. Executable Deployment Checklist: From Download and Quantization to Startup and Monitoring

    Local deployment should go beyond conceptual selection and become an actionable checklist. The following is a general workflow.

    1. Model selection

    Choose Dense models for stability, simplicity, and predictability. Choose MoE models if you want higher speed at a large total parameter count. Prefer models with strong community support, mature quantized builds, and good inference framework compatibility.

    2. Downloading and converting the model

    The commands below are general examples. Adjust them for the specific model repository and inference framework:

    1. # Download the model
    2. huggingface-cli download <model_id> --local-dir ./models/<model_name>
    3. # Convert to GGUF format
    4. python convert.py ./models/<model_name> --outfile model-f16.gguf --outtype f16
    5. # Quantize the model
    6. llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

    If you use frameworks such as vLLM, TensorRT-LLM, SGLang, Ollama, or LM Studio, the quantization formats and startup methods may differ. Before deployment, confirm that the framework supports the target model, quantization format, and context length.

    3. Startup parameter examples

    Here is an example of launching a vLLM-style OpenAI-compatible server:

    1. python -m vllm.entrypoints.openai.api_server \
    2. --model ./models/<model_name> \
    3. --max-model-len 8192 \
    4. --gpu-memory-utilization 0.90 \
    5. --dtype auto

    When running MoE models, also check:

    • Whether expert parallelism or tensor parallelism is enabled
    • Whether the batch size is appropriate for the GPU
    • Whether the context length exceeds what the hardware can handle
    • Whether the KV cache causes out-of-memory errors
    • Whether routing becomes imbalanced under multiple concurrent requests

    When running Dense models, check:

    • Whether the number of inference threads matches the CPU and GPU
    • Whether batching improves throughput
    • Whether long context causes the KV cache to grow sharply
    • Whether Q8 is more stable than Q4
    • Whether output formatting is controllable

    4. Monitoring metrics

    After launch, monitor at least these metrics:

    • tokens/s: tokens generated per unit of time
    • Time to first token: the speed users perceive first
    • P95 latency: tail latency that reflects the real user experience
    • P99 latency: extreme outlier latency
    • OOM count: memory or VRAM exhaustion events
    • Truncation rate: share of inputs or outputs that are cut off
    • Error rate: share of requests that fail or time out
    • Peak memory/VRAM usage: to gauge resource headroom
    • Queue length: to gauge concurrency pressure
    • Sampled output quality score: to check whether the quantized model still meets business requirements

    5. Common troubleshooting

    Insufficient memory or VRAM:

    • Switch from Q8 to Q4
    • Reduce the context length
    • Lower the number of concurrent requests
    • Reduce the batch size
    • Use lower precision or a more aggressive quantization format
    • Check whether the KV cache is too large
    • Make sure other large model services are not running at the same time

    Sudden throughput drops:

    • Check whether the CPU is a bottleneck
    • Check whether the inference framework supports the target model
    • Check whether batching is disabled or misconfigured
    • Check whether MoE expert routing is imbalanced
    • Check whether the context is too long
    • Check for heavy repetitive generation or abnormal sampling parameters

    Declining output quality:

    • Compare Q4 with Q8
    • Compare with the original FP16 or BF16 precision
    • Check whether the prompts have changed
    • Check sampling parameters such as temperature, top_p, and repetition penalty
    • Check whether the output is being truncated
    • Check whether long-text inputs exceed the context window

    Truncated output:

    • Increase the maximum output length
    • Increase the context length, but evaluate the KV cache impact
    • Split long tasks into multiple passes
    • Add summarization and rewriting logic at the application layer

    Service crashes:

    • Increase memory or VRAM headroom
    • Cap the maximum concurrency
    • Add request timeouts
    • Add automatic restart on OOM
    • Add health checks
    • Establish a model version rollback process

    8. Common Mistakes and Acceptance Criteria

    Here are the most common mistakes in local LLM deployment:

    1. Looking only at total parameters and ignoring activated parameters
    2. Looking only at activated parameters and overlooking that all expert weights must be loaded
    3. Treating a speed screenshot from one device as a general performance guarantee
    4. Estimating hardware requirements without specifying quantization precision
    5. Overlooking the impact of context length on the KV cache
    6. Ignoring batching, concurrency, and runtime headroom
    7. Testing only single requests without measuring P95 latency
    8. Focusing only on speed without accepting output quality against business requirements
    9. Not monitoring OOM events, truncation rate, and error rate
    10. Having no fallback, rollback, or manual override plans

    A practical acceptance criterion is this. With defined hardware, quantization format, context length, concurrency level, and sampling parameters, the service should meet the business’s targets for:

    • tokens/s
    • P95 latency
    • output quality
    • stability
    • cost budget

    Conclusion

    MoE and Dense architectures each have their strengths. MoE uses sparse activation to improve compute efficiency, but its weights still need to be fully loaded. Dense uses all parameters for every inference, delivering stable, linear, and predictable output.

    The real key to local deployment is not chasing the largest parameter count. It is matching activated parameters, quantization precision, context length, KV cache, target throughput, and hardware resources.

    View parameter count and model capability rationally. Validate quantization with real business samples, and use monitoring metrics to keep the service stable. That is how large models can move into production with costs under control.