MoE Model Selection: Memory, Cost, and Evaluation
A practical guide to choosing a Mixture-of-Experts model based on published specs, reproducible benchmarks, and real deployment cost. It explains why small active parameters do not mean small GPU memory, how to estimate weight memory and KV cache, which engineering metrics to watch, how to read public scores, and a checklist for production selection.
What to Evaluate: Provenance vs. Performance
Recent discussion of a lightweight Mixture-of-Experts (MoE) model often focuses on which company built it and how large its parameter count is. Source and community attention matter for visibility, but they help little with selection. Whether a model is fit for a business workload depends on three kinds of information: the official model card and announcements, reproducible evaluation results, and the real cost of deployment.
This article does not rely on unverified claims about identity. It covers only what can be derived from public configuration and general principles.
MoE Architecture: Less Compute per Step Is Not Less Memory
An MoE model splits the feed-forward network into multiple experts. For each token, the router activates only a few of those experts. Compute per generated step is therefore set by the active parameters, not the total parameter count.
Take a model with 320B total parameters and 18B active parameters:
- Compute per generated token is close to that of an 18B dense model. At roughly 2 FLOPs per parameter, that is about 36 GFLOPs per token.
- Weights for every expert still have to live somewhere. Weight memory depends only on total parameters and storage precision, not on active parameters.
A common deployment mistake is treating “small active parameters” as “small GPU memory.” That seriously underestimates hardware needs.
How to Estimate GPU Memory
Weight memory is total parameters times bytes per parameter. For 320B total parameters:
| Precision | Bytes per parameter | Weight memory (approx.) |
|---|---|---|
| FP16 / BF16 | 2 | 640 GB |
| INT8 | 1 | 320 GB |
| INT4 | 0.5 | 160 GB |
Those figures cover weights only. A real deployment also needs headroom for:
- KV cache: 2 × layers × KV heads × head dimension × sequence length × concurrency × bytes per element. The factor of 2 covers keys and values. Long context and high concurrency make this grow quickly.
- Activations and framework overhead: the inference runtime’s memory pools, communication buffers, and similar costs.
- Peak memory: plan for the peak under long context and full concurrency, not the average.
KV head count and layer count must come from the model config file. They should not be guessed from experience.
How you place the model changes the GPU count. With INT4 weights at 160 GB, four 80 GB GPUs (320 GB total) leave about 160 GB after loading weights for KV cache and framework overhead—a workable starting point. To use fewer GPUs, some experts can be offloaded to CPU memory or SSD. That has a clear cost: every access to an expert that is not resident moves data, and both latency and throughput drop. Offloading fits offline batch jobs, not interactive serving.
Engineering Challenges: Three Metrics to Keep Watching
- Load balance: measure the share of tokens each expert receives. If a few experts stay overloaded, throughput falls. A useful signal is busiest-expert load divided by average load.
- Communication overhead: with experts spread across GPUs, cross-device all-to-all traffic can become the bottleneck. Record the share of per-step time spent on communication.
- Peak GPU memory: watch the peak after weights are loaded and under long-context concurrency, and leave margin for it.
How to Read Benchmark Scores
Public benchmarks are a starting point, but they usually have three problems:
- Data contamination: test items may already be in the training corpus, which inflates scores.
- Narrow coverage: tests such as MMLU and GSM8K emphasize memorized knowledge and calculation. They cover multi-turn consistency and long-document reasoning only weakly.
- Inconsistent settings: the same model can score very differently under different prompt templates, sampling temperatures, or few-shot setups.
If a report says a model is “roughly on par” with a flagship closed model, check at least the model version, evaluation date, benchmark name, exact scores, prompt template, and decoding parameters before you choose it. A claim missing those details is a lead, not a decision.
A Developer Selection Checklist
The table below is a set of evaluation directions. Pass thresholds should come from your own business baseline.
| Check | How to collect it | What to watch |
|---|---|---|
| Code pass rate | Run unit tests from a private set in the target language and framework | Break results down by language and framework |
| Long-context retrieval | Hide key facts in documents of different lengths and measure recall | Bucket results by context length |
| Refusal quality | Use a compliance set and count both false refusals and missed refusals | Record both error types |
| P95 latency | Load-test across a concurrency ramp; record time to first token and total time | Whether the service-level target is met |
| Quantization quality drop | Compare pre- and post-quantization results on the same test set | How large a drop is acceptable |
Quantization reduces memory, but precision loss varies widely by task. Whether INT4 is usable should be decided on your own test set. If the quality drop is outside the acceptable range, fall back to INT8.
Open Weights and the Commercial Model
Open weights let smaller teams obtain a strong base model at relatively low cost. Vendors release weights to attract developers and build toolchain and deployment experience, then earn revenue from API access, private deployment, and enterprise support. That pattern is generally helpful to users, with two caveats:
- Fast version churn: after a base-model update, previously trained LoRA adapters may break because the underlying representations changed. Retrain or revalidate before upgrading.
- Inconsistent interfaces: tokenizers, chat templates, and tool-call formats differ across models. Migration means changing prompts and output parsing.
If you plan to use live interaction data to improve a model, obtain explicit user consent first, remove identifying information, and follow applicable personal-data rules. Data collected without permission should not enter training.
Order of Selection
Choose from business constraints, not from reputation. A practical order:
- Confirm the license allows commercial use, and read the usage limits in the model card.
- Run a small evaluation on real business samples and record the metrics in the table above.
- Estimate hardware from total parameters, precision, and target concurrency, then decide among a cloud API, private deployment, or a mix.
- Keep a regression set and rerun it whenever you change the model or the quantization method.
For RAG, customer support, and coding assistance, an MoE model with fewer active parameters can be cheaper per inference step. Its weight-memory requirement is not small. Cost should be total cost of ownership—hardware, power, operations, and migration—not only compute per token.
