Selecting and Deploying 32B Open-Source LLMs
Where 32B open-source models sit between capability and cost, how base, reasoning, and thinking-mode behavior differ, and what to check when estimating VRAM, serving locally, and measuring multi-GPU throughput.
1. Where 32B models sit
“32B” means about 32 billion parameters. This size sits between capability and cost. It handles multi-step reasoning, long text, and complex instructions better than smaller models, without the multi-GPU servers that much larger models often require. For teams that need a private deployment and cannot send data outside their environment, it is usually one of the first sizes worth evaluating.
Parameter count alone does not determine quality. Two 32B models can differ sharply in training data, post-training, and reasoning ability. Choose based on evaluation results for the target task. Use parameter scale only to estimate resource needs.
2. How to tell three model types apart
Publishers do not name model types consistently. Read the training-stage description and evaluation results in the model card instead of relying on the name. The split below is by behavior.
| Type | Typical traits | Suitable tasks |
|---|---|---|
| Base model | Large-scale pretraining, sometimes with instruction tuning; relatively stable general Q&A and text generation | Knowledge Q&A, content generation, a starting point for fine-tuning |
| Reasoning model | Extra training for math, code, and multi-step reasoning; outputs often include a long reasoning trace | Code generation, math, multi-hop questions |
| Thinking mode | The same model can be set, via a parameter or template, to think longer; quality may improve, but latency and token use rise | Complex planning and tasks that need repeated checking |
Thinking mode needs a separate note. It is not a separate model. It is an inference setting. Turning it on produces more tokens, slower responses, and higher cost. For simple Q&A, it is usually not worth it.
A 9B release is a separately published smaller variant, suited to edge devices, modest servers, or latency-sensitive lightweight services. Quantization is a deployment technique, not the origin of a 9B model. Quality differences between the two sizes should be tested separately.
3. Local deployment steps
1. Estimate VRAM
A rough estimate of weight memory is parameter count times bytes per parameter. A 32B model needs about 64 GB in FP16, about 32 GB in INT8, and about 16 GB in INT4. That is weights only. The KV cache grows with context length and concurrency, and the framework adds overhead, so real demand is clearly higher than the weight size. After INT4 quantization, a 24 GB GPU can usually support only a short context and low concurrency. Whether it is usable has to be measured.
2. Prepare the environment
- Match the GPU driver and CUDA version to the inference engine, and record the versions.
- Take weights, tokenizer, and config from the same model repository so a version mismatch does not produce bad output.
- Isolate dependencies in a virtual environment or container, and pin versions.
- Read secrets from environment variables. Do not put them in code or config files.
3. Start the service
Using vLLM as an example, the start command is below. Change paths and arguments for your environment, and take version details from the official documentation.
export MODEL_PATH=/data/models/your-32b-modelvllm serve "$MODEL_PATH" \--tensor-parallel-size 2 \--max-model-len 8192 \--gpu-memory-utilization 0.90 \--host 127.0.0.1 \--port 8000 \--api-key "$API_KEY"
--tensor-parallel-size should match the number of GPUs in use. --host 127.0.0.1 allows access only from the local machine. For access on an internal network, put the service behind an authenticated gateway. Do not bind 0.0.0.0 and expose it directly to the public internet.
After startup, run one smoke test against the API:
curl http://127.0.0.1:8000/v1/chat/completions \-H "Authorization: Bearer $API_KEY" \-H "Content-Type: application/json" \-d '{"model":"/data/models/your-32b-model","messages":[{"role":"user","content":"Introduce yourself in one sentence"}],"max_tokens":64}'
4. Deployment checklist
| Check | Pass criteria |
|---|---|
| VRAM use | Peak stays under the VRAM limit, with headroom |
| Single-request latency | Meets the business target at the intended concurrency, measured as P95 or P99 |
| Throughput | Load-tested with a realistic request mix; record tokens processed per second |
| Abnormal output rate | Empty output, garbled text, and format errors stay within an acceptable range |
| Port and access | The service is not exposed directly to the public internet, and the API requires authentication |
| Logs and monitoring | Cover latency, throughput, GPU memory, and error codes |
5. What to expect from multi-GPU parallelism
Tensor parallelism can run one model across several GPUs, but throughput usually does not scale linearly with GPU count. Inter-GPU bandwidth (for example, NVLink versus PCIe), batch size, and free VRAM all limit the gain. Measure on the target hardware. Do not extrapolate directly from the number of cards.
4. Comparing options
Selection should weigh task complexity, latency requirements, and cost together. The comparison table that follows in the source was not included here, so no extra rows or figures are added.
