Running a 32B Model Locally on a CNY 10,000 Full-Build Budget: GPU Selection Guide

    This guide explains how to choose a GPU for running a 32B-parameter model locally with a full-build budget of CNY 10,000. It estimates VRAM requirements for Q4_K_M weights and KV cache, compares 24 GB candidates including the used RTX 3090, RTX 4090, RX 7900 XTX, Tesla P40, and dual RTX 3060 12 GB cards, and covers a sample full-build budget, power sizing, PCIe and storage impact, plus llama.cpp and Ollama launch settings.

    1. Budget Scope and Target

    The CNY 10,000 budget in this article refers to the total price of the complete build, including the GPU, CPU, motherboard, memory, storage, power supply, case, and cooling. If the CNY 10,000 is reserved only for the GPU, more cards are available, but the remaining components still need to be budgeted separately.

    The target is to run a 32B-parameter model on a single GPU using 4-bit quantization in GGUF Q4_K_M format, with a context length of roughly 4K to 8K.

    2. How Much VRAM the Model Needs

    Model weight size is approximately the number of parameters multiplied by the number of bytes per parameter. For a 32B model, common precision levels are roughly:

    • FP16: about 64 GB
    • INT8: about 32 GB
    • Q4_K_M: about 19 to 20 GB

    The theoretical value for 4-bit is 16 GB, but GGUF Q4_K_M keeps some layers at higher precision and adds overhead for embedding layers, so the actual file is slightly larger than 16 GB.

    The same method can estimate models of other sizes:

    • 7B to 8B: FP16 about 15 GB, Q4 about 4.5 to 5 GB. An 8 GB GPU can run the Q4 version.
    • 14B: FP16 about 28 to 30 GB, Q4 about 9 GB. A 16 GB GPU can run the Q4 version; FP16 needs more than 30 GB.
    • 32B: FP16 about 64 to 66 GB, Q4 about 19 to 20 GB. A 24 GB GPU can run the Q4 version, but with very little headroom.

    These figures cover only the weights. Inference also requires KV cache, compute buffers, and system VRAM overhead.

    3. KV Cache and Context Length

    KV cache stores the attention keys and values for processed tokens, and it grows linearly with context length. The space per token depends on the number of layers, KV heads, head dimension, and cache precision.

    For example, Qwen2.5-32B has 64 layers, 8 KV heads, and a head dimension of 128. With FP16 cache, each token takes about 256 KB. A 4K context uses about 1 GB, 8K uses about 2 GB, and 16K uses about 4 GB.

    After a 24 GB GPU loads Q4 weights, about 3 GB remains. After deducting the CUDA runtime and compute buffers, which usually take 1 to 1.5 GB, only about 2 to 2.5 GB is available for KV cache. As a result, the practical usable context is usually between 4K and 8K; the exact value should be measured with your own framework.

    If VRAM cannot hold the model, the inference framework will place some layers in system memory for computation, and speed will drop noticeably. During text generation, each generated token requires reading the model weights, so the upper speed limit is mainly determined by VRAM bandwidth.

    Take the RTX 3090 as an example. It has 936 GB/s of bandwidth, and the model is about 20 GB. The theoretical upper limit is roughly 936 divided by 20, or 46 token/s. Dual-channel DDR4-3200 has about 51 GB/s of bandwidth. If most layers are computed in system memory, the theoretical upper limit is only about 2.5 token/s. These are theoretical limits and real performance will be lower, but the order-of-magnitude difference is real. Therefore, keep as many layers in VRAM as possible.

    4. Candidate GPUs

    24 GB is the threshold for running a 32B Q4 model. The following GPUs are not recommended: 16 GB cards such as the RTX 4080 Super, RTX 4070 Ti Super, and Intel Arc A770, as well as the RTX 3060 12 GB.

    1. RTX 3090 (Used)

    Specifications: 24 GB GDDR6X, 384-bit, 936 GB/s memory bandwidth, 350 W TDP.
    Reference price: about CNY 4,500 to 5,500.
    Pros: Strong value, mature CUDA ecosystem, and ready to use after installing llama.cpp or Ollama. It can fit within the full-build budget.
    Risks: Mining cards are common, some cards have very high memory hotspot temperatures, and thermal pads or paste may already be aged.

    2. RTX 4090 (Used)

    Specifications: 24 GB GDDR6X, 384-bit, 1,008 GB/s memory bandwidth, 450 W TDP.
    Reference price: about CNY 12,000 to 15,000.
    Pros: Newer architecture, and prompt processing and generation are usually faster.
    Limitation: The price of a single card already exceeds the CNY 10,000 full-build budget. It becomes a reasonable choice when the budget is raised to around CNY 20,000.

    3. RX 7900 XTX (Used)

    Specifications: 24 GB GDDR6, 384-bit, 960 GB/s memory bandwidth, 355 W TDP.
    Reference price: about CNY 4,500 to 6,000.
    Limitation: ROCm is available on Linux, while Windows typically uses the Vulkan backend. Compatibility and performance differ from CUDA, so confirm that your inference tool supports this path before buying.

    4. Tesla P40 (Used)

    Specifications: 24 GB GDDR5, 346 GB/s memory bandwidth, 250 W TDP.
    Reference price: about CNY 1,000 to 1,800.
    Limitation: Passive cooling, so you need to add your own fan or duct. The Pascal architecture is old, and support for it in newer drivers and frameworks is gradually tightening. Confirm that the current version can run before buying. Bandwidth is only about one third of the RTX 3090, so generation will be noticeably slower.

    5. Two RTX 3060 12 GB Cards

    Two cards provide 24 GB in total, but model layers must be split across cards and data must pass through PCIe, so speed is lower than with a single card. Case space and power supply requirements are also higher. This article does not recommend this as a first choice.

    Overall: choose a used RTX 3090 first when budget is the priority; choose the RTX 4090 when budget allows; consider the P40 if you are willing to tinker and only care about VRAM capacity; consider the RX 7900 XTX if you prefer the AMD platform.

    The prices above are rough references for the used market and will change over time. Check recent completed sales before buying and use actual prices as the reference.

    5. Full-Build Budget Table (RTX 3090 Example)

    • GPU: used RTX 3090, CNY 4,500 to 5,500
    • CPU: Ryzen 7 5700X or equivalent 8-core/16-thread processor, CNY 800 to 1,000. With full GPU inference, CPU core count has little impact.
    • Motherboard: B550 or equivalent motherboard supporting PCIe 4.0 x16, CNY 600 to 900
    • Memory: 64 GB DDR4-3200 or DDR5 kit, CNY 700 to 1,200. Memory prices fluctuate significantly, so check current prices before buying.
    • Storage: 2 TB PCIe 4.0 NVMe SSD, CNY 500 to 700
    • PSU: 850 W 80 Plus Gold fully modular power supply, CNY 500 to 650
    • Case and cooling: CNY 300 to 500

    Total: about CNY 7,900 to 10,450, with a midpoint around CNY 9,000. If the price is high, memory can be reduced to 32 GB. With full GPU inference, 32 GB is already enough; 64 GB is mainly for CPU-offloaded layers and other services.

    Power can be estimated as follows: full-load power is approximately GPU TDP plus CPU TDP plus about 100 W for other components. With a 3090 and 5700X, that is about 350 + 105 + 100 ≈ 555 W. Leaving about 30% headroom requires at least 720 W, so an 850 W Gold power supply has enough margin.

    6. Real Impact of PCIe and Storage

    With single-GPU full inference, PCIe 4.0 x16 is sufficient. After the model is loaded, inference mainly depends on VRAM bandwidth, and PCIe has little impact. When choosing a motherboard, just make sure the GPU can operate in x16 mode; there is no need to pay extra for PCIe 5.0.

    Storage affects model loading speed. For a model file of about 20 GB, NVMe sequential read at 3500 MB/s gives a theoretical load time of about 6 seconds. SATA SSD at about 550 MB/s gives a theoretical time of about 37 seconds. The difference is tens of seconds, not minutes. Actual loading is also affected by system cache and file fragmentation.

    7. Inference Frameworks and Launch Parameters

    llama.cpp or Ollama based on it is recommended. Parameter names may change between versions, so use --help first to confirm.

    llama.cpp example:

    1. llama-server -m qwen2.5-32b-instruct-q4_k_m.gguf -ngl 99 -c 8192 -np 1 --port 8080

    Parameter notes: -ngl 99 puts as many layers on the GPU as possible. -c 8192 sets the context to 8K. -np 1 sets one parallel slot to avoid multiple sessions sharing the context.

    Ollama example: create a Modelfile:

    1. FROM ./qwen2.5-32b-instruct-q4_k_m.gguf
    2. PARAMETER num_ctx 8192
    3. PARAMETER num_gpu 99

    Then run ollama create and ollama run. Ollama’s default context is relatively short, so num_ctx needs to be set explicitly.