Practical Guide to Downloading and Local Deployment of Open-Source LLM Weights
Open-source Large Language Model (LLM) weights are typically published in the Hugging Face format, comprising configuration files, tokenizer files, and weight shards. After acquiring the files, developers must complete a series of steps including license verification, downloading, integrity checking, environment setup, inference testing, and service deployment. This article provides a model-agnostic, step-by-step guide applicable to most open-source weights distributed in this format.
Open-source Large Language Model (LLM) weights are typically released in the Hugging Face format, comprising configuration files, tokenizer files, and several weight shards. Once the files are obtained, developers need to sequentially complete license verification, downloading, integrity checking, environment setup, inference testing, and service deployment. This article is not specific to a single model; the steps apply to most open-source weights published in this format.
1. Verify Licenses and Access Permissions Before Downloading
Open source does not equate to unrestricted commercial use. Before downloading, check the license on the model page, focusing on three key aspects: commercial use allowance, attribution requirements, and whether derivative models must use the same license. Some models are classified as gated, requiring you to apply on the page and agree to the terms before downloading them using a personal access token.
Tokens should be stored in environment variables, not hardcoded in scripts or committed to repositories:
export HF_TOKEN="your_access_token"huggingface-cli login --token "$HF_TOKEN"
Newer versions of huggingface_hub might indicate that huggingface-cli is deprecated. In this case, follow the prompt to use the new command; the parameters remain largely the same.
2. Download Weights and Verify Integrity
Using huggingface-cli to download is recommended. It supports resuming broken downloads; if the network is interrupted, simply re-running the same command will continue the download:
pip install -U huggingface_hubhuggingface-cli download <organization_name>/<model_name> --local-dir /data/models/<model_name>
Alternatively, you can use Git. First, run git lfs install, then git clone. Large files are downloaded via Git LFS. If the large files are only a few hundred bytes after cloning, it means the LFS files were not pulled. Running git lfs pull will complete the process.
After downloading, file integrity must be verified. In the model page’s file list, each large file is marked with a SHA256 value. Calculate and compare them locally:
sha256sum /data/models/<model_name>/*.safetensors
If a hash does not match, delete the file and re-download it; do not load it directly. Additionally, prioritize the .safetensors format. This format only stores tensor data and does not execute arbitrary code when loaded. The .bin format is based on pickle, which can execute malicious code and should only be loaded from trusted sources. If a repository requires trust_remote_code to be enabled, review the custom code first.
3. Estimate VRAM Requirements
Estimate VRAM before deployment. The VRAM consumed by the weights themselves is approximately the number of parameters multiplied by the bytes per parameter: 2 bytes for FP16, 1 byte for INT8, and 0.5 bytes for INT4.
The table below shows the theoretical values for weights only, in GB:
| Model Size | FP16 | INT8 | INT4 |
|---|---|---|---|
| 7B | ~14 | ~7 | ~3.5 |
| 13B | ~26 | ~13 | ~6.5 |
| 70B | ~140 | ~70 | ~35 |
Actual usage also includes KV Cache, activations, CUDA context, and framework overhead. The KV Cache grows with context length and the number of concurrent requests. In long-context or high-concurrency scenarios, it may consume more VRAM than the weights themselves. Therefore, the values in the table represent the lower limits. Before going live, perform stress testing with actual context lengths and concurrency levels, and reserve at least a 20% VRAM margin.
4. Environment Preparation
It is recommended to use a Linux system with NVIDIA drivers matching your installed CUDA version. Use a virtual environment to isolate dependencies:
python -m venv llm-envsource llm-env/bin/activatepip install torch transformers accelerate
Follow the PyTorch official website’s selection results for installation commands to ensure they match your local CUDA version. device_map="auto" relies on the accelerate library; missing it will cause a direct error. If an older version is installed, run pip install -U accelerate to upgrade. For long-term stable operation, consider fixing dependencies in a Docker image to avoid issues caused by host environment differences.
5. Inference Code Example
The following example can run in both multi-GPU and single-GPU environments:
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizermodel_path = "/data/models/<model_name>"tokenizer = AutoTokenizer.from_pretrained(model_path)if tokenizer.pad_token_id is None:tokenizer.pad_token = tokenizer.eos_tokenmodel = AutoModelForCausalLM.from_pretrained(model_path,torch_dtype=torch.float16, # In newer versions of transformers, this parameter may be renamed to dtype; follow your installed versiondevice_map="auto",)model.eval()prompt = "Briefly describe the core technical principles of large language models."inputs = tokenizer(prompt, return_tensors="pt").to(model.device)with torch.no_grad():outputs = model.generate(**inputs,max_new_tokens=256,do_sample=True,temperature=0.7,top_p=0.9,pad_token_id=tokenizer.pad_token_id,)# Only decode the newly generated part, excluding the input promptnew_tokens = outputs[0][inputs["input_ids"].shape[1]:]print(tokenizer.decode(new_tokens, skip_special_tokens=True))
Note the following when using the code:
device_map="auto"assigns model layers to available GPUs. In a multi-GPU setup, input tensors should be placed onmodel.devicerather than hardcoded ascuda.- Chat models should use the tokenizer’s built-in chat template to construct inputs, otherwise output quality may degrade:
messages = [{"role": "user", "content": prompt}]text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(text, return_tensors="pt").to(model.device)
- In a single-GPU environment, you can remove
device_mapand executemodel.to("cuda")after loading. max_new_tokensdetermines the maximum generation length per run. The larger the value, the more VRAM it consumes.
