Practical Guide to Downloading and Local Deployment of Open-Source LLM Weights

    Open-source Large Language Model (LLM) weights are typically published in the Hugging Face format, comprising configuration files, tokenizer files, and weight shards. After acquiring the files, developers must complete a series of steps including license verification, downloading, integrity checking, environment setup, inference testing, and service deployment. This article provides a model-agnostic, step-by-step guide applicable to most open-source weights distributed in this format.

    Open-source Large Language Model (LLM) weights are typically released in the Hugging Face format, comprising configuration files, tokenizer files, and several weight shards. Once the files are obtained, developers need to sequentially complete license verification, downloading, integrity checking, environment setup, inference testing, and service deployment. This article is not specific to a single model; the steps apply to most open-source weights published in this format.

    1. Verify Licenses and Access Permissions Before Downloading

    Open source does not equate to unrestricted commercial use. Before downloading, check the license on the model page, focusing on three key aspects: commercial use allowance, attribution requirements, and whether derivative models must use the same license. Some models are classified as gated, requiring you to apply on the page and agree to the terms before downloading them using a personal access token.

    Tokens should be stored in environment variables, not hardcoded in scripts or committed to repositories:

    1. export HF_TOKEN="your_access_token"
    2. huggingface-cli login --token "$HF_TOKEN"

    Newer versions of huggingface_hub might indicate that huggingface-cli is deprecated. In this case, follow the prompt to use the new command; the parameters remain largely the same.

    2. Download Weights and Verify Integrity

    Using huggingface-cli to download is recommended. It supports resuming broken downloads; if the network is interrupted, simply re-running the same command will continue the download:

    1. pip install -U huggingface_hub
    2. huggingface-cli download <organization_name>/<model_name> --local-dir /data/models/<model_name>

    Alternatively, you can use Git. First, run git lfs install, then git clone. Large files are downloaded via Git LFS. If the large files are only a few hundred bytes after cloning, it means the LFS files were not pulled. Running git lfs pull will complete the process.

    After downloading, file integrity must be verified. In the model page’s file list, each large file is marked with a SHA256 value. Calculate and compare them locally:

    1. sha256sum /data/models/<model_name>/*.safetensors

    If a hash does not match, delete the file and re-download it; do not load it directly. Additionally, prioritize the .safetensors format. This format only stores tensor data and does not execute arbitrary code when loaded. The .bin format is based on pickle, which can execute malicious code and should only be loaded from trusted sources. If a repository requires trust_remote_code to be enabled, review the custom code first.

    3. Estimate VRAM Requirements

    Estimate VRAM before deployment. The VRAM consumed by the weights themselves is approximately the number of parameters multiplied by the bytes per parameter: 2 bytes for FP16, 1 byte for INT8, and 0.5 bytes for INT4.

    The table below shows the theoretical values for weights only, in GB:

    Model Size FP16 INT8 INT4
    7B ~14 ~7 ~3.5
    13B ~26 ~13 ~6.5
    70B ~140 ~70 ~35

    Actual usage also includes KV Cache, activations, CUDA context, and framework overhead. The KV Cache grows with context length and the number of concurrent requests. In long-context or high-concurrency scenarios, it may consume more VRAM than the weights themselves. Therefore, the values in the table represent the lower limits. Before going live, perform stress testing with actual context lengths and concurrency levels, and reserve at least a 20% VRAM margin.

    4. Environment Preparation

    It is recommended to use a Linux system with NVIDIA drivers matching your installed CUDA version. Use a virtual environment to isolate dependencies:

    1. python -m venv llm-env
    2. source llm-env/bin/activate
    3. pip install torch transformers accelerate

    Follow the PyTorch official website’s selection results for installation commands to ensure they match your local CUDA version. device_map="auto" relies on the accelerate library; missing it will cause a direct error. If an older version is installed, run pip install -U accelerate to upgrade. For long-term stable operation, consider fixing dependencies in a Docker image to avoid issues caused by host environment differences.

    5. Inference Code Example

    The following example can run in both multi-GPU and single-GPU environments:

    1. import torch
    2. from transformers import AutoModelForCausalLM, AutoTokenizer
    3. model_path = "/data/models/<model_name>"
    4. tokenizer = AutoTokenizer.from_pretrained(model_path)
    5. if tokenizer.pad_token_id is None:
    6. tokenizer.pad_token = tokenizer.eos_token
    7. model = AutoModelForCausalLM.from_pretrained(
    8. model_path,
    9. torch_dtype=torch.float16, # In newer versions of transformers, this parameter may be renamed to dtype; follow your installed version
    10. device_map="auto",
    11. )
    12. model.eval()
    13. prompt = "Briefly describe the core technical principles of large language models."
    14. inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    15. with torch.no_grad():
    16. outputs = model.generate(
    17. **inputs,
    18. max_new_tokens=256,
    19. do_sample=True,
    20. temperature=0.7,
    21. top_p=0.9,
    22. pad_token_id=tokenizer.pad_token_id,
    23. )
    24. # Only decode the newly generated part, excluding the input prompt
    25. new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
    26. print(tokenizer.decode(new_tokens, skip_special_tokens=True))

    Note the following when using the code:

    • device_map="auto" assigns model layers to available GPUs. In a multi-GPU setup, input tensors should be placed on model.device rather than hardcoded as cuda.
    • Chat models should use the tokenizer’s built-in chat template to construct inputs, otherwise output quality may degrade:
    1. messages = [{"role": "user", "content": prompt}]
    2. text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    3. inputs = tokenizer(text, return_tensors="pt").to(model.device)
    • In a single-GPU environment, you can remove device_map and execute model.to("cuda") after loading.
    • max_new_tokens determines the maximum generation length per run. The larger the value, the more VRAM it consumes.