How Beginners Can Fine-Tune Large Language Models: Starting with LoRA

    A practical introduction to fine-tuning large language models with LoRA. Learn how to estimate hardware requirements, choose between full-parameter tuning, LoRA, and QLoRA, set initial hyperparameters, and prepare clean, representative datasets with a properly separated validation set.

    The Short Answer

    • For your first fine-tuning project, start with a parameter-efficient method such as LoRA instead of full-parameter fine-tuning.
    • Data quality usually affects the final result more than model size.
    • Set aside a validation set before training. Looking only at training loss means training without knowing how the model performs on unseen data.

    1. Evaluate Your Hardware Before Choosing a Method

    Fine-tuning requires significantly more resources than inference. With mixed precision and an Adam-style optimizer, full-parameter training requires roughly 16 bytes of memory per parameter for weights, gradients, and optimizer states, excluding activations. By this estimate, full-parameter training for a 30B-parameter model requires about 480 GB, which is far beyond the capacity of a typical workstation.

    By comparison, inference with FP16 weights requires approximately 60 GB for the same model. The resource requirements therefore differ by several multiples.

    You can use the following formulas for a rough estimate:

    1. Memory required for full-parameter training ≈ Number of parameters × 16 bytes (excluding activations)
    2. Memory required for inference weights ≈ Number of parameters × 2 bytes (FP16)

    The first step in method selection is therefore estimation. Start by identifying the model’s parameter count, estimate the memory required for full-parameter training, and compare it with the hardware available to you. If the estimate exceeds your device’s limits, switch to a parameter-efficient method.

    A distillation project from the author’s team provides a useful comparison. The project used 1,869 training samples and 401 validation samples to fully fine-tune a small model for 21-dimensional regression output. It achieved a median validation error of approximately 0.55. This shows that full-parameter fine-tuning can be practical for a small model and a relatively small dataset. Applying the same approach to a large language model, however, can quickly exceed the budget and hardware capacity of a beginner.

    2. LoRA: Freeze the Base Model and Train Low-Rank Updates

    LoRA works by freezing the original weight matrix (W) and adding two smaller matrices, (A) and (B), to the target layer. The forward computation becomes:

    1. h = Wx + BAx

    The rank (r) of (A) and (B) is much smaller than the dimensions of the original matrix. During training, only (A) and (B) are updated while the original weights remain unchanged.

    Consider a 4096 × 4096 attention projection matrix:

    • Full parameter count: 4096 × 4096 ≈ 16.77 million
    • LoRA parameters with r = 8: 4096 × 8 + 8 × 4096 = 65,536, or approximately 0.39% of the full parameter count

    This design provides three direct benefits:

    1. Lower memory usage. Optimizer states are maintained only for the small number of added parameters, making it possible to train a 7B-scale model on a consumer-grade GPU.
    2. Lower experimentation cost. If the result is unsatisfactory, you can retrain with a different configuration without modifying the original base model files.
    3. Better preservation of base-model capabilities. Because most parameters remain frozen, the risk of overwriting general-purpose capabilities is reduced. This makes LoRA suitable for adjusting formats, styles, and task workflows.

    QLoRA extends LoRA by quantizing the base model weights to 4-bit precision. For a 7B model, FP16 weights require approximately 14 GB, while 4-bit quantization reduces that to about 3.5 GB, excluding quantization constants and other overhead. When GPU memory is limited, this can make training feasible.

    The trade-offs are that training is usually slower and quantization error introduces some uncertainty. For this reason, QLoRA is often best considered when standard LoRA cannot run within the available hardware constraints.

    3. How to Set Initial Hyperparameters

    The goal of the first training run is to complete and validate the entire workflow, not to achieve optimal performance immediately. A reasonable starting point is:

    • Rank (r): Use 8 or 16. A higher rank increases representational capacity but can also increase the risk of overfitting. There is no need to start with a large value.
    • Learning rate: Use a relatively small value. For LoRA, the 1e-4 range is a common starting point. Adjust it later based on validation curves.
    • Number of epochs: Start with 1 to 3 epochs.
    • Batch size: Use the largest small batch size that your GPU memory allows, and use gradient accumulation to increase the effective batch size.

    A possible first-run configuration is:

    1. method: lora
    2. lora_rank: 8
    3. lora_alpha: 16
    4. target_modules: [attention_q, attention_v]
    5. learning_rate: 1.0e-4
    6. epochs: 2
    7. batch_size: 4
    8. grad_accumulation_steps: 8
    9. eval_every_steps: 50
    10. early_stop_patience: 3

    The exact field names vary by training framework. The configuration above is intended to illustrate the meaning of the parameters.

    After the first run, check whether the model can consistently produce a parseable format and complete the basic task. Reaching this point is already one of the most valuable steps in the process.

    After that, change only one variable at a time and record the results. This is the only reliable way to determine the actual effect of each modification.

    4. Data Is the Main Variable in Fine-Tuning

    Data problems account for a large share of failed fine-tuning projects. Method selection matters, but switching to a more complex method rarely helps when the dataset itself is poor. Data preparation can be divided into three areas.

    1. Keep the Data Clean

    • Cover the target task accurately. Instructions and answers should correspond to the tasks you actually want the model to perform. Randomly collecting generic question-and-answer pairs from public corpora to increase the dataset size can dilute the task signal. The model may learn an average style of response rather than the intended behavior.
    • Remove duplicate samples. Duplicates can cause the model to overemphasize certain patterns and can make evaluation results look artificially high.
    • Remove contradictory samples. If the same input is paired with different outputs, the model may learn an ambiguous average answer.
    • Use a consistent format. Field names, separators, and label conventions should remain consistent. Otherwise, the model has to learn multiple formats at the same time.

    The following example illustrates deduplication and format filtering:

    1. def clean_samples(samples):
    2. seen = set()
    3. cleaned = []
    4. for s in samples:
    5. # normalize is a text-normalization function, such as trimming whitespace
    6. # and standardizing full-width and half-width characters.
    7. key = normalize(s["instruction"]) + "|" + normalize(s["input"])
    8. if key in seen:
    9. continue
    10. # is_valid_format checks that required fields are present
    11. # and that the output follows the target format.
    12. if not is_valid_format(s):
    13. continue
    14. seen.add(key)
    15. cleaned.append(s)
    16. return cleaned

    Detecting contradictory samples is more difficult than removing duplicates. It usually requires manual review or rules that identify multiple outputs associated with the same input.

    2. Match Dataset Size to the Task

    A few hundred samples are rarely enough for a model to learn an unfamiliar domain. A focused, high-quality dataset of one to two thousand samples is a healthier starting point.

    In the distillation example mentioned earlier, 1,869 samples were effective largely because the task was narrow and the target was clearly defined. Quantity is necessary, but it is not sufficient. The more diverse and scattered the task is, the more samples are typically required.

    3. Set Aside a Representative Validation Set in Advance

    A declining training loss only shows that the model is fitting the training samples. It does not prove that the model performs better on new inputs. Validation metrics should be the basis for judging performance.

    In practice, keep the following points in mind:

    • Split the data before training begins. Validation samples must not be used for training or repeatedly treated as training data during hyperparameter tuning.
    • The validation set should follow the same distribution as the training set. It should cover the same types of scenarios and approximately the same proportions. If the training data focuses on several specific scenarios while the validation set contains only different scenarios, its metrics will no longer provide a meaningful reference.
    • When splitting by time or source, prevent data leakage. In particular, avoid allowing the same underlying record or closely related information to appear across both the training and validation sets.