How to Train a 20-Million-Parameter Small Language Model from Scratch

    This guide walks through training a small language model of roughly 20–30 million parameters from scratch, covering the full pipeline: data preparation, tokenizer training, pretraining, supervised fine-tuning (SFT), preference optimization with DPO, and evaluation and deployment. It includes a recommended architecture and hyperparameters, runnable code and commands, a troubleshooting checklist, advanced optimization techniques, and compliance and rollback guidance. It's aimed at people who want to understand language models hands-on and at small vertical-domain tasks, offline inference, and low-latency local deployment—not complex multi-step reasoning or high-compliance production systems.

    Bottom Line First: What This Guide Delivers

    If you want to genuinely understand how language models work—rather than just calling an off-the-shelf model through an API—you can train a small language model of roughly 20 million parameters by hand. The goal isn’t to replace billion-parameter cloud models. It’s to complete the full loop of “data preparation, tokenization, pretraining, instruction fine-tuning, preference optimization, and evaluation and deployment” on hardware you can realistically afford.

    The scope of this guide:

    • Target model: A Transformer decoder architecture with roughly 20–30 million parameters.
    • Target hardware: A minimum of one 24GB consumer or workstation GPU. With one 80GB GPU, you can train larger batches and longer contexts more comfortably.
    • Final deliverables: A base text model, plus an instruction-tuned model capable of simple Q&A, classification, extraction, and summarization.
    • Prerequisites: Basic Python syntax, and a grasp of PyTorch tensors and gradients.
    • Suitable scenarios: Teaching and reproduction, small vertical-domain tasks, offline inference, low-latency local deployment, and model-principle validation.
    • Unsuitable scenarios: Complex multi-step reasoning, high-risk Q&A, strongly regulated production services such as finance or healthcare, and any automated system with extremely high reliability requirements.

    In one sentence: the payoff of training from scratch isn’t “building the most powerful model,” but gaining interpretability, controllability, and debuggability.

    1. Where Small Language Models Fit

    A 20-million-parameter model should not be marketed as “capable of anything.” Its strengths are that it is lightweight, transparent, and customizable. It fits tasks like:

    1. Text classification: ticket routing, review sentiment, simple intent detection.
    2. Information extraction: pulling names, timestamps, order IDs, or device names from short text.
    3. Instruction-following demos: simple summarization, formatted output, fixed-template Q&A.
    4. Offline assistants: a local prompt assistant embedded in a device, with its scope deliberately limited.
    5. Teaching and experiments: observing tokenization, embeddings, attention, loss curves, and backpropagation.

    It is not suited to complex logical reasoning, deep analysis of long documents, high-risk Q&A, or stable high-concurrency production services. Compared with cloud LLM APIs, a small model’s advantages are local deployment, data control, adjustable latency, and predictable cost; its disadvantages are limited knowledge coverage, weaker generalization, and a greater tendency toward repetition, hallucination, or off-topic answers on complex questions.

    So the sensible positioning is: locally distill or narrow down large-model capabilities for a task, rather than trying to match a general-purpose model across the board.

    2. Core Architecture: Even a Small Model Needs a Complete Skeleton

    Modern small language models are typically built on the Transformer decoder. A minimal trainable structure includes:

    1. Tokenizer: splits text into subword units. BPE or ByteLevel BPE is recommended. For Chinese, pay attention to vocabulary coverage so common characters, punctuation, digits, and units aren’t fragmented too finely.
    2. Embedding layer: maps tokens to vectors. Vocabulary size is usually 16,000 to 32,000.
    3. Positional encoding: rotary position embeddings (RoPE) are recommended, though learnable positional embeddings also work.
    4. Causal attention: each token can only see the content before it.
    5. Feed-forward network (FFN): strengthens each layer’s nonlinear representational capacity.
    6. Output layer: produces a probability distribution over the vocabulary to predict the next token.

    Attention logic can be expressed in pseudocode:

    1. import torch
    2. import torch.nn.functional as F
    3. def causal_attention(Q, K, V):
    4. d_k = K.size(-1)
    5. scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    6. # Causal mask: block future tokens
    7. seq_len = Q.size(-2)
    8. mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1).bool()
    9. scores = scores.masked_fill(mask, float('-inf'))
    10. weights = F.softmax(scores, dim=-1)
    11. return torch.matmul(weights, V)

    For a minimal reproduction, fix the random seed (for example torch.manual_seed(42)) and first validate output shapes, gradient health, and whether loss decreases on a small hand-built batch.

    3. The Full Training Pipeline at a Glance

    Training a small model isn’t a single script run—it advances in stages, and each stage should have inputs, outputs, and acceptance criteria.

    Stage Input Output Acceptance check
    Data preparation Raw text, domain corpora, license notes Cleaned training corpus Deduplication rate, length distribution, sensitive-content sampling
    Tokenizer training Clean text Tokenizer files Chinese character coverage, average token length
    Pretraining Unlabeled text Base language model Validation loss, perplexity, readability sampling
    Supervised fine-tuning Instruction–response data Instruction model Instruction-following rate, format correctness, manual spot checks
    Preference alignment Preference-pair data Aligned model Preference win rate, safety checks, regression evaluation
    Deployment evaluation Model weights Inference-ready model Latency, memory usage, task pass rate

    3.1 Data Preparation

    Pretraining data can come from compliant open-source corpora, your own domain documents, or data you’re explicitly licensed to use. At minimum, do the following:

    • Remove HTML, ads, repeated paragraphs, and low-quality text.
    • Deduplicate approximately so repeated sampling doesn’t cause overfitting.
    • Measure text lengths and keep a distribution suited to training.
    • De-identify personal information such as phone numbers, ID numbers, addresses, and account numbers.
    • Retain data sources, license types, and cleaning scripts for auditing and rollback.

    For SFT, use jsonl with one sample per line. For example:

    1. {"messages":[{"role":"user","content":"Rewrite this sentence in a formal tone: don't come tomorrow"},{"role":"assistant","content":"Please refrain from coming tomorrow."}]}

    For DPO, use preference pairs:

    1. {"prompt":"Help me write a short meeting reminder","chosen":"Please submit your project weekly report before 17:00 today.","rejected":"Don't forget the weekly report, don't leave it until tomorrow."}

    3.2 Tokenizer Training

    Recommended target vocabulary size: 16,000 to 32,000. For Chinese, 32,000 is usually more stable than 16,000. Example command:

    1. python train_tokenizer.py \
    2. --files ./corpus/clean/*.txt \
    3. --vocab_size 32000 \
    4. --min_frequency 2 \
    5. --special_tokens "<pad>|<unk>|<bos>|<eos>"

    Acceptance criteria: common Chinese characters, punctuation, digits, and domain terms should be reasonably covered, and the encoded text length should not balloon abnormally.

    3.3 Pretraining

    The goal of pretraining is to learn the statistical regularities of language. A recommended small-scale configuration:

    Parameter Recommended value
    Vocabulary size 32,000
    Model dimension d_model 320
    Layers 8
    Attention heads 8
    FFN dimension 1,280
    Context length Start at 1,024; up to 2,048 with enough memory
    Optimizer AdamW
    Learning rate Around 3e-4; experiment from 1e-4 to 5e-4
    Warmup 2%–5% of total steps
    Total training tokens Start from 100M–500M
    Batch strategy Small per-GPU micro batch, raised to a larger global batch with gradient accumulation

    Example launch command:

    1. python train_pretrain.py \
    2. --data_path ./corpus/pretrain/*.txt \
    3. --tokenizer_path ./tokenizer \
    4. --vocab_size 32000 \
    5. --d_model 320 \
    6. --n_layer 8 \
    7. --n_head 8 \
    8. --block_size 1024 \
    9. --total_tokens 200000000 \
    10. --micro_batch_size 8 \
    11. --global_batch_tokens 65536 \
    12. --learning_rate 3e-4 \
    13. --warmup_ratio 0.03 \
    14. --output_dir checkpoints/pretrained \
    15. --bf16

    On convergence time: avoid absolute claims like “converges in a few hours.” Actual time depends on GPU model, memory, data volume, framework implementation, parallel strategy, and training objective. Roughly, training 100M tokens on one 24GB GPU may take anywhere from a few hours to a few dozen hours; training 500M tokens or using longer contexts increases this noticeably.

    Pretraining acceptance criteria:

    • Training loss decreases steadily, with no long plateaus or violent swings.
    • Validation loss keeps dropping and falls below the random-initialization baseline.
    • Sampled generations shouldn’t contain large amounts of gibberish or entirely meaningless tokens.
    • For Chinese, check for abnormally frequent characters, punctuation, and digits.

    3.4 Supervised Fine-Tuning (SFT)

    The goal of SFT is to turn a model that can “continue text” into one that “answers according to instructions.” Recommended data size: 2,000 to 20,000 high-quality instruction samples.

    Recommended configuration:

    • Epochs: 2 to 3.
    • Learning rate: 1e-4 to 2e-4.
    • Context length: 1,024 or 2,048.
    • Compute loss only on assistant replies, so user input doesn’t dominate training.
    • Hold out 10%–20% as a validation set to watch for overfitting.

    SFT acceptance metrics:

    • Format correctness: can it output the required JSON, tables, lists, and so on?
    • Instruction-following rate: around 80% on a manually labeled validation set is generally usable.
    • Refusal rate: it should decline clearly unsafe or out-of-scope requests rather than mechanically completing them.
    • Manual spot checks: review at least 50 outputs; don’t judge by loss alone.

    3.5 Preference Alignment (DPO)

    DPO doesn’t require separately training a complex reward model, which suits small and mid-sized teams. The input is preference pairs of “better response” and “worse response.” Recommended size: 2,000 to 10,000 pairs.

    Recommended configuration:

    • beta: start at 0.1; adjustable between 0.05 and 0.3.
    • Learning rate: 5e-6 to 2e-5, usually much smaller than for SFT.
    • Epochs: 1 to 3.
    • Monitor preference accuracy rather than just loss.

    DPO acceptance:

    • On a manually or script-labeled preference set, the chosen response’s win rate should exceed the baseline.
    • Check for over-refusal, templated responses, sycophancy, or repetition.
    • Run a regression test on original capabilities, so instruction-following improves without base language quality degrading.

    4. Hands-On: From Random Weights to a Conversational Model

    Below is an executable order. Don’t stop at just loading a model and doing Q&A.

    Step 1: Set Up the Environment

    Recommended environment:

    • Python 3.10 or newer.
    • PyTorch 2.x.
    • CUDA 12.x or a version matching your GPU driver.
    • transformers, datasets, tokenizers, accelerate, safetensors.
    • Optional: wandb or tensorboard for logging loss.

    Example installation:

    1. python -m venv env
    2. source env/bin/activate
    3. pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
    4. pip install transformers datasets tokenizers accelerate safetensors tensorboard

    If your network is slow, you can point pip at a package mirror closer to your region, but verify checksums and source credibility.

    Step 2: Train the Tokenizer and Validate

    First run a tokenizer smoke test on 10,000 to 50,000 clean text samples. Verify that encode/decode round-trips the original text, and confirm that special tokens and Chinese coverage behave correctly.

    Step 3: Start a Small-Scale Pretraining Run

    First run the training loop end-to-end on 100,000 to 1,000,000 tokens and check:

    • Does data loading throw errors?
    • Is there learning-rate warmup?
    • Does loss decrease?
    • Is memory stable?
    • Can checkpoints save and resume?

    Once the smoke test passes, scale up to 100M–500M tokens.

    Step 4: Run SFT and DPO

    Only after pretraining is complete should you do instruction fine-tuning. Don’t skip the validation set, and don’t judge results from just a few examples. Build a fixed evaluation set covering:

    • Information extraction accuracy.
    • Classification accuracy.
    • Format-following rate.
    • Safety refusal rate.
    • Human readability score.

    Step 5: Deploy and State the Limits

    Before deploying, make the model’s boundaries explicit. You can declare in the system prompt that “this system handles only simple text tasks and is not suitable for high-risk decisions.” For production, add input filtering, output review, and human fallback mechanisms.

    5. Common Issues and Troubleshooting Checklist

    5.1 What to Do When You Run Out of Memory

    Handle it in this order:

    1. Lower the micro batch size.
    2. Lower the context length.
    3. Enable gradient accumulation.
    4. Enable gradient checkpointing.
    5. Use mixed precision, such as bf16 or fp16.
    6. Use CPU offload, though it slows things down.
    7. For inference, use quantization such as INT8 or INT4.

    5.2 What to Do When Loss Becomes NaN

    Check in sequence:

    • Is the learning rate too high?
    • Is there dirty data or abnormally long sequences?
    • Is fp16 causing overflow? Try bf16.
    • Is gradient clipping missing?
    • Are the data labels or mask misaligned?
    • Is some token ID outside the vocabulary range?

    5.3 What to Do When Loss Doesn’t Decrease

    Don’t blindly add epochs. First check:

    • Is the data fully duplicated?
    • Is the data format wrong?
    • Does the tokenizer match?
    • Is the learning rate too low?
    • Is warmup missing?
    • Are you training on very few samples?
    • Do you need more data or a higher proportion of task-relevant corpus?

    5.4 What to Do When Chinese Tokenization Coverage Is Poor

    You can:

    • Increase the vocabulary to 32,000 or 48,000.
    • Add domain terms to the tokenizer training.
    • Check whether you’re using an incompatible English pretrained tokenizer.
    • Run targeted evaluation on mixed Chinese-English text.

    5.5 What to Do When DPO Preference Pairs Are Noisy

    Check:

    • Is the difference between chosen and rejected too small?
    • Is there labeling inconsistency?
    • Is length bias making the model only learn to write longer answers?
    • Is the preference data inconsistent with the task goal?
    • Is overfitting causing more refusals?

    6. Advanced Optimization: Stronger and Cheaper

    6.1 Knowledge Distillation

    You can use a larger teacher model to generate high-quality answers and have the small model learn from them. Note that teacher outputs must be filtered by humans or rules—don’t let them directly pollute the training set.

    6.2 Quantization and Pruning

    Quantization converts FP16 weights to INT8 or INT4, reducing memory usage and inference cost. Pruning removes redundant weights, but small models don’t have many parameters to begin with, so it’s usually better to prioritize quantization and inference-engine optimization.

    6.3 The Limits of Mixture-of-Experts (MoE)

    MoE can reduce the compute per activated token, but it isn’t always a fit for tiny models. MoE may bring:

    • Higher total parameter count.
    • Increased memory usage.
    • Uneven routing load.
    • Greater inference-implementation complexity.
    • Added cost of tuning the number of experts and activation ratio.

    So evaluate it together with memory, inference latency, number of experts, activation ratio, and quantization and pruning. If the goal is resource-constrained deployment, MoE isn’t necessarily the first choice.

    6.4 Systematic Evaluation

    Don’t judge a model by “it feels good.” Build a fixed test set that includes:

    • Classification accuracy.
    • Extraction F1.
    • Format correctness.
    • Instruction-following rate.
    • Safety pass rate.
    • Inference latency.
    • Peak memory usage.
    • Output repetition rate.

    7. Compliance, Risk, and Failure Rollback

    Training a language model is not only an engineering problem but also a compliance one. At minimum:

    • Training data must have a legal source or clear authorization.
    • Personal data should be de-identified so privacy can’t be re-identified.
    • Filter pornographic, violent, illegal, and otherwise harmful content.
    • Retain data-cleaning records so nothing becomes untraceable.
    • Model outputs can’t be trusted by default; high-risk scenarios need human review.
    • When an experiment fails, keep the original configuration and checkpoints for rollback.
    • Don’t let a single training run overwrite the production model.

    If training quality regresses, use these rollback strategies:

    1. Roll back to the last stable checkpoint.
    2. Reduce the number of training tokens.
    3. Lower the learning rate.
    4. Check whether the data mix was contaminated by a single domain.
    5. Expand the validation set and re-confirm the target metrics.
    6. Tag model versions, such as v0.1-base-pretrain, v0.2-sft-clean.

    8. Conclusion

    Training a small language model from scratch is one of the most solid ways to understand large models. It turns attention, loss functions, gradient updates, data distributions, and optimizers from abstract concepts into an observable, tunable, reproducible experiment. A 20-million-parameter model can’t replace a general-purpose model, but it’s enough to carry you through a complete language-model engineering loop.

    The right attitude isn’t chasing “the most powerful model at the lowest cost,” but using verifiable steps to understand how a model learns to speak from random weights—and to know where its capability boundaries lie.