Training a Transformer from Scratch on CPU: The Complete End-to-End Workflow

    This article walks through what it actually takes to train a Transformer model from scratch entirely on CPU — no pre-trained weights, no shallow fine-tuning. It defines a concrete reference configuration (~110M parameters, 6-layer decoder-only, sequence length 256), sets out realistic wall-clock timings across different hardware, and explains the core mechanics behind multi-head attention, feed-forward layers, layer normalization, and autoregressive language modeling. You will find a step-by-step run guide with safety and download-verification notes, a troubleshooting table for common failures, the essential training loop in code, and a post-run checklist. It is aimed at readers who want to move from "using tools" to "understanding how training works."

    1. What “Training from Scratch” Actually Means: Making Training Observable

    While most discussion happens at the application layer — how to call an API, how to fine-tune a model — “training from scratch” describes a deeper path. Instead of loading pre-trained weights or making shallow modifications, you run the entire pipeline yourself: data preparation, forward computation, backpropagation, parameter updates, and evaluation.

    The value here is not in producing a commercially usable model. It is in turning the abstract idea of “training” into something measurable. You can watch the train_loss curve fall, track perplexity on the validation set, and see the model’s output gradually shift from gibberish toward something resembling real sentences. This is the step from “knowing how to use a tool” to “understanding the principles.”

    2. Three Major Obstacles and How the Teaching Approach Addresses Them

    Obstacle Typical symptom How the teaching approach responds
    Environment setup Framework, compute backend, and GPU driver versions conflict with one another Pre-packaged runtime with pinned framework and dependency versions, bundled with its own data
    Code barrier Training scripts span four layers: data, model, optimizer, and the loop itself Compress the main line into five steps: data → forward → loss → backward → update
    Compute cost Requires a professional accelerator card, which individuals struggle to afford Keep the parameter count around 100M, the sequence length at 256, and make it run on pure CPU

    3. Model Configuration and How Timing Is Measured

    Before discussing how long training takes, you must fix the configuration; otherwise, any timing number is meaningless. This article uses the following setup as its reference baseline:

    Item Value
    Layers 6-layer decoder-only Transformer
    Hidden dimension d_model 1024
    Feed-forward dimension 4096 (4×d_model)
    Attention heads 16 (64 dimensions each)
    Sequence length 256
    Vocabulary size 16000 (BPE)
    Parameter count ~110 million
    Training corpus 50 MB of plain text, ~10 million tokens
    Batch size 32, ~1220 steps/epoch, 3 epochs ≈ 3700 steps

    Timing is conditional, not a promise. Under the configuration above:

    • Intel i5-8250U (4 cores / 8 threads, 15W TDP): roughly 40–90 minutes
    • A desktop-class 8-core CPU (e.g., i7-10700): roughly 15–30 minutes
    • Reduce the sequence length to 128 and epochs to 1 (~1200 steps), and you can bring it down to around the 10-minute mark

    For consistency, timing should be measured as wall-clock time (using time python train.py or time.perf_counter() inside Python), and you should also record the framework version (torch.__version__), torch.get_num_threads(), whether MKL-DNN is enabled, and whether the data has already been cached. A “runs in a few minutes” claim without these conditions is not comparable.

    4. Technical Principles: The Complete Skeleton Inside a Small Model

    Multi-Head Attention: Computes the relevance between positions across the sequence in parallel across 16 representation subspaces. This is what distinguishes the Transformer from an RNN, and it is the foundation of context modeling.

    Feed-Forward Network (FFN): Applies a nonlinear transformation independently to each position. Attention decides “who to focus on”; the FFN decides “how to process the information.”

    LayerNorm and Residual Connections: LayerNorm stabilizes the input distribution of each layer, while the residual connection gives gradients an identity path. Together they let a 6-layer (or even deeper) network train stably.

    Autoregressive Language Modeling: The objective is to predict the next token. Cross-entropy measures the gap between the predicted distribution and the actual token, and backpropagation uses that error to update the parameters.

    During training you must record two kinds of metrics at the same time: train_loss (degree of fit) and validation metrics (such as val_loss, or perplexity ppl = exp(val_loss), degree of generalization). Watching only the training loss fall can hide overfitting — with a corpus of just 10 million tokens, 3 epochs is already close to saturation.

    5. How to Run It Step by Step (with Safety Notes)

    Confirm four things before you start: you have legal rights to use the data (your own text, public-domain corpora, or a clearly licensed open dataset); the package comes from a verifiable official source; you have a fresh, empty directory to work in; and the machine has more than 5 GB of free disk space and more than 8 GB of memory.

    1. Download the environment. Download only from the project’s official repository or release page, over HTTPS, and verify the SHA256 checksum provided on the page. Do not use third-party mirrors or forwarded file-sharing links directly.
    2. Verify and extract. After the checksum passes, extract into a newly created empty directory (for example, D:\transformer-demo\) rather than on top of an existing project directory — files with the same names inside the package will overwrite your existing content. On Windows, pay attention to SmartScreen warnings and your antivirus scan results; on macOS, watch for Gatekeeper blocks. Do not disable security mechanisms or grant administrator privileges just to “make it run.”
    3. Start training. Example executables: train_demo.exe (Windows), ./train_demo.sh (macOS/Linux). Common arguments: --epochs 3 --batch-size 32 --seq-len 256 --data ./data/corpus.txt --out ./ckpt. The bundled data is usually a single plain-text file (.txt) in UTF-8 encoding; keep the same format when you swap in your own corpus.
    4. Confirm the outputs and logs. Model weights are written to ./ckpt/ by default (e.g., ckpt_epoch3.pt), with the vocabulary and config in the same directory. Run logs go to ./logs/train.log, and any stack trace from a crash is written there too.
    5. Verify that it actually succeeded. Check three things: the loss decreases steadily in the training log; a non-empty weight file is generated under ./ckpt/; and when you load that weight with the same inference script, the generated text is clearly different from that of a randomly initialized model.

    Common issues to compare against:

    Symptom Common cause Fix
    Hangs with CPU maxed out Thread count exceeds physical cores Set torch.set_num_threads(number of physical cores)
    loss stays nan Learning rate too high, or abnormal characters in the data Lower the learning rate to 3e-4, clean the corpus
    Killed by out-of-memory Batch size or seq_len too large batch 32→16, seq 256→128
    Empty weight file Training did not reach the save point Check the --out path and --epochs
    Inference output still gibberish Loaded a wrong / untrained checkpoint Confirm the weight source and step count

    6. What the Training Loop Looks Like

    The code below is illustrative and cannot be run as-is: it omits the initialization of the model definition, dataloader, criterion, and optimizer, and only demonstrates the control flow. The minimum dependencies are pip install torch numpy tiktoken, and a complete, runnable example is available in Karpathy’s nanoGPT or the official PyTorch tutorials.

    1. import torch
    2. from torch.utils.data import DataLoader
    3. # The following four objects must be defined/initialized yourself; they are not shown in this illustrative code
    4. # model: nn.Module, containing embedding + 6 decoder blocks + lm_head
    5. # dataloader: returns (x, y), where y is x shifted right by one (the next token)
    6. # criterion: nn.CrossEntropyLoss()
    7. # optimizer: torch.optim.AdamW(model.parameters(), lr=3e-4)
    8. for epoch in range(num_epochs):
    9. for x, y in dataloader: # x: input context, y: target tokens
    10. logits = model(x) # forward: the prediction distribution at each position
    11. loss = criterion(
    12. logits.view(-1, vocab_size), # flatten the batch and sequence dimensions
    13. y.view(-1)
    14. )
    15. optimizer.zero_grad() # clear gradients from the previous step
    16. loss.backward() # backpropagation
    17. torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) # gradient clipping
    18. optimizer.step() # update parameters
    19. print(f"epoch {epoch} train_loss={loss.item():.4f}")

    These five actions — data, forward, loss, backward, update — are the shared backbone of training every model.

    7. Checklist After a Successful Run

    • Data source and licensing are documented
    • Loss keeps decreasing in the training log
    • Validation loss / perplexity is recorded, with no obvious overfitting
    • The weight file has been generated and can be reloaded reliably
    • Inference samples are reproducible (fixed random seed and sampling parameters)
    • The timing methodology is recorded (hardware, thread count, framework version, step count)

    8. Boundaries and Transferable Value

    Between a 100-million-parameter model and a hundred-billion-parameter model, the gap is not just orders of magnitude — it also involves data quality, training strategy, alignment techniques, and systems engineering. The incoherent text a small model produces actually confirms the scaling law that “data volume, parameter count, and compute together determine capability.” Once you understand a complete training pipeline, you will have the right intuition when you later approach large-scale training, data engineering, or tuning work.

    One last point must be stated clearly: a model trained locally does not produce factual output — it may reproduce biases or errors from the corpus, and it must not be used in scenarios that require factual accuracy. What is truly dangerous in technical learning is not “not knowing how to call a tool,” but having neither run through a complete pipeline once nor figured out what the hardware, the data, and the code each contribute.