LLM Parameter Count vs. Model Size: A Complete Conversion Guide

    A practical guide to counting parameters in neural networks and converting those counts into real GPU memory footprint. It breaks down how to count parameters layer by layer—convolutional, fully connected, embedding, and Transformer—shows how the data type (FP32, FP16/BF16, INT8, INT4) determines the bytes each parameter occupies, explains why training and inference memory differ so sharply, covers quantization as a way to shrink model size, and clarifies when high-speed interconnect actually becomes necessary.

    1. Start With a Real Question

    During a recent discussion, a colleague raised a very common question: with roughly 300 GB of VRAM available, do you really need high-speed interconnect bandwidth? And what parameter scale—tens of billions, hundreds of billions, or trillions—does that 300 GB correspond to?

    Answering this requires separating two things: how many parameters a model has, and how much space those parameters occupy in VRAM/memory. The gap between them is a simple but often overlooked relationship:

    Model size ≈ Parameter count × Bytes per parameter

    Below, we first explain how to count parameters for different layer types, then work through this conversion in detail, and finally return to the interconnect question.

    2. Layer by Layer: How Parameters Are Actually Counted

    Parameter count is fundamentally the total number of elements in all trainable weights (including biases). Different layer types have different weight tensor shapes, so the counting method differs too.

    2.1 Convolutional Layers

    The weights of a convolutional layer form a four-dimensional tensor with shape [out_channels, in_channels, kernel_height, kernel_width], plus one bias per output channel. So:

    1. params = (kernel_height × kernel_width × in_channels + 1) × out_channels

    Take the first layer of an early classic deep convolutional network as an example: a three-channel input image, an 11×11 kernel, and 96 kernels in total:

    1. (11 × 11 × 3 + 1) × 96 = (363 + 1) × 96 = 34944

    That is about 35,000 parameters for the layer. Notice that the product of kernel size and channel count dominates a single layer’s parameter count.

    2.2 Fully Connected Layers

    Fully connected layers come in two cases.

    Case 1: A fully connected layer that follows a convolutional layer. Here the input is the feature map produced by the convolution, “flattened” into a long vector whose length is output_space_side² × channels, then mapped to F outputs:

    1. params = (output_space_side² × channels + 1) × F

    Using the same example, the first fully connected layer takes a feature map with side length 6 and 256 channels, and outputs a dimension of 4096:

    1. (6² × 256 + 1) × 4096 = (9216 + 1) × 4096 = 37752832

    About 37.75 million parameters—showing that fully connected layers are often where parameters pile up.

    Case 2: A fully connected layer that follows another fully connected layer. Here the input dimension is simply the previous layer’s output dimension F_{l-1}, and the output dimension is F_l:

    1. params = (F_{l-1} + 1) × F_l

    For the final classification layer, with input 4096 and output 1000:

    1. (4096 + 1) × 1000 = 4097000

    2.3 Embedding Layers

    In large language models, the embedding layer is the most easily underestimated component. Take a classic bidirectional pre-trained language model as an example: its embeddings usually consist of three parts plus a LayerNorm—token embeddings, sentence/segment-type embeddings, and position embeddings.

    Assume a vocabulary size of 30522, a sequence length of 512, and a hidden dimension of 1024:

    • Token embedding Embedding(30522, 1024) → 30522 × 1024
    • Segment-type embedding Embedding(2, 1024) → 2 × 1024
    • Position embedding Embedding(512, 1024) → 512 × 1024
    • LayerNorm weight and bias → 1024 × 2

    Total:

    1. (30522 + 2 + 512) × 1024 + 1024 × 2 = 31782912

    About 31.78 million parameters. The key point here: embedding parameter count = vocabulary size × hidden dimension. The larger the vocabulary and the higher the hidden dimension, the heavier this part becomes—for models with very large vocabularies, it can account for a significant share of total parameters.

    2.4 Transformer and Self-Attention

    The core of the Transformer is self-attention. The input is a sequence of word vectors that pass through three separate linear transformations—Q, K, and V—and then a single output linear transformation after concatenation.

    Whether you use self-attention (Q = K = V from the same source) or ordinary attention (Q, K, V from different sources), the parameter counting is the same. The reason: before attention is computed, Q, K, and V each pass through their own independent linear mapping with their own weight matrices, so the parameter count is independent of whether the sources are shared.

    Let the hidden dimension be hidden, and let each linear layer have weights shaped [hidden, hidden] with a bias. Then each of the four linear layers—Q, K, V, and the output projection—contains hidden² + hidden parameters. With hidden = 1024, a single linear layer is about 1024 × 1024 + 1024 ≈ 1.05 million, and the four together total about 4.2 million.

    Going further, a complete Transformer layer also includes a feed-forward network (usually two layers, with the intermediate dimension expanded to 4× hidden) and two LayerNorms. Ignoring biases, a classic approximation is commonly used in engineering for quick estimation:

    1. params per Transformer layer ≈ 12 × hidden²
    2. total params ≈ num_layers × 12 × hidden² + embedding params

    This 12 × num_layers × hidden² rule of thumb is the most commonly used formula for estimating the size of dense Transformer models, and it helps you quickly gauge roughly what scale a model is in.

    3. From Parameter Count to Model Size: The Core Conversion

    3.1 Bytes per Parameter Depends on the Data Type

    Parameter count is a “count,” while model size is a “byte count.” The bridge between them is how many bytes each parameter uses:

    • Single-precision float FP32: 4 bytes
    • Half-precision FP16 / BF16: 2 bytes
    • 8-bit integer INT8: 1 byte
    • 4-bit quantization INT4: 0.5 byte

    3.2 Conversion Formula and Examples

    The conversion formula is straightforward:

    1. model size (bytes) = parameter count × bytes per parameter

    Take a large model with roughly 130 billion (130B) parameters as an example:

    • FP16/BF16 inference: 130 × 10⁹ × 2 = 260 GB
    • FP32 inference: 130 × 10⁹ × 4 = 520 GB

    So the same model can double its memory footprint purely because of precision differences. This is an important reason inference defaults to half precision: as long as the precision loss is acceptable, it directly halves both the footprint and bandwidth pressure.

    One caveat: the industry uses two conventions for GB—based on 10⁹ (commonly used in vendor specs) and based on 2³⁰ (i.e., GiB, commonly shown by operating systems). By the 2³⁰ convention, 130B at half precision is about 242 GiB, slightly different from 260 GB. It’s best to keep the convention consistent when evaluating.

    3.3 Answering “How Big a Model Fits in 300GB”

    Back to the opening scenario. If this 300 GB is VRAM used purely for weights during inference:

    • Half precision (2 bytes): 300 ÷ 2 ≈ 150 billion parameters, i.e., on the order of 150B
    • Single precision (4 bytes): 300 ÷ 4 ≈ 75 billion parameters, i.e., on the order of 75B

    Conclusion: 300 GB at half precision roughly corresponds to a “hundred-billion-parameter-class” model, a common tier among today’s mainstream large models; at single precision it drops to “between tens of billions and hundreds of billions.” So to the colleague’s question—“tens of billions, hundreds of billions, or trillions”—the answer is usually hundreds of billions, and almost certainly at half precision.

    4. The Overlooked Overhead: Training and Inference Memory Are Not the Same

    The conversion above only accounts for the “weights themselves.” Real usage depends on the scenario.

    4.1 Training: Gradients + Optimizer States Are the Real Bulk

    During training, in addition to weights, you also need to store gradients and optimizer states. With the most common adaptive optimizer plus mixed-precision training, a classic estimate is about 16 bytes per parameter:

    1. half-precision weights 2 + half-precision gradients 2 + FP32 master weights 4 + first moment 4 + second moment 4 ≈ 16 bytes/param

    In other words, training a 130B model can push the “model states” alone close to the 2 TB level, far beyond a single card’s capacity—which is the fundamental reason training almost always requires multi-card parallelism.

    4.2 Inference: Don’t Forget the KV Cache

    Inference doesn’t need an optimizer, but fitting the weights is only the first step—you also need to cache K and V for attention, i.e., the KV Cache. Its size is proportional to batch size × sequence length × num_layers × hidden dimension, and it grows rapidly with long context and high concurrency, sometimes taking up more VRAM than the weights themselves. So when evaluating inference memory, you can’t look only at model size; you also need to reserve space for activations and the KV Cache.

    5. Quantization: Fitting the Same Model in Fewer Bytes

    Since model size is directly determined by “parameter count × bytes per parameter,” besides throwing more VRAM at the problem, there’s another path: reducing the bytes per parameter—that is, quantization. Compressing weights from FP16 to INT8 or even INT4 roughly halves or quarters the model size, letting models that previously didn’t fit squeeze into limited VRAM. The cost is possible precision loss, usually controlled through post-training quantization or quantization-aware training. For VRAM-constrained deployment scenarios, quantization is often the most cost-effective move.

    6. Back to High-Speed Interconnect: When Is It Really Needed?

    Now we can answer the opening question head-on. When a model (or the total memory required for training) exceeds a single device’s capacity, you have no choice but to shard the model across multiple devices for parallelism—whether by splitting a single layer (tensor parallelism) or by splitting across layers (pipeline parallelism), both generate large amounts of data exchange between devices. This communication is frequent and concentrated; if the interconnect bandwidth between devices is insufficient, communication becomes the bottleneck, compute units sit idle waiting, and overall throughput drops rather than rises. Therefore:

    • If a single card/node can hold the model (e.g., medium-model inference after quantization), high-speed interconnect is usually not a hard requirement;
    • Once you need to shard across cards or nodes, especially at high degrees of parallelism, high-speed interconnect becomes almost essential.

    Back to the 300 GB scenario: if it is a single device’s capacity and can hold the target model, there’s no big problem; but if it is aggregated across multiple cards and needs tensor parallelism to run at all, then the importance of high-speed interconnect rises significantly.

    7. Summary

    Breaking a complex problem into three steps makes it clear:

    1. Count parameters layer by layer—convolutions by kernel size and channel count, fully connected layers by input/output dimensions, embeddings by vocabulary size and hidden dimension, and Transformers via the quick 12 × num_layers × hidden² estimate;
    2. Convert to model size using “parameter count × bytes per parameter,” where precision determines the byte count;
    3. Separate training from inference—training adds gradients and optimizer states, inference adds the KV Cache—and make good use of quantization to shrink the footprint.

    Once this chain is clear—whether you’re evaluating VRAM, choosing precision, or deciding whether to adopt high-speed interconnect—you have a solid basis for every decision.