Understanding LLM Evaluation Data: Parameters, Benchmarks, and Hallucination Rates

    New LLM releases often mix parameter counts, benchmark scores, and hallucination rates in the same discussion, even though these metrics describe different aspects of a model. This article explains why engineering teams should separate them during model selection, how total and activated parameters differ in MoE architectures, and why long context windows still incur substantial KV-cache memory costs. It also includes the KV-cache size formula and a Python function signature for estimating memory use at 1 million tokens.

    1. Why Model Release Data Needs to Be Read Separately

    Every new model release brings parameter counts, benchmark scores, hallucination rates, and other numbers at the same time. They describe different objects. Parameters describe model structure; benchmark scores describe performance on specific tasks; hallucination rates describe output reliability. Comparing the three as if they were the same thing can easily lead to wrong conclusions. For engineering teams doing model selection, a more reliable approach is to first define which capabilities the business needs, then check item by item whether the numbers correspond to those capabilities.

    2. Parameter Scale: Total Parameters and Activated Parameters

    Mixture-of-Experts (MoE) architectures split a model into multiple expert sub-networks, and each token passes through only some of them. This makes it necessary to distinguish two numbers:

    • Total parameters: the number of all weights, which determines storage and GPU memory requirements.
    • Activated parameters: the number of weights actually involved in a single forward computation, which roughly determines the compute per token.

    Take a hypothetical model with 1000B total parameters and 40B activated parameters. Its per-token compute is close to that of a 40B dense model, but deployment still requires loading all weights. At 2 bytes per parameter (FP16 or BF16), the weights alone require about 2 TB of GPU memory, not including KV cache and runtime overhead. Therefore, a smaller activated parameter count usually means lower per-token compute, but it does not mean a lower hardware threshold.

    3. Long Context: The Real Cost of 1 Million Tokens

    The context window is the maximum number of tokens a model can process at once. A large window does not mean the model can use it well every time, nor does it mean resource overhead can be ignored. During inference, the model must store K and V vectors for historical tokens, i.e., the KV cache. Its GPU memory usage grows linearly with sequence length. The formula is:

    KV cache bytes = 2 × layers × KV heads × head dimension × sequence length × batch size × bytes per element

    The following script uses hypothetical structural parameters to estimate the cache footprint for 1 million tokens:

    1. def kv_cache_gb(layers, kv_heads, head_dim, seq_len, batch=1, bytes_per_elem=2):
    2. # factor 2