Where Do Large Language Model Capability Differences Come From? A Technical Analysis of Key Approaches

    Large language model capabilities are shaped by more than a single benchmark score. This article examines how model architecture, training data, training procedures, inference engineering, long-context design, code training, multimodal processing, and model compression affect cost, latency, reliability, and real-world performance. It also presents practical evaluation methods, including needle-in-a-haystack retrieval, pass@k, sandboxed code execution, and task-specific multimodal testing.

    Introduction

    Differences in large language model capabilities are rarely explained by a single metric. Two models may both claim to support long context, generate code, or understand images, while relying on completely different underlying implementations. Those implementation choices determine inference cost, response latency, and output stability—all of which affect whether a model is suitable for real-world applications.

    This article does not compare specific products. Instead, it examines the key technical factors behind capability differences and explains how to evaluate them.

    1. Four Technical Sources of Capability Differences

    Large language model capabilities are mainly shaped by four categories of factors:

    • Model architecture: This includes the implementation of attention mechanisms, positional encoding methods, and whether the model uses a Mixture-of-Experts (MoE) architecture. Architecture determines the upper limit of long-sequence efficiency and how much knowledge can be represented under a given compute budget.
    • Training data: Corpus size and quality, language distribution, and the proportions of code, mathematics, and multimodal data all matter. Data composition often has a more direct impact on a specific capability than architectural changes.
    • Training process: After pretraining, models typically go through instruction tuning, preference learning, reinforcement learning, or related stages. These stages affect how the model interprets instructions, follows output formats, and behaves around refusals and safety boundaries.
    • Inference engineering: Quantization, distillation, KV-cache management, and batch scheduling can significantly change deployment performance. The same model may have several-fold differences in latency and cost depending on how it is deployed.

    Understanding the relationship between these four layers helps explain where a claimed capability comes from and predict the situations in which it may fail.

    2. Long-Context Capability: The Bottleneck Is Attention Computation

    The computational complexity and memory usage of standard self-attention grow quadratically with sequence length. When the context window expands from 4K to 128K tokens, the number of elements in the attention matrix increases by approximately 1,000 times because (32^2 = 1,024).

    Long-context support therefore relies on several categories of techniques:

    1. Positional encoding extrapolation: Interpolation or scaled rotary positional embeddings (RoPE) can help the model retain positional awareness beyond its training length.
    2. Attention optimization: Block-wise computation, sliding-window attention, and sparse attention reduce computational complexity. Some approaches also share attention results across layers.
    3. Staged training: The model is first trained on shorter sequences and then gradually exposed to longer ones, allowing it to learn increasingly distant dependencies.
    4. Grouped-Query Attention (GQA): Multiple query heads share the same group of key-value heads, substantially reducing KV-cache requirements.

    The KV cache is one of the main sources of GPU memory usage during long-context inference. Its size can be estimated as follows:

    1. KV-cache bytes per token = 2 × number of layers × number of KV heads × head dimension × bytes per element

    Consider a model with 32 layers, 32 KV heads, a head dimension of 128, and FP16 values, where each element uses 2 bytes. Each token requires 524,288 bytes, or approximately 512 KB. Processing 128K tokens would therefore require about 64 GiB for the KV cache alone.

    This example shows that even when the model weights fit in GPU memory, long-context inference may still exhaust available resources because of the cache.

    It is also important to note that “supporting a 128K context window” only describes the maximum input length. It does not mean that the model can effectively use all information within that window. Evaluation should include needle-in-a-haystack tests: place a fact at different positions in a long document and check whether the model can retrieve it accurately. Testing should cover multiple positions and context lengths.

    3. Code Capability: Data, Training Signals, and Evaluation

    Improvements in code generation generally depend on three areas.

    First, data processing. Code datasets need to be deduplicated, filtered for low-quality files, checked for licensing issues, and balanced across programming languages. Excessive duplication can cause the model to memorize specific fragments and reduce its ability to generalize.

    Second, training signals. In addition to conventional next-token prediction, common approaches include:

    • Fill-in-the-Middle (FIM) training: The model completes code between a preceding and following context, which better matches code completion in an editor.
    • Instruction data: This covers tasks such as writing a function from a requirement, explaining code, and fixing an error.
    • Execution feedback: Unit tests are run, and only samples that pass the tests are retained for training or used as reward signals.

    Third, evaluation methods. The most common metric for code generation is pass@k: the probability that at least one of (n) generated samples passes the tests when (k) samples are considered. Chen et al. introduced an unbiased estimator in 2021. A commonly used implementation is:

    1. from math import comb
    2. def pass_at_k(n: int, c: int, k: int) -> float:
    3. """
    4. n: total number of samples generated for each problem
    5. c: number of samples that pass the tests
    6. k: number of samples considered during evaluation
    7. """
    8. if n - c < k:
    9. return 1.0
    10. return 1.0 - comb(n - c, k) / comb(n, k)
    11. # Example: 20 samples are generated, and 5 pass the tests
    12. print(round(pass_at_k(20, 5, 1), 3)) # 0.25
    13. print(round(pass_at_k(20, 5, 5), 3)) # approximately 0.84

    When executing model-generated code, use an isolated sandbox. Network access, file-system permissions, CPU time, and memory should all be restricted to prevent malicious or faulty code from affecting the host environment.

    4. Multimodal Models: How Visual Information Enters a Language Model

    Most multimodal large language models consist of three main components:

    • A vision encoder converts images into vector features.
    • A projection layer or connector maps those visual features into the representation space of the language model.
    • The language model interprets the combined information and generates a response.

    This architecture introduces several engineering challenges:

    • The trade-off between resolution and cost: Higher-resolution images produce more visual tokens. A single high-resolution image may generate hundreds or even thousands of tokens, consuming context-window capacity and inference compute.
    • Text recognition depends on retained detail: Reading tables, receipts, or small text requires high-resolution input or dynamic-resolution tiling.
    • Video understanding often relies on frame sampling: When key frames are extracted at fixed intervals, the model sees only part of what happens between sampled frames. Frame rate and the total number of frames directly affect temporal reasoning accuracy.

    Multimodal evaluation should be divided by task—for example, chart question answering, document extraction, interface understanding, and scene description. Accuracy should be reported separately for each task rather than allowing a single aggregate score to conceal specific weaknesses.

    5. Model Compression: Sparse Activation, Quantization, and Distillation

    The goal of model compression is to reduce inference cost while keeping quality degradation within an acceptable range. Three common approaches are sparse activation, quantization, and distillation.

    5.1 Mixture-of-Experts (MoE)

    MoE architectures divide the feed-forward network into multiple expert subnetworks. A gating network selects only a small number of experts for each token. As a result, the model can have a large total parameter count while activating only a fraction of those parameters for any individual token.

    The core routing process is to calculate a score for each expert, apply softmax to obtain probabilities, and then select the top (K) experts. The following is a simplified example:

    1. import torch
    2. import torch.nn.functional as F
    3. def route_tokens(hidden, gate_weight, k=2, temperature=1.0):
    4. """
    5. hidden: [number of tokens, hidden dimension]
    6. gate_weight: [hidden dimension, number of experts]
    7. Returns the selected expert indices and normalized weights for each token.
    8. """
    9. logits = hidden @ gate_weight # Score for each token-expert pair
    10. probs = F.softmax(logits / temperature, dim=-1)
    11. topk_probs, topk_idx = torch.topk(probs, k, dim=-1)
    12. topk_probs = topk_probs / topk_probs.sum(dim=-1, keepdim=True)
    13. return topk_idx, topk_probs

    One frequently misunderstood detail is the role of the temperature coefficient. Temperature changes the shape of the softmax distribution: a higher temperature produces a flatter distribution, while a lower temperature produces a sharper one. However, softmax is a monotonic transformation, so temperature scaling does not change the ranking of the expert scores.