Scaling LLMs Without More Parameters: Parameter Density and Post-Training

    This article explains how large language models can continue improving without increasing total parameters, architecture, or activated parameters. It introduces parameter density as an analytical framework, then examines LoRA, the compression-as-intelligence perspective, MoE architectures, the different demands of memory-heavy and reasoning-heavy tasks, and the role of post-training. It also offers practical guidance for developers on model selection, fine-tuning, evaluation, and deployment.

    Key takeaway: When parameters stay fixed, parameter density increases

    Over the past few years, large model scaling has often been viewed as a linear expansion of more parameters, more data, and more compute. However, as base models have become more mature, an increasingly important question has emerged: can a model continue to improve when the total parameter count, architecture, and even the number of activated parameters remain unchanged?

    The answer is: under specific conditions, yes. In this context, “becoming stronger” does not mean adding more parameters. Instead, it means improving the model’s effective capability density on target tasks. This article refers to this phenomenon as parameter density scaling.

    Parameter density can be roughly defined as the effective capability a model produces per unit of total parameters or per unit of activated parameters on a given task.

    It is important to note that parameter density is not a unified or standardized metric. Rather, it is an analytical framework that helps explain differences in resource allocation across training, post-training, and inference.

    A public example is DeepSeek-R1. According to its publicly available technical report, DeepSeek-R1 uses DeepSeek-V3-Base as its base model and relies primarily on reinforcement learning for post-training. The report shows significant improvements on the mathematics and code reasoning benchmarks listed in the report.

    Three boundaries should be clarified:

    • The source is the publicly available DeepSeek-R1 technical report.
    • The base model is DeepSeek-V3-Base.
    • The evaluation scope is limited to the mathematics and code benchmarks listed in the original report and should not be directly extrapolated to all business tasks.

    This example shows that post-training may lead to substantial capability gains even when the parameter count remains unchanged. However, this should not be treated as a universal rule.

    What LoRA tells us: the intrinsic dimension of adaptation may be low

    Low-Rank Adaptation, commonly known as LoRA, is a class of parameter-efficient fine-tuning methods. Its basic approach is to freeze most of the pretrained model weights and train only a small set of low-rank update matrices.

    The proportion of trainable parameters in LoRA is usually very low, yet the method can still produce noticeable gains on specific tasks.

    This phenomenon is often used to suggest that the parameter space of a model is not completely saturated. A more precise interpretation comes from the original LoRA paper and related research on low intrinsic dimensionality: the weight updates required for task adaptation may have a relatively low intrinsic dimension.

    In other words, the number of effective degrees of freedom required to complete a specific adaptation task may be limited.

    However, another boundary is needed here: the effectiveness of LoRA does not directly prove that the pretrained weights themselves are largely redundant, nor does it prove that all knowledge has already been maximally compressed.

    It only shows that, for many downstream tasks, adapting model behavior may not require modifying all weights.

    For developers, this offers a practical implication: when compute resources are limited, low-rank or localized fine-tuning can often be a cost-effective approach.

    Compression as intelligence: a cautious interpretation

    A common claim in the industry is that “compression is intelligence.”

    This claim can be cautiously interpreted as follows: intelligence may arise from efficiently encoding information, extracting useful features, and expressing regularities in a low-complexity form. A system that can express more regularities with fewer parameters or fewer symbols may often demonstrate stronger generalization ability.

    However, “compression is intelligence” is not a law that can be cited directly. More precisely, it is a viewpoint held by some researchers regarding the mechanism of intelligence.

    If a model can still improve efficiency through compression, this does not necessarily mean that its parameters are redundant. It may also mean that the training objective, task distribution, or post-training method has not yet fully unlocked the value of the existing parameters.

    Therefore, parameter density should not be understood as a physical compression ratio. It is better understood as task utility density.

    The same model size, trained with different data, different post-training methods, and different inference strategies, may show very different effective capabilities.

    MoE architecture: separate total parameters, activated parameters, and inference compute

    Modern large models often adopt the Mixture of Experts, or MoE, architecture. Understanding MoE models requires distinguishing three concepts.

    • Total parameters: This determines the upper bound of model capacity. It can be understood as how much knowledge the model can store.
    • Activated parameters: This refers to the number of parameters actually involved in the forward computation for each token. It determines the approximate FLOPs of a single forward pass. It does not necessarily equal the total parameter count.
    • Effective compute depth or test-time compute: This refers to the additional computation invested during inference, such as reasoning steps, sampling, search, tool calls, or additional verification. If this concept is used, it should be clarified that it is not traditional model depth, but rather extra computation performed during inference.

    Therefore, a large total parameter count in an MoE model does not mean that every token uses all parameters. Similarly, a low activated parameter count does not necessarily mean that the model “thinks shallowly.”

    A lower number of activated parameters means lower compute cost per forward pass, but the model may still achieve stronger performance through reasoning chains, repeated sampling, reward-model filtering, or tool calls during inference.

    Dimension Determining factor Observable metric or engineering implication
    Total parameters Upper bound of model capacity; knowledge storage capacity Model size, memory footprint, potential knowledge coverage
    Activated parameters Per-token forward computation cost Inference FLOPs, latency, throughput
    Training data and post-training Task adaptation quality; reasoning capability Domain benchmarks, reasoning benchmarks, alignment quality
    Inference-time reasoning steps or test-time compute Compute invested during inference Reasoning length, sampling count, tool calls, cost ceiling

    When evaluating an MoE model, it is not sufficient to look only at total parameters or only at activated parameters. A comprehensive evaluation should consider capacity, activated compute, training data quality, post-training methods, and inference-stage cost.

    Memory-heavy tasks and reasoning-heavy tasks: a conditional distinction

    A common observation is that memory-related tasks and reasoning-related tasks depend on resources in different ways.

    Memory-related tasks, such as factual retrieval and knowledge question answering, usually depend more on parameter count and knowledge injection because the model needs to “store” information.

    Reasoning-related tasks, such as mathematics, code generation, and multi-step planning, usually depend more on high-quality training data, task environments, and post-training because the model needs to “learn” how to solve problems.

    However, this is not an absolute dichotomy.

    Reasoning ability may also benefit from a larger base model, because stronger representation capability may provide a better foundation. Conversely, memory ability is not built solely through parameter scaling; data quality, knowledge structure, and retrieval mechanisms also matter.

    A more cautious statement is as follows:

    When training data is relatively fixed and post-training is insufficient, simply continuing to increase total parameters may yield diminishing returns on some tasks.

    On some reasoning benchmarks, increasing parameters may not reliably improve performance. In some cases, improvement may become less significant due to mismatched data distributions, unclear optimization objectives, or differences in evaluation sets.

    Whether negative returns occur depends on the specific task, the specific data, and the specific evaluation methodology. It cannot be generalized.

    Post-training: improving task fit on the existing parameter set

    If the base model already has enough capacity to support most tasks, then the focus of competition shifts from “who can stack more parameters” to “who can achieve higher effective capability with the same parameters.”

    Post-training is a key lever in this shift.

    Without changing the model architecture, total parameter count, or activated parameter count, post-training can mainly improve parameter density through two types of methods:

    1. More complex task environments: allowing the model to repeatedly practice long-horizon dependencies, multi-step reasoning, tool calls, code execution, or real-world business scenarios, so that the weights better match the target task distribution.
    2. Stronger or more refined reward signals: using supervised fine-tuning, preference optimization, or reinforcement learning to guide the model to adjust its output tendencies, making the existing parameters more effective on target tasks.

    A common misunderstanding should be avoided here: post-training does not simply turn “redundant parameters” into “capability parameters,” nor does it automatically make the model stronger by deleting redundancy.

    A more reasonable description is that post-training adapts the weights to the target task distribution through task environments and reward signals while keeping the original parameters unchanged.

    This process can improve performance on specific tasks, but it may also produce inconsistent effects: improvements in some capabilities may come with degradation in overall capability.

    At the same time, large-scale reinforcement learning is not inexpensive. It usually requires substantial sampling, verifiers, reward models, data construction, and compute support.

    Without reasonable reward design, problems such as reward hacking, benchmark gaming, formalistic answers, and reduced generalization may occur.

    Therefore, “a model can improve without increasing parameters” does not mean that improvement is low-cost. Instead, it means that investment shifts from parameter expansion toward data, post-training, and inference configuration.

    Practical implications for developers: from parameter-only evaluation to density-aware evaluation

    Understanding parameter density has practical value for model selection, fine-tuning, and deployment evaluation.

    First, do not evaluate a model only by its total parameter count.

    At the same parameter scale, differences in training data quality, post-training strategy, inference-time compute, and task adaptation may lead to significant capability gaps.

    A more reasonable evaluation should include:

    • Total parameter count
    • Activated parameter count
    • Inference latency
    • Task-specific metrics
    • Knowledge coverage
    • Stability
    • Safety boundaries

    Second, for memory and knowledge-related tasks, parameter scale, knowledge injection, and retrieval augmentation should still be taken seriously.

    For reasoning and multi-step tasks, developers should pay more attention to high-quality data, task environments, reward signals, and test-time compute.

    Third, parameter-efficient fine-tuning remains valuable.

    Methods such as LoRA are effective because the weight updates required for many adaptation tasks have a relatively low intrinsic dimension.

    However, the rank should not be made as low as possible. If the rank is too low, the model may underfit. If the rank is too high, it may increase training cost and memory pressure. The appropriate rank should be selected based on task complexity, data scale, and validation results.

    Fourth, before deployment, developers should guard against “better metrics but weaker capabilities.”

    If post-training over-optimizes a single reward, reward hacking or alignment tax may occur. It is advisable to run regression tests on independent evaluation sets and also monitor general capability, safety, format stability, and long-tail task performance.

    Common misconceptions

    Does a larger MoE model with more total parameters always perform better?

    Not necessarily. Total parameters represent the upper bound of capacity, but performance on a single task also depends on activated parameters, training data, routing quality, post-training, and inference compute configuration.

    Does doubling the parameter count always improve reasoning ability?

    Not necessarily. Reasoning ability is affected by data quality, task distribution, reward signals, and evaluation methods. On some tasks, increasing parameters may yield diminishing returns and may be less effective than targeted post-training.

    Is a lower rank always better for low-rank adaptation?

    No. If the rank is too low, it may limit expressive capacity and lead to insufficient task adaptation. If the rank is too high, it may increase training cost and memory pressure. The rank should be validated according to task complexity.

    Conclusion

    From LoRA to MoE, and then to post-training and reinforcement learning, a new understanding of large model scaling is emerging: capability growth does not necessarily come only from parameter expansion. It may also come from improvements in parameter density.

    With the same base model, the same architecture, and the same parameter count, training data, post-training methods, and inference strategies may still become key variables affecting model capability.

    Therefore, “a model can become stronger without increasing parameters” should not be interpreted as a universal promise. It should be viewed as an engineering phenomenon: under suitable base models, suitable tasks, and suitable post-training conditions, a model can continue to improve by increasing task adaptation density.