Accelerating Diffusion Language Models with Feature-Space Distribution Matching

    This article explains how feature-space distribution matching can improve the quality of diffusion language models under low-step sampling. It covers the limitations of token-level denoising objectives, the differences between discrete and continuous diffusion, the use of Maximum Mean Discrepancy (MMD) as a global regularizer, and practical considerations for training, feature selection, sampling cost, and evaluation.

    1. The Speed–Quality Trade-Off in Diffusion Language Models

    Diffusion language models generate text through an iterative denoising process. They typically start from noise or a masked representation and progressively transform an incomplete intermediate state into coherent text over multiple steps. Unlike autoregressive models, which generate tokens sequentially from left to right, diffusion language models have the potential to generate tokens in parallel and refine their predictions iteratively. This has made them an active area of research.

    However, these models face a fundamental trade-off: generation quality is strongly tied to the number of sampling steps. With more steps, the model has more opportunities to correct errors and produce coherent text. With fewer steps, denoising may be incomplete, and local prediction errors can accumulate across the iterative chain. The result may include semantic drift, repetition, broken grammar, or inconsistencies with the broader context.

    In practical terms, reducing the number of steps forces the model to coordinate an entire sequence in less time. Existing training objectives often teach the model what to predict at the next step, but do not directly constrain what distribution the complete generated sequence should follow.

    This raises an important question: can training introduce a global constraint that helps preserve text quality during low-step sampling? One possible approach is to match the distribution of generated samples with that of real samples in feature space. In effect, this gives the model a global reference point. It evaluates not only whether individual token predictions are correct, but also whether the complete generated sequence resembles real text in its representation space.

    2. Two Technical Approaches to Diffusion Language Modeling

    Diffusion language models can broadly be divided into two categories based on the type of representation being modeled.

    2.1 Discrete Diffusion

    Discrete diffusion defines the noising and denoising process directly in token space. During training, some tokens in a sentence may be replaced with a special mask symbol, and the model must recover the missing tokens from their context. This process is similar to a fill-in-the-blank task: the model learns to predict the correct token at a masked position given incomplete context.

    The advantage of discrete diffusion is that it closely reflects the discrete nature of text. Its training objective is clear and can be integrated relatively easily with existing language modeling tasks. However, the denoising process often relies on independent or conditional predictions for individual positions, so the training signal remains primarily local. Even if each position is predicted reasonably well at one step, the distribution of the complete sequence may still deviate from that of real text.

    2.2 Continuous Diffusion

    Continuous diffusion first maps text into a sequence of continuous vectors. Noise is added and removed in this continuous representation space, after which the result is decoded back into text. This approach can take advantage of established noise schedules and samplers from continuous diffusion modeling. Its representation space is also more suitable for continuous optimization.

    Nevertheless, continuous diffusion language models face a similar decoding gap. Denoising errors in the continuous space, mapping errors introduced by the decoder, and the exposure bias associated with low-step sampling can all affect the final text. Whether the model uses a discrete or continuous approach, focusing only on local predictions at each position or step leaves the model without a mechanism for globally calibrating the distribution of complete generations.

    3. Limitations of Existing Objectives: Local Correctness Does Not Guarantee Natural Text

    The main training objectives used by diffusion language models can generally be described as predicting the next state or recovering tokens that have been corrupted. These objectives share several limitations.

    First, the supervision signal is localized. The model is corrected for a prediction at a particular position or timestep, but it receives no direct evaluation of the generated sequence as a whole. This is similar to teaching someone to write by correcting spelling mistakes one character at a time without explaining whether the entire passage is coherent or well structured. Correct individual tokens do not necessarily produce natural text.

    Second, training and low-step sampling may have different distributions. During training, the model encounters noisy states produced by a predefined schedule. During inference, aggressively reducing the number of sampling steps changes the distribution of intermediate states. A local denoising loss does not explicitly constrain this distribution shift, allowing errors to accumulate during low-step sampling.

    Third, the overall balance between diversity and realism is not directly constrained. If the model optimizes only token-level likelihood, it may favor frequent and conservative phrases. Under low-step sampling, it may also degenerate into repetitive patterns. A constraint at the distribution level can complement token-level supervision.

    A natural improvement is therefore to add a distribution-level regularizer to the existing denoising loss, encouraging generated samples to remain close to real samples in feature space.

    4. Core Idea: Use Feature-Space Distribution Matching as a Global Constraint

    Feature-space distribution matching compares real and generated text after mapping them into vector representations with a feature extractor, rather than comparing the raw token sequences directly. The reason is that token space is discrete and sparse, making direct distribution comparison difficult. Feature space can capture more stable global information, including semantics, syntax, and context.

    One commonly used distribution metric is Maximum Mean Discrepancy (MMD). MMD uses a kernel function to map samples into a reproducing kernel Hilbert space and compares the mean embeddings of the two sample distributions. Intuitively, the closer the two distributions are in feature space, the smaller the MMD value.

    Let the representations of real and generated text be (a) and (b), respectively, and let (k) denote the kernel function. The squared MMD can be written as:

    [
    \mathrm{MMD}^2(a,b)=
    \mathbb{E}[k(a,a’)]+
    \mathbb{E}[k(b,b’)]-
    2\mathbb{E}[k(a,b)]
    ]

    During training, MMD can be added to the total loss as a regularization term:

    [
    L{\text{total}}=
    L
    {\text{denoise}}+
    \lambda L_{\text{dist}}
    ]

    Here, (L{\text{denoise}}) is the original denoising or masked-token prediction loss, (L{\text{dist}}) is the feature-space distribution matching loss, and (\lambda) controls the relative contribution of the two objectives.

    The model must therefore learn both local denoising and global distribution alignment. This global constraint is particularly useful for low-step sampling because it helps address the situation in which each individual denoising step appears reasonable, while the resulting sequence as a whole is unnatural.

    5. Training Design: Include Short-Step Sampling in the Optimization Process

    Adding a distribution matching term is not enough by itself. The training process must also expose the model to the results of low-step sampling. One practical training loop can include the following steps:

    1. Sample a batch of real text and add noise using the standard diffusion process.
    2. Run the model’s denoising prediction and calculate the token-level or step-level denoising loss.
    3. Use a sampler with fewer steps to generate a batch of text representations, simulating the short-step inference scenario.
    4. Encode the real and generated text with a feature extractor.
    5. Calculate the Maximum Mean Discrepancy between the two groups of representations.
    6. Compute a weighted sum of the distribution matching loss and the denoising loss, then update the model through backpropagation.

    Short-step sampling introduces additional computational overhead. In practice, the cost can be controlled by computing the distribution loss periodically, caching representations, applying the loss to only part of each batch, or using a lightweight feature extractor.

    The feature extractor can share some layers with the main model or use a separate encoder. Sharing parameters is more efficient, while an independent encoder provides greater flexibility at a higher cost. The key requirement is that the representations should reliably reflect the overall semantics of the text rather than only shallow lexical patterns.

    The following framework-agnostic pseudocode illustrates the process:

    1. function train_step(batch):
    2. x_real = batch.text_tokens
    3. t = sample_timestep()
    4. noise = sample_noise_like(x_real)
    5. x_noisy = add_noise(x_real, t, noise)
    6. pred = model(x_noisy, t)
    7. loss_denoise = denoise_loss(pred, x_real, t)
    8. if should_compute_distribution_loss(step):
    9. x_fake = short_step_sampler(model, steps=K)
    10. feat_real = feature_encoder(x_real)
    11. feat_fake = feature_encoder(x_fake)
    12. loss_dist = mmd_rbf(feat_real, feat_fake, bandwidth)
    13. else:
    14. loss_dist = 0
    15. loss = loss_denoise + lambda_dist * loss_dist
    16. update(model, loss)
    17. return loss
    18. function mmd_rbf(a, b, bandwidth):
    19. k_aa = mean(kernel(a, a, bandwidth))
    20. k_bb = mean(kernel(b, b, bandwidth))
    21. k_ab = mean(kernel(a, b, bandwidth))
    22. return k_aa + k_bb - 2 * k_ab

    The central idea is to optimize not only whether denoising predictions are correct, but also whether the overall representations produced by short-step sampling resemble those of real text. The parameter (K) controls the intensity of the low-step simulation and allows the model to gradually adapt to reduced-step inference.

    6. Key Engineering Considerations

    6.1 Choose Feature Layers for Both Semantics and Stability

    The effectiveness of distribution matching depends heavily on the selected feature space. Features from shallow layers may mainly capture word forms and local combinations, making them insufficient for constraining global semantics. Features from very deep layers may overfit the training objective or reduce diversity.

    In practice, different layers can be evaluated, or features from multiple scales can be concatenated. The feature extractor should remain relatively stable during training. Frequent changes, especially early in training, may cause the distribution loss to fluctuate.

    6.2 Tune the Kernel Bandwidth and Loss Weight

    MMD is sensitive to the kernel bandwidth. If the bandwidth is too small, the comparison becomes overly local and gradients may become unstable. If it is too large, distribution differences may be overly smoothed and the constraint may become weak.

    Using multiple bandwidths or adapting the bandwidth based on batch statistics can be considered. The distribution loss weight (\lambda) should also be kept under control. If it is too large, it may overwhelm the denoising objective and reduce generation diversity. A practical strategy is to start with a small weight and monitor both generation quality and diversity on a validation set.

    6.3 Control the Cost of Short-Step Sampling

    Running short-step sampling at every training step can significantly increase computational cost. Possible approaches include periodic computation, applying the loss to only a subset of samples, using a smaller number of approximate sampling steps, or maintaining a lightweight sampling cache.

    For large-scale training, the distribution loss can be computed less frequently or introduced through a curriculum. The early stage can focus on establishing basic denoising capability, while the weight of short-step distribution matching is gradually increased later.

    6.4 Evaluate More Than Perplexity

    The goal of distribution matching is to improve the overall quality of generated text, so evaluation should cover multiple dimensions. These may include coherence under low-step sampling, repetition rate, semantic consistency, diversity, and human readability.

    Quality curves across different sampling step counts should also be compared. This helps determine whether the method genuinely improves the low-step region rather than merely maintaining similar quality at higher step counts. Conclusions should not be based on a single metric.

    6.5 Consider Numerical Stability and Distributed Aggregation

    MMD involves pairwise kernel computations. For large batches, memory usage and numerical stability must be considered. In distributed training, local sufficient statistics can be computed first and then aggregated across devices, avoiding the need to transfer large representation matrices directly.

    Normalizing the kernel matrix and applying numerical clipping can also help stabilize training.

    7. Conclusion

    One of the central challenges of diffusion language models is preserving the overall naturalness of generated text as the number of sampling steps is reduced. When training relies only on token-level or step-level denoising losses, the model can fall into a “locally correct but globally drifting” failure mode.

    Feature-space distribution matching provides a complementary solution. By using statistical measures such as Maximum Mean Discrepancy, it encourages the representation distribution of generated samples to approach that of real text, introducing a global perspective into training.

    This approach is not intended to replace the original denoising objective. Instead, it adds global calibration to local supervision. Practical implementation requires careful feature-space selection, control of short-step sampling costs, tuning of the kernel bandwidth and loss weight, and multidimensional evaluation of low-step generation quality.

    For developers seeking to improve diffusion language model inference efficiency without sacrificing usable text quality, feature-space distribution matching is a general and promising direction.