LLM Identity Confusion: Distillation and Data Contamination Explained

Recently, developers testing large language models have noticed a curious behavior: when they ask a model who it is, it may answer with the name of another model. This “identity confusion” is often treated as a product defect, but technically it is a predictable outcome of several training and data governance issues. This article explains the phenomenon through three technical dimensions: model distillation, training data exhaustion, and synthetic data contamination.

1. The Essence of the Phenomenon: Where Style and Identity Come From

An LLM’s output style, common phrasing, and self-description are not arbitrary. They are learned statistical patterns from training corpora. When the training data contains a large amount of content generated by other models, the model also absorbs their wording habits, sentence templates, and identity labels. As a result, the answer to “Who am I?” depends heavily on which model identities appeared in the training corpus. This point is the foundation for understanding the technical details that follow.

2. Two Technical Routes for Model Distillation

Model distillation is a common engineering technique in which a smaller “student model” learns capabilities from a larger “teacher model.” Its goal is to preserve the teacher model’s capabilities as much as possible while compressing model size and reducing inference cost. In practice, it is usually divided into two categories:

White-box Distillation: Fitting Internal Representations Closely

White-box distillation aims to imitate the teacher model’s internal behavior in depth. It aligns not only the final outputs, but also the teacher model’s probability distributions (logits), attention allocation, and even neuron activation patterns. These methods often perform better, but they have a hard prerequisite: full access to the teacher model’s weights and internal structure. Therefore, they are generally applicable only to open-source models. Their typical loss function adds a measure of probability distribution difference beyond standard cross-entropy.

# Core idea of white-box distillation (illustrative pseudocode)
# The student model learns not only hard labels, but also the teacher model's soft probability distribution
student_logits = student_model(inputs)
teacher_logits = teacher_model(inputs)  # Requires teacher model weights
# Temperature-softened probability distributions
soft_student = softmax(student_logits / temperature)
soft_teacher = softmax(teacher_logits / temperature)
# KL divergence aligns probability distributions; cross-entropy aligns true labels
loss = alpha * kl_divergence(soft_student, soft_teacher) \
     + (1 - alpha) * cross_entropy(student_logits, hard_labels)

Black-box Distillation: Learning Only from Input-Output Pairs

When the teacher model’s weights are unavailable, teams must fall back to black-box distillation. In this case, the student model is trained with supervised learning based only on “input-output” pairs and does not access any internal information from the teacher model. This approach is easier to implement and more general, but it also means that the student model can only imitate the teacher’s surface behavior and struggles to reproduce its deeper reasoning ability. A common implementation is to batch-generate question-answer data through an API and use it to train an in-house model.

It is important to emphasize that both approaches are technical methods; neither is inherently right or wrong, and mutual distillation within the industry is already a common practice. The real problem comes from the lack of governance at the data level.

3. Training Data Exhaustion and Synthetic Data Contamination

As high-quality human-written corpora gradually approach exhaustion, training newer models increasingly relies on synthetic data generated by existing models. This creates two connected problems:

• Data self-circulation: If synthetic data lacks quality filtering over time, the training set is diluted by large amounts of low-quality model-generated output, forming a loop in which “models eat model outputs.”
• Model collapse: In theory, when a model is recursively trained on outputs from previous models, data diversity gradually declines, tail information is lost, and generated results become increasingly homogeneous and mediocre. This is one of the fundamental reasons why some later versions show degradation in literary quality and creativity: originally high-quality in-house corpora are contaminated by mediocre outputs from external models.

Therefore, synthetic data is not unusable. The key is whether strict source labeling and quality filtering mechanisms are in place.

4. How to Understand and Mitigate Identity Confusion

For developers and enterprise users, the following approaches can help:

1. Clarify the source and proportion of training data: When building in-house corpora, try to retain high-quality human-authored data as an “anchor,” and apply deduplication, quality scoring, and source labeling to external synthetic data so that it does not take up an excessively high proportion.

2. Build data traceability and audit capabilities: With general cloud capabilities such as logging services and object storage, fully record corpus sources, generation batches, and cleaning rules. If model outputs show anomalies, it becomes easier to locate the specific data batch involved.

3. Apply identity constraints at inference time: Defining the model’s role and identity boundaries through a system prompt can largely correct misaligned self-descriptions. When necessary, monitoring and alerting capabilities can be combined to observe and intercept abnormal self-identification outputs.

4. Accept the phenomenon as reasonable and focus on actual effectiveness: Identity confusion usually does not affect the model’s task completion ability. More important is the capability degradation caused by data contamination. Rather than focusing on “who the model says it is,” it is better to ensure output quality through continuous evaluation and data governance.

Conclusion

When an LLM “misrecognizes itself,” it may appear to be identity confusion on the surface, but it reflects a deeper issue in training paradigms and data governance. It reminds us that, when both compute and data are increasingly constrained, a model’s capability ceiling depends not only on scale, but also on the quality and source control of training data. For developers, understanding distillation mechanisms, guarding against synthetic data contamination, and building a traceable data governance system are key to maintaining high-quality model outputs over the long term.