画板 1

    LLM Self-Identification Failures: Distillation Contamination, Identity Alignment, and Governance

    5 hr ago0 views

    This article explains why large language models can misidentify themselves, especially when system prompts are removed or weakened. It breaks identity behavior into three sources—pretraining statistics, identity alignment data, and runtime system prompts—and provides a reproducible probe-based evaluation method. It then analyzes white-box and black-box distillation, the limits of data cleaning, diversity degradation from recursive training on synthetic data, and practical governance through data lineage, identity alignment datasets, preference optimization, and release gates.

    1. Deconstructing the Phenomenon: Where Does a Model’s “Self-Knowledge” Come From?

    A frequently discussed issue is that a leading general-purpose LLM, when not guided by a system prompt, may identify itself as belonging to another model family. Such outputs are often treated as slips or curiosities. From an engineering perspective, however, they are technical defects that can be located, measured, and repaired.

    A model’s self-identification is not controlled by a single switch. It is shaped by three overlapping sources:

    1. Identity statistical priors from pretraining corpora. During pretraining, a model sees large amounts of text such as “Who am I?” question-answer pairs, model self-descriptions, technical blogs, and code comments. These texts are frequent, templated, and highly similar in phrasing, so they can be encoded into model weights as strong statistical preferences.

    2. Identity alignment data from post-training. Post-training often injects explicit identity declaration samples, including developer attribution, capability boundaries, and refusal strategies. This is the only strong signal that can directly “correct the name” of the model.

    3. Runtime system prompts. System prompts have the highest priority, but they are a one-time contextual override within the context window. They do not change the weight-level prior.

    This leads to a key conclusion: a system prompt can suppress an incorrect identity, but it cannot repair an incorrect identity. Once the system prompt is removed, the language is switched, multi-turn elicitation is applied, or the sampling temperature is increased, the weight-level prior can resurface. This also explains why the issue is often observed only in a “bare model” state.

    A reproducible evaluation workflow:

    1. Input: model M under evaluation, with the system prompt set to empty
    2. 1. Build an identity probe set P covering Chinese, English, Japanese, and multilingual cases, including direct identity questions, role-play elicitation, multi-turn tracing, and capability-boundary follow-ups
    3. 2. For each probe, sample N = 8 times at temperature 0.7 and record the distribution of identity statements
    4. 3. Compute three metrics:
    5.    - Identity consistency rate = number of correct self-identifications / total samples
    6.    - Cross-identity rate = proportion of responses claiming membership in another model family
    7.    - Avoidance rate = proportion of refusals or vague statements
    8. 4. Run the probe set once at temperature 0 and once at temperature 0.7; the difference reflects how strongly the prior is suppressed

    If the model is consistent at temperature 0 but crosses identity boundaries after the temperature is increased, it means the system prompt and identity alignment data are working, but the weight-level prior is still contaminated. If both temperature settings produce identity violations, the issue originates in the pretraining stage.

    2. White-Box Distillation: Fitting Distributions, Not Copying Text

    White-box distillation assumes access to the teacher model’s weights and intermediate activations. Its goal is not to memorize what the teacher said, but to make the student’s probability distribution approximate the teacher’s probability distribution.

    The core mechanisms include three parts:

    • Soft-label fitting. The student does not learn from one-hot labels. It learns from the full probability distribution output by the teacher, with a temperature coefficient used to amplify information in low-probability regions.
    • Intermediate-layer alignment. Hidden representations are constrained with mean squared error, while attention maps are aligned through mapping constraints so that internal representation structures become similar.
    • Loss weighting. Distillation loss and true-label loss are mixed using a coefficient to prevent the student from completely losing its discriminative ability for task labels.

    Example pseudocode that expresses only the mathematical structure:

    1. def distill_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.7):
    2.     # Teacher gradients are not propagated
    3.     with no_grad():
    4.         t_soft = softmax(teacher_logits / T)
    6.     s_soft = log_softmax(student_logits / T)
    7.     kl = sum(t_soft * (log(t_soft) - s_soft)) * (T * T)   # Temperature squared is used to correct the gradient scale
    9.     ce = cross_entropy(student_logits, labels)
    10.     return alpha * kl + (1 - alpha) * ce

    The white-box path is limited by weight availability. It usually occurs only between open-weight models or between enterprise-owned models. However, it provides the most thorough channel for transmitting identity contamination. The teacher model’s self-referential habits, disclaimers, refusal boundaries, and even specific formulaic response structures can be fitted into the student’s weights as high-frequency short sequences. If the teacher model’s identity prior is already contaminated, the student can replicate the same error in the same location. This is a cross-generation form of contamination propagation.

    3. Black-Box Distillation: Synthetic Data Engineering at Scale

    Black-box distillation does not access weights or attention maps. It only uses input-output pairs from the teacher model. A typical pipeline is:

    1. Question pool construction -> Teacher sampling (answers + chain of thought) -> Rule-based and model-based filtering
    2. -> Difficulty stratification and deduplication -> Supervised fine-tuning / preference optimization -> Evaluation regression

    From a data engineering perspective, black-box distillation is essentially large-scale synthetic data production. Its value proposition is clear:

    • Expand pretraining or post-training corpora to fill gaps in high-quality data;
    • Reduce annotation costs by using the teacher’s reasoning chains as process supervision signals;
    • If one party can obtain high-quality domain data that it otherwise could not access, it can transfer that domain knowledge through the teacher model’s outputs.

    In scale, domain-specific distillation is typically in the million-to-ten-million sample range, while large-scale distillation from open-weight teachers can reach hundreds of millions of samples. This scale leads to a commonly overlooked conclusion: synthetic data at this volume cannot be cleaned completely.

    Common cleaning methods and their limitations:

    Cleaning method What it can filter What it cannot filter
    Exact or near deduplication Fully duplicate and near-duplicate samples Semantically equivalent samples expressed differently
    n-gram overlap detection Literal overlapping fragments with known corpora Structural and stylistic fingerprints
    Perplexity filtering Obvious low-quality, garbled, or truncated text Fluent but stylistically homogeneous teacher outputs
    Discriminator scoring Samples with factual errors Samples with incorrect identity statements

    In other words, literal decontamination is feasible, but removing stylistic and identity-level fingerprints is very difficult. This is why multiple models may show highly similar features in fixed opening phrases, specific itemized structures, or refusal templates. It is not coincidence; it is residue from data lineage.

    A popular but incorrect claim should also be clarified: distillation benefits are not determined by a model’s “intelligence tier.” Teacher outputs provide a data distribution, and the student gains the ability to fit that distribution. Whether a higher-capability model needs to learn from a lower-capability model depends on whether the target domain data is covered, not on model ranking. Using model capability levels to defend or condemn distillation behavior is technically unsound.

    4. Data Cannibalization: Mechanisms Behind Reduced Literary Quality and Diversity

    When the proportion of human-authored content in public corpora declines and the proportion of model-generated text increases, recursive training can cause observable degradation. The mechanism can be broken down into three steps:

    1. Tail loss. Teacher sampling often uses temperature truncation or nucleus sampling, systematically discarding low-probability but valid expressions. When the student fits this distribution, the tail contracts further.
    2. Mean reversion. Each generation of training pulls the distribution closer to the mode, variance decreases generation by generation, and the final result appears as if “all answers were written by the same model.”
    3. Homogenization feedback. High-frequency sentence patterns gain higher probability through repeated exposure, forming a self-reinforcing loop.

    A set of metrics can quantify this in engineering practice: decreasing distinct-1/distinct-2 scores, increasing self-BLEU, increasing sentence-template hit rates, and decreasing semantic variance across multiple samples for the same prompt. For Chinese creative-writing tasks, typical symptoms include more formulaic language, repeated imagery, and templated paragraph structures. This is usually not “capability degradation,” but the projection of corpus diversity degradation on the generation side.

    Possible mitigation directions:

    • Preserve sufficient high-quality human-authored corpus and set a minimum ratio that cannot be displaced by synthetic data;
    • Introduce diversity regularization or multi-teacher sampling to avoid convergence to a single-point distribution;
    • Preserve high-temperature sampling channels and explicitly track diversity metrics in evaluation instead of only looking at accuracy;
    • Apply difficulty stratification to synthetic data, prioritizing samples that the student “cannot answer” rather than the teacher’s most fluent outputs.

    5. Identity Alignment: From Metrics to Training

    Governance of identity issues should be treated as a data governance problem, not a prompt engineering problem.

    Data side:

    • Establish a data ledger that records the source, license, whether content is model-generated, and the identity of the generating model for each corpus item;
    • Label and isolate suspected model-generated corpora so they cannot directly enter identity alignment datasets;
    • Deploy a style fingerprint library at the corpus entrance, covering high-frequency opening sentences, fixed itemized structures, and specific disclaimer phrases. Matches should be down-weighted or sent for review;
    • Identity alignment datasets must be explicit, diverse, and multilingual. They should cover four question types: “Who am I?”, “Who developed me?”, “What is my knowledge cutoff?”, and “What can I not do?” They should also avoid repeatedly stacking content using a single template.

    Training side:

    1. Stage 1: Pretraining / continued pretraining -> control the synthetic corpus ratio and preserve a minimum share of human-authored data
    2. Stage 2: Identity alignment SFT -> inject explicit identity statements covering multilingual and multi-turn cases
    3. Stage 3: Preference optimization -> add an identity consistency term to the reward signal and apply negative rewards to out-of-scope identity statements
    4. Stage 4: Regression evaluation -> include identity consistency rate, cross-identity rate, and diversity metrics in release gates

    Example reward shaping:

    1. def reward(response, task_score, identity_ok):
    2.     # Identity violation is a hard constraint and is not included in weighted scoring
    3.     if not identity_ok:
    4.         return -1.0
    5.     return task_score

    Runtime side: The system prompt serves as the final fallback, but it should be treated as a masking layer. If evaluation shows that the model is correct only when a system prompt is added, the issue should be traced back and fixed in the data layer, rather than solved by continuously lengthening prompts.

    6. Industry Perspective: Transparency and Quality First

    Model outputs entering public corpora is an objective reality today, and cross-distillation is common across research and development processes in many organizations. Treating it as behavior unique to one party does not match technical reality. What deserves investment are three engineering tasks: traceable corpus sources, identifiable synthetic data, and regression-ready evaluation systems. Keeping the discussion at the level of moral accusations does not help solve the shared problem of declining data-pool quality. For enterprises with self-development needs, when building their own data loop on model platforms such as Baidu AI Cloud, data lineage management and identity-consistency evaluation should also be moved earlier into pipeline design, rather than remediated only after identity errors appear online.

    7. Conclusion

    A model “misidentifying itself” is a defect that can be attributed, measured, and repaired. It can be attributed to pretraining corpora being contaminated by outputs from other models, measured through identity probes and statistical metrics, repaired through identity alignment data and preference optimization, and defended in the long term through data lineage and decontamination pipelines. The accompanying decline in literary quality is essentially diversity contraction caused by recursive training on synthetic data. Understanding that these two phenomena share the same underlying data mechanism is more valuable than debating who is distilling from whom.

    09040904

    Top articles