Identity Perception Bias in Large Language Models: Causes, Diagnosis, and Mitigation
When a large language model misidentifies itself—especially under an empty system prompt—the problem is rarely a flaw in the model's "self-awareness" (which it doesn't have). Instead, it reflects identity anchoring signals embedded in training data, fine-tuning samples, alignment preferences, and inference pipelines. This article explains the four main sources of identity contamination, presents a reproducible diagnostic workflow combining black-box probes, pipeline audits, and data tracing, and outlines practical mitigation strategies including system prompt hardening, output validation, data cleaning, and evaluation gating.
Why Models “Get Their Identity Wrong” Without a System Prompt
In one test scenario, an inference call was made with a completely empty system prompt, and the model was asked only a basic question—“Who are you?”—yet it responded with a description of a different model’s identity. On the surface this looks like “the model said the wrong thing,” but it actually exposes the fragility of identity perception in large language models: the answer to “who am I” does not come from a stable sense of self, but from identity anchoring signals that repeatedly appear in training data, fine-tuning samples, alignment preferences, and the inference chain.
Any discussion of this issue should first distinguish two levels. First, does the model possess factual self-awareness? Clearly, no. Second, when the model generates identity-related answers, is it dominated by certain high-probability templates? This second question is the one that can actually be diagnosed and addressed in engineering practice. An empty system prompt does not guarantee a clean inference context: the call chain may contain default templates, gateway-injected content, residual conversation history, cache hits, evaluation framework prompt templates, or routing fallback strategies. Any of these can cause the model to “see” identity clues that are invisible to the user.
Four Sources of Identity Anchoring
1. Self-Referential Text in Training Corpora
During pretraining, large models are exposed to large amounts of Q&A, dialogue, documentation, and character introduction texts. If the corpus frequently contains self-referential expressions like “I am [X] assistant” or “developed by [X] team,” the model learns a statistical association: when users ask “who are you,” the answer should follow a certain assistant-identity template. With insufficient cleaning, identity introductions from other models, evaluation answers, or user paraphrases may enter the corpus and become identity anchoring signals.
This problem does not necessarily manifest as verbatim memorization. More often, the joint probability of “identity word—assistant tone—capability description” gets raised. Even if the model never copies a specific passage, it may produce similar identity statements under low-constraint conditions.
2. Contamination in Fine-Tuning and Preference Data
Instruction fine-tuning datasets typically contain large numbers of “user question—assistant answer” samples. If answers carrying another model’s identity markers slip in during labeling, collection, or synthesis, the model further reinforces that identity template during fine-tuning. Preference alignment data carries the same risk: if preference samples include identity self-descriptions, the model learns that stating that identity is a more assistant-style response.
Keyword filtering alone is not enough for mitigation, because identity statements can be paraphrased into variants like “I am an assistant trained by a certain team.” Instead, establish source labeling, sample review, and identity-field anonymization mechanisms, and run targeted sampling checks on identity-related Q&A samples.
3. Synthetic Data and Teacher Output Leakage
In synthetic data production, distillation-style training, or answer expansion workflows, other models’ outputs are often used as training signals. If those outputs retain the teacher model’s identity self-descriptions, the student model may learn the teacher’s identity. This type of leakage is subtle: individual samples look normal, but the problem gets concentrated and amplified on high-trigger questions like “who are you.”
Synthetic data should therefore undergo identity anonymization and consistency rewriting: keep the task answers while stripping teacher identity. Also record generation sources, prompt templates, and cleaning versions for traceability.
4. Evaluation Data and Inference Pipeline Leakage
Evaluation sets often include identity questions, role-play scenarios, and safety boundary probes. If the training set and evaluation set are not strictly isolated, evaluation prompt templates may leak into training data, causing benchmark contamination. Additionally, default system prompts in the inference chain, gateway templates, poorly designed cache keys, and incorrect conversation history concatenation can cause empty-system-prompt requests to actually carry identity information.
Impact Assessment: More Than Wrong Answers
Identity perception bias introduces multiple risks. First, product identity consistency suffers—users may develop misunderstandings about the service provider, responsibility boundaries, and capability scope. Second, audit and compliance risks increase: if identity-related answers reach external content, responsibility attribution becomes unclear. Third, evaluation validity degrades: once identity Q&A is contaminated, model capability assessments may be distorted. Fourth, safety policies may be bypassed—attackers can exploit identity confusion to induce the model to produce content outside product boundaries.
A Reproducible Diagnostic Workflow
A combined approach of “black-box probing + pipeline auditing + data tracing” is recommended. Black-box probes observe the phenomenon; pipeline audits rule out engineering-level injection; data tracing locates root causes on the training side.
The pseudocode below expresses the experimental design only and does not represent any specific product interface:
define identity_probe(model, question_set, system_prompt, sample_count):results = []for question in question_set:for i in range(1, sample_count):answer = model.generate(system_prompt=system_prompt,user_question=question,temperature=0.7)results.append(classify_identity(answer))return distribution_stats(results)Experiment groups:Group A: empty system promptGroup B: system prompt explicitly declares product identity and boundariesGroup C: system prompt contains a conflicting identity; observe whether the model holds the boundaryGroup D: insert identity-inducing questions in multi-turn conversations
During diagnosis, record the full call chain for every request: system prompt version, gateway template version, conversation history, cache keys, routing strategy, and model version. If the empty-prompt group shows a high proportion of foreign identities while the explicit-prompt group converges significantly, the problem lies in inference-layer injection or insufficient system prompt constraints. If both groups show the issue, prioritize reviewing training data and fine-tuning samples.
Mitigation Strategies and Best Practices
1. Declare Identity and Boundaries Clearly in the System Prompt
The system prompt should consistently declare the product identity, capability scope, refusal boundaries, and identity Q&A conventions, avoiding generic phrasing that could be confused with other assistants. System prompt versions should be managed through release processes, and cache keys must include the prompt version to prevent stale caches from polluting new requests.
2. Build Identity Consistency Output Validation
Add lightweight validation on the output side: for identity-related questions, extract identity fields from the answer and compare them against a product identity whitelist. On mismatch, trigger rewriting, refusal, or human handoff. Validation rules should cover multiple languages, paraphrases, and multi-turn inducement—not just fixed keyword matching.
3. Targeted Cleaning of Training and Synthetic Data
Establish source labeling and identity-field anonymization pipelines for pretraining, fine-tuning, preference, and synthetic data. Run targeted sampling checks on identity Q&A samples, rewrite identity self-descriptions in synthetic data while preserving task information and stripping teacher identity, and apply deduplication and leakage detection to high-frequency assistant templates and evaluation prompt templates.
4. Evaluation Isolation and Regression Gates
Training sets, evaluation sets, and prompt templates should be strictly isolated. Add identity consistency regression tests before release, covering empty system prompts, explicit system prompts, conflicting prompts, multi-turn inducement, and cache hit scenarios. Release only when the identity consistency rate meets the threshold.
5. Monitoring, Alerting, and Continuous Governance
Monitor the distribution of identity-related answers in production across model versions, prompt versions, and tenant dimensions, with alerts triggered when abnormal proportions appear. Combine logging services and monitoring/alerting capabilities to build a closed loop, and regularly review data source and inference pipeline changes.
Conclusion
A model calling itself by another’s identity under an empty system prompt is not evidence of emergent self-awareness—it is the result of identity anchoring signals being amplified through data and engineering pipelines. Developers should treat identity consistency as a testable, governable engineering quality metric: clean identity contamination from training and fine-tuning data, while also constraining system prompts, caching, routing, and output validation. Only by incorporating identity perception into continuous regression testing and monitoring can models reliably answer “who am I” across complex call chains.
