AI Chat Services Leaking Real Phone Numbers: Root Causes and Engineering Safeguards

1. Problem Abstraction: Why an Incorrect Contact Becomes a Privacy Incident

A concerning pattern has emerged in recent public discussions: when users ask a generative AI chat service how to contact a certain type of service provider, the model may generate a phone number or instant messaging account that looks plausible but actually belongs to a real individual. Callers and people sending friend requests do not know that the number is not an official customer service contact; they simply trust the “answer” provided by the chat service. As a result, the number owner is continuously harassed, and the company’s customer service channels are also affected by the incorrect information.

On the surface, this looks like “the model gave a wrong answer.” In reality, it is a composite risk created by privacy leakage, factual hallucination, and product interaction design flaws. It differs from a traditional database breach or API privilege escalation: there are no obvious signs of system intrusion, and no complete batch of user records is directly exposed. The leakage occurs at the moment the model generates text, and it is random, fragmented, and difficult to reproduce. For this reason, traditional security systems based on perimeter protection and permission control are unlikely to work alone.

From an engineering perspective, there are at least three types of affected parties:

1. The number owner: A private contact method is incorrectly exposed, creating risks of harassment, scam attempts, or identity misuse.
2. The user asking the question: The user receives incorrect customer service guidance and may submit account information to the wrong party, creating secondary risk.
3. The service provider: Official customer service channels are confused with personal contacts, damaging brand trust and compliance accountability at the same time.

Therefore, the solution cannot stop at “making the model hallucinate less.” It should establish a complete protection chain covering data collection, retrieval augmentation, model output, application interaction, operational response, and compliance auditing.

2. Root Causes: Why the Model Reveals Real Phone Numbers

2.1 PII Mixed into Training and Retrieval Data

The factual sources for generative AI services usually include training corpora, public web pages, business documents, knowledge bases, and web snippets retrieved temporarily during retrieval-augmented generation. Public web pages may contain personal phone numbers, email addresses, instant messaging accounts, and enterprise contact information. Some forums, Q&A pages, recruitment pages, and second-hand marketplace pages even display private numbers as “contact information.” If personal information identification and filtering are missing during collection and cleaning, these snippets may enter the data space accessible to the model.

A more hidden problem is that personal numbers often do not appear as a “telephone number” field. Instead, they are embedded in natural language, for example: “If you have questions, contact Mr. X at…” Structured field filtering alone is unlikely to cover this. Text-level PII identification, source auditing, and retention policies are needed together.

2.2 Retrieval Augmentation Mistakes “Relevant Snippet” for “Authoritative Fact”

Retrieval-augmented generation can improve freshness, but it also introduces new risks: the retrieval system recalls snippets by semantic similarity, not by authority. A web page may be recalled simply because it contains terms such as “customer service,” “contact information,” or “payment issue.” If a personal number is mixed into that page, the model may organize it into the answer. Without source allowlists, authority scoring, freshness checks, and conflict resolution, retrieval augmentation can amplify “relevant text” into an “official answer.”

2.3 Entity Disambiguation Failure Combined with Hallucination

“Customer service for a payment app,” “a partner of a credit card company,” and “a real individual” are three different entities. The model needs to perform entity linking, relationship judgment, and contact ownership verification. Once entity disambiguation fails, the model may mistake “a person related to the business” for “the official customer service of the business.” When the model fills in missing facts, it can generate a number that is correctly formatted and delivered in a credible tone but attributed to the wrong entity. This type of error is harder for users to identify than obvious gibberish.

2.4 Output-Format Pressure and Product Interaction Flaws

User questions often contain strong format pressure: “Give me a customer service phone number” or “Just give me the contact information.” Under instruction-following pressure, the model tends to provide a specific string rather than answer “I cannot confirm that.” If the product interface then presents the model response as “smart customer service guidance,” users are even more likely to treat it as official information. The lack of policies such as “mask sensitive fields,” “prioritize official channels,” and “refuse when uncertain” is a direct product-level cause of such incidents.

2.5 Memory, Context, and Multi-Turn Conversations Expand Exposure

If the chat service has memory or long-context capabilities, contact information previously entered by the user, contacts in business documents, or numbers in historical conversations may be referenced again in later responses. Multi-turn conversations can also lead the model to gradually “rationalize” an incorrect entity: one turn guesses a number, and the next turn continues to generate customer service explanations around that number, eventually forming what appears to be complete but incorrect guidance.

2.6 Missing Output-Side PII Policy and Response Loop

Many systems perform content safety review on the input side but lack detection and handling for personal information on the output side. Even when a phone number is detected, the system may not distinguish among “the user’s own number provided voluntarily,” “a public enterprise switchboard,” “a third-party private number,” and “an instant messaging account.” Without classification, differentiated policies cannot be defined. Without logs, alerts, and a complaint response loop, low-frequency but high-impact leakage events cannot be discovered.

3. Engineering Safeguards: From “Less Hallucination” to “Auditable”

3.1 Data Governance: Control PII Sources First

Establish data source classification and PII lifecycle management:

• Collection stage: Perform text-level PII detection on web pages, documents, and knowledge bases to identify sensitive fields such as phone numbers, email addresses, instant messaging accounts, and identity document numbers.
• Ingestion stage: Mask or isolate private contact information by default and retain only official enterprise channels. For customer service information that must be retained, require auditable, traceable, and updatable sources.
• Update stage: Establish deletion and correction mechanisms to support data subject requests and removal of incorrect information.
• Audit stage: Record data sources, collection time, authorization basis, and masking policies to avoid “unexplainable data residue.”

3.2 Retrieval Augmentation: Authoritative Source Allowlists and Fact Verification

For highly sensitive facts such as contact information, semantic similarity alone should not be relied on. The following strategies can be used:

• Build an official channel knowledge table and allow only contact information from official documentation or verified knowledge bases to enter answers.
• Apply source authority scoring, freshness scoring, and entity consistency checks to recalled snippets.
• When multiple sources conflict, prioritize official sources. When confirmation is impossible, output “Please check through official channels” instead of generating a number.
• Set up dedicated intents for contact-information questions and route them to rule-based retrieval or structured queries rather than free-form generation.

3.3 Output-Side PII Detection and Policy-Based Handling

The output side should distinguish different sensitivity levels and take different actions. A pseudocode example is shown below:

function output_policy(text, intent, source_trust):
    hits = regex_match(text, [PHONE_PATTERN, EMAIL_PATTERN, IM_ACCOUNT_PATTERN])
    hits += ner_extract(text, ["PHONE", "EMAIL", "IM_ACCOUNT", "PERSON_NAME"])
    if hits is empty:
        return allow(text)
    if intent == "official_contact" and source_trust == "official":
        return allow_with_reference(text)
    if hits contains "PHONE" or "IM_ACCOUNT":
        return replace_with_message(
            "Unable to confirm the ownership of this contact information. Please use official channels."
        )
    return mask_sensitive_fields(text)

The key is not simple regex replacement. The policy should incorporate intent, source trustworthiness, sensitive field type, and whether the user provided the information voluntarily. Private numbers should not be output by default. Official switchboards must be accompanied by source and update time. Numbers that cannot be confirmed should trigger a refusal or guidance to official channels.

3.4 Model Layer: Uncertainty Expression and Refusal Capability

The model should be trained and constrained to explicitly express uncertainty when contact information cannot be confirmed from a trusted source and not to generate a specific number. This can be achieved through prompt strategies, supervised fine-tuning, preference optimization, and evaluation set construction. Evaluation sets should include adversarial samples such as “induced requests for numbers,” “multi-turn follow-up questions,” “fake official tone,” and “requests to provide contact information directly,” and should measure sensitive-field output rate, incorrect attribution rate, and refusal accuracy.

3.5 Application Layer: Official Channel Priority and Human Review

The product interface should clearly distinguish “official channels” from “model-generated content.” In high-risk scenarios involving accounts, payments, healthcare, or legal matters, contact information should come only from structured official knowledge tables. Numbers in model responses should be masked by default, and users should be directed to official channels after clicking. For customer service requests that cannot be confirmed, provide human review or official lookup entry points rather than allowing the model to improvise.

3.6 Operations Monitoring: Red Teaming, Log Alerts, and Complaint Loop

Establish a continuous red teaming mechanism to regularly probe system outputs with questions involving sensitive contact information. Logs should record intent, recalled sources, sensitive-field hits in outputs, and handling actions, but should not record unnecessary complete personal information. Monitoring alerts should focus on signals such as abnormal increases in sensitive-field output rate, concentrated refusals for a certain intent, and complaints mentioning “incorrect number attribution.” After receiving user reports, the system should support rapid source identification, removal of incorrect snippets, knowledge table updates, and regression testing.

3.7 Compliance and Audit: Minimal Necessity and Explainability

Personal information processing should follow principles such as minimal necessity, purpose limitation, notice and consent, and deletion and correction. For contact-information outputs, the system should be able to answer: where the information came from, why it was allowed to be output, what masking was applied, who reviewed it, and when it was updated. Audit records do not need to store sensitive original text, but they should store source identifiers, policy versions, and handling results so that issues are traceable, reproducible, and improvable.

4. Implementation Checklist

1. Has text-level PII identification and source auditing been performed on training, retrieval, and knowledge base data?
2. Are contact-information questions routed through dedicated intents and authoritative source queries rather than free-form generation?
3. Does the output side detect phone numbers, email addresses, and instant messaging accounts and apply differentiated policies?
4. Is there a default policy to refuse or direct users to official channels when confirmation is impossible?
5. Are human review and official channel redirects enabled for high-risk scenarios?
6. Has a closed loop been established for red team evaluation, log alerts, complaint response, and regression testing?
7. Can each sensitive output explain its source, policy, and update time?

5. Conclusion

Real phone numbers leaked by AI chat services are not a single model defect. They are the result of failures across data, retrieval, entity resolution, product policy, and operational loops. For developers, the most effective path is not to pursue a model that never makes mistakes, but to remove highly sensitive facts such as contact information from free-form generation and entrust them to auditable authoritative sources, enforceable output policies, and closed-loop operational mechanisms. Only when “refuse when uncertain, do not output sensitive fields by default, and prioritize official channels” becomes default system behavior can intelligent experiences improve while privacy boundaries are protected.