Causes and Protection Strategies for Account Information Leaks in Generative AI Services

A personal data protection authority in South Korea recently disclosed that an operator of a generative AI service had voluntarily reported a security incident on October 31. Account-related information associated with approximately 100 South Korean companies may have been affected. Based on the preliminary information released so far, an external security party may have exploited a vulnerability in third-party software to enter the operator’s internal systems and access data related to some enterprise accounts.

The relevant regulators are still conducting a preliminary review and have not yet formally launched a full investigation. Because the incident involves an overseas company, the next steps may include cross-border document requests, evidence verification, confirmation of the scope of impact, and damage assessment. The final conclusions will therefore depend on the results of the formal investigation.

The potentially exposed data appears to include enterprise account identifiers, names, email addresses, and other business contact information. So far, there is no evidence that prompts, generated content, or payment information was accessed.

The significance of this incident extends beyond determining whether a particular service experienced a data breach. It illustrates several common security challenges facing generative AI services:

• Third-party software supply chain risk;
• Excessive permissions in internal systems;
• Centralized storage of enterprise account information;
• The complexity of responding to cross-border data incidents; and
• The effectiveness of security monitoring and customer notification mechanisms.

1. How to Understand the Incident

Based on the information disclosed so far, the incident can be divided into four broad stages.

First, an external security actor appears to have exploited a vulnerability in a commercial third-party software component to gain an entry point into the service operator’s internal environment. The key issue may not have been a defect in the AI model itself. Instead, the vulnerability may have existed in a backend management application, operations and maintenance system, identity component, or data-processing platform that supported the service.

Second, after entering the internal environment, the attacker may have accessed data associated with enterprise accounts. Account information is often distributed across user management systems, customer relationship management systems, ticketing platforms, identity systems, and notification services. If these systems trust one another too broadly, an attacker may obtain valuable identity data without compromising the core model service.

Third, after detecting abnormal activity, the operator isolated the affected systems and worked with the relevant supplier to remediate the vulnerability. Rapid isolation, suspension of high-risk interfaces, credential rotation, and preservation of logs are important measures for limiting ongoing impact.

Fourth, the operator notified regulators and some customers. Regulators then assessed whether the incident met the conditions for a formal investigation and began working to confirm the affected parties, data categories, access period, and potential losses.

It is important to distinguish exposure of account information from disclosure of model conversations. Security incidents must be assessed according to the type of data involved and the available evidence of access. The fact that accounts were affected does not by itself prove that business content, prompts, or payment data were stolen.

2. Why Third-Party Software Can Become a High-Risk Entry Point

Modern enterprises rarely depend entirely on systems developed in-house. Identity management, log analysis, ticketing, remote operations, customer notifications, and data synchronization commonly rely on external software and services. These tools improve development and operational efficiency, but they also expand the attack surface.

Third-party software risk generally comes from several factors.

2.1 Patch delays create an exposure window

After a supplier releases a security patch, an enterprise may still need to test it, assess compatibility, and schedule deployment. During this period, attackers may already be exploiting the vulnerability. For internet-facing management systems, delayed patching can significantly increase the time available for exploitation.

2.2 Software often receives more access than it actually needs

Some operations components require access to multiple databases, logging systems, and internal interfaces to support automation. If permissions are not based on the principle of least privilege, compromising one component may allow an attacker to move laterally across several business systems.

2.3 Dependencies are difficult to inventory completely

Enterprises use not only software purchased directly from suppliers, but also related components, plugins, scripts, and service interfaces. A vulnerability in a low-profile extension can still affect a critical business environment.

2.4 Security responsibilities can be unclear

The supplier is responsible for fixing defects in the software, while the enterprise is responsible for deployment, configuration, and monitoring. If the parties have not clearly defined responsibilities for alerts, notifications, patching, and forensic support, information may be delayed when an incident occurs.

Third-party risk management should therefore extend beyond a security questionnaire during procurement. It should cover onboarding, deployment, upgrades, monitoring, and retirement.

3. Why Enterprise Account Information Requires Strong Protection

Names, email addresses, and account identifiers may appear less sensitive than payment data, but they can still be highly valuable to attackers.

First, account information can help attackers create more convincing phishing messages. If an attacker knows a company’s name, an employee’s name, and a business email address, they can forge notifications, imitate password reset messages, or conduct more targeted social engineering.

Second, account identifiers can be used to correlate records across different systems. When multiple systems use the same email address, username, or internal identifier, attackers may combine data to reconstruct organizational structures, employee relationships, and business processes.

Enterprise accounts may also be associated with access permissions, membership relationships, and administrative activity records. If exposed account information is not followed by credential rotation or stronger authentication, the risk may expand from privacy exposure to unauthorized access.

Organizations should therefore avoid treating “non-core data” as data that does not require protection. Data protection should consider identifiability, linkability, exploitability, and potential effects on business continuity.

4. Key Protection Measures for Generative AI Services

4.1 Establish a clear data classification system

Organizations should at least distinguish among public data, internal data, sensitive personal information, business confidential information, and regulated data. Each category should have appropriate requirements for storage, transmission, access, and auditing.

For account information, organizations should also define sensitivity at the field level. A name or business email address may be ordinary identity data on its own, but its combination with an organizational role, access permission, or customer number can significantly increase its sensitivity.

4.2 Apply least privilege and layered isolation

Internal management systems, customer account systems, model invocation systems, and payment systems should not be trusted by default or allowed to communicate freely. Network segmentation, identity separation, fine-grained authorization, and short-lived credentials can reduce the risk of lateral movement.

Permission design should follow these principles:

• Deny access by default and grant it only when required;
• Separate administrator privileges from routine operational privileges;
• Limit service accounts to the necessary data tables and interfaces;
• Require additional authentication or approval for high-risk operations;
• Automatically revoke unused accounts and keys;
• Record the operator, time, source, and result of access to important data.

4.3 Strengthen third-party software lifecycle management

Organizations should maintain a software asset inventory that records the software name, version, deployment location, supplier, owner, access permissions, and data scope. For internet-facing systems, the organization should also define vulnerability severity levels, remediation deadlines, and temporary mitigation measures.

When an upgrade cannot be completed immediately, organizations can tighten access controls, disable unnecessary functions, restrict source addresses, add application-layer protections, and increase anomaly monitoring. These compensating controls, however, should not replace final patch deployment.

4.4 Improve logging and anomaly detection

Security logs should cover authentication, permission changes, data queries, bulk exports, API calls, configuration changes, and abnormal network access. Logging should not merely record events. Logs should also support centralized search, correlation, analysis, and alerting.

For account information systems, the following activities deserve particular attention:

• Large numbers of different enterprise accounts queried within a short period;
• Access from unusual geographic regions or network sources;
• Bulk exports performed at night or during holidays;
• Service accounts suddenly accessing data they have never previously accessed;
• Large-scale reads immediately following a privilege escalation;
• Multiple failed authentication attempts followed by a successful login;
• Deleted or altered logs, or breaks in the event timeline.

4.5 Minimize the processing of sensitive fields

If a business process only needs to verify whether a user exists, it should not return the user’s full name and email address. If an operator only needs to locate an account, the interface can use masked values, such as displaying only part of an email address or using an internal hashed identifier.

Data minimization reduces the potential impact of an incident. Even if an attacker obtains a database, the data may not directly reveal complete and usable identity information.

5. Incident Response When a Similar Event Is Detected

Step 1: Establish the facts

Organizations should distinguish among the following conditions:

• A vulnerability exists in a system;
• Abnormal access has been detected;
• Data has been confirmed as accessed; and
• Data has been confirmed as downloaded, copied, or disclosed.

This distinction helps avoid premature conclusions while preventing unclear communications.

Step 2: Contain the incident immediately

Possible actions include isolating affected hosts, disabling suspicious accounts, revoking access tokens, rotating keys, shutting down high-risk interfaces, and restricting external access.

Isolation should be performed carefully. Systems and logs should not be deleted or casually cleaned up, because doing so may compromise later forensic analysis.

Step 3: Preserve evidence

Organizations should retain authentication logs, network traffic records, database access logs, configuration change records, vulnerability scan results, and communications with suppliers. Evidence should preferably use a consistent time reference, and modification permissions should be restricted.

Step 4: Assess the impact

The assessment should answer the following questions:

• Which systems were affected?
• When did the attacker obtain the ability to access them?
• Which data was actually accessed?
• Was the data downloaded, copied, or distributed further?
• Which customers, employees, or partners may be affected?
• Does the incident involve regulated personal information?
• Is there a risk that credentials were misused?

Step 5: Provide legally required notifications

Organizations should notify regulators, customers, and relevant internal personnel according to applicable laws and contractual obligations. Notifications should be accurate, measured, and supported by evidence.

A notification should explain the confirmed facts, potential impact, actions already taken, and steps that users should follow. Organizations should avoid statements such as “completely secure” or “there is absolutely no risk” when such claims cannot be verified.

Step 6: Review and improve

After the incident, the organization should determine whether the root cause involved vulnerability management, permission configuration, missing monitoring, inadequate supplier oversight, or immature response procedures. Improvement actions should be assigned to specific owners and tracked through deadlines and acceptance criteria.

6. Practical Checklist for Development and Operations Teams

Development and operations teams can include the following items in their routine reviews:

1. Is there a complete inventory of software and dependencies?
2. Can the organization quickly identify which systems are affected by a particular component?
3. Do all service accounts use least-privilege access?
4. Are keys, tokens, and administrator credentials rotated regularly?
5. Are limits applied to bulk queries and bulk exports?
6. Are abnormal logins and privilege escalations monitored and reported?
7. Can systems be isolated quickly without compromising evidence?
8. Are suppliers assessed regularly from a security perspective?
9. Is customer account information masked and stored according to data sensitivity?
10. Have notification, forensic, and recovery procedures been validated through exercises?

If an organization uses common cloud capabilities such as object storage, message queues, container platforms, logging services, or monitoring and alerting systems, it should also review public access settings, service account permissions, network boundaries, key management, log retention periods, and cross-environment access controls.

Cloud services do not automatically eliminate security responsibilities. Configuration errors and uncontrolled permissions remain common sources of risk.

Conclusion

This account information incident demonstrates that the security boundary of a generative AI service is not limited to its models and APIs. It also includes identity systems, backend management platforms, third-party software, the broader supply chain, and operational processes.

Even though there is currently no evidence that prompts, generated content, or payment information were affected, exposure of account-related data may still enable phishing, identity impersonation, and subsequent data correlation.

For enterprises, the most effective protection is not simply adding another security product. It is building a complete cycle that covers asset inventory, data classification, least-privilege access, supplier management, continuous monitoring, and incident response.

Organizations can detect similar incidents earlier, reduce their impact, and improve recovery capability only when they implement clear technical and procedural controls for who can access data, what they accessed, when they accessed it, why the access occurred, and how abnormal activity will be handled.