Why does AI create a new attack surface for personal data?
GDPR Article 5(1)(f) requires appropriate security to ensure integrity and confidentiality of personal data, and Article 32 requires technical and organisational measures appropriate to the risk. Taiwan's Personal Data Protection Act Article 27 requires non-government agencies holding personal data files to adopt proper security measures. These duties do not change when the processing tool is AI, but AI opens three paths that did not exist before:
- Input leakage: staff paste customer lists, medical records or CVs into an external generative AI; the data leaves company control and may be retained or used for training by the provider.
- Retrieval leakage: an internal AI assistant connected to HR, customer service or CRM data reveals personal data to the wrong people if retrieval is not filtered by the user's original permissions or if it is steered by prompt injection.
- Model leakage: a model trained or fine-tuned on personal data can be attacked by membership inference (was this person in the training data?) or model inversion. EDPB Opinion 28/2024 states that for a model to be considered anonymous, the likelihood of direct extraction and of obtaining personal data through queries must both be insignificant, and lists resistance to such attacks as an assessment element.
Which AI attacks expose personal data, and where do they come in?
| Attack | Entry point | Personal data exposed | Reference |
|---|---|---|---|
| Staff input into external AI | Generative AI on personal accounts, browser extensions | Customer, employee and patient data | OWASP LLM02 Sensitive Information Disclosure |
| Direct and indirect prompt injection | Customer service bots, AI assistants that read mail or documents | Any personal data the system can retrieve | NIST prompt injection; OWASP LLM01 |
| Retrieval permission bypass | Vector databases, RAG knowledge bases | HR or customer data the user is not entitled to see | OWASP LLM08 Vector and Embedding Weaknesses |
| Membership inference, model inversion, training data extraction | Models trained or fine-tuned on personal data, inference APIs | Individuals in the training data | NIST privacy attacks; EDPB Opinion 28/2024 |
| System prompt leakage | Prompts containing customer data or internal rules | Personal data and settings in the prompt | OWASP LLM07 |
| AI-assisted social engineering | Phishing mail and cloned voices made with generative AI | Personal data reachable after account takeover | NIST misuse |
Which controls apply, and what evidence should you keep?
The obligation column follows the official texts; the controls are our practical recommendations.
| Obligation | Control | Evidence |
|---|---|---|
| GDPR Art. 25: data protection by design and by default | AI gateway detects and masks personal data before sending; conversation logs not kept by default | Gateway rules, masking test records |
| GDPR Art. 32; PDPA Art. 27: security measures | RAG retrieval filtered by the user's original permissions; least-privilege data access for AI agents; abnormal query alerts | Permission design, alert records |
| GDPR Art. 28: processors | Processor terms with external AI services: no training on your data, retention limits, sub-processors, incident notification | Contracts, supplier assessment |
| GDPR Art. 35: data protection impact assessment | DPIA before a new AI use goes live, including prompt injection and membership inference scenarios | DPIA report |
| GDPR Art. 33; PDPA Art. 12: breach notification | AI-caused leaks included in incident classification and notification | Incident records |
| EDPB Opinion 28/2024: claims that a model is anonymous | Extraction and inference attack tests and documentation behind any anonymity claim | Test reports, assessment records |
How does ISO 27701 turn AI privacy risk into a system that keeps running?
ISO/IEC 27701 is a privacy information management system (PIMS). It makes the controls above someone's ongoing responsibility rather than a one-off project:
- Records of processing: register each AI use as a processing activity with purpose, categories of personal data, AI service provider, retention and cross-border transfers.
- Risk assessment: merge the AI attack list on this page into the privacy risk assessment and reassess when a new AI use starts or the model changes.
- Supplier management: bring AI service providers into processor assessment and contract management.
- Data subject rights: when a customer asks for access or erasure, make sure AI conversation logs, vector databases and fine-tuning data are in scope.
- Internal audit and management review: sample gateway logs and RAG permission settings to confirm the controls actually run.
Who does what across IT, legal, business units and AI suppliers, and where do Winners and Ahura fit?
- Privacy lead / DPO: maintain records of processing, lead DPIAs, decide whether an incident must be notified.
- IT and security: build the AI gateway, RAG permission filtering, logging and alerts.
- Business units: explain data categories and purpose when proposing an AI use, and do not open external AI services on their own.
- Legal and procurement: sign processor terms with AI providers confirming no training, deletion on request and audit rights.
Winners Consulting determines GDPR and PDPA applicability, brings AI uses into the ISO 27701 system, and designs the DPIA method and controls. Ahura (POC version) is fully on-premises, keeping assessment and sign-off records in the company's own environment with an eIDAS-compatible evidence chain.
Already running ISO 27701 or a GDPR programme: where do you start with AI?
- Find shadow AI: list the AI services and extensions staff actually use, and block services that train on submitted data.
- Extend records of processing: register one entry per AI use with data categories and provider.
- Run a DPIA on the highest-risk use: usually the internal AI assistant connected to customer or HR data, with prompt injection and retrieval permission bypass in the tests.
With these three done, the next internal audit will find AI inside the system.
AI attack defence for other services
Official sources
- Regulation (EU) 2016/679 (GDPR) — EUR-Lex
- EDPB Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models
- Personal Data Protection Act — Laws & Regulations Database of the R.O.C. (Taiwan) (in Chinese)
- ISO/IEC 27701 — Privacy information management — ISO
- NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- OWASP Top 10 for LLM Applications 2025 — OWASP GenAI Security Project
