ISO 27701:2025 PIMS & GDPR Privacy Compliance Consulting × AI attack defence

Personal data and AI attacks: stop leakage and reconstruction

Updated 2026-11-10

The short answer

AI adds three leak paths for personal data: staff entering it into external AI services, internal AI assistants revealing it to the wrong people through prompt injection or poor permissions, and the model itself being attacked by membership inference or model inversion. Countermeasures are an AI gateway that masks personal data, retrieval filtered by the user's original permissions, processor terms that forbid training on your data, and a DPIA with attack testing before go-live, all run continuously inside an ISO 27701 privacy information management system.

Companion page: service overview: ISO 27701:2025 PIMS & GDPR Privacy Compliance ConsultingWhat this service is, who it is for and what we deliver are on the service overview page.

Why does AI create a new attack surface for personal data?

GDPR Article 5(1)(f) requires appropriate security to ensure integrity and confidentiality of personal data, and Article 32 requires technical and organisational measures appropriate to the risk. Taiwan's Personal Data Protection Act Article 27 requires non-government agencies holding personal data files to adopt proper security measures. These duties do not change when the processing tool is AI, but AI opens three paths that did not exist before:

  • Input leakage: staff paste customer lists, medical records or CVs into an external generative AI; the data leaves company control and may be retained or used for training by the provider.
  • Retrieval leakage: an internal AI assistant connected to HR, customer service or CRM data reveals personal data to the wrong people if retrieval is not filtered by the user's original permissions or if it is steered by prompt injection.
  • Model leakage: a model trained or fine-tuned on personal data can be attacked by membership inference (was this person in the training data?) or model inversion. EDPB Opinion 28/2024 states that for a model to be considered anonymous, the likelihood of direct extraction and of obtaining personal data through queries must both be insignificant, and lists resistance to such attacks as an assessment element.

Which AI attacks expose personal data, and where do they come in?

AttackEntry pointPersonal data exposedReference
Staff input into external AIGenerative AI on personal accounts, browser extensionsCustomer, employee and patient dataOWASP LLM02 Sensitive Information Disclosure
Direct and indirect prompt injectionCustomer service bots, AI assistants that read mail or documentsAny personal data the system can retrieveNIST prompt injection; OWASP LLM01
Retrieval permission bypassVector databases, RAG knowledge basesHR or customer data the user is not entitled to seeOWASP LLM08 Vector and Embedding Weaknesses
Membership inference, model inversion, training data extractionModels trained or fine-tuned on personal data, inference APIsIndividuals in the training dataNIST privacy attacks; EDPB Opinion 28/2024
System prompt leakagePrompts containing customer data or internal rulesPersonal data and settings in the promptOWASP LLM07
AI-assisted social engineeringPhishing mail and cloned voices made with generative AIPersonal data reachable after account takeoverNIST misuse

Which controls apply, and what evidence should you keep?

The obligation column follows the official texts; the controls are our practical recommendations.

ObligationControlEvidence
GDPR Art. 25: data protection by design and by defaultAI gateway detects and masks personal data before sending; conversation logs not kept by defaultGateway rules, masking test records
GDPR Art. 32; PDPA Art. 27: security measuresRAG retrieval filtered by the user's original permissions; least-privilege data access for AI agents; abnormal query alertsPermission design, alert records
GDPR Art. 28: processorsProcessor terms with external AI services: no training on your data, retention limits, sub-processors, incident notificationContracts, supplier assessment
GDPR Art. 35: data protection impact assessmentDPIA before a new AI use goes live, including prompt injection and membership inference scenariosDPIA report
GDPR Art. 33; PDPA Art. 12: breach notificationAI-caused leaks included in incident classification and notificationIncident records
EDPB Opinion 28/2024: claims that a model is anonymousExtraction and inference attack tests and documentation behind any anonymity claimTest reports, assessment records

How does ISO 27701 turn AI privacy risk into a system that keeps running?

ISO/IEC 27701 is a privacy information management system (PIMS). It makes the controls above someone's ongoing responsibility rather than a one-off project:

  • Records of processing: register each AI use as a processing activity with purpose, categories of personal data, AI service provider, retention and cross-border transfers.
  • Risk assessment: merge the AI attack list on this page into the privacy risk assessment and reassess when a new AI use starts or the model changes.
  • Supplier management: bring AI service providers into processor assessment and contract management.
  • Data subject rights: when a customer asks for access or erasure, make sure AI conversation logs, vector databases and fine-tuning data are in scope.
  • Internal audit and management review: sample gateway logs and RAG permission settings to confirm the controls actually run.

Who does what across IT, legal, business units and AI suppliers, and where do Winners and Ahura fit?

  • Privacy lead / DPO: maintain records of processing, lead DPIAs, decide whether an incident must be notified.
  • IT and security: build the AI gateway, RAG permission filtering, logging and alerts.
  • Business units: explain data categories and purpose when proposing an AI use, and do not open external AI services on their own.
  • Legal and procurement: sign processor terms with AI providers confirming no training, deletion on request and audit rights.

Winners Consulting determines GDPR and PDPA applicability, brings AI uses into the ISO 27701 system, and designs the DPIA method and controls. Ahura (POC version) is fully on-premises, keeping assessment and sign-off records in the company's own environment with an eIDAS-compatible evidence chain.

Already running ISO 27701 or a GDPR programme: where do you start with AI?

  1. Find shadow AI: list the AI services and extensions staff actually use, and block services that train on submitted data.
  2. Extend records of processing: register one entry per AI use with data categories and provider.
  3. Run a DPIA on the highest-risk use: usually the internal AI assistant connected to customer or HR data, with prompt injection and retrieval permission bypass in the tests.

With these three done, the next internal audit will find AI inside the system.

AI attack defence for other services

Official sources

Ask a consultant on LINE

No forms: the button opens LINE with the page you are reading filled in. Send it and a Winners Consulting consultant will reply. The first diagnosis is free.

Winners Consulting track record

Winners Consulting Services is a Taiwan RegTech platform provider and enterprise risk management consultancy.

Trusted long-term by 20+ companies, we work as an industry-focused, long-term partner to our clients.

Named client engagements, published with client permission:

See case studies

Personal data and AI attacks: stop leakage and reconstruction