Most AI agents never train a model, so the real risks are a provider using your prompts, your own fine-tuning on raw records, and copies left in logs, traces, embeddings and plug-ins. Classify the data, use business or API terms that exclude training in a signed agreement, redact identifiers before the prompt, retrieve documents at question time instead of fine-tuning on them, control and redact your logs, and keep a written map of every system the text passes through. Then test with a planted fake record rather than trusting the settings page.
What does it mean for data to be in an AI agent's training set?
People use one phrase for three different worries, and the fixes are different for each.
- The model provider trains on your prompts. What you send to the model becomes part of a future model.
- You train on your own data. You fine-tune a model on customer records, tickets or contracts, and the model can later repeat them.
- Data lands somewhere you did not plan. Logs, traces, embeddings, analytics tools and plug-ins keep copies, and none of them is "training" but all of them are exposure.
Most agents never train a model at all. They send text to a model that is already trained, and fetch your documents when they need them. For those, the realistic risk is the first and third, and it is mostly solved by choices about vendors, settings and logging rather than by anything exotic.
Where does sensitive data leak from an AI agent?
| Place | What can happen | Control |
|---|---|---|
| Consumer AI tools | Staff paste customer data into a free chat app whose settings allow training | Provide an approved tool; block or policy-ban the rest |
| Model provider API | Prompts are retained for a period, or used for training under some terms | Business or API terms that exclude training, in a signed agreement; ask about zero retention |
| Fine-tuning data | The model memorises and later repeats names, numbers or contract terms | Do not fine-tune on raw records; redact first, or retrieve instead |
| Logs and tracing tools | Full prompts and answers stored in an observability or evaluation service | Redact before logging, set retention, check that vendor's terms too |
| Vector store and embeddings | A copy of your content sits in a database anyone with access can query | Access control per user, encryption, the same retention rules as the source |
| Plug-ins and third-party tools | Another vendor receives the text when the agent calls it | List every subprocessor in the data path and review each one |
How do you keep sensitive data out of an AI agent, step by step?
- 01
Classify the data before you design anything
Decide which fields are sensitive: personal data, health or payment information, credentials, contract terms. What the agent never needs, it should never receive.
- 02
Choose provider terms, not provider promises
Major model providers say that data sent through their business and API products is not used for training by default, but terms change and consumer products differ. Read the current terms, get a data processing agreement, and ask about retention and zero-retention options.
- 03
Redact before the prompt
Strip or replace names, account numbers and identifiers before text reaches the model, and put them back in your own code afterwards. The model rarely needs the real value to do the job.
- 04
Retrieve instead of fine-tuning
If the agent needs your documents, fetch the relevant passages at question time with access control, rather than baking them into model weights, where you cannot remove or restrict them.
- 05
Control what you log
Keep the logs you need to debug and audit, redact them, set a retention period and apply the same access rules as the source data.
- 06
Write down the data path
One diagram: where the text starts, every system it passes through, who runs each, and where it is stored. It is what your security reviewer and your customer will ask for.
The choice between retrieval and training is covered in fine-tuning vs RAG vs prompt engineering, and it is the biggest privacy decision most projects make.
Is it ever safe to fine-tune on company data?
Sometimes, with limits. Fine-tuning is reasonable for style, format and narrow classification tasks, where the training examples can be redacted or synthetic. It is a poor fit for facts that are private or change often.
What to check first
- Can you remove each person's data from the training set, and would you need to retrain to honour a deletion request?
- Who can use the fine-tuned model, and is it separate from models other customers use?
- Have you tested whether it repeats training examples when prompted?
- Would retrieval give the same quality without storing the data in the model?
How do you check that sensitive data is actually staying out?
Do not rely on the settings page. Test the data path you drew.
- Plant a unique fake record, such as a made-up customer with a distinctive name, then search your logs, traces, vector store and exports to see everywhere it appears.
- Ask each vendor in writing how long prompts are kept, who can see them and how deletion works.
- Try to extract it: ask the agent about the fake record from an account that should not see it.
- Review the list again whenever you add a tool, change a model or switch a provider.
Access leaks are a separate problem from training leaks; the controls for agents that read untrusted text are in AI agent security, and the EU side is in the EU AI Act checklist for chatbots.
How do you keep sensitive data out of an AI agent's training set?
Send the model only what it needs, redacted where possible. Use business or API terms that exclude training, in a signed agreement. Retrieve your documents at question time instead of fine-tuning on them, control and redact your logs, and keep a written map of every system the text passes through. Then test it with a planted record rather than trusting the settings.
If you want an agent built with that data path designed in from the start, our security page explains how we handle client data, and our AI agent development team builds them.
