AI agents can work in regulated industries like healthcare, but mainly on administrative and support tasks: scheduling, benefits questions, intake summaries, claims paperwork and coding suggestions that a person approves. Clinical decisions such as triage, diagnosis and dosing can make the software a regulated medical device. Treat the data path as the main design problem: agreements with every vendor that sees protected health information, minimum data, redaction, audit logs and access control. Start with a low-risk task, involve compliance early, run in shadow mode and measure before widening.
Can AI agents work in regulated industries like healthcare?
Yes, but the work you can safely give them is narrower than the work they can technically do. In healthcare, an agent that books appointments, summarises intake forms or drafts a prior-authorisation packet for a person to check is an administrative tool. An agent that tells a patient what their symptoms mean or what dose to take is making clinical decisions, and is treated very differently.
Two questions decide most of it: what kind of data does the agent touch, and what happens if it is wrong? This guide uses healthcare as the main example, with notes on finance and legal at the end.
What rules apply to an AI agent in healthcare?
| Rule | When it applies | What it means for the agent |
|---|---|---|
| HIPAA (US) | The agent handles protected health information for a covered entity or its business associate | A business associate agreement with every vendor in the data path, minimum-necessary access, audit logs, safeguards |
| GDPR (EU, UK) | The agent processes health data about people in those regions | Health data is a special category: a lawful basis, usually a data protection impact assessment, strict limits on use |
| Medical device rules (FDA, EU MDR) | The software is intended to diagnose, treat, monitor or drive clinical decisions | It may need to be cleared or certified as a medical device. Intended use, not technology, decides this |
| EU AI Act | The system is high-risk, for example a medical device or a safety component of one | Risk management, data governance, human oversight, logging and documentation |
This is a summary, not legal advice; the rules differ by country, state and use, and your compliance lead or counsel should confirm which apply. For the EU side of the picture, see the EU AI Act checklist for chatbots and our compliance page.
Which healthcare tasks suit an AI agent, and which do not?
| Task | Risk level | How to approach it |
|---|---|---|
| Appointment booking, reminders, rescheduling | Lower | Agent acts within clear rules; escalate anything unusual |
| Insurance and benefits questions | Lower | Answer from plan documents with citations; refuse when unsure |
| Intake form summaries and call notes | Medium | Draft for clinician review; the clinician signs off |
| Prior authorisation and claims paperwork | Medium | Assemble the packet; a person submits it |
| Medical coding suggestions | Medium | Suggest codes with the evidence; a certified coder approves |
| Triage, symptom assessment, diagnosis, dosing | High | Avoid unless you are building a regulated medical device with clinical oversight |
Most healthcare organisations get the most value, soonest, from the first four rows. They are also where the regulatory burden is lowest, because the agent supports staff rather than replacing a clinical judgement.
How do you handle patient data in an AI agent?
- Sign a business associate agreement with every vendor that can see protected health information, including the model provider, the cloud host and any logging service. Several large cloud and model providers will sign one for specific eligible services, so confirm it for the exact service you use, not the company in general.
- Send the minimum. If the agent can do the job with a first name and an appointment time, do not send the whole record.
- De-identify where you can, and keep the key that maps it back in your own system.
- Keep data in the region your obligations require, and know where each subprocessor stores it.
- Log who asked what and what the agent returned, redact the logs, and set a retention period.
- Give each user access only to the records they are entitled to see, enforced in retrieval, not in the prompt.
The same principles apply to any sensitive data; our guide on keeping sensitive data out of an AI agent's training set goes through the data path in more detail.
Where must a human stay in the loop?
Anywhere a wrong answer could harm a patient or breach the rules. In practice that means three things.
The agent knows what it must not answer
It refuses to give medical advice, tells people to contact a clinician, and shows emergency instructions when a message suggests an urgent problem. Test those refusals with real phrasings before launch.
A person approves anything consequential
Clinical notes, claims, and messages that go to patients with new clinical content are drafted by the agent and approved by a named person. The pattern is in escalating an AI agent to a human.
The agent's limits are tested, not assumed
Build an evaluation set from real, de-identified questions, including awkward ones, and re-run it after every change. Our go-live checklist shows what we check before an agent talks to real users.
What about finance, legal and other regulated sectors?
The same two questions apply. The details differ:
- Finance: decisions about credit, advice or trading are regulated, and you generally need to explain them. Use agents for research, document handling and drafting, and keep the decision with a licensed person.
- Legal: confidentiality and privilege matter most, and citations need to be checkable. See RAG for law firms.
- Insurance: coverage decisions stay with licensed staff. See RAG chatbots for insurance agencies.
How should a regulated organisation start with AI agents?
- 01
Start with a non-clinical, low-risk task
Scheduling, benefits questions or document summaries. They prove the approach and build trust with staff and compliance.
- 02
Involve compliance on day one
Agree the data path, vendors and agreements with your privacy and compliance leads before building, not at launch.
- 03
Run in shadow mode first
Let the agent draft while people do the work as normal, and compare. Move to draft-and-approve only when the numbers justify it.
- 04
Keep an audit trail
Record inputs, sources, outputs and approvals so you can answer a regulator, an auditor or a patient.
Can AI agents handle regulated industries like healthcare?
Yes, for administrative and support work with a person approving anything consequential. Avoid clinical decisions unless you are building a regulated device. Treat the data path as the main design problem: agreements with every vendor, minimum data, redaction, audit logs and access control. Start small, involve compliance early and measure before widening.
A note on where we stand: we have not signed a business associate agreement to date, and we say so on our security page. For work involving protected health information we would raise that at scoping and tell you honestly whether we can meet your requirements. For non-PHI administrative work, our AI agent development team can start straight away.
