Explainerby Ahsan Ahmad

Can AI agents work in regulated industries like healthcare? What is allowed, what needs a BAA, and where the line sits

Summary

AI agents can work in regulated industries like healthcare, but mainly on administrative and support tasks: scheduling, benefits questions, intake summaries, claims paperwork and coding suggestions that a person approves. Clinical decisions such as triage, diagnosis and dosing can make the software a regulated medical device. Treat the data path as the main design problem: agreements with every vendor that sees protected health information, minimum data, redaction, audit logs and access control. Start with a low-risk task, involve compliance early, run in shadow mode and measure before widening.

Can AI agents work in regulated industries like healthcare?

Yes, but the work you can safely give them is narrower than the work they can technically do. In healthcare, an agent that books appointments, summarises intake forms or drafts a prior-authorisation packet for a person to check is an administrative tool. An agent that tells a patient what their symptoms mean or what dose to take is making clinical decisions, and is treated very differently.

Two questions decide most of it: what kind of data does the agent touch, and what happens if it is wrong? This guide uses healthcare as the main example, with notes on finance and legal at the end.

What rules apply to an AI agent in healthcare?

Rules that commonly apply to AI agents in healthcare
RuleWhen it appliesWhat it means for the agent
HIPAA (US)The agent handles protected health information for a covered entity or its business associateA business associate agreement with every vendor in the data path, minimum-necessary access, audit logs, safeguards
GDPR (EU, UK)The agent processes health data about people in those regionsHealth data is a special category: a lawful basis, usually a data protection impact assessment, strict limits on use
Medical device rules (FDA, EU MDR)The software is intended to diagnose, treat, monitor or drive clinical decisionsIt may need to be cleared or certified as a medical device. Intended use, not technology, decides this
EU AI ActThe system is high-risk, for example a medical device or a safety component of oneRisk management, data governance, human oversight, logging and documentation

This is a summary, not legal advice; the rules differ by country, state and use, and your compliance lead or counsel should confirm which apply. For the EU side of the picture, see the EU AI Act checklist for chatbots and our compliance page.

Which healthcare tasks suit an AI agent, and which do not?

Healthcare tasks by suitability for an AI agent
TaskRisk levelHow to approach it
Appointment booking, reminders, reschedulingLowerAgent acts within clear rules; escalate anything unusual
Insurance and benefits questionsLowerAnswer from plan documents with citations; refuse when unsure
Intake form summaries and call notesMediumDraft for clinician review; the clinician signs off
Prior authorisation and claims paperworkMediumAssemble the packet; a person submits it
Medical coding suggestionsMediumSuggest codes with the evidence; a certified coder approves
Triage, symptom assessment, diagnosis, dosingHighAvoid unless you are building a regulated medical device with clinical oversight

Most healthcare organisations get the most value, soonest, from the first four rows. They are also where the regulatory burden is lowest, because the agent supports staff rather than replacing a clinical judgement.

How do you handle patient data in an AI agent?

  • Sign a business associate agreement with every vendor that can see protected health information, including the model provider, the cloud host and any logging service. Several large cloud and model providers will sign one for specific eligible services, so confirm it for the exact service you use, not the company in general.
  • Send the minimum. If the agent can do the job with a first name and an appointment time, do not send the whole record.
  • De-identify where you can, and keep the key that maps it back in your own system.
  • Keep data in the region your obligations require, and know where each subprocessor stores it.
  • Log who asked what and what the agent returned, redact the logs, and set a retention period.
  • Give each user access only to the records they are entitled to see, enforced in retrieval, not in the prompt.

The same principles apply to any sensitive data; our guide on keeping sensitive data out of an AI agent's training set goes through the data path in more detail.

Where must a human stay in the loop?

Anywhere a wrong answer could harm a patient or breach the rules. In practice that means three things.

The agent knows what it must not answer

It refuses to give medical advice, tells people to contact a clinician, and shows emergency instructions when a message suggests an urgent problem. Test those refusals with real phrasings before launch.

A person approves anything consequential

Clinical notes, claims, and messages that go to patients with new clinical content are drafted by the agent and approved by a named person. The pattern is in escalating an AI agent to a human.

The agent's limits are tested, not assumed

Build an evaluation set from real, de-identified questions, including awkward ones, and re-run it after every change. Our go-live checklist shows what we check before an agent talks to real users.

What about finance, legal and other regulated sectors?

The same two questions apply. The details differ:

  • Finance: decisions about credit, advice or trading are regulated, and you generally need to explain them. Use agents for research, document handling and drafting, and keep the decision with a licensed person.
  • Legal: confidentiality and privilege matter most, and citations need to be checkable. See RAG for law firms.
  • Insurance: coverage decisions stay with licensed staff. See RAG chatbots for insurance agencies.

How should a regulated organisation start with AI agents?

  1. 01

    Start with a non-clinical, low-risk task

    Scheduling, benefits questions or document summaries. They prove the approach and build trust with staff and compliance.

  2. 02

    Involve compliance on day one

    Agree the data path, vendors and agreements with your privacy and compliance leads before building, not at launch.

  3. 03

    Run in shadow mode first

    Let the agent draft while people do the work as normal, and compare. Move to draft-and-approve only when the numbers justify it.

  4. 04

    Keep an audit trail

    Record inputs, sources, outputs and approvals so you can answer a regulator, an auditor or a patient.

Can AI agents handle regulated industries like healthcare?

Yes, for administrative and support work with a person approving anything consequential. Avoid clinical decisions unless you are building a regulated device. Treat the data path as the main design problem: agreements with every vendor, minimum data, redaction, audit logs and access control. Start small, involve compliance early and measure before widening.

A note on where we stand: we have not signed a business associate agreement to date, and we say so on our security page. For work involving protected health information we would raise that at scoping and tell you honestly whether we can meet your requirements. For non-PHI administrative work, our AI agent development team can start straight away.

Tell us what you are building

A free strategy session: what is buildable, what it costs, what we would not attempt.