Which AI use cases in healthcare are high risk under the EU AI Act?

Article 6(1) and Annex I: software that analyzes a medical image to detect or prioritize a disease finding is itself, or is a safety component of, a device in scope of the EU Medical Device Regulation, and typically needs a notified body conformity assessment as software as a medical device (the FDA's AI Enabled Medical Device List shows US market authorization for devices in this category, listing authorized stroke triage devices from Viz.ai and Aidoc's BriefCase triage devices), which makes it high risk under the EU AI Act regardless of Annex III. The radiologist's own diagnostic read stays a human decision; the AI narrows and reorders the queue. Annex I high risk classification under Article 6(1) applies from 2 August 2028 (Article 113(c)); until then, Article 4 (AI literacy obligations) and Article 5 (prohibited practices), which bind the hospital as a deployer, already apply.

Article 6(1)(a) and (b) and Annex I: a system is high risk when it is a safety component of, or itself is, a product covered by EU harmonization legislation listed in Annex I, and that product is required to undergo a third party conformity assessment under that legislation. Software that predicts sepsis or deterioration to guide treatment typically qualifies as a Class IIa or higher medical device under the EU Medical Device Regulation, Rule 11, which brings it into the high risk tier, though MDR Article 5(5) provides an in house exemption that can apply to a tool a health institution builds and uses only within its own organization.

Software that creates diagnostic report text from a medical image is likely to be medical device software under the EU Medical Device Regulation. A product that needs third party conformity assessment under that regulation is high risk through Article 6(1) and Annex I of the EU AI Act, whether or not it appears in Annex III. That Annex I route applies from 2 August 2028 (Article 113(c)). Until then Article 4 (AI literacy) and Article 5 (prohibited practices) already bind the hospital as a deployer. The radiologist's signed interpretation stays a human decision.

Annex III point 5(d): AI systems intended to evaluate and classify emergency calls or to dispatch or set priority for emergency first response services (police, fire, medical aid) are high risk. Pure transcription that performs a narrow procedural or preparatory task may fall outside it under the Article 6(3) exceptions, but alerts that influence triage are in scope. An AI agent that speaks with callers directly, for example on a non emergency line, must also tell them they are interacting with AI (Article 50).

A scribe that only transcribes and summarizes for a clinician to review is not listed in Annex III and is usually minimal risk, although the provider of a system that generates text can still owe the Article 50(2) duty to mark output as AI generated, unless an exception such as an assistive function for standard editing applies. If the product qualifies as medical device software under the EU Medical Device Regulation and needs a notified body assessment, for example because it suggests diagnoses or treatment, it becomes high risk under Article 6(1) and Annex I. Health data in audio and notes falls under GDPR Article 9 in every case.

Booking, rescheduling and reminders carry transparency duties: patients must be told they are dealing with AI (Article 50(1)). It becomes high risk if a public authority, or a provider acting on its behalf, uses it to evaluate eligibility for healthcare services (Annex III point 5(a)), or if it acts as an emergency healthcare patient triage system (Annex III point 5(d)). Clinical triage may also make it a medical device, which is high risk under Article 6(1) when the device needs a notified body assessment. Keep the agent to scheduling and use risk scores only to offer support.

The tier depends on what the system is scoped to do. A design limited to occupancy and discharge forecasting and to sequencing bed assignments for patients already admitted is operational decision support for hospital logistics, outside Annex III. Annex III point 5(d) covers AI used "to dispatch, or to establish priority in the dispatching of, emergency first response services", including medical aid and emergency healthcare patient triage systems. On a plain reading, that point can apply when a system dispatches, or sets the priority of dispatching, ambulance or critical care transport itself (work similar to what the Johns Hopkins center's Lifeline transport staff do for helicopter and ambulance transfers), or when it assesses the clinical urgency of an emergency patient, that is, triage. Sequencing which already admitted ED patient gets the next ward bed, and deciding whether to accept an inter hospital transfer request on capacity grounds, are not listed activities under 5(d) as written; whether either counts as dispatching or triage in a given deployment is a case by case legal question, not a settled fact, and should be assessed with counsel before relying on this tier. For public hospitals, Annex III point 5(a) (access to essential public services, including healthcare) can also be relevant. Scoping the system to bed sequencing and transfer acceptance only, and keeping every ambulance dispatch and ED triage decision with clinical staff outside the AI's recommendation, is what keeps a deployment in the lower tier.

Prescreening for research that staff verify is not listed in Annex III and is usually minimal risk. The Article 2(6) exclusion covers only systems developed and put into service for the sole purpose of scientific research and development, so an operational recruitment tool used across a health system usually falls inside the Act. If the software recommends trials to a clinician as a treatment option for an individual patient, it may qualify as medical device software under the Medical Device Regulation; where that needs a notified body assessment, it is high risk under Article 6(1). Processing health records for research falls under GDPR Article 9 and national research rules.

A voice agent that talks to patients must tell them they are interacting with an AI system (Article 50). If the software is itself a medical device, or a safety component of one, that needs a third party conformity assessment under the EU Medical Device Regulation, it is high risk under Article 6(1) and Annex I. A script that only collects answers for a clinician to read is closer to limited risk, while a system that classifies symptoms and decides who is reviewed can move into device territory. Dora is described as a UKCA medical device and LOLA as a class I medical device under the earlier EU directive.

The patient never interacts with the AI system directly, only with the clinician who reviews, edits and sends the reply, so Article 50(1), which covers systems intended to interact directly with natural persons, does not apply here; UC San Diego Health's patient facing disclosure is a voluntary practice, not an Article 50(1) duty. A copilot that only drafts for a clinician to review is not listed in Annex III and is usually minimal or limited risk, although the provider can still owe the Article 50(2) duty to mark the drafted text as AI generated, unless the exception for an assistive function with standard editing where the deploying clinician retains final responsibility applies. It would move toward high risk under Article 6(1) and Annex I only if the software qualifies as a medical device under the EU Medical Device Regulation and needs a notified body assessment, for example because it suggests a diagnosis or treatment rather than only drafting correspondence.

People must be told they are interacting with an AI system, and synthetic voice or text must be identifiable as such (Article 50). Reminding people of existing bookings and disclosure alone are limited risk. A missed appointment score used by or for a public authority to grant, reduce, revoke or reclaim access to healthcare or other essential public assistance and services, for example deciding who is offered funded transport, can fall within Annex III point 5(a), and profiling of natural persons within Annex III rules out the Article 6(3) exemption. Using the score only to decide who gets extra reminders or support does not by itself place it outside Annex III when that support is itself the assistance being granted.

Education from approved content is a limited risk system with transparency duties (Article 50), and an avatar that copies a real clinician's face and voice is synthetic content that must be disclosed as artificially generated. If the avatar assessed symptoms, triaged patients or advised on treatment it could be medical device software under the Medical Device Regulation. That makes it high risk under Article 6(1) and Annex I of the AI Act when the device needs a third party conformity assessment, and brings medical device conformity rules.

Bedside voice charting only transcribes and maps a nurse's own observations for the nurse to confirm, which is not listed in Annex III and is usually minimal risk. The end of shift note drafter generates a structured document from the patient's chart, so the provider of that generative text can owe the Article 50(2) duty to mark the output as AI generated, unless an exception such as an assistive function for standard editing applies. Neither deployment makes a clinical decision or profiles the patient today, but a version that summarized or flagged clinical risk, rather than only structuring what already happened, could need assessment as medical device software under Article 6(1) and Annex I of the EU Medical Device Regulation. Health data captured by voice or generated in a note falls under GDPR Article 9 in every case.

Answering onboarding questions and orchestrating provisioning is limited risk: under Article 50(1) the assistant must be designed so that employees are told they are interacting with AI, unless that is obvious. It becomes high risk under Annex III point 4(b) if it is used to make decisions on the terms or termination of the work relationship, to allocate tasks based on individual behavior or personal traits, or to monitor and evaluate new hires' performance or behavior, for example to judge probation.

Answering policy questions and starting routine requests is limited risk, with the Article 50 duty to disclose AI. It becomes high risk under Annex III point 4 if it is used to make or support decisions on recruitment, promotion, termination, allocating tasks based on individual behavior or personal traits, or the monitoring and evaluation of workers; an employer deploying it then must also inform workers' representatives and the affected workers before use (Article 26(7)). Sensitive topic detection should work on what the employee writes: inferring emotions of people in the workplace from biometric data such as voice or facial expressions is prohibited under Article 5(1)(f), except for medical or safety reasons.

Annex III point 5(a) makes AI high risk when it is used by or on behalf of public authorities to evaluate eligibility for essential public assistance benefits and services, including healthcare services, or to grant, reduce or revoke them, which can cover statutory health schemes run by or for public bodies. Point 5(c) covers risk assessment and pricing in life and health insurance, not claim review. A copilot for a private insurer's claim review, where people decide, is usually outside Annex III; for public schemes, Article 6(3) may exempt a system that only performs a preparatory task, unless it profiles natural persons. GDPR rules on health data (Article 9) and on solely automated decisions (Article 22) apply in every case.

Claims fraud, waste and abuse detection is not itself listed in Annex III. Annex III point 5(a) only covers AI used by or on behalf of public authorities to evaluate a natural person's eligibility for essential public assistance benefits and services, or to grant, reduce or revoke them, which can cover statutory health schemes run by or for public bodies such as Medicare; it does not cover a private health insurer denying a member's own claim. For a private insurer, only Annex III point 5(c), risk assessment and pricing in life and health insurance, can make a system high risk, and claims fraud, waste and abuse detection by itself is not covered by it. Provider level detection that never decides a patient's own entitlement to care stays outside Annex III either way.

Triage of phishing, endpoint, network and cloud alerts for an organization's own cyber defense is not listed in Annex III. Recital 55 of the AI Act says that components intended to be used solely for cybersecurity purposes should not qualify as safety components, so the agent does not fall under Annex III point 2 (critical infrastructure), and for this scope the tier is minimal. The design changes that when the agent triages identity, data loss prevention, insider risk or user behavior alerts in a way that scores or monitors individual employees: monitoring and evaluating the behavior of persons in a work relationship falls under Annex III point 4(b), so that scope needs its own high risk assessment before it goes live. The Article 50(1) duty to disclose AI interaction does not apply because it is obvious to a reasonably well informed analyst that they are working with an AI agent. An operator that lets AI act autonomously on network or operational technology controls should assess that design separately, and reading employees' emails and sign in data remains subject to data protection law.

A generator of synthetic tabular test data is not listed in Annex III and does not interact with people, so it is minimal risk with only the AI literacy duty of Article 4. When the system generates synthetic text, images, audio or video, such as documents or conversation transcripts, Article 50(2) requires its provider to mark the output in a machine readable format as artificially generated. When synthetic data is used to train, validate or test a high risk system, such as credit scoring, it falls under that system's data governance duties in Article 10.

Used only for practice and feedback, the simulator is limited risk. Article 50 requires that people know they are interacting with AI unless that is obvious from the context, as it usually is in a training session, and the provider must mark synthetic voice or text output as AI generated in a machine readable format. It becomes high risk under Annex III point 4(b) if its scores are used to evaluate the performance of workers or to decide on their promotion or termination, and can fall under point 3(b) when a vocational training institution uses it to evaluate learning outcomes. Inferring trainees' emotions from voice or face in the workplace is prohibited under Article 5(1)(f), except for medical or safety reasons.

As a pure assist tool for agents it is minimal risk; the customer does not interact with the AI. It becomes high risk under Annex III point 4(b) if its data is used to monitor and evaluate individual agents' performance, and inferring agents' emotions at work is prohibited under Article 5(1)(f).

Article 50(1): members must be told they are talking to an AI system. Answering benefit and claims status questions is not itself an Annex III use. It would become high risk under Annex III point 5(c) if the same system assessed or priced an individual's health insurance risk, or under point 5(a) if it decided eligibility for a public health scheme; keep those decisions in the plan's existing underwriting and adjudication process, which this use case only informs the member about.

Article 50(1): people must be told they are interacting with an AI system unless that is obvious. Finding locations and booking appointments does not fall under any Annex III category. If a healthcare version starts to triage patients by urgency, or a public body uses it to decide eligibility for a public service, reassess it against Annex III point 5.

Article 50(1) requires an assistant that talks with people to make clear they are interacting with AI, unless that is obvious from the context. It is not listed in Annex III. The agent does allocate work, but it routes tickets to resolver and assignment groups based on the content of the request, not to individual workers based on their behavior or personal traits or characteristics, so Annex III point 4(b) does not apply. It also does not decide on recruitment, promotion, credit or access to essential services. Any use that assigns work to individual analysts, or monitors and evaluates them from their behavior or performance (including through the agent's logs), would need its own assessment.

Drafting letters for human approval is not listed in Annex III. The decision the letter communicates may come from a separate high risk system, such as credit scoring (Annex III point 5(b)) or a public body's eligibility decision on benefits (point 5(a)); the drafting tool does not make that decision. Article 50(2) requires the provider of an AI system that generates text to mark the output as artificially generated, which puts this on the limited risk (transparency) tier; this includes an organization that builds its own drafting tool. Article 50(2) does not apply where the AI has only an assistive function for standard editing and does not substantially alter the input data or the semantics of the output.

Producing information videos is not listed in Annex III. Providers of tools that generate synthetic video must mark the output as AI generated, and deployers must disclose deep fakes, such as an avatar modeled on a real staff member (Article 50(2) and (4)). A system that also decided who gets a benefit or service would need a separate assessment under Annex III point 5.

Assigning billing and statistical codes from clinical documentation is not listed in Annex III and does not decide on a person's access to care, so no specific AI Act obligations apply beyond AI literacy. Health data processing falls under GDPR Article 9, and in the United States under HIPAA and the payment integrity rules of public payers. Minimal under the AI Act does not mean low stakes: the Veterans Health Administration classifies its computer assisted coding deployment on this page as high impact in the 2025 US federal AI use case inventory, even though coders select every code.

Matching a company's own incoming payments to its own open invoices is a back office finance operation. It is not listed in Annex III and does not decide a natural person's creditworthiness or eligibility for a service, so it is minimal risk and the AI literacy duty of Article 4 applies. Using deduction or payment behavior to score an individual sole trader's creditworthiness would need a fresh risk assessment.

Drafting and triaging code fixes for an organization's own software is not an Annex III use, and developers, not the public, interact with the system. The software being fixed remains subject to its own security and resilience rules, whoever wrote the fix.

Classifying the organization's own purchase lines into categories is not listed in Annex III and is used internally by procurement staff, so no specific obligations apply beyond AI literacy. The data can still contain personal data, for example in purchasing card and expense lines, which brings GDPR duties. Using the classified card and expense lines to monitor or evaluate individual employees would move the system towards Annex III point 4 (employment and worker management) and a high risk assessment.

Processing public text and broadcasts to find health events is not a use listed in Annex III and makes no decision about a person. GDPR still applies where social media posts or other personal data are processed, and the assessment would change if the system were used to profile individuals.