Image source: Adobe Stock/Sutthiphong

News • AI expert points out risks and benefits

Dangerous chatbots? How LLMs should be regulated for healthcare use

Generative chat tools based on large language models (LLMs), such as ChatGPT or Google’s MedPaLM have great medical potential, but there are inherent risks associated with their unregulated use in healthcare.

In a new paper published in Nature Medicine, a team led by Prof. Stephen Gilbert addresses one of the most pressing international issues of our time: How to regulate LLMs in general and specifically in health.

portrait of stephen gilbert
Prof. Stephen Gilbert


“Large language models are neural network language models with remarkable conversational skills. They generate human-like responses and engage in interactive conversations. However, they often generate highly convincing statements that are verifiably wrong or provide inappropriate responses. Today there is no way to be certain about the quality, evidence level, or consistency of clinical information or supporting evidence for any response. These chatbots are unsafe tools when it comes to medical advice and it is necessary to develop new frameworks that ensure patient safety”, said Prof. Stephen Gilbert, Professor for Medical Device Regulatory Science at Else Kröner Fresenius Center for Digital Health at TU Dresden. 

Most people research their symptoms online before seeking medical advice. Search engines play a role in decision-making process. The forthcoming integration of LLM-chatbots into search engines may increase users’ confidence in the answers given by a chatbot that mimics conversation. It has been demonstrated that LLMs can provide profoundly dangerous information when prompted with medical questions. LLM’s underlying approach has no model of medical “ground truth”, which is dangerous. 

Current LLM-chatbots do not meet key principles for AI in healthcare, like bias control, explainability, systems of oversight, validation and transparency

Stephen Gilbert

Chat interfaced LLMs have already provided harmful medical responses and have already been used unethically in ‘experiments’ on patients without consent. Almost every medical LLM use case requires regulatory control in the EU and US. In the US their lack of explainability disqualifies them from being ‘non devices’. LLMs with explainability, low bias, predictability, correctness, and verifiable outputs do not currently exist and they are not exempted from current (or future) governance approaches. In this paper the authors describe the limited scenarios in which LLMs could find application under current frameworks, they describe how developers can seek to create LLM-based tools that could be approved as medical devices, and they explore the development of new frameworks that preserve patient safety. “Current LLM-chatbots do not meet key principles for AI in healthcare, like bias control, explainability, systems of oversight, validation and transparency. To earn their place in medical armamentarium, chatbots must be designed for better accuracy, with safety and clinical efficacy demonstrated and approved by regulators,” concludes Prof. Gilbert. 

Source: Technische Universität Dresden


Read all latest stories

Related articles


News • Discerning good algorithms from bad ones

Medical AI evaluation is surprisingly patchy, study finds

In just the last two years, artificial intelligence has become embedded in scores of medical devices that offer advice to ER doctors, cardiologists, oncologists, and countless other health care…


News • Development of medical care

New research: how healthcare will change over the next 50 years

AI, robotics, climate change and more: Swedish researchers outline emerging technologies and trends that may define what the healthcare industry looks like in the next 50 years.


Article • “Embracing the future: Making robots for humans” at ICRA

Conference unveils the latest developments in robotics and automation

Ever-more sophisticated robots are advancing healthcare across a wide range of areas. Latest developments and research are being showcased in London from May 29 to June 2 at the 40th International…

Related products

3DQuorum SmartSlices

Artificial Intelligence

Hologic · 3DQuorum SmartSlices

Hologic, Inc.
Advanced intelligent Clear-IQ Engine for MR

Artificial Intelligence

Canon · Advanced intelligent Clear-IQ Engine for MR

Canon Medical Systems Europe B.V.
AI-Rad Companion

Artificial Intelligence

Siemens Healthineers · AI-Rad Companion

Siemens Healthcare GmbH
Aquilion Exceed LB

Oncology CT

Canon · Aquilion Exceed LB

Canon Medical Systems Europe B.V.
Aquilion Lightning

20 to 64 Slices

Canon · Aquilion Lightning

Canon Medical Systems Europe B.V.
Subscribe to Newsletter