ChatGPT does a surprisingly solid job at prescribing the right meds, but...

Researchers from the University of Liverpool have tested whether the AI-powered chatbot ChatGPT could be used to make decisions about prescribing patients with antibiotics.

In a letter published in The Lancet, academics from the Institute of Systems, Molecular and Integrative Biology show that, while artificial intelligence can’t yet replace the family doctor, there is clear potential for technology to play a role in clinical practice. The researchers presented ChatGPT with eight hypothetical infection scenarios which people would commonly consult their doctor about (such as a chest infection). They then assessed the advice delivered by the technology for its appropriateness, consistency and its impact on patient safety.

With the rise of antibiotic resistance posing a significant threat to global health, the ability of AI to provide accurate and safe treatment advice could revolutionise the way we approach patient care
Alex Howard

The assessment found that ChatGPT understood the scenarios and provided coherent answers, including disclaimers, and signposting patients to sources of advice. It also appeared to understand the need to only prescribe antibiotics when there was evidence of bacterial infection. However, ChatGPT provided unsafe advice in complex scenarios and where important information was not explicitly provided.

Interestingly, the AI tended to focus on the type of antibiotic prescribed in each scenario rather than other factors, reflecting the assumptions often initially made by doctors during consultation.

Following the experiment, the researchers have now developed a checklist for standards that AI should meet in order to be considered for use in clinical practice in the future. Co-author of the letter, Dr Alex Howard said, “It was fascinating to see the potential of artificial intelligence in healthcare demonstrated through this experiment testing ChatGPT’s ability to give antibiotic treatment advice. With the rise of antibiotic resistance posing a significant threat to global health, the ability of AI to provide accurate and safe treatment advice could revolutionise the way we approach patient care. We look forward to further exploration of this technology and its implications for the future of healthcare.”

Source: University of Liverpool

23.02.2023