© lassedesignen – stock.adobe.com
News • Socio-technical risks of speech-to-text systems
AI scribes in clinical settings: How reliable are they?
New paper points out five key risks of clinical speech-to-text, shows importance of human review in AI-driven physicians’ documentation
AI speech-to-text had a moment in the spotlight on the medical drama series, “The Pitt,” just earlier this year. “Studies show that you can spend 80% less time charting,” says character Baran Al-Hashimi, a doctor showing her peers a new AI speech-to-text tool. But a skeptical colleague raised concerns after the tool misheard the name of a patient’s medication.
While the show is fictional, it poses a real question: Do the risks of these AI tools outweigh the reward? Nelly Elsayed, associate professor at the University of Cincinnati, wanted to not only answer this question but find specific solutions to advancing safe and reliable AI systems in medical documentation. Her paper, “Socio-technical risks of clinical speech-to-text systems: Transparency, privacy, and reliability challenges in AI-driven documentation,” was just published in the International Journal of Medical Informatics. The study examined a multitude of existing research, ethical guidelines and government regulations to spot how AI’s adoption is outpacing its oversight. “These AI tools, the quality is improving, but there are other aspects we have to care about to make them more efficient and transparent,” Elsayed said.
We need to have a human in the loop to check whether the text is exactly what has been spoken
Nelly Elsayed
The study examined a multitude of existing research, ethical guidelines and government regulations to spot how AI’s adoption is outpacing its oversight. “These AI tools, the quality is improving, but there are other aspects we have to care about to make them more efficient and transparent,” Elsayed said.
The paper points out five key risks of clinical speech-to-text:
- Inconsistent disclosure and consent practices
- Decreased performance for accented and disordered speech
- Extraneous noises at clinical facilities lowering the accuracy of AI
- Lack of human review over AI-generated text leading to unchecked mistakes
- Unclear accountability for errors: Is the software or the clinician responsible?
Elsayed said that simply having a human review data before it is finalized can reduce a fair amount of these concerns. “We need to have a human in the loop to check whether the text is exactly what has been spoken,” Elsayed explained. “And that test needs to be done for the entire text, not just for the first couple statements.”
Elsayed also found that there are a multitude of factors in real-life settings that can throw off an AI’s ability to record properly. AI is often trained in an “ideal” setting, with none of the bustle of a real doctor’s office like machines beeping and doctors chatting. Without training large-language models in specific scenarios, for different accents and for speech disorders, they cannot be reliable in real life.
Another way to curb error is to train the clinicians on the software before they adopt it in their practices. “The organization developing the system needs to give guidelines for the doctor, what they can use, what they cannot use and what to look out for,” she said.
Source: University of Cincinnati
17.07.2026
