Augmented Reality in the OR: matching man and machine

However, this is not because the physicians balk at the new technology, he explains, but because the technology is not yet mature enough. “The point is not,” he underlines, “to add a computer graphics layer to the data sets but to enhance the accuracy of the visualization”.

Not every surgery needs AR support, Professor Hansen says, as surgeons can do appendix surgery without an advanced navigation system. AR is needed for more complex interventions that are performed less frequently. 

But what exactly is AR? “AR means enhancing reality with an additional layer of virtual information. The additional layer should be at the correct location, i.e. geometrically registered; it should be interactive and be available in real-time. In addition to visual information, AR can offer audio, haptic, olfactory and even gustatory information,” Professor Hansen explains and adds that there are three different AR modalities: video see-through, optical-see-through and projector-based AR.

Video see-through

In video see-through a video feed of the real world is displayed on a screen with digital information added. This modality is widely used in healthcare, particularly in research projects and clinical studies but also “routinely in laparoscopy and endoscopy since the devices can be easily upgraded to present additional information”, says Professor Hansen. CAScination, for example, is one of the laparoscopy providers. But video see-through has also been adopted in many fields outside medicine: if you are interested in an article in the Ikea catalogue, with the help of a mobile phone and an app you can see what it will look like it in your own home.

“The disadvantage of video see-through is the limited resolution of the camera”, Professor Hansen explains but adds that “technological progress will sooner or later overcome this hurdle. Another issue, however, is the screen between the user and the patient”.

Optical see-through

In optical see-through systems the images of the real world are not displayed on a screen but are projected onto a slanted semi-transparent mirror. Stryker has started to use optical-see-through in ENT surgery.

The Microsoft HoloLens headset is a widely used optical see-through device that provides surgeons in real-time with vital data and diagnostic images. The disadvantage of body-worn AR solutions is the fact that the glasses are still rather large. Wearing them over an extended period of time is uncomfortable, particularly if the surgeon is uses regular glasses or a magnifier to improve vision.

Moreover, the optical see-through medium absorbs light which makes reality appear darker than it actually is – something the surgeon has to get used to. For AR headsets to conquer the OR, Professor Hansen says, “they have to become smaller, more functional and they have to be adapted better to surgical use”.

Projector-based AR

Projector-based AR does not rely on a body-worn or hand-held device. Rather the projector is integrated in the OR environment and “the information is projected onto the objects of interest. Thus, reality is augmented with digital information”, Professor Hansen explains.

This modality is still in an early research phase and no devices are commercially available. According to Professor Hanse, “the projectors generate too much air and heat to be installed in an OR”. Another potential problem with the use of a single projector is the fact that the surgeon or another OR team member may obstruct the beam. LAP uses projector-based AR for radiation therapy planning or for patient positioning in CT and MRI.

Interdisciplinary cooperation

Most overlays are not 2D images but 3D data sets that can consist of several layers such as skin, surface of different organs, vascular systems and other data. “The components and their interactions are complex and we are trying to figure out how the surgeons can best interact with these data sets. Since the surgeon usually only has one hand available, voice activation or a combination of hand and foot movements would be ideal”.

In order to be able to answer all these questions and to develop useful AR systems, interdisciplinary cooperation is crucial, Professor Hansen underlines. In addition to surgeons, IT experts and above all psychologists play important roles. “This is about a man-machine interface; thus we need to clarify a number of psychological issues. When new technologies are launched too early, people might trust and rely on these systems too much. We all know stories about people who drove right into rivers because they blindly relied on their navigation system”. Another issue is distraction: the AR overlays distract from what’s happening in reality. Thus a surgeon might be so focused on the digital information that he overlooks a bleeding that occurs unexpectedly.

Training

Immersive technologies are helpful training tools. While AR images might not be overly important for an experienced surgeon, they can be crucial for junior physicians. “I expect Virtual Reality to assume a major role in surgical training. Simulators can guide trainees through highly immersive multi-user environments. They can be exposed to different scenarios and train behaviour in complex situations or the communication with the OR team”, Professor Hansen explains.

Professor Hansen and his team are experimenting with this kind of training in a joint project with the University Hospital Mainz. Funded by the German Federal Ministry of Education and Research, they combine VR systems with relevant simulation and training software. The physicians wear a headset which allows them to enter a multi-user environment. This VR scenario is linked to a laparoscopy or anaesthesia simulator to train certain situations. An example: during surgery a vessel clip drops and the patient loses so much blood that the vital parameters reach critical levels. The success of the intervention depends to a large extent on the communication between the surgeon and the anaesthetist – a scenario that can be trained in VR.

Dr Christian Hansen, Professor for Virtual and Augmented Reality at Otto von Guericke University Magdeburg.

In Professor Hansen‘s OR of the future robots that are much smaller than today’s systems will be used. They are equipped with miniaturized instruments that the surgeon can insert via natural body orifices. Moreover, in hybrid ORs mobile and multimodal imaging systems will be routinely used. “Thanks to modern interfaces advanced visualization technologies will be standard; they will allow different data sets to be superimposed as AR images”.

Profile:
Professor Dr Christian Hansen studied computational visualistics with a focus on medicine at Otto von Guericke University in Magdeburg. From 2006 to 2013 he worked in different research & development areas at Fraunhofer Institute for Digital Medicine MEVIS in Bremen. In September 2013, Hansen was appointed Junior Professor for Computer-Assisted Surgery at the Institute for Simulation Graphics / Computer Science Department at the Otto von Guericke University Magdeburg an. In September 2020 he took on his new role as Adjunct Professor for Virtual and Augmented Reality.

17.12.2021

• surgery (583)
• virtual (71)