Panoramic view of a virtual operating room for training laparoscopic...
Panoramic view of a virtual operating room for training laparoscopic interventions with the help of software simulation.

Source: University Magdeburg

Article • Visualization

Augmented Reality in the OR: matching man and machine

One of the crucial future technologies in surgery is Augmented Reality (AR). Most experts agree that AR will increase safety and efficiency, improve surgical training and decrease costs. Currently, “AR is not being used to the extent possible”, regrets Dr Christian Hansen, Professor for Virtual and Augmented Reality at Otto von Guericke University Magdeburg, Germany.

Report: Sascha Keutel

However, this is not because the physicians balk at the new technology, he explains, but because the technology is not yet mature enough. “The point is not,” he underlines, “to add a computer graphics layer to the data sets but to enhance the accuracy of the visualization”.

Not every surgery needs AR support, Professor Hansen says, as surgeons can do appendix surgery without an advanced navigation system. AR is needed for more complex interventions that are performed less frequently. 

But what exactly is AR? “AR means enhancing reality with an additional layer of virtual information. The additional layer should be at the correct location, i.e. geometrically registered; it should be interactive and be available in real-time. In addition to visual information, AR can offer audio, haptic, olfactory and even gustatory information,” Professor Hansen explains and adds that there are three different AR modalities: video see-through, optical-see-through and projector-based AR.

Navigation support using a tablet computer (video-see-through AR). The planned...
Navigation support using a tablet computer (video-see-through AR). The planned puncture site as well as the needle orientation and depth are displayed by means of a crosshair visualization.

Source: University Magdeburg

Video see-through

In video see-through a video feed of the real world is displayed on a screen with digital information added. This modality is widely used in healthcare, particularly in research projects and clinical studies but also “routinely in laparoscopy and endoscopy since the devices can be easily upgraded to present additional information”, says Professor Hansen. CAScination, for example, is one of the laparoscopy providers. But video see-through has also been adopted in many fields outside medicine: if you are interested in an article in the Ikea catalogue, with the help of a mobile phone and an app you can see what it will look like it in your own home.

“The disadvantage of video see-through is the limited resolution of the camera”, Professor Hansen explains but adds that “technological progress will sooner or later overcome this hurdle. Another issue, however, is the screen between the user and the patient”.

Training of laparoscopic interventions with the help of a VR simulator and...
Training of laparoscopic interventions with the help of a VR simulator and special joysticks as input devices.

Source: University Medical Center of the University Mainz

Optical see-through

In optical see-through systems the images of the real world are not displayed on a screen but are projected onto a slanted semi-transparent mirror. Stryker has started to use optical-see-through in ENT surgery.

The Microsoft HoloLens headset is a widely used optical see-through device that provides surgeons in real-time with vital data and diagnostic images. The disadvantage of body-worn AR solutions is the fact that the glasses are still rather large. Wearing them over an extended period of time is uncomfortable, particularly if the surgeon is uses regular glasses or a magnifier to improve vision.

Moreover, the optical see-through medium absorbs light which makes reality appear darker than it actually is – something the surgeon has to get used to. For AR headsets to conquer the OR, Professor Hansen says, “they have to become smaller, more functional and they have to be adapted better to surgical use”.

Projector-based AR in an AR test lab. The projected representation can be...
Projector-based AR in an AR test lab. The projected representation can be adapted by the user with the help of a tracked 3D input device.

Source: University Magdeburg

Projector-based AR

Projector-based AR does not rely on a body-worn or hand-held device. Rather the projector is integrated in the OR environment and “the information is projected onto the objects of interest. Thus, reality is augmented with digital information”, Professor Hansen explains.

This modality is still in an early research phase and no devices are commercially available. According to Professor Hanse, “the projectors generate too much air and heat to be installed in an OR”. Another potential problem with the use of a single projector is the fact that the surgeon or another OR team member may obstruct the beam. LAP uses projector-based AR for radiation therapy planning or for patient positioning in CT and MRI.

Projector-based AR in an AR test lab. The projected representation can be...
Projector-based AR in an AR test lab. The projected representation can be adapted by the user with the help of a foot and head tracking system. With the tracked pointer in the hand of the user, specific objects can be selected.

Source: University Magdeburg

Interdisciplinary cooperation

Most overlays are not 2D images but 3D data sets that can consist of several layers such as skin, surface of different organs, vascular systems and other data. “The components and their interactions are complex and we are trying to figure out how the surgeons can best interact with these data sets. Since the surgeon usually only has one hand available, voice activation or a combination of hand and foot movements would be ideal”.

In order to be able to answer all these questions and to develop useful AR systems, interdisciplinary cooperation is crucial, Professor Hansen underlines. In addition to surgeons, IT experts and above all psychologists play important roles. “This is about a man-machine interface; thus we need to clarify a number of psychological issues. When new technologies are launched too early, people might trust and rely on these systems too much. We all know stories about people who drove right into rivers because they blindly relied on their navigation system”. Another issue is distraction: the AR overlays distract from what’s happening in reality. Thus a surgeon might be so focused on the digital information that he overlooks a bleeding that occurs unexpectedly.

VR scenario for anesthesia for communication training in the event of...
VR scenario for anesthesia for communication training in the event of complications during surgical interventions.

Source: University Magdeburg


Immersive technologies are helpful training tools. While AR images might not be overly important for an experienced surgeon, they can be crucial for junior physicians. “I expect Virtual Reality to assume a major role in surgical training. Simulators can guide trainees through highly immersive multi-user environments. They can be exposed to different scenarios and train behaviour in complex situations or the communication with the OR team”, Professor Hansen explains.

Professor Hansen and his team are experimenting with this kind of training in a joint project with the University Hospital Mainz. Funded by the German Federal Ministry of Education and Research, they combine VR systems with relevant simulation and training software. The physicians wear a headset which allows them to enter a multi-user environment. This VR scenario is linked to a laparoscopy or anaesthesia simulator to train certain situations. An example: during surgery a vessel clip drops and the patient loses so much blood that the vital parameters reach critical levels. The success of the intervention depends to a large extent on the communication between the surgeon and the anaesthetist – a scenario that can be trained in VR.

Dr Christian Hansen, Professor for Virtual and Augmented Reality at Otto von Guericke University Magdeburg.

In Professor Hansen‘s OR of the future robots that are much smaller than today’s systems will be used. They are equipped with miniaturized instruments that the surgeon can insert via natural body orifices. Moreover, in hybrid ORs mobile and multimodal imaging systems will be routinely used. “Thanks to modern interfaces advanced visualization technologies will be standard; they will allow different data sets to be superimposed as AR images”.

Professor Dr Christian Hansen studied computational visualistics with a focus on medicine at Otto von Guericke University in Magdeburg. From 2006 to 2013 he worked in different research & development areas at Fraunhofer Institute for Digital Medicine MEVIS in Bremen. In September 2013, Hansen was appointed Junior Professor for Computer-Assisted Surgery at the Institute for Simulation Graphics / Computer Science Department at the Otto von Guericke University Magdeburg an. In September 2020 he took on his new role as Adjunct Professor for Virtual and Augmented Reality.


More on the subject:
Read all latest stories

Related articles


News • Virtual distraction

Use of VR during wide-awake surgery helps ease anxiety

Patients immersed in virtual reality, while undergoing wide-awake surgery experience more joy and less anxiety than those in a traditional operating room setting, according to a new study.


Article • Augmented reality in the OR

AR helps surgeons to see, feel and understand

X-ray vision, context-sensitive guidance, coordinator, training assistant and more: augmented reality (AR) has hit the OR. While still in its infancy AR does grow rapidly and has already shown…


Article • Medica 2021 product presentation

Medtech megatrends and more from Taiwan

Video streaming of surgeries, augmented reality displays and at-home assessment of sleep apnea: Medtech companies from Taiwan will showcase their newest products at Medica in Düsseldorf. Ahead of…

Related products

Subscribe to Newsletter