Article • Augmented Reality in the operating theatre
Virtual data merges with a real body
Medical Augmented Reality (AR) assistance systems overlay information onto a surgeon’s field of view. This technology is complex and expensive. Therefore, the procedure must offer a big advantage compared to conventional treatment and diagnostic methods to qualify for standard use.
Report: Anja Behringer
The objective is a system that shows a surgeon a 3-D image of inside the body plus instruments used during surgery – and not on an additional screen but with a direct view of the patient.
For the system to improve matters it must be easy to use, show relevant information and be easy to integrate into established workflow. Computer scientists merge existing and processed data with camera images of the real environment.
The C-arm, which with the help of AR lowers radiation exposure for patients by factor 40, has proved its value for some years. The challenges that medical technicians still faced three years ago, such as problems with imaging, reliable and precise tracking or issues around data preparation and visualisation have considerably reduced. Real-time imaging and visualisation are now so refined that AR systems can be utilised for an increasing number of applications.
During our interview, Dr Ulrich Eck, Senior Research Scientist for Computer Aided Medical Procedures and Augmented Reality at the Department of Informatics, Technical University of Munich, discussed some future applications.
‘Minimally invasive procedures use endoscopes for imaging,’ he began. ‘The surgeon mainly operates with the help of images transmitted by the endoscope. As the image data is acquired and visualised electronically this type of platform is particularly suitable for the visualisation of additional information, such as pre-operative image data (CT/MRI/PET), or intra-operative image data (US/OCT).
Data glasses for use in the operating theatre
‘Planning data can also be visualised for interventions. One particular challenge is the provision of relevant information for the surgeon for every step, at just the right time. The AR system must automatically detect in which phase the current procedure is.’
‘The use of data glasses during surgery is an interesting concept, but there are still some unanswered questions, such as issues of ergonomics and sterilisation. The integration of data glasses also changes workflows – similar to the use of navigation systems. Effort and result must have a meaningful ratio, and it’s not obvious which interventions are most suitable for this.
‘In summary, there are currently no data glasses/HMD for use in the operating theatre which, along with technical criteria, such as screen resolution and contrast, system latency, precision and quality of visualisation, also meet the requirements for ergonomics (light, removable, comfortable, no cables) and operating theatre specific requirements (sterilisable, reusable, robust, cost efficient). The development of an HMD suitable for the operating theatre needs close cooperation between manufacturers of data glasses, researchers in augmented reality and medical experts. The first studies in this area are currently being carried out.’
Asked for which application we can expect the early implementation in the operating theatre at a reasonable cost, Eck spoke of medical devices manufacturers who ‘now have the first devices in their range which integrate the concept of the camera-augmented C-arm (CamC) developed in our department, into their products’.
X-ray images projected onto the patient’s body
‘CamC is intended to help surgeons work faster, with more precision and with reduced radiation exposure for patients and staff. ‘In future, we can expect augmented reality enhancements of products for specific types of application, such as in neuro-surgery.
The idea of projecting a patient’s X-ray images onto their body during surgery makes sense even to lay people. Eck explains the complexity: ‘An X-ray image projects a 3-D space (the body) onto a 2-D plane. When an X-ray image is projected onto the body the spatial correlation between image content- and visualisation is no longer accurate, because it results in the impression that the information is located on the user’s skin. Only when the projector projects from the perspective of the X-ray source and the user views the patient from this direction is all the information of an X-ray image visualised correctly. This is the idea that our CamC-system is based on, albeit with a screen and not a projector.’
Detailed 3-D reconstruction of the respective body region (static or dynamic) is an essential prerequisite for the correct projection of X-ray images onto the body. Only then can the image be projected without distortion.
‘In simple cases,’ Eck concludes, ‘projection of an X-ray image onto a body part, such as a flat hand, offers added value as the spatial correlation between the surface of the hand and the image corresponds well enough. We believe that, in most other cases, this type of projection does not deliver any noteworthy advantage.’
Profile
In 2012, Ulrich Eck PhD began his Computer and Information Science studies at the University of South Australia. After gaining his PhD in 2016, for his thesis ‘Precise Co-Location of Haptic Devices in Visuo-Haptic Augmented Reality’, he became a Senior Research Scientist at the Technical University of Munich. He manages research in the NARVIS laboratory at the Chair for Computer Aided Medical Procedures. His research interests include medical augmented reality, medical simulation, haptic simulators, computer vision and human/computer interaction.
14.11.2017
