Head-mounted display prototype
Head-mounted display used to develop prototypes for medical AR systems. The displays has been developed in Munich – but is not conceived for use in the operating theatre (too many cables, too heavy)

Article • Augmented Reality in the operating theatre

Virtual data merges with a real body

Medical Augmented Reality (AR) assistance systems overlay information onto a surgeon’s field of view. This technology is complex and expensive. Therefore, the procedure must offer a big advantage compared to conventional treatment and diagnostic methods to qualify for standard use.

Report: Anja Behringer

The objective is a system that shows a surgeon a 3-D image of inside the body plus instruments used during surgery – and not on an additional screen but with a direct view of the patient. 

For the system to improve matters it must be easy to use, show relevant information and be easy to integrate into established workflow. Computer scientists merge existing and processed data with camera images of the real environment. 

The C-arm, which with the help of AR lowers radiation exposure for patients by factor 40, has proved its value for some years. The challenges that medical technicians still faced three years ago, such as problems with imaging, reliable and precise tracking or issues around data preparation and visualisation have considerably reduced. Real-time imaging and visualisation are now so refined that AR systems can be utilised for an increasing number of applications.

During our interview, Dr Ulrich Eck, Senior Research Scientist for Computer Aided Medical Procedures and Augmented Reality at the Department of Informatics, Technical University of Munich, discussed some future applications.

‘Minimally invasive procedures use endoscopes for imaging,’ he began. ‘The surgeon mainly operates with the help of images transmitted by the endoscope. As the image data is acquired and visualised electronically this type of platform is particularly suitable for the visualisation of additional information, such as pre-operative image data (CT/MRI/PET), or intra-operative image data (US/OCT). 

Cam-C-Projekt
Cam-C-Projekt

Data glasses for use in the operating theatre

‘Planning data can also be visualised for interventions. One particular challenge is the provision of relevant information for the surgeon for every step, at just the right time. The AR system must automatically detect in which phase the current procedure is.’

‘The use of data glasses during surgery is an interesting concept, but there are still some unanswered questions, such as issues of ergonomics and sterilisation. The integration of data glasses also changes workflows – similar to the use of navigation systems. Effort and result must have a meaningful ratio, and it’s not obvious which interventions are most suitable for this.

‘In summary, there are currently no data glasses/HMD for use in the operating theatre which, along with technical criteria, such as screen resolution and contrast, system latency, precision and quality of visualisation, also meet the requirements for ergonomics (light, removable, comfortable, no cables) and operating theatre specific requirements (sterilisable, reusable, robust, cost efficient). The development of an HMD suitable for the operating theatre needs close cooperation between manufacturers of data glasses, researchers in augmented reality and medical experts. The first studies in this area are currently being carried out.’

Asked for which application we can expect the early implementation in the operating theatre at a reasonable cost, Eck spoke of medical devices manufacturers who ‘now have the first devices in their range which integrate the concept of the camera-augmented C-arm (CamC) developed in our department, into their products’.

X-ray images projected onto the patient’s body

Ulrich Eck PhD
Ulrich Eck PhD

‘CamC is intended to help surgeons work faster, with more precision and with reduced radiation exposure for patients and staff. ‘In future, we can expect augmented reality enhancements of products for specific types of application, such as in neuro-surgery.

The idea of projecting a patient’s X-ray images onto their body during surgery makes sense even to lay people. Eck explains the complexity: ‘An X-ray image projects a 3-D space (the body) onto a 2-D plane. When an X-ray image is projected onto the body the spatial correlation between image content- and visualisation is no longer accurate, because it results in the impression that the information is located on the user’s skin. Only when the projector projects from the perspective of the X-ray source and the user views the patient from this direction is all the information of an X-ray image visualised correctly. This is the idea that our CamC-system is based on, albeit with a screen and not a projector.’

Detailed 3-D reconstruction of the respective body region (static or dynamic) is an essential prerequisite for the correct projection of X-ray images onto the body. Only then can the image be projected without distortion.

‘In simple cases,’ Eck concludes, ‘projection of an X-ray image onto a body part, such as a flat hand, offers added value as the spatial correlation between the surface of the hand and the image corresponds well enough. We believe that, in most other cases, this type of projection does not deliver any noteworthy advantage.’


Profile

In 2012, Ulrich Eck PhD began his Computer and Information Science studies at the University of South Australia. After gaining his PhD in 2016, for his thesis ‘Precise Co-Location of Haptic Devices in Visuo-Haptic Augmented Reality’, he became a Senior Research Scientist at the Technical University of Munich. He manages research in the NARVIS laboratory at the Chair for Computer Aided Medical Procedures. His research interests include medical augmented reality, medical simulation, haptic simulators, computer vision and human/computer interaction.


14.11.2017

More on the subject:
Read all latest stories

Related articles

Photo

News • Tech from Taiwan

Augmented reality is the future of surgery

Physicians have been performing surgery with the assistance of x-ray technology for almost half a century. While this technology has been progressing steadily, its limitations continue to be a major…

Photo

Article • A valuable tool for reconstruction

Augmented reality lets surgeons ‘see’ inside limbs

Researchers at Imperial College London (ICL) have shown how the Microsoft HoloLens headset can be used during reconstructive lower limb surgery. Surgeons at London’s St Mary’s Hospital are using…

Photo

News • Orthopaedic surgery

2D and 3D MRIs add value for ACL surgery

Magnetic resonance imaging (MRI) can reliably establish measurements for anterior cruciate ligament (ACL) “footprints” that are critical to the placement of grafts for reconstruction surgery.

Related products

Canon – Alphenix Sky+

Single Plane

Canon – Alphenix Sky+

Canon Medical Systems Europe B.V.
Canon - Alphenix Sky+ High Definition Detector

Single Plane

Canon - Alphenix Sky+ High Definition Detector

Canon Medical Systems Europe B.V.
Canon Electron Tubes & Devices – X-ray Image Intensifier

Accessories / Complementary Systems

Canon Electron Tubes & Devices – X-ray Image Intensifier

Canon Electron Tubes & Devices Co., Ltd
DK Medical – Prostar

Surgical II-C-Arms

DK Medical – Prostar

DK Medical Systems Co., Ltd
Dunlee - 3D printed pure tungsten anti-scatter grids

Accessories / Complementary Systems

Dunlee - 3D printed pure tungsten anti-scatter grids

Dunlee – Philips Medical Systems DMC GmbH
Fujifilm - FDX Visionary-C and CS

Surgical Flat Panel C-arms

Fujifilm - FDX Visionary-C and CS

FUJIFILM Europe GmbH
Subscribe to Newsletter