Mixed reality – new image technology in experimental use

Henrik Brun; Egidijus Pelanis; Ola Wiig; Javier Armando Luzon; Sigurd Birkeland; Rahul Prasanna Kumar; Åsmund Avdem Fretland; Kathrine Rydén Suther; Bjørn Edwin; Ole Jacob Elle

Mixed reality – new image technology in experimental use

Henrik Brun, Egidijus Pelanis, Ola Wiig, Javier Armando Luzon, Sigurd Birkeland, Rahul Prasanna Kumar, Åsmund Avdem Fretland, Kathrine Rydén Suther, Bjørn Edwin, Ole Jacob Elle

See All Articles

Henrik Brun

E-mail: hbrun@ous-hf.no

Henrik Brun, MD, senior consultant at the Department of Paediatric Cardiology, Oslo University Hospital, Rikshospitalet, and postdoctoral fellow at the Intervention Centre, Oslo University Hospital, Rikshospitalet

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

See All Articles

Egidijus Pelanis

Egidijus Pelanis, MD, researcher at the Intervention Centre, Oslo University Hospital, Rikshospitalet, and PhD candidate at the Institute of Clinical Medicine, University of Oslo.

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

See All Articles

Ola Wiig

Ola Wiig, MD, senior consultant at the Division of Orthopaedic Surgery, Oslo University Hospital, Rikshospitalet.

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

See All Articles

Javier Armando Luzon

Javier Armando Luzon, MD, specialty registrar at the Department of Gastrointestinal Surgery, Akershus University Hospital, and clinical PhD candidate and lecturer at the Institute of Clinical Medicine, University of Oslo.

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

See All Articles

Sigurd Birkeland

Sigurd Birkeland, MD, senior consultant at the Department for Cardiothoracic Surgery, Oslo University Hospital, Rikshospitalet.

The author has completed the ICMJE form and declares no conflicts of interest.

See All Articles

Rahul Prasanna Kumar

Rahul Prasanna Kumar, engineer and postdoctoral fellow at the Intervention Centre, Oslo University Hospital, Rikshospitalet

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

See All Articles

Åsmund Avdem Fretland

Åsmund Avdem Fretland, MD, senior consultant at the Intervention Centre and Unit for Liver and Pancreatic Surgery, Oslo University Hospital, Rikshospitalet.

The author has completed the ICMJE form and declares no conflicts of interest.

See All Articles

Kathrine Rydén Suther

Kathrine Rydén Suther, MD, senior consultant at Section of Paediatric Radiology, Department of Radiology, Rikshospitalet, Division of Radiology and Nuclear Medicine, Oslo University Hospital.

The author has completed the ICMJE form and declares no conflicts of interest.

See All Articles

Bjørn Edwin

Bjørn Edwin, MD, senior consultant at the Intervention Centre and Unit for Liver and Pancreatic Surgery, Oslo University Hospital, Rikshospitalet, and professor at the Institute of Clinical Medicine, University of Oslo.

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

See All Articles

Ole Jacob Elle

Ole Jacob Elle, head of Section for Medical Cybernetics and Image Processing, the Intervention Centre, Oslo University Hospital, and professor at the Department of Informatics, University of Oslo.

The author has completed the ICMJE form and declares the following conflict of interest: He is a co-inventor of technology licensed to the company HoloCare AS and owns shares in the company indirectly via Inven2 AS.

Article

New methods for holographic visualisation provide a true three-dimensional experience of medical images. The technique is generating great interest among surgeons.

We are accustomed to viewing diagnostic medical images on two-dimensional (2D) screens, but the relationship between structures is easier to see in a three-dimensional (3D) presentation. Virtual reality imaging technology is constantly evolving. One example is mixed reality, where the real and virtual worlds merge, enabling a holographic image to be viewed through special glasses in a hybrid of reality and virtual reality.

Improvements in image quality and software have simplified the production of detailed 3D images that can facilitate the understanding of localisation and shape of pathological changes in the body. These images are typically displayed on a flat computer screen with a limited depth perception. Holographic images of organs can enable a faster and deeper understanding of both the problem and its solution.

Holographic images of malformations of the skull were described in 1988 (1). Experimental articles on spinal surgery (2) and 3D ultrasound of the heart (3) appeared in the 1990s, and the shift from 2D to 3D initially took place in the field of surgery. Planning in three dimensions is challenging in, for example, heart surgery (4), and the use of holograms in the simulation of difficult procedures has therefore been suggested for training purposes.

Early holographic technology was cumbersome, and the data capacity was limited. The equipment needed for a virtual reality 3D experience has become more user-friendly and effective, and is used in many disciplines. We describe the development of mixed reality technology in medical diagnostics and treatment and how the work in many disciplines can be adapted.

Mixed reality in clinical studies

Mixed reality with holographic glasses has been in experimental clinical use at Oslo University Hospital, Rikshospitalet, and Akershus University Hospital since 2017. Areas under study are liver surgery, congenital heart defects, colon cancer and hip disorders (Figure 1). CT, MR and ultrasound images are used as a basis for the holograms, and the use of artificial intelligence in image processing is increasing. Mixed reality also opens up new communication methods; for example, a treatment provider using holographics can reach patients who are isolated or far away.

Planning, navigation and visualisation

Surgery for congenital heart defects

The planning and implementation of surgery for complex congenital heart defects are highly dependent on good-quality images. Critical structures such as coronary arteries and large blood vessels need to be moved, and heart valves reconstructed and repaired, sometimes in neonates. Traditionally, the planning of such procedures has been based on surgeons' ability to produce a 3D mind image of the patient's anatomy from cross-sectional images. This ability varies from individual to individual (5). Even with detailed terminology, communicating the problem and solution can be a challenge.

3D reconstruction of CT and MR images on a flat screen has simplified the understanding of, for example, aortic malformations. Holograms and 3D printouts of individual heart models have enabled real 3D perception. Augmented reality is now being compared to 3D printouts as a way of defining ranges of application (6). In 2017, Oslo University Hospital received glasses with mixed reality software. With the help of computer engineers, it was possible to study the data models used for 3D printouts in the form of holograms.

Over 50 patients were enrolled in a study of the benefits of holographic visualisation before paediatric cardiac surgery. Preliminary results show that the technology is easy to use and provides a good depth perception (7). Experience from individual cases shows that holograms can simplify the understanding of complex congenital heart malformations. However, there are still some challenges with the method in relation to, for example, the use of distance measurement. Thin and moving structures, such as heart valves, are time-consuming to model. Holographic modelling of implants for the closure of ventricular septal defects is now being tested. Further developments are aimed at merging image modalities and navigating holograms during interventions. The use of various technologies for 3D visualisation of congenital heart defects has been described in a recent review (6).

Liver surgery

Laparoscopic liver resections were introduced more than 20 years ago and are now an established surgical technique for the removal of tumours in metastatic colorectal cancer. The principle is to remove the tumour with sufficient margins and save as much healthy liver as possible. The patient retains the majority of the liver, thereby reducing the risk of complications. About half of these patients relapse, and the method simplifies their further treatment (8). Surgeons' options for planning liver surgery are currently limited.

A patient-specific 3D model of the liver, including blood vessels and tumours, shows relevant preoperative information and provides an overview of the relationship between anatomical structures. Advanced software in 3D modelling can facilitate preoperative planning and the communication of complex surgery plans. One of the benefits of mixed reality is that the equipment is mobile. The holograms can be used flexibly in interdisciplinary meetings and in operating theatres. We found that it is quicker to locate a liver tumour using a holographic model than MRI (9). The solution is now used in research to plan complex procedures, but may in the future become part of everyday clinical practice.

The next step is to use this technology during surgery as an interactive map with continuous updates. This leads on to full functionality for navigation, where surgical instruments are tracked and shown in the holographic 3D model as an interactable surgical map.

Tumour surgery in the colon

For curative purposes, the tumour in the colon segment is removed keeping the mesentery intact containing regional lymph nodes. At Akershus University Hospital, extended mesenterectomies are performed for small and large bowel cancer, where mesenteric blood vessels are ligated at their source. The goal of this surgical method is to remove more lymph nodes as part of a more radical treatment in the hope of reducing the risk of relapse.

Large variations in blood vessel anatomy require detailed mapping, and this is provided in the form of high-resolution, contrast-enhanced CT imaging, which produces a detailed 3D representation of the characteristics. Typically, these reconstructions are displayed on paper or screen, but they can now be presented as 3D-printed models or holograms (10). Testing is underway to determine whether mixed reality can also be used, and results to date find that the 3D models can easily show the anatomical structures both pre- and intraoperatively. The goal is safer navigation during cancer surgery and more customised treatment as part of a tailored oncological surgical approach (11).

The segmentation method that creates 3D models needs to be validated prior to clinical use, including with the help of artificial intelligence in the future.

Orthopaedic surgery

Deformities of the hip can cause pain and reduced mobility, which are difficult to diagnose and investigate. X-rays are normally taken in two planes in addition to a CT examination. To better understand the anatomy, the images can be reconstructed in 3D, but to date, these images have been displayed on a screen in 2D, without any information on movement. Optimal surgical outcome requires surgeons to understand the dynamic pathology and how the femoral head and the acetabulum interact, and which movements are restricted and painful for the patient.

Holographic technology with mixed reality (12) enables us to perform an examination of the joints that shows how the skeletal structures move when examining the patient. CT images of the pelvis and hips are used to make a hologram of these areas.

The hologram is then placed on the patient's femur, hip joint and pelvis using markers and an optical tracking system, and it moves as the examiner changes the position of the patient's leg. This enables the examiner to see the movement restrictions and possible causes, and allows the patient to indicate which movements and joint positions are painful. This generates new information about the patient's hip disorder and provides a better basis for assessing what treatment may be effective.

We have used this examination technique on selected patients with late sequelae of paediatric hip diseases. This technology has enormous potential within orthopaedic surgery. We are working on methods that can be used in surgical planning, where holographic images of bones and joints can be used to direct the virtual surgical cutting plane to be positioned through the bone and to reconstruct skeletal deformities in mixed reality. 'Real-time X-ray vision' can enable faster, more accurate, less invasive and safer surgery, and the potential for surgical application is therefore far-reaching.

Conclusion

Mixed reality is a technology that can display holograms from medical imaging as part of real-time interactions (for example, with the patient, the doctor and the instruments). Effects such as X-ray vision and 3D navigation during surgery are achieved through tracking technology. The use of mixed reality in medicine is still in an early stage of development and is assumed to have enormous potential. Automatic production of holograms using artificial intelligence can increase the clinical benefit, but the method needs to be validated within each clinical area in order to identify possible limitations. Holographic telemedicine is an exciting area of development that can reduce the need for travel by healthcare personnel and patients. Virtual doctor appointments during epidemics can reduce the spread of infection and the need for personal protective equipment.

The patient who participated in the video recording has given consent for the article and video to be published.

Published: 23 November 2020

Tidsskr Nor Legeforen 23 November 2020

doi:

10.4045/tidsskr.20.0498

Received 3.6.2020, first revision submitted 7.9.2020, accepted 21.10.2020.

Published: 23 November 2020

Tidsskr Nor Legeforen 2020

doi: 10.4045/tidsskr.20.0498

Received 3.6.2020, first revision submitted 7.9.2020, accepted 21.10.2020.

PDF

Print

Mixed reality – new image technology in experimental use

Mixed reality in clinical studies

Planning, navigation and visualisation

Surgery for congenital heart defects

Liver surgery

Tumour surgery in the colon

Orthopaedic surgery

Conclusion

Recent Articles