3D visualisation of breast reconstruction using Microsoft HoloLens

Abstract: The purpose of the project is to create a Mixed Reality (MR) application for the 3D visualisation of the result of a breast reconstruction surgery. The application is to be used before surgery to facilitate communication between patient an surgeon about the expected result. To this purpose Microsoft HoloLens is used, which is a pair of Mixed Reality (MR) glasses developed and manufactured by Microsoft that has a self-contained, holographic rendering computer. For the development of the MR application on the Hololens, MixedRealityToolkit-Unity is used which is a Unity-based toolkit available. The goal of the application is that the user can scan a torso of a patient, render a hologram of the torso and attach to it a prefabricated breast which possibly follows the patient's specification. To prepare a prefabricated breast, a 3D model of the breast is first created in the 3D modelling software Blender. It then gets its texture from a picture taken with the HoloLens camera. The picture is cropped to better fit the model and uploaded as a 2D texture which is then attached to the prefabricated breast, which is imported into Unity. To scan objects, the HoloLens’s operating system feature Surface Observer is used. The resulting mesh from the observer is cropped using a virtual cube, scaled, moved and rotated by the user. The cropped mesh is then smoothed using the Humphrey's Classes smoothing algorithm. To fuse the smoothed mesh with the prefabricated breast model, the Unity components: Colliders and Transforms are used. On a collision the breast's transform parent is set to the mesh’s transform, making the objects transforms behave depending on each other. The MR application has been developed and evaluated. The evaluation results show that the goal has been achieved successfully. The project demonstrates that the Microsoft HoloLens is well suited for developing such medical applications as breast reconstructive surgery visualisations. It can possibly be extended to other surgeries such as showing on a patient’s body how the scar will look after a heart surgery, or a cesarean section.