Machine Learning Based Video Editing Toolbox for Automatic Summary of Medical Videos

Abstract: Recording of medical procedures makes it possible for medical staff to review their work, learn from the mistakes and produce material for education. Medical videos tend to be long, which has an impact on usability and raises issues concerning memory storage. The aim of this master thesis is to make the material more user-friendly with an intelligent toolbox based on machine learning and image analysis techniques. The tools divide the video into chapters with a k-means++ clustering technique, detect when an X-ray source is active with ROI based processing, track camera movements by comparing frames with the optical flow algorithm by Gunnar Farnebäck and identify when medical instruments are present using an artificial neural network. Based on the information from these tools a combined timeline and an automatic summary is created. The results indicate that the chapter tool is especially promising when the videos include pre and post procedure sections, since these are easier to separate. The ROI based tool detects all the frames with an active X-ray. The neural network performs well on classifying frames containing an instrument, but requires annotated data. The majority of camera movements are found, but the algorithm sometimes fails to detect zoom in the video. This thesis is intended as a proof of concept of the potential in automatic processing of medical videos. The tools can create reference points to important sequences. More data and evaluation of the tools are necessary for the further development of an automatic summary system.