Combining Register Data and X-Ray Images for a Precision Medicine Prediction Model of Thigh Bone Fractures

Abstract: The purpose of this master thesis was to investigate if using both X-ray images and patient's register data could increase the performance of a neural network in discrimination of two types of fractures in the thigh bone, called atypical femoral fractures (AFF) and normal femoral fractures (NFF). We also examined and evaluated how the fusion of the two data types could be done and how different types of fusion affect the performance. Finally, we evaluated how the number of variables in the register data affect a network's performance. Our image dataset consisted of 1,442 unique images from 580 patients (16.85% of the images were labelled AFF corresponding to 15.86% of the patients). Since the dataset is very imbalanced, sensitivity is a prioritized evaluation metric. The register data network was evaluated using five different versions of register data parameters: two (age and sex), seven (binary and non-binary) and 44 (binary and non-binary). Having only age and sex as input resulted in a classifier predicting all samples to class 0 (NFF), for all tested network architectures. Using a certain network structure (celled register data model 2), in combination with the seven non-binary parameters outperforms using both two and 44 (both binary and non-binary) parameters regarding mean AUC and sensitivity. Highest mean accuracy is obtained by using 44 non-binary parameters. The seven register data parameters have a known connection to AFF and includes age and sex. The network with X-ray images as input uses a transfer learning approach with a pre-trained ResNet50-base. This model performed better than all the register data models, regarding all considered evaluation metrics.        Three fusion architectures were implemented and evaluated: probability fusion (PF), feature fusion (FF) and learned feature fusion (LFF). PF concatenates the prediction provided from the two separate baseline models. The combined vector is fed into a shallow neural network, which are the only trainable part in this architecture. FF fuses a feature vector provided from the image baseline model, with the raw register data parameters. Prior to the concatenation both vectors were normalized and the fused vector is then fed into a shallow trainable network. The final architecture, LFF, does not have completely frozen baseline models but instead learns two separate feature vectors. These feature vectors are then concatenated and fed into a shallow neural network to obtain a final prediction. The three fusion architectures were evaluated twice: using seven non-binary register data parameters, or only age and sex. When evaluated patient-wise, all three fusion architectures using the seven non-binary parameters obtain higher mean AUC and sensitivity than the single modality baseline models. All fusion architectures with only age and sex as register data parameters results in higher mean sensitivity than the baseline models. Overall, probability fusion with the seven non-binary parameters results in the highest mean AUC and sensitivity, and learned feature fusion with the seven non-binary parameters results in the highest mean accuracy.