Prediction of the gain in classification performance from combining multiple imaging modalities

Abstract: In this work, we investigate the relationship between different image modalities and classification performance, aiming to predict the potential gain in classification accuracy when combining multiple modalities. We analyze mathematical and statistical measures and develop novel reconstruction measures (RMSE and RSSIM) to assess information distribution between different image modalities. Our study focuses on four classification problems with varying complexity, ranging from binary classification on the CIFAR-10 dataset, two degraded versions of CIFAR-10 dataset to a biomedical dataset for cancer detection, with 291 models trained and evaluated. We first evaluate the correlations between basic measures (MSE, SSIM, PCC, and MI), ourreconstruction measures (RMSE and RSSIM) and classification performance gain. We concludethat these measures meter different image properties and are potential predictor candidates for performance prediction in image processing tasks. In the case of the relatively simple binary classification task using unmodified CIFAR-10 images, we were able to identify certain correlations. Additionally, we attempt end-to-end classification gain prediction using a neural network model, aiming to generalize classification performance from one subset of problems to another. Our findings indicate that performance in similar classification tasks does not generalize well to new tasks, even when the image data is comparable. In conclusion, our study reveals the limitations of using mathematical, statistical, and reconstruction measures for predicting classification performance improvement when incorporating additional image modalities. Nevertheless, our novel reconstruction measure shows potential for further research in assessing the quality and importance of different image modalities, and our findings provide insights into the complex relationship between image properties and classification performance. Future work may explore the application of the reconstruction measure for other image processing tasks.