Contrastive Multimodal Image Representations for Deformable Image Registration

Abstract: Image registration is the process of finding correspondences between images of the same or of similar scenes. Multimodal image registration is the specific case where the images are captured with different imaging systems. In deformable imag e registration, the images are assumed to be misaligned with local deformations and therefore the registration must also be performed locally. Intensity and Spatial Information-Based Deformable Image Registration (INSPIRE) is a robust framework for monomodal deformable image registration. However, relying on the absolute similarity of the image intensities, it is not suited for multimodal scenarios where this assumption does not hold. A recently developed method named Contrastive Multimodal Image Representation for Registration (CoMIR) can be used totransform multimodal image pairs into the same representational space. CoMIR is a type of image representation that is learned through contrastive learning and where the corresponding CoMIRs for a multimodal image pair share enough similarity in order to successfully apply rigid monomodal image registration on them. This project combines CoMIR and INSPIRE in order to perform multimodal deformable image registration. Furthermore, it introduces two new constraints to the contrastive loss called affine equivariance and deformable equivariance. We evaluate the quality of CoMIR representations, as well as the success rate of registration for the combination of CoMIR and INSPIRE on three datasets:  (i) A satellite dataset of Near-Infrared (NIR) and RGB images, (ii) A Magnetic Resonance Imaging (MRI) dataset of T1 and T2 2D images of brains, and (iii) a histological dataset of second-harmonics generation (SHG) and bright-field (BF) microscopy images of tissue samples. We observe that training a CoMIR model with affine and/or deformable equivariance yields CoMIRs with a much higher level of detailthan what is achieved with rotational equivariance or no imposed equivariance. The proposed multimodal deformable registration method achieves near monomodal registration performance on the satellite NIR to RGB dataset and outperforms a popular software for multimodal deformable image registration on all three datasets.