Ultrasound neural style transfer using domain specific features

Abstract: Ultrasound imaging is a widely used fast, low-cost, and non-invasive modality for monitoringfetal development during pregnancy and identifying potential problems or other injuries.However, interpreting the images may be difficult due to the noisy appearance and requiresextensive training of sonographers. Additionally, finding volunteer patients available forultrasound screening, particularly if screening for rare diseases, poses a significant challenge.To alleviate these problems, ultrasound simulation is often used as an alternative as it is able tocreate artificial scenarios without the need for volunteers. However, the creation of realisticsimulated ultrasound images is a nontrivial task and many simulated images end up lookingsynthetic. Deep learning models have proven effective in generating simulated ultrasoundimages, but labeled datasets are scarce. To mitigate this challenge, an UltrasoundSemi-supervise Contrastive Learning method (USCL) was created and trained on 23.000images with ResNet18 as the chosen backbone. The main objective of this study was toimprove the realism of the simulated ultrasound images using these domain-specific featuresextracted from the USCL when used in combination with neural style transfer. The neural styletransfer program extracted features from real ultrasound images and transferred them tosimulated ultrasound images, employing two different ultrasound datasets on two differentbackbones (VGG19 and ResNet18). The results show that ResNet18 pretrained with ultrasoundimages got worse results than VGG19 pretrained on ImageNet, both in terms of metrics andevaluating the realism of the images. Several factors could be behind these outcomes, includingthe unsuitability of ResNet18 for style transfer due to the residual connections. However, recentadvancements have shown improvements regarding the robustness of the ResNet18 modelwhich may counter this problem. Furthermore, USCL which considers two ultrasound imagesfrom different videos a negative pair, may have impacted the training of the model negativelydue to this reasoning. Notably, a new USCL version called Meta USCL was released which has shown promising results. Future research should consider employing the newly released MetaUSCL model when performing neural style transfer as well as improving the robustness of ResNet18 to generate more realistic simulated ultrasound images.