Cycle-GAN for removing structured foreground objects in images

Abstract: The TRACAB Image Tracking System is used by ChyronHego for the tracking of ball and players on football fields. It requires the calibration of the cameras around the arena which is disrupted by fences and other mesh structures that are positioned between the camera and the field as a safety measure for the public. The purpose of this work was the implementation of a cycle consistent Generative Adversarial Network (cycle-GAN) for removing the fence from the image using unpaired data. Cycle-GANs are part of the state-of-the-art of image-to-image translation and can solve this kind of problem without the need of paired images. This makes it an exciting and powerful method and, according to the latest investigations in the current work, it has never been used for this kind of application before. The model was able to strongly attenuate, and in some cases completely remove, the net structure from images. To quantify the impact of the net removal a homography matching was performed. Then, it was compared with the homography associated to the baseline of blurring the image with a gaussian filter and the original image without the use of any filter. The results showed that the identification of key-points was harder on synthetic images than on the original image with or without small Gaussian filters, but it showed a better performance against images blurred with filters with a standard deviation of 3 pixels or more. Despite the performance not being better than the baseline in all the cases it always added new key-points, and sometimes, it was able to find correct homographies where the baseline could not. Therefore, the cycle-GAN model proved to complement the baseline.