Image Synthesis Using CycleGAN to Augment Imbalanced Data for Multi-class Weather Classification

Abstract: In the last decade, convolutional neural networks have been used to a large extent for image classification and recognition tasks in a number of fields. For image weather classification, data can be both sparse and unevenly distributed amongst labels in the training set. As a way to improve the performance of the classifier, one often used traditional augmentation techniques to increase the size of the training set and help the classifier to converge towards a desirable solution. This can often be met with varying results, which is why this work intends to investigate another approach of augmentation using image synthesis. The idea is to make use of the fact that most datasets contain at least one label that is well represented. In weather image datasets, this is often the sunny label. CycleGAN is a framework which is capable of image-to-image translation (i.e. synthesizing images to represent a new label) using unpaired data. This makes the framework attractive as it does not put any unnecessary requirements on the data collection. To test the whether the synthesized images can be used as an augmentation approach, training samples in one label was deliberately reduced sequentially and supplemented with CycleGAN synthesized images. The results show adding synthesized images using CycleGAN can be used as an augmentation approach, since the performance of the classifier was relatively unchanged even though the number of images was low. In this case it was as few as 198 training samples in the label that represented foggy weather. Comparing CycleGAN to traditional augmentation techniques, it proved to be more stable as the number of images in the training set decreased. A modification to CycleGAN, which used weight demodulation instead of instance normalization in its generators, removed artifacts that otherwise could appear during training. This improved the visual quality of the synthesized images overall.