Real-Time Video Super-Resolution : A Comparative Study of Interpolation and Deep Learning Approaches to Upsampling Real-Time Video

Abstract: Super-resolution is a subfield of computer vision centered around upsampling low-resolution images to a corresponding high-resolution counterpart. This degree project investigates the suitability of a deep learning method for real-time video super-resolution. Following earlier work in the field, we use bicubic interpolation as a baseline for comparison. The deep learning method selected is specifically suited towards real-time super-resolution and consists of a motion compensation network and an upsampling network. The deep learning method and bicubic interpolation are compared by quantitatively evaluating the methods against each other in quality metrics and performance metrics. Suitable quality metrics are selected from earlier works to provide increased comparability of results, namely peak signal-to-noise ratio and structure similarity index. The performance metrics are: number of operations for a single upsampled frame, latency, throughput, and memory requirements. We apply the methods to a highly challenging publicly available dataset specifically engineered towards video super-resolution research. To further investigate the deep learning method, we propose a few modifications and study the effect on the metrics. Our findings show that the deep learning models outperform bicubic interpolation in the quality metrics, while bicubic interpolation outperformed the deep learning models in the performance metrics. We also find no significant quality metric improvement associated with having a motion compensation network for this dataset, suggesting that the dataset might be too complex for the motion compensation network. We conclude that the deep learning method exhibits real-time capabilities as the method has a throughput of around 500 frames per second for full HD super-resolution. Additionally, we show that by modifying the deep learning method, we achieve similar latency as bicubic interpolation without sacrificing throughput or quality.