Audio Detection for Preparation of Video Rate Controller

Abstract: Modern video streaming may use the protocol H.264 to encode video in high quality. The protocol takes advantage of similarities between consecutive frames to lower the bit rate. If there are differences between frames the bit rate will therefore rise. To control the fluctuations in bit rate a rate controller (RC) can be used together with the encoder. The RC manages the quantisation levels of the encoder. Rapid unforeseen changes may require the RC to drastically increase the quantisation parameter (QP) or even skip frames. This paper evaluates if audio event detection can be used to prepare the RC for incoming changes by creating a warning signal that acts on the RC. Two methods are investigated; restricting the minimum allowed QP during detections, and allowing higher bit rates temporarily. The paper also examines a number of methods for performing audio event detection. Three types of Gaussian mixture models (GMM) are implemented and tested. Further, two envelope based algorithms are used. Finally, one GMM and one envelope algorithm are used together with the minimum QP scheme to test a combined solution. Results show that both envelopes and GMMs can be used to perform detection useful for the purposes of RC preparation. When acted on by the warning system the RC lowers the amount of frame skips on most tests. Further, the peak quantisation needed during an event is often lower. Investigation of the resulting video quality shows that the quality score is lower in terms of VMAF and SSIM before and after the event happens. However, during the event the quality is roughly the same and sometimes slightly higher. When a real audio detection is used for preparation the frame skips are improved, but there are side effects in terms of quality related to false positives in the audio detection.