Predictive Uncertainty Estimates in Batch Normalized Neural Networks

Abstract: Recent developments in Bayesian Learning have made the Bayesian view of parameter estimation applicable to a wider range of models, including Neural Networks. In particular, advancements in Approximate Inference have enabled the development of a number of techniques for performing approximate Bayesian Learning. One recent addition to these models is Monte Carlo Dropout (MCDO), a technique that only relies on Neural Networks being trained with Dropout and L2 weight regularization. This technique provides a practical approach to Bayesian Learning, enabling the estimation of valuable predictive distributions from many models already in use today. In recent years however, Batch Normalization has become the go to method to speed up training and improve generalization. This thesis shows that the MCDO technique can be applied to Neural Networks trained with Batch Normalization by a procedure called Monte Carlo Batch Normalization (MCBN) in this work. A quantitative evaluation of the quality of the predictive distributions for different models was performed on nine regression datasets. With no batch size optimization, MCBN is shown to outperform an identical model with constant predictive variance for seven datasets at the 0.05 significance level. Optimizing batch sizes for the remaining datasets resulted in MCBN outperforming the comparative models in one further case. An equivalent evaluation for MCDO showed that MCBN and MCDO yield similar results, suggesting that there is potential to adapt the MCDO technique to the more modern Neural Network architecture provided by Batch Normalization.