Non-Parametric Calibration for Classification

Abstract: Many applications for classification methods not only require high accuracy but also reliable estimation of predictive uncertainty. This is of particular importance in fields such as computer vision or robotics, where safety-critical decisions are made based on classification outcomes. However, while many current classification frameworks, in particular deep neural network architectures, provide very good results in terms of accuracy, they tend to incorrectly estimate their predictive uncertainty. In this thesis we focus on probability calibration, the notion that a classifier’s confidence in a prediction matches the empirical accuracy of that prediction. We study calibration from a theoretical perspective and connect it to over- and underconfidence, two concepts first introduced in the context of active learning. The main contribution of this work is a novel algorithm for classifier calibration. We propose a non-parametric calibration method which is, in contrast to existing approaches, based on a latent Gaussian process and specifically designed for multiclass classification. It allows for the incorporation of prior knowledge, can be applied to any classification method that outputs confidence estimates and is not limited to neural networks. We demonstrate the universally strong performance of our method across different classifiers and benchmark data sets from computer vision in comparison to existing classifier calibration techniques. Finally, we empirically evaluate the effects of calibration on querying efficiency in active learning.