Machine Unlearning and hyperparameters optimization in Gaussian Process regression

Abstract: The establishment of the General Data Protection Regulation (GDPR) in Europe in 2018, including the "Right to be Forgotten" poses important questions about the necessity of efficient data deletion techniques for trained Machine Learning models to completely enforce this right, since retraining from scratch such models whenever a data point must be deleted seems impractical. We tackle such a problem for Gaussian Process Regression and define in this paper an efficient exact unlearning technique for Gaussian Process Regression which completely include the optimization of the hyperparameters of the kernel function. The method is based on an efficient retracing of past optimizations by the Resilient Backpropagation (Rprop) algorithm through the online formulation of a Gaussian Process regression. Furthermore, we develop an extension of the proposed method to the Product-of-Experts and Bayesian Committee Machines types of local approximations of Gaussian Process Regression, further enhancing the unlearning capabilities through a random partitioning of the dataset. The performance of the proposed method is largely dependent on the regression task. We show through multiple experiments on different problems that several iterations of such optimization can be recomputed without any need for kernel matrix inversions, at the cost of saving intermediate states of the training phase. We also offer different ideas to extend this method to an approximate unlearning scheme, even further improving its computational complexity.