Optimizing L2-regularization for Binary Classification Tasks

Abstract: An Artificial Neural Network (ANN) is a type of machine learning algorithm with widespread usage. When training an ANN, there is a risk that it gets overtrained and cannot solve the task for new data. Methods to prevent this, such as L2-regularization, introduce hyperparameters that are time-consuming to optimize. In this thesis, I investigate a hypothesis which postulates how the optimal L2-regularization strength for a binary classification task depends on the number of input dimensions and available training patterns. First I generating binary classification tasks consisting of Gaussian clouds of different size in different numbers of dimensions. Several networks were then trained with varying L2-regularization strengths to see which ones achieved the lowest validation strength on the tasks. Results were promising in favor of the hypothesis. No statistical significance is shown, but results were similar to the behaviour predicted by the hypothesis when the number of training patterns was in a certain interval.