Trainable Activation Functions For Artificial Neural Networks

Abstract: Artificial Neural Networks (ANNs) are widely used information processing algorithms based roughly on biological neural networks. These networks can be trained to find complex patterns in datasets and to produce certain output signals given a set of input signals. A key element of ANNs are their so-called activation functions, which control the signal strengths between the artificial neurons in a network, and which are normally chosen from a standard set of functions. This thesis investigates the performance of small networks with a new activation function, named the Diversifier, that differs from the common ones in that its shape is trainable, while the others are generally not. Additionally, a new method is introduced that helps to avoid the well known issue of overtraining. In the end it was shown that networks with the Diversifier performed slightly better compared to networks using two of the most common activation functions, the rectifier and the hyperbolic tangent, trained on two different datasets. There have been articles covering explorations of different kinds of trainable activation functions, including, a trainable rectifier in a convolutional neural network (CNN) [5]. They also reported an improvement in performance. However, none of the ones read introduced something similar to the Diversifier.