Applying the Maxout Model to Increase the Performance of the Multilayer Perceptron in Shallow Networks

Abstract: The Maxout network is an alternative to artificial neural networks that use fixed activation functions. By adding extra layers of linear nodes the Maxout network is able to learn both the relationship between the hidden nodes and the activation function that they use. The idea was presented as a natural companion to the dropout technique when training large convolutional networks, showing state-of-the-art results on several benchmark datasets. In this project we apply the Maxout method, without dropout, as a substitute to a sigmoidal activation function used in small networks with only one hidden layer. We show that the Maxout method can improve the performance of some classification problems while also decreasing the computer run-time needed for training. The classification problems used were artificially created to be non-linear multi-dimensional problems. The Maxout method was also tested on a highly complex regression problem and showed to yield as good results as the sigmoidal activation function, while taking a lot shorter time to train.