Performance Analysis Between Combinations of Optimization Algorithms and Activation Functions used in Multi-Layer Perceptron Neural Networks

Abstract: Background:- Artificial Neural networks are motivated from biological nervous system and can be used for classification and forecasting the data. Each neural node contains activation function could be used for solving non-linear problems and optimization function to minimize the loss and give more accurate results. Neural networks are bustling in the field of machine learning, which inspired this study to analyse the performance variation based on the use of different combinations of the activation functions and optimization algorithms in terms of accuracy results and metrics recall and impact of data-set features on the performance of the neural networks. Objectives:- This study deals with an experiment to analyse the performance of the combinations are performing well and giving more results and to see impact of the feature segregation from data-set on the neural networks model performance. Methods:- The process involve the gathering of the data-sets, activation functions and optimization algorithm. Execute the network model using 7X5 different combinations of activation functions and optimization algorithm and analyse the performance of the neural networks. These models are tested upon the same data-set with some of the discarded features to know the effect on the performance of the neural networks. Results:- All the metrics for evaluating the neural networks presented in separate table and graphs are used to show growth and fall down of the activation function when associating with different optimization function. Impact of the individual feature on the performance of the neural network is also represented. Conclusions:- Out of 35 combinations, combinations made from optimizations algorithms Adam,RMSprop and Adagrad and activation functions ReLU,Softplus,Tanh Sigmoid and Hard_Sigmoid are selected based on the performance evaluation and data has impact on the performance of the combinations of the algorithms and activation functions which is also evaluated based on the experimentation. Individual features have their corresponding effect on the neural network.