Developing a spiking neural model of Long Short-Term Memory architectures

Abstract: Current advances in Deep Learning have shown significant improvements in common Machine Learning applications such as image, speech and text recognition. Specifically, in order to process time series, deep Neural Networks (NNs) with Long Short-Term Memory (LSTM) units are widely used in sequence recognition problems to store recent information and to use it for future predictions. The efficiency in data analysis, especially when big-sized data sets are involved, can be greatly improved thanks to the advancement of the ongoing research on Neural Networks (NNs) and Machine Learning for many applications in Physics. However, whenever acquisition and processing of data at different time resolutions is required, a synchronization problem for which the same piece of information is processed multiple times arises, and the advantageous efficiency of NNs, which lack the natural notion of time, ceases to exist. Spiking Neural Networks (SNNs) are the next generation of NNs that allow efficient information coding and processing by means of spikes, i.e. binary pulses propagating between neurons. In this way, information can be encoded in time, and the communication of information is activated only when the input to the neurons change, thus giving higher efficiency. In the present work, analog neurons are used for training and then they are substituted with spiking neurons in order to perform tasks. The aim for this project is to find a transfer function which allows a simple and accurate switching between analog and spiking neurons, and then to prove that the obtained network performs well in different tasks. At first, an analytical transfer function for more biologically plausible values for some neuronal parameters is derived and tested. Subsequently, the stochastic nature of the biological neurons is implemented in the neuronal model used. A new transfer function is then approximated by studying the stochastic behavior of artificial neurons, allowing to implement a simplified description for the gates and the input cell in the LSTM units. The stochastic LSTM networks are then tested on Sequence Prediction and T-Maze, i.e. typical memory-involving Machine Learning tasks, showing that almost all the resulting spiking networks correctly compute the original tasks. The main conclusion drawn from this project is that by means of a neuronal model comprising of a stochastic description of the neuron it is possible to obtain an accurate mapping from analog to spiking memory networks, which gives good results on Machine Learning tasks.