Use of Stochastic Switching in the Training of Binarized Neural Networks

Abstract: Most prior research into the field of Binarized Neural Networks (BNNs) has been motivated by a desire to optimize Deep Learning computations on traditional hardware. These methods rely on bit-wise operations to drastically decrease computation time, however to address the resulting loss in accuracy it is common practice to reintroduce continuous parameters and train using batch normalization. Here a separate application for BNNs is investigated to find if it is a feasible application to build and train BNNs on hardware based on binary memristors with stochastic switching. To do this BNNs have been trained without batch normalization and multiple new algorithms have been proposed which attempt to utilize the stochastic switching ability in place of storing multiple bits of information per weight. The results show that BNNs without batch normalization are limited to lower accuracies than conventional CNNs, and it is essential to include an element of gradient accumulation for stochastic switching to work. To adress instability issues with stochastic switching based training an undo function is introduced which is shown to stabilize training well. Future research should look into accumulating gradients using stochastically switching memristors.