Implementation and Visualization of Importance sampling in Deep learning

Abstract: Artificial neural networks are networks made up of thousands and sometimes millions or more nodes also referred to as neurons. Due to the sheer scale of a network, the task of training the network can become very compute-intensive. This is because all samples need to be evaluated through the network during training, and the gradients need to be updated based on each sample`s loss. Like humans, neural networks find some samples more difficult to interpret correctly than others. By feeding the network with more difficult samples while avoiding samples it has already mastered the training process can be executed more efficiently. In the medical field neural networks are among other use cases used to identify malignant cancer in tissue samples. In such a use case being able to increase the performance of a model by 1-2 percentage units could have a huge impact on saving lives by correctly discovering malignant cancer. In this thesis project, different importance sampling methods are evaluated and tested on multiple networks and datasets. The results show how importance sampling can be utilized to faster reach a higher accuracy and save time. Not only are different importance sampling methods evaluated but also different thresholds and methods to determine when to start the importance sampling.