Stabilizing Side Effects of Experience Replay With Different Network Sizes for Deep Q-Network

Abstract: This report investigates the effects of two different types of batch selection used for traininga Deep Reinforcement Learning agent in games. More specifically, the impact of thedifferent methods were tested for different sizes of Deep Neural Networks while using theDeep Q-Network (DQN) algorithm. The two methods investigated were Random batchselection and Combined Experience Replay (CER). Random batch selection is the mostcommonly used method while CER is a more recent method with low additionalcomputational cost. These two methods were tested on the two classic games Snake andSuper Mario Bros, using DQN and a variety of Deep Neural Network Sizes. It was seen thatthe CER method improved stability between the different network sizes while not reducingthe learning rate compared to the Random method. This reduces the difficulty of tuning theDeep Neural Network size while trying to optimise the agent.