Quantum Reinforcement Learning for Sensor-Assisted Robot Navigation Tasks

Abstract: Quantum computing has advanced rapidly throughout the past decade, both from a hardware and software point of view. A variety of algorithms have been developed that are suitable for the current generation of quantum devices, which are referred to as noisy intermediate-scale quantum devices. Amongst them is the variational quantum algorithm, a hybrid quantum-classical algorithm that optimizes the parameters of a parameterized quantum circuit using optimization methods from classical machine learning. Previous research has shown that this approach requires fewer trainable parameters and fewer time steps to find a solution to certain problems compared to various classical machine learning algorithms. This is particularly interesting in the case of reinforcement learning, which is a powerful type of machine learning, although its algorithms are notoriously difficult to train. They often require many training steps to converge to a solution, leading to large computational costs. Therefore, this work investigates the effect of replacing the deep neural networks of the Deep Q-learning algorithm with various parameterized quantum circuits. For the first time, a simulated TurtleBot equipped with a LiDAR sensor is trained to move through three environments containing fixed obstacles, each with a different size and complexity. The influence of different configurations of input data on the performance of both classical and quantum models is investigated. Furthermore, the expressibility, entanglement capability and effective dimension of the quantum circuits were computed to investigate the correlation between these metrics and the performance of the quantum models. Finally, the effect of depolarizing noise on the performance of one of the quantum models was investigated. Results showed that in the smallest environment, the quantum algorithm had a higher chance to converge and required fewer time steps to do so compared to a classical network containing a similar number of trainable parameters. In addition, the best quantum model performed equally well compared to the largest classical model, even though the latter has an order of magnitude more trainable parameters. In general, the data encoding strategy had a strong impact on the performance of the quantum models. Furthermore, there seemed to be no strong correlation between the quantum metrics and the performance of the quantum models. Finally, depolarizing noise at the a relatively low error rate did not influence the performance of the tested model. These results constitute a useful and practical base for further research into training sensor-assisted agents using quantum reinforcement learning. Further investigation into the the performance of the models in larger and more complex environments, as well as their performance in non-static environments, is required. Additionally, the correlation between the quantum metrics, especially the normalized effective dimension, and the quantum model performances requires further analysis. Lastly, noise models incorporating two-qubit gate errors, as well as different types of noise, should be further investigated.