AI in Simulated 3D Environments : Application to Cyber-Physical systems

Abstract: Over the past several years, UAVs (unmanned aerial vehicles) and autonomous systems in general, have become hot topics, both in academia and in industry. Indeed, the opportunities for the application of such technologies are vast, with the military and the infrastructure industry being the two most eminent cases. Up until recently, autonomous systems showed quite little flexibility, their actions originating in well-defined programs that executed and replicated a given task, without much ability to adapt to new conditions in the surrounding environment. However, recent advances in AI and Machine Learning have made it possible to train computer algorithms with unprecedented effectiveness, which opened the door to having cyber-physical systems that can show intelligent behaviour and decision-making capabilities. Using simulated environments, one is now able to train such systems to exhibit decent performance in tasks whose complexity stumped state-of-the-art algorithms less than a decade ago. An approach that has proved extremely successful is Reinforcement Learning (RL). In this thesis, we used it (along with other AI techniques) to train a virtual flying drone to perform two different tasks. The first one consists in having the drone fly towards a predefined object, no matter where it is placed. The second one is to have it fly in a manner that would allow for the exploration of an unknown environment. We were able to combine both tasks: to find and head towards a specific target within an unknown environment, by using only the relative position of the drone to its taking off point and its camera, therefore without any environment specific information. After a process of trial and error, we developed a framework for exploration on a plane, excluding the movement on the yaw axis. In order to perform such tasks with a deep Q network model we had to retrieve a depth image, the relative position of the drone and a segmented image. The results presented herein demonstrate that a drone can be trained to be reasonably performant in the aforementioned tasks. It was achieved up to 81% accuracy on an unknown test environment for the first task while achieving 98% accuracy for the training environment on the same task. It holds the promise for doing the same with other cyber-physical systems and for more complex tasks.