A Novel SLAM Quality Evaluation Method

Abstract: Autonomous vehicles have grown into a hot topic in both research and industry. For a vehicle to be able to run autonomously, it needs several different types of systems to function properly. One of the most important of them is simultaneous localization and mapping (SLAM). It is used for estimating the pose of the vehicle and building a map of the environment around it based on sensor readings. In this thesis we have developed an novel approach to measure and evaluate the quality of a landmark-based SLAM algorithm in a static environment. The error measurement evaluation is a multi-error function and consists of the following error types: average pose error, maximum pose error, number of false negatives, number of false positives and an error relating to the distance when landmarks are added into the map. The error function can be tailored towards specific applications by settings different weights for each error. A small research concept car with several different sensors and an outside tracking system was used to create several datasets. The datasets include three different map layouts and three different power settings on the car’s engine to create a large variability in the datasets. FastSLAM and EKF-SLAM were to test the proposed SLAM evaluation method. A comparison to just the pose error was made to asses if our method can provide more information concerning establishing SLAM quality. Our results show that the pose error is often a good enough indicator of SLAM quality. However it can occasionally be misleading with errors related to mapping (location of landmarks, false negative and false positive landmarks). By using the method presented in this thesis, errors relating to the mapping will be more easily detected than by looking at the pose error.