System for comparing topic suggestion algorithms using multiple evaluation properties

Abstract: Recommender systems are core components for many companies. These companies constantly improve their functionality aiming to maximize user satisfaction from their products. The only evaluation of the recommender systems usually done by the companies is performed in an online experiment on actual data leaving them without any offline tools to consider and postponing the quality assessment to the time when the algorithm is deployed and used in production. In this work, we describe a software evaluation tool for a selected recommender algorithm applicable for offline cases. We discuss different properties that are important for the assessment of the chosen algorithm, present the user behavior that best reflects expected real life attitude, debate various data sets (available on the Internet and provided by the company) suitable for the offline evaluation. We introduce an extensible software tool for offline assessment that is integrated into test environment created and maintained by Salesforce.com. The tool aims to be exible allowing data sets, user behavior and metrics to be easily switched or used for evaluation of other recommender algorithms. We also describe a set of recommendations on how the selected algorithm could be improved supporting these enhancement suggestions with an evaluation performed using the implemented tool.