Expert Knowledge Elicitation for Machine Learning : Insights from a Survey and Industrial Case Study

Abstract: While machine learning has shown success in many fields, it can be challenging when there are limitations with insufficient training data. By incorporating knowledge into the machine learning pipeline, one can overcome such limitations. Therefore, eliciting expert knowledge can play an important role in the machine learning project pipeline. Expert knowledge can come in many forms, and it is seldom easy to elicit and formalize it in a way that is easily implementable into a machine learning project. While it has been done, not much focus has been on how. Furthermore, the motivations for why knowledge was elicited in a particular way as well as the challenges that may exist with the elicitation, are not always focused on either. Making educated decisions for knowledge elicitation can therefore be challenging for researchers. Hence, this work aims to explore and categorize how expert knowledge elicitation has been done by researchers previously. This was done by developing a taxonomy that was then used for analyzing articles. A total of 43 articles were found, containing 97 elicitation paths that were categorized in order to identify trends and common approaches. The findings from our study were used to provide guidance for an industrial case in its initial stage to show how the taxonomy presented in this work can be applied in a real-world scenario.