Replicating Data Mining Techniques for Development: A Case Study of Corruption

Abstract: Data Mining has a reputation in social science for lacking statistical rigour. This study challenges this reputation and argues that, whilst such a method (as with any other) can be abused, it has particular promise as a tool to be used for monitoring and explorative research, especially by smaller development organisations. Drawing on recent advances in adapting commercial ‘Big Data’ techniques for use in international development, this study uses an example data set of global news reports to measure the level of discussion about corruption using a Text Mining methodology. The methodology outlined holds particular promise for tracking the dissemination of ideas and concepts, although it is heavily dependent on contextual interpretation and the quality of the data set used.