On the Quality of Grey Literature and its use in Information Synthesis during Systematic Literature Reviews

Abstract: Context: The Internet has become a vital channel for disseminating and accessing scientific literature for both the academic and industrial research needs. Nowadays, everyone has wide access to scientific literature repositories, which comprise of both “white” and “Grey” literature. The “Grey” literature, as opposed to “white” literature, is non-peer reviewed scientific information that is not available using commercial information sources such as IEEE or ACM. A large number of software engineering researchers are undertaking systematic literature reviews (SLRs) to investigate empirical evidence in software engineering. The key reason to include grey literature during information synthesis is to minimize the risk of any bias in the publication. Using the state of the art non-commercial databases that index information, the researchers can make the rigorous process of searching empirical studies in SLRs easier. This study explains the evidence of Grey literature while performing synthesis in Systematic Literature Reviews. Objectives: The goals of this thesis work are, 1. To identify the extent of usage of Grey Literature in synthesis during systematic literature reviews. 2. To investigate if non-commercial information sources primarily Google Scholar are sufficient for retrieving primary studies for SLRs. Methods: The work consists of a systematic literature review of SLRs and is a tertiary study and meta-analysis. The systematic literature review was conducted on 138 SLRs’ published through 2003 until 2012 (June). The article sources used are IEEEXplore, ACM Digital Library, Springer-Link and Science Direct. Results: For each of the selected article sources such as ACM, IEEEXplore, Springer-link and Science Direct, we have presented results, which describe the extent of the usage of Grey literature. The qualitative results discuss various strategies for systematic evaluation of the Grey literature during systematic literature review. The quantitative results comprise of charts and tables, showing the extent of Grey literature usage. The results from analysis of Google Scholar database describe the total number of primary studies that we are able to find using only Google Scholar database. Conclusion: From the analysis of 138 Systematic Literature Reviews (SLRs’), we conclude that the evidence of Grey literature in SLRs is around 9%. The percentage of Grey literature sources used in information synthesis sections of SLRs is around 93.2%. We were able to retrieve around 96 % of primary studies using Google Scholar database. We conclude that Google Scholar can be a choice for retrieving published studies however; it lacks detailed search options to target wider pool of articles. We also conclude that Grey literature is widely available in this age of information. We have provided guidelines in the form of strategies for systematic evaluation of Grey literature.