Potential metrics for Agile and Lean: Systematic Literature Review and Survey

Abstract: Context: Despite continuously increasing importance of Agile and Lean in software development, the number of studies that investigate on use of metrics relevant to Agile or Lean are limited and yet few studies implements was unclear. Unclear is which are the prominent metrics that are useful in industries, and their purpose of usage. Objectives: Main goal of this study is to find the metrics useful in Agile and Lean practicing industries; that are evaluated in industries by systematically identifying all the metrics from empirical evidence found in Literature as well as verifying which of them are prominently being used in industries. In addition, the purpose of using these metrics in industries are reported, and causes for dissatisfaction on use of some of the identified metrics among surveyed companies are investigated and reported. Methods: Two research methodologies are used; Systematic Literature Review (SLR) and Industrial Survey. SLR is performed using snowballing as search approach to select primary studies. SLR is used to identify all the metrics that are useful for Agile and Lean software development. Rigor and relevance analysis is performed to assess the quality of the resulted primary studies. Industrial survey was conducted in order to verify and extend the empirical evidence exists in Literature regarding metrics by finding which of them are more prominently being used. Moreover causes for dissatisfaction over outcome of metrics use for process improvements were observed by performing comparative analysis between unsatisfied respondents results and satisfied respondents results. Results: In total 20 metrics were identified from the studies having high rigor and high relevance. Moreover 11 out of these 20 metrics were identified to be prominently being used in industries using survey and other 9 metrics are found useful for Agile or Lean methods but need more awareness. Evidence from both SLR and survey shows that most of these identified or potential metrics are used for time associated purposes which are predictability, tracking, forecasting or planning, and very little evidence found for metrics that are being used directly for quality purpose. It was observed that some of the surveyed respondents who answered not satisfied with the metrics being used are not aware of the potential benefits these metrics can offer in Agile or Lean settings. Conclusion: Evidence from both SLR and survey shows that the identified 20 metrics are very important and useful for Agile or Lean methods. 11 out of these 20 metrics are prominently being used by industries and evidence shows for other 9 metrics are also useful for Agile but needs more awareness for industries to realize their potential benefits in large scale. Also, more evidence is found for metrics that are used for time related purposes which are being dominant and important in industries than quality focused metrics. Therefore, it is important for industries not only to know which metrics are appropriate for Agile or Lean but also to have a deep understating of metrics behaviors. This will help to realize the level predictability these metric’s offer in order to make right assumptions or planning.