Benchmarking of ESA Risk Management Tool

Abstract: Projects in the space domain are extremely complex and often characterized as being “one-off”, which makes it very difficult to make good predictions on future developments. Furthermore, space projects are also associated with state-of-the-art technology, high costs and long development time. Risks are therefore a threat to project success because they may have negative effects on the project cost, schedule and technical performance. In order to properly manage these potential risks, a Risk Management (RM) process is implemented as a support in the various project phases. The RM discipline provides a framework for an iterative process that shall be used for finding, dealing and working proactively with project risks. To facilitate the proper implementation of the RM activities ESA has developed, together with a software company, a Risk Management system, Alpha. This is a customized solution and a question has been raised whether or not the customized solution is the best approach. Some reasons being that ESA has to take all the cost and initiative for further development of the system. As an alternative, a COTS solution could be implemented. In order to find out what direction to take a question needs to be initially answered: Are there any available COTS solutions that are capable in supporting ESA’s RM process and replacing the existing solution? This report has tried to answer the question through conducting a survey of commercially available RM tools and through performing a benchmarking exercise where four COTS solutions were compared with the existing customized solution. The survey was based on some of the major RM elements suggested by the ECSS standard for RM with the purpose to find and down select four tools to be included in the benchmark. The benchmark was in general based on a framework for software requirements suggested by Soren Lausen and interviews with relevant ESA personal but in particular based on the ECSS standard for RM. Several capable solutions where found in the survey but an analysis of the results separated four tools (Alpha, Beta, Gamma and Delta) to be included in the benchmark. The systems acquired the following score for the mandatory requirements: Alpha 80 %, Beta 67 %, Gamma 79 % and Delta 74 %.From these results and the benchmark the author drew the following conclusions: Alpha is performing well and the tool is superior, compared to the evaluated commercial tools, in terms of reporting requirements. Two tools, Gamma and Delta, are able to compete with Alpha in terms of overall performance. Gamma scored almost equally with Alpha and if the tool supports the claimed configuration possibilities it could achieve a perceivable increase in score. This basically means that if ESA would change strategy and implement a commercial tool instead, the agency would not have to sacrifice anything in terms of relative performance (there would of course be gains/sacrifices for specific requirements). Beta is currently not able to compete in terms of performance with the other tools. The approach of sending the requirement list as an enquiry to Epsilon’s developer did not provide sufficient information to include the system in the benchmark. However, the impression is that the tool may be capable in supporting ESA’s RM process. Alpha scored 80 % for the mandatory requirements, which means that there is a possibility to improve the system. In chapter six suggestions of further development of the tool has been presented which could increase Alpha’s score to 97 %. While Alfa is definitely a good candidate for continuing supporting the ESA RM process, especially if the suggestions for further developing the tool are taken into consideration, the author’s opinion is that the agency should further investigate the possibility of implementing a new RM system. A recommended approach is to extend the investigation described in this report by including additional parameters (e.g. one of them also being financial) with the objective to improve the current process and find a suitable vendor competent in both the IT and RM fields. In this investigation a third party consultant could be included with experience in the area of IT but most importantly also in the RM field. Such project could be beneficiary for ESA both from an IT and from a RM point of view. The author’s opinion is that the developed methodology used in this study has provided ESA with proper results and could be used in similar future studies. However, as the requirements were assessed quantitatively an assessment of the importance or criticality of each requirement would have provided even better results.