Evaluating the cloud : On benchmarking and choosing a cloud

Abstract: The introduction of the cloud changes the way we look upon data and computations. It is no longer clear what "our system" is or if data is really "our data" as it isn’t stored on our own disks and computations are not performed on our own processors. While this shift brings many benefits, there are also problems. Questions regarding performance variation, security, loss of control, data and code lock-in etcetera are still a concern. As such, any cloud deployment carries a certain risk. In order to better understand this risk customers have to evaluate various cloud providers from a multitude of angles, which is a time consuming and difficult task. In this report we show how a public cloud can be evaluated and benchmarked for a number of key parameters by first assessing how a number of common workloads fare in a cloud setting and then by performing a technical benchmark of Windows Azure and Amazon EC2. Discussions regarding the non-technical aspects to a cloud are also provided. Lastly, a real world situation in which a medium sized organization chose a cloud provider is examined and analyzed. One of the key findings from the technical evaluation, aside from the actual benchmarking results, was that it is not possible to perform a time and effort efficient evaluation of a cloud platform. We find that as the task of cloud evaluation is so complex and costly, most cloud customers are likely to choose platform based on non-technical factors only, such as costs and general preferences. We believe this to be a potentially harmful development. Regarding workloads, we found that security-critical and computationally intensive applications don’t do well in a cloud setting. However, applications which utilize scalability or need to be geographically spread can find the cloud to suit their needs very well. Generally, workloads that are of a sensitive nature or requires serious performance should avoid the cloud for now. In conclusion, we believe that much work needs to be done in order to ease the process of evaluating and choosing cloud platform. There is a lack of openness and standardization which is hurting both business and customers. Considering the many benefits that cloud computing brings, such as scalability and low price; we also think that if these issues were properly dealt with, cloud computing has the potential to truly transform the way we consume and develop programs and data to the better.