IP Router Testing, Isolation and Automation

Abstract: Context. Test Automation is a technique followed by the present software development industries to reduce the time and effort invested for manual testing. The process of automating the existing manual tests has now gained popularity in the Telecommunications industry as well. The Telecom industries are looking for ways to improve their existing test methods with automation and express the benefit of introducing test automation. At the same time, the existing methods of testing for throughput calculation in industries involve measurements on a larger timescale, like one second. The possibility to measure the throughput of network elements like routers on smaller timescales gives a better understanding about the forwarding capabilities, resource sharing and traffic isolation in these network devices. Objectives. In this research, we develop a framework for automatically evaluating the performance of routers on multiple timescales, one second, one millisecond and less. The benefit of introducing test automation is expressed in terms of Return on Investment, by comparing the benefit of manual and automated testing. The performance of a physical router, in terms of throughput is measured for varying frame sizes and at multiple timescales. Methods. The method followed for expressing the benefit of test automation is quantitative. At the same time, the methodology followed for evaluating the throughput of a router on multiple timescales is experimental and quantitative, using passive measurements. A framework is developed for automatically conducting the given test, which enables the user to test the performance of network devices with minimum user intervention and with improved accuracy. Results. The results of this thesis work include the benefit of test automation, in terms of Return on Investment when compared to manual testing; followed by the performance of router on multiple timescales. The results indicate that test automation can improve the existing manual testing methods by introducing greater accuracy in testing. The throughput results indicate that the performance of a physical router varies on multiple timescales, like one second and one millisecond. The throughput of the router is evaluated for varying frame sizes. It is observed that the difference in the coefficient of variance at the egress and ingress of the router is more for smaller frame sizes, when compared to larger frame sizes. Also, the difference is more on smaller timescales when compared to larger timescales. Conclusions. This thesis work concludes that the developed test automation framework can be used and extended for automating several test cases at the network layer. The automation framework reduces the execution time and introduces accuracy when compared to manual testing. The benefit of test automation is expressed in terms of Return on Investment. The throughput results are in line with the hypothesis that the performance of a physical router varies on multiple timescales. The performance, in terms of throughput, is expressed using a previously suggested performance metric. It is observed that there is a greater difference in the Coefficient of Variance values (at the egress and ingress of a router) on smaller timescales when compared to larger timescales. This difference is more for smaller frame sizes when compared with larger frame sizes.