Scalable Hyperparameter Opimization: Combining Asynchronous Bayesian Optimization With Efficient Budget Allocation

Abstract: Automated hyperparameter tuning has become an integral part in the optimization of machine learning (ML) pipelines. Sequential model based optimization algorithms, such as bayesian optimization (BO), have been proven to be sample efficient with strong final performance. However, the increasing complexity and training times of ML models requires a shift from sequential to asynchronous, distributed hyperparameter tuning. The literature has come up with different strategies to modify BO to work in an asynchronous setting. By combining asynchronous BO with budget allocation strategies, poor performing trials are stopped early to free up expensive resources for other trials, improving the efficient use of resources and hence scalability further. Maggy is an open-source asynchronous hyperparameter optimization framework built on Spark that transparently schedules and manages hyperparameter trials. In this thesis, we present new support for a plug and play API to arbitrarily combine asynchronous Bayesian optimization algorithms with budget allocation strategies, like Hyperband or Median Early Stopping. This combines the best of both worlds and provides high scalability through efficient use of resources and strong final performance. We experimentally evaluate different combinations of asynchronous Bayesian Optimization with budget allocation algorithms and demonstrate its competitive performance and ability to scale.