Designing a Performant Ablation Study Framework for PyTorch

Abstract: PyTorch is becoming a really important library for any deep learning practitioner, as it provides many low-level functionalities that allow a fine-grained control of neural networks from training to inference, and for this reason it is also heavily used in deep learning research, where ablation studies are often conducted to validate neural architectures that researchers come up with. To the best of our knowledge, Maggy is the first open-source framework for asynchronous parallel ablation studies and hyperparameter optimization for TensorFlow, and in this work we added important functionalities such as the possibility to execute ablation studies on PyTorch models as well as the generalization of feature ablation on any data type. This work also shows the main challenges and interesting points of developing a framework on top of PyTorch and how these challenges have been addressed in the extension of Maggy.