Training Multi-Task Deep Neural Networks with Disjoint Datasets

Abstract: This work examines training neural networks which are capable of learning multiple tasks. We propose an architecture trained on KITTI and Cityscapes, which respectively include only the annotations for 2D object detection and semantic segmentation. We propose 4 methods for training with disjoint datasets, and show the difference in performance with hyperparameters taken from literature as well as a hyperparameter search. We show the feasibility of training with disjoint datasets. We observe that the best strategy for training is using multiple forward passes and summing the gradients. By using multi-task learning we note an increase in mean average precision for the 2D object detection task but a decrease in mean intersection over union for the semantic segmentation task.