Investigation of dynamic control ML algorithms on existing and future Arm microNPU systems

Abstract: In this thesis, dynamically controlled machine learning algorithms running on state of the art Arm microNPUs, with an attached Cortex-M CPU, were investigated. The machine learning framework used was Tensorflow and different subsets of it, such as Tensorflow Lite micro. Compiling the network to run on the microNPU was done with the use of an open-source compiler called, Vela. In order to investigate the dynamic support - the algorithm, MTCNN, as proposed by K. Zhang et, al. [1] was implemented on the aforementioned hardware; MTCNN was chosen due to its dynamic properties, as the algorithm structure changes depending on the input it receives. Where some parts of the algorithm may not be executed as frequently, or at all, depending on the result of earlier stages. Full dynamic support was possible on the CPU through custom kernel implementations. Several of the components needed for full use of dynamic control, on the microNPU, were unsupported throughout the toolchain. Because of this, MTCNN was divided into several parts (removing the need for dynamic support) to further investigate the performance gain by natively supporting the dynamic control operators on the microNPU. The results obtained clearly outlines a general performance gain by adding dynamic control support and the ability to run and schedule an algorithm, as a single ML model. However, for MTCNN in particular it was concluded that a major speedup was achieved by executing the static parts of the algorithm on the microNPU. The dynamic parts, with regards to MTCNN, amounts for a small percentage of the total run-time.