MISK-Moves

Abstract: Move-to-play is a musical instrument for persons with both cognitive and physical impairments, who would have trouble playing traditional instruments. Everyone, no matter their abilities, are given the chance to play and control music by moving their own body. This project is part of the MISK project, which is a collaboration between Certec, Furuboda folkhögskola and Eldorado resurscenter. To make this possible, a machine learning model for the detection of people and their limbs, pose estimation, is used. This enables the translation of body movement into data used for controlling music software, which is made as another part of the MISK project. It runs under Max 8 by Cycling '74. To make it easy and affordable for other creators to recreate the project, relatively cheap hardware has been used, a NVIDIA Jetson Nano and a Raspberry Pi V2 camera. It is a system-on-module, which compact size and low power consumption makes placement even in small compartments easy. The machine learning model used is ResNet-18 pose, which proved to have adequately sufficient precision and performance to track people and their limbs in real time. It has not mattered whether a person has been standing or sitting, playing the instrument has worked equally well in both cases. Move-to-play has some latency however, up to about 150 ms. Users will have to be selective about what or how the instrument should control the music, since different people have different sensitivity to the connection between their own actions, latency and the effect of their actions on the music.