Capturing Detailed Hand Motion Using the Kinect Sensor and Max-Sum Belief Propagation

Abstract: Recent research indicates that several neurological diseases that affect motor functions could be diagnosed using analysis of detailed arm and hand motion. This analysis has earlier been carried out manually by looking for certain mo- tion patterns in patients and animals performing a skilled reaching task. In this thesis we investigate the possibility of performing these tests in a more auto- mated fashion by implementing image analysis methods for capturing arm and hand motion data from RGBD recordings. We have used the Microsoft Kinect sensor to capture motion both on a precise level, describing movements around individual joints of the hand, and on a coarser level, finding directions and po- sitions of the lower and upper arm. Our methods take advantage of both the RGB photos, detecting skin colour and finding arm/hand pixels, and the depth images, constructing 3D point clouds that we try to match to a simple geometrical model of the hand. Our approach is to model each phalanx of the hand individually, draw hypotheses for each of these around their pose from the previous frame and then optimize to find the most likely hand configuration using a Belief Propagation based algorithm. We present results from running our algorithms on a few test sequences. The algorithm works well under favourable circumstances but has problems giving the correct pose for example when fingers occlude each other. Possible additions to the framework that might help to overcome these issues are also discussed.