Computer Vision and Machine Learning for a Spoon-feeding Robot : A prototype solution based on ABB YuMi and an Intel RealSense camera

Abstract: A lot of people worldwide are affected by limitations and disabilities that make it hard to do even essential actions and everyday tasks, such as eating. The impact of robotics on the lives of elder people or people having any kind of inability, which makes it hard everyday actions as to eat, was considered. The aim of this thesis is to study the implementation of a robotic system in order to achieve an automatic feeding process. Different kinds of robots and solutions were taken into account, for instance, the Obi and the prototype realized by the Washington University. The system considered uses an RGBD camera, an Intel RealSense D400 series camera, to detect pieces of cutlery and food on a table and a robotic arm, an ABB-YuMi, to pick up the identified objects. The spoon detection is based on the pre-trained convolutional neural network AlexNet provided by MATLAB. Two detectors were implemented. The first one can detect up to four different objects (spoon, plate, fork and knife), the second one can detect only spoon and plate. Different algorithms based on morphology were tested in order to compute the pose of the objects detected. RobotStudio was used to establish a connection between MATLAB and the robot. The goal was to make the whole process as automated as possible. The neural network trained on two objects reached 100% of accuracy during the training test. The detector based on it was tested on the real system. It was possible to detect the spoon and the plate and to draw a good centered boundary box. The accuracy reached can be considered satisfying since it has been possible to grasp a spoon using the YuMi based on a picture of the table. It was noticed that the lighting condition is the key factor to get a satisfying result or to miss the detection of the spoon. The best result was archived when the light is uniform and there are no reflections and shadows on the objects. The pictures which get a better result for the detection were taken in an apartment. Despite the limitations of the interface between MATLAB and the controller of the YuMi, a good level of automation was reached. The influence of lighting conditions in this setting was discussed and some practical suggestions and considerations were made.