Sensor Fusion for Dynamic Privacy Masking

Abstract: This master thesis investigates the possibility of combining a regular visual surveillance camera with a thermal infrared surveillance camera monitoring the same view of a scene to detect and classify heat radiating objects such as human beings. A method is presented for the application of privacy masking of a window while detecting heat radiating objects between the cameras and the window and preventing them from being masked out. The proposed method is based on determining if an object is present in both the visual and the IR frame. Registration of image pairs is done using thin plate spline interpolation. Foreground segmentation is done using a Mixture of Gaussians method. The proposed method looks at connected components from the foreground segmentation and for each component determines if it should be excluded from the mask. Classification is done by thresholding scores obtained by matching features in corresponding IR-visual frame pairs. Three measures for classifying heat radiating objects and reflections in an IR image are also proposed. The classification routine, when combined with the proposed measures, achieves a 98.9% true positive rate and a true negative rate of 99.7%.