Estimating lighting from unconstrained RGB images using Deep Learning in real-time for superimposed objects in an augmented reality application

Abstract: Modern deep learning enables many new possibilities for automation. Within augmented reality, deep learning can be used to infer the lighting to accurately render superimposed objects with correct lighting to mix seamlessly with the environment.  This study aims to find a method of light estimation from RGB images by investigating Spherical Harmonic coefficients and how said coefficients could be inferred for use in an AR application in real-time. The pre-existing method employed by the application estimates the light by comparing two points cheek-to-cheek on a face. This fails to accurately represent the lighting in many situations, causing users to stop using the application. This study investigates a deep learning model that shows significant improvements in regards to the lighting estimation while also achieving fast inference time. The model results were presented to respondents in a survey and was found to be the better method of the two in terms of light estimation. The final model achieved 19 ms in inference time and 0.10 in RMS error.