AI-assisted Image Manipulation with Eye Tracking

Abstract: Image editing tools can pose a challenge for motor impaired individuals who wish to perform image manipulation. The process includes many steps and can be difficult given a lack of tactile input such as mouse and keyboard. To increase the availability of image editing for motor impaired individuals, the potential for new tools and modalities have to be explored. In this project, a prototype was developed, which allows the user to edit images using eye tracking and deep learning models, specifically the DALL-E 2 model. This prototype was then tested on users who rated its functionality based on a set of human-computer interaction principles. The quality of the results varied a lot depending on the eye movements of the user, and the provided prompts. The results of the user testing found that there was potential for an editing tool implementing eye tracking and AI assistance, but that it requires further iteration and time to learn how to use. Most users enjoyed the experience of using the prototype and felt that continued experimentation would lead to improved results.